• JOIN ON table1.yyy = table2.xxx
• Some typical SDSS joins:
  • SDSS/BOSS : join specobj and photoobj
  • APOGEE : join apogeeStar, aspcapStar, apogeeObject
• Note example queries on SDSS web site, and also in casJobs interface!

• SkyServer
• Database access in left column:
  • some simple searches that make the SQL for you: SQS (spectroscopic query search), IRSQS (IR spectroscopy query search)
  • general SQL query form: SqlSearch
  • interface for larger queries: casJobs

• Make some SDSS query:
  • get at least four output numerical quantities, possibilities:
    • SDSS/BOSS: galaxies within some region of the sky, get RA/DEC/redshift velocity dispersion from specobj, magnitudes from PhotoObj
    • APOGEE: stars within some region of the sky, get RA/DEC/nvisits/radial velocity/vscatter from apogeeStar, stellar parameters/abundances from aspcapStar
  • start with HTML query, then get both CSV and FITS output files on your computer
  • Copy your output csv and fits files to your subdirectory on the astronomy server:

    scp {filename} visitor@astronomy.nmsu.edu:{dirname}/
    

Data files

• text files: create/inspect with standard editor. Columns separated by some delineator:
  • spaces
  • commas (CSV files)
• binary files: more compact, but not inspectable with an editor, need to use a program to read
  • FITS files ( name.fits)
    • image files
    • tables

Interactive languages and plotting

• Python
• IDL
• TOPCAT
• others!

Python

Ways to run Python:

• Interactively: strongly recommend using ipython (enhanced interactive python interface), to enable automatic plot display use: ipython --matplotlib
• Write python commands in a file (standard extension .py)
  • Run the commands: python file.py
  • While running ipython: %run file.py or import file.py
• iPython notebooks: allow you to intersperse text and python, including execution of code

• print (intrinsically in interactive mode!)
• arithmetic; +, -, *, /, ** (be careful about variable types and truncation!)
• variables : single values, lists
• arrays (though numpy)
  • initialize array using numpy.zeros([ny,nx]) (after import numpy) or numpy.ones([ny,nx])
  • turn a list into an array using numpy.array(listvar)
  • array aritmetic is implicit: if one does arithmetic between a constant and an array, it applies to each element automatically; if one does arithmetic between arrays, it does it element-by-element (if the shapes don't match, then it generates an error!)
• strings: concatenation, strip, etc.
• variables as objects: they have attributes and methods. In ipython, use {object}. to view all of the possibilities
• control statements: if / else
  • Note that indentation defines the blocks!
• loops: for
• functions: used to avoid repetition of code for repeated calculations, also for code organization and readability, example:

  def addconst(a,const=1) :
      return a+const
  
  addone(5)
  

  • Note that functions can return values that can be assigned to variables
  • functions can take arguments, either positional or by keyword; keyword arguments can have default values
• Advantages of writing code in a file, rather than just doing it at command line
  • it can be saved and rerun!
  • it can be commented
    • # at the beginning of the line indicates a comment
    • text between triple-quotes (''' or """) allows for multi-line comments
    • all functions should have docstrings between triple-quotes!

• Start ipython interactively: ipython
• Define some numerical variables, and do some arithmetic with them
• Define some string variables, and do some operations with them
• Write some operations in a file, and then execute the file from the command line, and from within ipython

• a module is a collection of routines (functions); a package may be a single module, but may also contain multiple modules
• import module
• import module as shortname
• from packagename import module
• Finding modules
  • system-installed modules
  • user modules: PYTHONPATH environment variable
• Frequently used modules:
  • matplotlib : plotting routines
  • numpy : numerical functions, provides array functionality and operations
  • scipy : many scientific analysis routines
  • astropy : rapidly developing set of routines for astronomical analysis/calculation
• Convenience feature of ipython: startup files in ~/.ipython/profile_default/startup

• text files: using astropy.io.ascii or numpy.loadtxt
  • from astropy.io import ascii
  • var = ascii.read('filename')
  • import numpy as np
  • var = np.loadtxt('filename')
• FITS files: using astropy.io.fits
  • from astropy.io import fits
  • var = fits.open('filename')
  • FITS files may have multiple extensions, so this gives a FITS ``header-data unit (HDU)'' for each dimension
  • Each HDU has a header (.header) and data (.data)
• Both routines return numpy structured arrays (for FITS tables)
  • for a given column name, access values using var['colname']
  • see columns automatically in ascii tables by typing variable name, or for either using columns method on returned object: var.columns
• Very useful function: numpy.where
• FITS image files
  • For wavelength calibrated spectra with uniform sampling, wavelength information may be in header, e.g. CRVAL1/CDELT1/NAXIS1 or for BOSS spectra, COEFF0/COEFF1/NAXIS1: wave=var.header['CRVAL1']+var.header['CDELT1']*np.arange(var.header['NAXIS1'])
  • if uniform sampling is on log10(wavelength) scale: wave=10**wave

• Using the data files you got from your SQL queries, read them into Python variables, both csv files and fits files

• remember, for automatic plot display, start ipython with --matplotlib!
• import matplotlib.pyplot as plt
• Simple plotting and overplotting:
  • plt.plot(x,y[,pointstyle]) (with pointstyle, draws points, otherwise connects points)
    • example pointstyles give color ([krgbcym]) and shape ([o+.^v<>sph]) : 'ro', 'g+', 'b.', 'c^', 'yv', ...
  • plt.plot(x,z[,pointstyle])
    • if plot window doesn't automatically show, restart ipython with --matplotlib, or use plt.show() to see plot
  • plt.clf() # to clear the plot
  • Using matplotlib window functions from the icons
  • Customizing plots with limits and labels:
    • plt.xlim(xmin,xmax) # x limits
    • plt.ylim(ymin,ymax) # y limits
    • plt.xlabel(xlabel)
    • plt.ylabel(xlabel)
    • plt.text(x,y,text) # add text at arbitrary location
    • plt.title(title) # put a title on the plot
• Subwindows
  • plt.clf()
  • plt.subplot(1,2,1) # (ny, ny, id)
  • plt.plot(x,y)
  • plt.subplot(1,2,2)
  • plt.plot(x,z)
• More sophisticated plot interface, allows for multiple figures to be open at a time, more control
  • fig=plt.figure()
  • ax1=fig.add_subplot(1,2,1) (shorthand: fig.add_subplot(121)) # note difference in function name from above
  • ax1.plot(x,y)
  • ax1.set_xlim(xmin,xmax) # note difference in function name from above
  • ax1.set_ylim(ymin,ymax) # note difference in function name from above
  • ax2=fig.add_subplot(1,2,2)
  • ax2.plot(x,y)
  • ax2.cla # to clear an axis
  • fig.tight_layout() # helps if you have things that overlap between plots
  • plt.draw()
• Packing more information into your plots: point colors and point sizes
  • scatter: allows you to code points by color and/or size, e.g.

       ax.scatter(x,y,c=z,vmin=zlo,vmax=zhi,size=t)
       ax.colorbar()
    

    will plot x vs y, color points according to z, size points by values in t, add colorbar

• Plot up your 4 dimensional data using point color and size!

IDL

• Interactive use
• Writing IDL procedures and running them

• Documentation
• Astronomy Users Library

• print
• variables and arrays
  • initialize arrays using var=fltarr(nx,ny) (or intarr)
• arithmetic
• strings
• conditionals
• loops
• functions (return a value) and procedures (don't return value, but can modify arguments)

• Start IDL: idl
• Define some numerical variables, and do some arithmetic with them
• Define some string variables, and do some operations with them
• Write some operations in a file, and then execute the file from the command line, and from within ipython

• Astronomy users library
• SDSS tools: idlutils, idlspec2d
• Finding routines: IDL_PATH environment variable

• IDL Users library readcol routine (library must be in IDL_PATH!)
  • each column is read into an array variable
  • readcol,'filename',c1,c2,c3,c4,c5,c6[,format=('fffff') (variable names for columsn can be anything you want, formats are fda for float (default), integer, character)
• IDL Users library mrdfits routine (library must be in IDL_PATH!)
  • var=mrdfits('filename'[,ext,head=head]) reads data from specified extension into var
  • table data is read into a structure variable, access elements using e.g. var.colname
  • help,var to see the columns

• As above for Python, but IDL

• plot,x,y[,xtit=xtitle,ytit=ytitle,xr=[xmin,max],yr=[ymin,ymax]]
• oplot

• plots command allows you to specify color (between 0-255) and size of a point

    plots,x,y,color=(z-zmin)*(zmax-zmin),size=size
  

• so far as I know, for color-coded and/or size-coded points, need to plot one point at a time in a loop over array elements:

• As above for Python, but IDL

TOPCAT

• Load a table
• Plot columns against each other
  • Code points by size (color?)
• Histogram