==== Observing ====

Observing is done using virtual destop on the song1m virtual machine.

There should be a terminal with at least 3 tabs, running:
  * aposong  : main control window
  * status : populates status window and databases
  * guider : process that waits for guide commands from aposong

In addition to this, processes must be running on the machines dome1m, pwi1m, and spec1m, but these are generally fairly stable, and only need to be checked in case of issues.

Under normal operations, robotic observing is accomplished with a single command in aposong:

<code>
robotic.observe(foc0,display=disp)
</code>
where foc0 is an inital focus guess, and disp is a pyvista display object if you want to see images as they are being taken.

The basic robotic workflow is:
  * touches an MJD file to show sequence has started
  * wait until sunset+dt_sunset  (dt_sunset can be set with keyword to robotic.observe, default=0)
  * wait until safe to open, as determined from either 2.5m open, 3.5m open or override set
  * open dome and mirror covers, and turn on telescope fans
  * wait until sunset (if not yet there)
  * open louvers
  * wait until nautical twilight+dt_nautical  (also set by keyword, default=0)
  * do an initial focus run (until initfoc=False keyword is given)
  * loop until morning nautical twilight
    * if unsafe to observe, close dome and wait 90 seconds
    * else if dome is closed and now safe to open, open dome
    * else if it has been more than dt_focus (default=1.5 unless specified by keyword), do a focus run
    * else choose best object to observe
      * if not the same as last object, acquire object in guider
      * perform requested observing sequence
  * close dome, turn off fans, close louvers, stop guider, park telescope and dome (all via domeclose())
  * take morning cals
  * create guider movies, focus plots, reduce data, and make web page
  * sends successful completion email with webcam snapshot attached, removes MJD file

Two cron jobs run at 8:00 each morning:
  * one runs copy command to sync /data/1m to astronomy /home/1m
  * second checks for existence of an MJD file and sends failure email if it finds one

==== Morning ====

After each night of observing, you should check:
  * webcam to ensure that dome is closed!
  * aposong window to see if robotic sequence completed normally, which will leave it at a python prompt
  * if sequence completed normally, check web pages to see what was observed, check focus curves, check throughput plots as desired
  * if sequence didn't complete normally, try to figure out what happened! May be useful to create the web pages, etc., manually, using <code> robotic.mkhtml(int(Time.now().mjd))</code>. Can also consult logs in /data/1m/logs/daily.log
 
==== Troubleshooting ====

If observing did not complete successfully:
  * check the webcam to ensure that dome is closed! 
  * if not closed, or if closed but telescope not in park position, issue domeclose() command from an aposong window (you may wish to start a new one if you want to leave the incompleted observing one open for inspection)
  * if dome does not close, may need to call APO!
  * to determine failure mode:
    * attempt to understand where script failed
    * check guider window to see if guider program is still running
    * check for any errors in status window
    * check pwi1m computer to see if Planewave had issues, either diconnected or one axis disabled
    * check spec1m computer to ensure ASCOM Remote and Alpyca server are running