Stellar Observations Network Group (SONG): SONG in New Mexico

You've loaded an old revision of the document! If you save it, you will create a new version with this data.

====== Software ======

===== Overview =====

Overall software design is to control all components from a main server computer, song1m, located in
the main computer room, running Linux. This communicates with the telescope/camera computer, pwi1m, and the dome computer, dome1m, via network communication; this is extendable to additional devices on additional computers. It is largely a client-server model, where the server devices are interfaced over the network through [[https://ascom-standards.org/|ASCOM/Alpaca]].

Devices can also be controlled locally from the computer to which they are attached, as this is useful for lower level functionality, debugging, etc.

A schematic of the software architecture is shown below (click to enlarge).
{{ :software:song_software_architecture.jpg?direct&400 |}}

===== Control software =====

A full set of Python commands that communicate with all of the component devices is implemented in the aposong.py module in the APOsong repository ([[https://github.com/holtzmanjon/APOsong|github.com/holtzmanjon/APOsong]]). The communication is implemented (for the most part) with the [[https://ascom-standards.org/alpyca/|Alpyca]] Python interface to  Alpaca devices.
Given that all of the devices are on the internal APO network, the main control computer, song1m, is located on this network, and remote operation can be done using VNC with an openVPN or tunneling connection. On song1m, the software environment is set up for the song user.

An interactive client can be run from a Python environment. APOsong provides convenience routines to connect to Alpaca devices and provide user-friendly commands to control the telescope, dome, camera etc., but you can also use all regular Python functionality, providing for a powerful environment. Within the APOsong software, interactive image display is enabled using the pyvista display tool and built-in, e.g. for the expose() command, but of course any other Python packages can be used interactively.

To start an ipython session and access the command set:

<code>
aposong
</code>

You can get a quick listing of commands using
<code>
commands()
</code>
and you can use help(command) or command? to see the docstrings.

This will also provide instantiations of all of the Alpaca devices: Telescope as T, Dome as D, Safetymonitor as S, Cameras as C[], Switches as SW[], Focussers as F[].

Documentation is available at http://aposong.readthedocs.io

===== Robotic software =====

Alternatively, a robotic client is implemented in the robotic.py module in this APOsong repository. This makes use of the command set provided by aposong.py, but adds in functionality for opening and closing the dome at appropriate times and under appropriate conditions, and selecting targets and exposure sequences from entries in a Postgres database on song1m.

For SONG operation, robotic operations are conducted through Postgres database syncs from Aarhus to the song1m_db machine. Requested observations are loaded into the obs_request_4 table in db_song with information about the
target and the observing sequence. As the request is started, completed, or aborted, the status is
communicated back through the obs_reqeust_status_4 table in db_apo. Other tables in db_apo communicate
the state of the APO system so that the conductor can choose appropriate targets.

===== song1m computer =====

This is the main computer interface. It is a virtual machine running on a box in the APO server room. Use the song account to log in. Once all remote software is started (see pwi1m, spec1m, and dome1m):
<code>
cd APOsong
aposong
</code>

This should show you a list of commands (which you can see again with the commands() function), and a list of connected devices.

Two other processes also need to be running on song1m:
- status  : updates status window and database with various information
- guider : process that handles guiding through socket connection to main aposong process

These are normally run in different tabs in a terminal on a VNC desktop.

===== pwi1m computer =====

pwi1m runs Windows. For normal operation, several things must be running:

* the PWI4 software to control the telescope: this provides
    * as interactive graphical interface
    * several ASCOM drivers for telescope, port 1 focuser, and mirror cover control
    * a custom PlaneWave API interface
  * ASCOM remote for communication with native Windows ASCOM drivers:
    * PWI-4 interface
    * PlaneWave Focuser (via PWI4)
    * PlaneWave Mirror Cover (via PWI4)
    * Atik guide camera
    * Atik eShel camera
    * QSI camera (for PlanWave port 1)
    * QSI FilterWheel 
  * A custom Alpaca server which supports Alpaca drivers for:
   * the Zaber focusing stage (Zaber XX) through an ASCOM Focuser device, 
   * the calibration stage (Thorlabs LTS 150 stage), commanded through an ASCOM Focuser device
   * the iodine stage (Thorlabs TLS 150 stage), commanded through an ASCOM Focuser device, 
   * the iodine cell temperature controller (Thorlabs TC300 controller), commanded through an ASCOM Switch device
   *  the Shelyak eShel calibration unit, through an ASCOM Switch device.

PlaneWave interface 4 is started from a desktop icon. It will open (after a minute or two) a graphical window (more details below). Need to ensure that telescope is connected and all motors/focuser are enabled.

ASCOM remote is stated from a desktop icon. It may need to be restarted if power is cycled to any of the devices that it interfaces to. It takes a minute or two to start and should show connections to all of the devices listed above.

A desktop icon has been set up to start the Alpyca server. This just starts an Anaconda Python shell and does a:
<code>
cd APOalpyca/device
python fpu_app.py
</code>

The system should then be ready for remote usage.

The three programs have been put into Startup folder so they should start by themselves if the computer is restarted (may take a minute or two).

=== Planewave software ===

The PlaneWave software is not normally used for regular operation, as we just use the ASCOM and direct PWI interface. However, more detailed information and more command functionality is available through the PWI graphical interface, e.g., for pointing models and telescope troubleshooting. Hence we provide some information here.

On pwi1m, the telescope is controlled through the PWI4 software. This also has the capability to control an ASCOM camera, necessary when using PWI4 to construct a pointing model.

On starting PWI4, devices are controlled from the left panel, which has five tabs: Mount/Focus/Rotate/M3/Temp. You connect to the telescope using the Connect button on the Mount tab, and
you can enable/disable the azimuth and altitude motors. On the Focus tab, you connect to the
focuser and enable/disable it, and on the Rotate tab, you connect and enable/disable the rotator. On
the Temp tab, you connect the OTA, which allows you to get temperature monitoring, control fans, etc. The M3 tab is used to position the tertiary to the Nasmyth ports: Port 1 is the port with the rotator and 
focuser.

{{scope:scope:software:mount.png?130}}
{{scope:scope:software:focus.png?130}}
{{scope:scope:software:rotate.png?130}}
{{scope:scope:software:m3.png?130}}
{{scope:scope:software:temp.png?130}}

You can slew the telescope using the SkyViewer tab: click on the desired location, then confirm with Goto
button. You can also just enter coordinates directly on the Goto tab. Tracking should be started after the slew. You can monitor the tracking performance with the Tracking tab. Temperatures can be viewed on the
Temperature tab. Finally, a camera can be controlled through the Camera tab; you need to select the camera (ASCOM or MaximDL), and then connect to it.

The PWI also has ASCOM drivers for Telescope and Focuser, which the control software uses. The PWI also 
has its own web interface (see https://planewave.com/files/software/PWI4/python/), which we use to get some information that is not available through the ASCOM 
driver.

=== Focus ===

=== Pointing model ===

=== ASCOM remote ===

ASCOM remote must be installed and configured to see the native ASCOM devices. This is achieved through the Setup button. Simply choose the type of device and select the correct driver.

=== Alpaca devices ===

Several devices are controlled via ASCOM Alpaca, as implemented in the [[https://ascom-standards.org/alpyca/|Alpyca]] package, using custom Alpaca drivers developed using the [[https://github.com/ASCOMInitiative/AlpycaDevice|AlpycaDevice]] templates:

* ASCOM [[https://ascom-standards.org/alpyca/alpaca.focuser.html|Focuser]] devices
  - built using Thorlabs 150 https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=10285
  - built using Zaber stage https://software.zaber.com/motion-library/api/py
 * ASCOM [[https://ascom-standards.org/alpyca/alpaca.switch.html|Switch]] devices 
  - TC300 https://www.thorlabs.com/thorproduct.cfm?partnumber=TC300, https://www.thorlabs.com/drawings/a037cd0c76c0bd79-A3C155F8-CDFD-BAC7-1B554FE246DB47DD/TC300-Manual.pdf
  - Shelyak eShel calibration unit, built using K8056 routine

Documentation of the Alpaca drivers can be found at [[https://apoalpyca.readthedocs.io/en/latest/|http://APOalpyca.readthedocs.io]].

===== spec1m computer =====

spec1m is the SONG spectrograph computer. It controls the spectrograph QHY detector and the Esatto focuser. It also controls two thermocouples that monitor the body temperature of the QHY and a relay that resets a watchdog timer if the body temperature is under a limit; otherwise the watchdog is not reset and the QHY camera power will be cut after 2-3 minutes.

Run ASCOM remote for QHY interface and Esatto focuser interface.

A desktop icon has been set up to start the Alpyca server for the thermocouple interface. This just starts an Anaconda Python shell and does a:
<code>
cd APOalpyca/device
python spectro_app.py
</code>

Since the QHY power is controlled by the thermocouple interface, the ASCOM remote interface won't connect until this is run: it will prompt you to connect the ASCOM device within 10 seconds.

The system should then be ready for remote usage.

The two programs (ASCOM remote and Alpyca server) have been put into Startup folder so they should start by themselves if the computer is restarted (may take a minute or two).

===== dome1m =====

Dome control is implemented through software running on a Raspberry Pi, dome1m.

Low-level functions are implemented in the APOAshDome.py module. An ASCOM Alpaca interface was implemented using the templates from the [[https://github.com/ASCOMInitiative/AlpycaDevice|AlpycaDevice]] package.
Software is archived at [[https://github.com/holtzmanjon/APOalpyca]].

APOAshDome controls dome shutter and rotation through relays, reads dome position through an encoder, and also reads dome home switch. For the encoder, the pulses are read using the pigpiod daemon, so that must be installed and started every time the computer is rebooted: 
<code>
sudo pigpiod -s 10
</code>
This has been implemented in /etc/rc.local on dome1m.

The dome software can be run locally from a python interface: from APOalpyca/device, start python and import APOAshdome. A Dome object can be instantiated through which properties and methods can be accessed: 
<code>
import APOAshDome
D = APOAshDome.APOAshDome()
...
</code>

For documentation of the methods, see [[https://apoalpyca.readthedocs.io/en/latest/|http://APOalpyca.readthedocs.io]].

dome1m also runs an ASCOM Alpaca SafetyMonitor, APOSafety. This returns True/False for issafe() by querying 25m and 35m dome status: if either is open, then issafe() returns True. APOSafety also has some custom supported actions to query 25m and 35m dome status independently, and has an Action('override',t) method that sets a timer within which issafe() returns True, regardless of 25m and 35m status.

The Alpyca servers runs on port 5555 on dome1m. To start the Alpaca devices after a reboot:
<code>
cd APOalpyca/device 
python dome_app.py
</code>

===== Windows installations =====

* Install Windows 11 from NMSU USB. Account SONG
  * Install TigerVNC 64-1.13.1 and TigerVNC-winvnc-1.13.1
  * Install ASCOM 6.6SP2 (installs .NET 3.5)
  * Install ASCOM Remote 6.7.1
  * Install Anaconda3
  * OpenVPN?
  * Cisco AnyConnect?
  * git for Windows

Modify timezone, location, display setting (for remote VNC), firewall as needed (for ASCOM Remote and TigerVNC), and power settings (don't let computer go to sleep).

For the spectrograph spec1m computer:
     QHY AllInOne StableVer.20220817
     pywinusb to control USB relay
     visual studio C++ for alpyca install to run watchdog
     pip install yoctopuce for thermocouple

For the telescope pwi1m computer:
  
  Camera drivers:
     Atik drivers
     QSI drivers (if needed)

Telescope software:
     Planewave PWI4
     PWTools ?
   
  eShel software:
     pip install pyserial
     Keyspan 19H USB-to-serial
     
  TC300:
     pip install pyserial
     Thorlabs TC300B software (for desktop control)
     
  Thorlabs LTS stage:
     pip install pythonnet
     Thorlabs Kinesis software (for desktop control)
     
  Zaber stage:
     pip install zaber_motion
     Zaber launcher software (for desktop control)
     
   
 ===== Linux installation =====

<code>
dnf install tcsh emacs git gnome-tweaks tigervnc-server gcc g++ strongswan
</code>

change selinux to Permissive and reboot

<code>
useradd --uid 1013 -d /home/song song
passwd song
add song user into /etc/tigervnc/vncserver.users
systemctl enable --now vncserver@:2
firewall-cmd --permanent --add-service=vnc-server
firewall-cmd --reload

</code>

set up ipsec.secrets, ipsec.conf
<code>
systemctl enable --now strongswan
</code>

anaconda
  install
  
add ssh-key
<code>
git clone APOsong.git
cd APOsong
pip install -e .
</code>
  alpyca, tkthread, importlib_resources, photutils, astroscrappy, astroplan, astroquery, influxdb_client
  
postgres 13
<code>
dnf install postgresql postgresql-server
</code>

pgadmin4
<code>
dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
dnf install -y https://ftp.postgresql.org/pub/pgadmin/pgadmin4/yum/pgadmin4-redhat-repo-2-1.noarch.rpm
dnf install pgadmin4
</code>

Need to set up correct password authetication in pg_hba.conf and postgresql.conf

influxdb
<code>
# RedHat and CentOS
# Add the InfluxData key to verify downloads
curl --silent --location -O \
https://repos.influxdata.com/influxdata-archive_compat.key \
&& echo "393e8779c89ac8d958f81f942f9ad7fb82a25e133faddaf92e15b16e6ac9ce4c  influxdata-archive_compat.key" \
| sha256sum --check - && cat influxdata-archive_compat.key \
| gpg --dearmor \
| tee /etc/pki/rpm-gpg/RPM-GPG-KEY-influxdata > /dev/null

# Add the InfluxData repository to the repository list.
basearch=x86_64
cat <<EOF | tee /etc/yum.repos.d/influxdata.repo
[influxdata]
name = InfluxData Repository - Stable
baseurl = https://repos.influxdata.com/stable/${basearch}/main
enabled = 1
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-influxdata
EOF

# Install influxdb
dnf install influxdb2

systemctl start influxdb
</code>

influxdb-client
<code> 
  # amd64
  cd /home/local/influx
  wget https://dl.influxdata.com/influxdb/releases/influxdb2-client-2.7.5-linux-amd64.tar.gz
  tar xvf influxdb2-client-2.7.5-linux-amd64.tar.gz
  cp influx /usr/local/bin
</code>

<code>
   influx config create --config-name aposong --org NMSU --host-url http://localhost:8086 --token ***
</code>

For fresh install from backup
<code>
rm -rf /var/lib/influxdb
systemctl restart influxdb
influxdb setup --token ADMIN_TOKEN
influx restore /path/to/backup/  --full
</code>

grafana

<code>
wget -q -O gpg.key https://rpm.grafana.com/gpg.key
sudo rpm --import gpg.key
wget -q -O gpg.key https://rpm.grafana.com/gpg.key
rpm --import gpg.key
vi /etc/yum.repos.d/grafana.repo
dnf install grafana
systemctl daemon-reload
systemctl start grafana-server
</code>

Copy existing grafana.db  to /var/lib/grafana

Edit summary

Note: By editing this page you agree to license your content under the following license: CC Attribution-Share Alike 4.0 International