apo-wiki

Attending: Nancy Chanover (NMSU), Bill Ketzeback (APO), Misty Bentz (GSU), Anne Verbiscer (UVa), Aleksandr Mosenkov (BYU), Kevin Schlaufman (JHU), Eric Bellm (UW), Jon Holtzman (NMSU), Moire Prescott (NMSU), Chip Kobulnicky (UWy), Michael Hayden (OU), Ben Williams (UW)

Nancy started by announcing the newest member of the 3.5m observing team: Harrison (Harry) Abbot started his position at APO a few weeks ago. We will introduce him at a later date, although some of you may meet him online (over TUI) in the meantime.

• JHU - The longitude/latitude/elevation of APO used to be easily findable at the top of the APO home page. This information is important for many users so please add it back in. Nancy agreed and will ensure that this gets done.
• NMSU - A user recently reported seeing a higher dark current rate with KOSMOS; the values were elevated from what they saw in November-December after the last repair. This is seen in longish (1200 sec) exposures, although it seems to be present in the Dark frames but not necessarily in the science frames (which leads to an over-subtraction when removing the dark current). The APO staff will take a look at this in more detail and will use this information as they develop their larger repair schedule.
• Oklahoma - nothing to report
• Wyoming - nothing to report
• Williams - no report
• BYU - nothing to report
• UVa - nothing to report from users; Anne thanked APO for keeping the legacy website up. We discussed the website later in the agenda.
• Georgia State - one user responded to Misty's request for feedback regarding having APO data made available in a public archive (after a proprietary period). The user said it would be really helpful for them as they try to locate historical data taken from APO.
• Colorado - no report
• Washington - nothing to report
• Colgate - no report

The detailed site report is included below, followed by additional information discussed during today's meeting.

3.5-m Telescope and Instruments Highlights, 01/14/26 – 02/02/26

1) Overview

The first half of January was typical wintery weather for APO, with a mix of clouds and 2 inches of fresh, powdery snow that melted within two days. During this same storm, we experienced our coldest weather of the season, down to -12C with winds above the closure limits, making for a couple of very cold nights. The last week of the month, however, Winter Storm Fern, which had affected much of the United States, left the observatory under a blanket of ice and 20+ inches (50 cm) of heavy, wet snow. Utility power to the site was interrupted. APO remained on emergency generator power for a week. Multiple trees were down on utility lines on the way to the observatory (and around the mountain), resulting in more than a dozen power poles damaged or sheared off over a 5-kilometer-long path. It took electric crews a week to replace poles and power lines in difficult terrain and deep snow before electricity was restored to communities along the Sunspot Highway to Timberon. Linemen from several electric companies were contracted to assist in the repairs. Utility power was restored to the Observatory this past Thursday evening (on January 29), but significant snow drifts and ice buildup remained on the 3.5m enclosure over the weekend. Staff have been orienting the enclosure to maximize sun exposure over the weekend to clear the roof.

We had an on-site documentary filmmaker from the Fisk Planetarium visit during this time period.

2) Operations

3.5m Telescope: The telescope has been performing nominally. We experienced another instance this month of a collimation failure due to software time-outs, when one of the actuators for the secondary mirror did not reach or move to the commanded position. The next move seemed to correct itself. We have identified additional instances of the same error over the past several months, which we are investigating further. The problem is intermittent, making troubleshooting difficult because it cannot be reproduced.

0.5m Telescope: Working as expected. Dcam and UVAcam are both available options for users.

KOSMOS: System is cold and stable. Instrument is performing nominally.

ARCTIC: The diffuser rotation mechanism is still unreliable. We have manufactured some new parts, but have not gotten the opportunity to install and test. The mechanism that moves the diffuser in and out of the optical path is still functional in the meantime. The L1 lens was cleaned, and a full vacuum servicing was performed in January. Work was delayed, and some planned tasks were cancelled due to weather conditions.

SoonerCam: The Agile instrument rotator is still not performing nominally, and we will continue troubleshooting to prepare for SoonerCam. SoonerCam parts are on hand, assembled, and software development has begun. Software development is in the early stages to learn the Hamamatsu and Brandy Wine software APIs.

ARCES: There was another intermittent failure of the thorium argon lamp power supply, resulting in lamp dropouts or weak lines. Another instance of trouble with the ThAr calibrations was not the lamp power supply, but the calibration flip mirror not moving into place. If identified by an echelle user during their cals, the observing specialists have procedures to identify which is occurring and adjust the power supply or move the mirror by hand. Users are advised to carefully examine their calibration frames.

DIS: System is powered down and warm. Decommissioning plans have begun. The instrument has been decabled, and we will be lowering it off the observing floor soon.

NICFPS: System is cooled and usable.

TripleSpec: System is cooled and usable.

APOLLO: The instrument is usable for laser ranging. Maintaining the instrument's temperature during extreme environmental conditions remains challenging. The team has experienced a streak of poor weather and software connection issues over the past month.

The recent winter storm that affected a large swath of the US dumped heavy rain for 6-8 hours, which then froze, followed by about 2 feet of snow. Trees were weighted down by frozen ice/snow, which led to a number of downed limbs, trees, and power poled. The site lost power and was on generator power for about a week. Fortunately, the generator worked as expected the entire week. We are now open and can observe again as of last night (Monday February 2).

There was a documentary filmmaker on site for an SDSS planetarium presentation.

There was a telescope issue (described above); we are still trying to determine if it is a hardware or software issue.

Yesterday we learned of the recent elevated dark current for KOSMOS (see above), which we will investigate ASAP.

A servicing of ARCTIC was completed last month, including cleaning the L1 lens and vacuum servicing. Additional planned work on the diffuser rotation mechanism was halted because of the bad weather. Those potential upgrades are still in the works.

The early software development for SoonerCam is still ongoing.

We are still having issues with the ARCES ThAr lamps; users should carefully examine their calibration frames and report any unusual behavior to the observing specialist. The ThAr and possibly quartz frames will be blank or have very low signal when things are not functioning correctly.There are two failure modes and the obs-specs know how to identify each one and what to do to remedy it.

There are no current issues with TripleSpec and NICFPS. The DIS and Agile decommissioning is still ongoing. APOLLO has been having some thermal issues due to the cold weather, which we are continuing to troubleshoot.

There is an open position for a mechanical engineer to join the 3.5m engineering group. This is at a more junior level than the Chief Telescope Engineer position that was advertised previously. Please share this advertisement with any qualified candidates! https://careers.nmsu.edu/jobs/mechanical-engineer-remote-locations-new-mexico-united-states

The Q1 schedule is posted. It is pretty full at this point, although there is an available B half (DD01) on February 11. Users interested in requesting this time should send the request (including a program ID if the project already has been assigned time this quarter, otherwise a short proposal) to Ben, Nancy, Russet, Amanda, and their institutional schedulers.

The allocations to institutional schedulers were sent out last night. Proposals are due on February 24. We are aware of one class group that plans to visit APO during Q2 but if there are others who have not yet notified us please do so ASAP.

The ARCSAT Q1 schedule is posted; there is one open week remaining at the end of the quarter.

Jon Holtzman led a discussion of the APO wiki (http://astronomy.nmsu.edu/apo-wiki/doku.php?id=start), which he has been cleaning up recently. While the instrument manuals on the APO website are fine for learning basic properties and operational procedures of the instruments, information about “features” or “issues” for each instrument (e.g those related to data processing or tricks/tips for observing in a very specific mode) are not held in a common repository despite the vast experience spread around the ARC user community. This would be particularly useful for new users, students, etc. The user wiki was designed for this, but it has been fairly stagnant lately. Jon is leading an effort to revitalize the wiki and populate it with things that users should be aware of and provide some descriptions of reduction issues and software options. Therefore Jon is trying to collect 1) a list of features/issues for each instrument, provided by users, and 2) a list of the software options for data reduction for each instrument (no judgment here - truly, what are people using?). He needs help populating these two sections for each instrument! If you have and information regarding either of these items, please email it to Jon Holtzman (holtz@nmsu.edu). This will ultimately help all users achieve the pest possible performance from the 3.5m facility instruments for their science.

Jon Holtzman is also working on incorporating modules into PypeIt for reducing spectra from 3.5m instruments. PypeIt is designed primarily for data from slit spectrographs, and we have received funding as part of an NSF award to develop a PypeIt implementation for KOSMOS, ARCES, and TripleSpec. Initial modules have been created for all three instrument; ARCES is challenging because of its short slit, but the versions for KOSMOS and TripleSpec are nearing usability for testing. The PypeIt group does an excellent job in version controlling their software; in order for a new module to be incorporated into the overall distribution version, new modules need to be supplied with test data so that any time modifications are made to the code base they can be applied to the test data. In order to move us closer towards being able to implement the new APO-specific modules, we are looking for users in the ARC community that have familiarity w/PypeIt for non-APO instruments and are willing to test out our development modules. There are lots of parameters that can be tuned, and we're looking for people who have experience in doing so. UC reps: please query your users and have them contact Jon if they are willing to help with this.

• SoonerCam: See above site report and subsequent discussion.
• KOSMOS grisms: We received a number of useful responses to the recent survey - THANK YOU! The low-res grism is the highest priority for a significant majority of the respondents so we will move forward with purchasing that grism immediately as we continue to seek funds for the purchase of additional grisms.
• ASPEN: No major updates this month. We still plan to hold the Preliminary Design Review in Q2 2026.
• Ocotillo: No major updates this month. The TEG will be hiring an engineer/technician at the BS/MS level very soon; interested parties should look for the job ad on the UW hiring site shortly.

Nancy stated that we'd like to understand what users are not able to find/do on the new website in an effort to encourage users to move to that one. One user suggested getting rid of the drop-down menus on the front page and restoring all the links to the front page. Or at minimum, reducing the number of clicks that it takes to reach specific information. Another user mentioned that there are no Exposure Time Calculators for any of the instruments. Bill stated that we have tried for the last 15 years to get ETCs for all instruments but these have not been completed. Another user mentioned that there is no front page that gives an overview of all the instrument. We concluded that an efficient way to address some of these issues is to schedule a hackathon where users could meet in real time with the web development team to discuss and implement suggestions. Nancy and Ben will discuss this further with the IT staff.

—-

Open action items from previous meetings:

• Joanne Hughes: put the KOSMOS MOS data reduction software on the wiki. Joanne is waiting for an update from her colleagues at the College of Idaho. Update: Nancy connected with the folks working on this while at the AAS meeting so we are getting closer to have this software in hand and made available to the

user community through the wiki. Status: Holtzman put link to CofI software on the wiki, and notes that this is a wrapper for lower level reduction stuff in pyVista. Additional descriptive text/examples would stil be welcome and good. CLOSED

• UC reps: remind your users to review the Q1 schedule and request OPEN/DD01 time if they can make a good case for it. Status: OPEN.
• UC reps: remind your KOSMOS users to complete this survey no later than Friday January 30 to register their final preferences for which new grism option(s) are most important for their science: https://forms.gle/Fr6LVu9o8Up782hW7 . Status: CLOSED.

New action items from this meeting:

• UC reps: ask your users to provide instrument usage/data reduction tips and tricks to Jon Holtzman for inclusion on the wiki.
• UC reps: ask your users if they have experience using PypeIt with non-APO instruments; if so please contact Jon.
• Nancy: ensure that observatory lat/long/elevation gets put back on new APO home page
• Nancy: explore web page hackathon idea with APO webdev team.

None.

The next Users Committee meeting will be on Tuesday March 03 at 10:30 MST.

Attending: Nancy Chanover (NMSU), Kristian Finlator (NMSU), Bill Ketzeback (APO), Michael Hayden (OU), Aleksandr Mosenkov (BYU), Ben Williams (UW), Misty Bentz (GSU), Kevin Schlaufman (JHU), Anne Verbiscer (UVa), Amanda Townsend (APO), Adam Kowalski (CU), Chip Kobulnicky (UWy), Sarah Tuttle (UW), Shane Thomas (APO)

• Wyoming - nothing to report
• Williams - nothing to report (by email)
• Washington - nothing to report (by email)
• UVa - Anne is currently unable access the observing schedule on the new website and does not see a way to access the old website as was promised last month (this was discussed later in the meeting)
• Oklahoma - nothing to report
• NMSU - nothing to report
• JHU - nothing to report
• Georgia State - nothing to report
• Colorado - nothing to report
• Colgate - no report
• BYU - Aleksandr reported that a user who tried to sign up for an account on the new website discovered that BYU was not listed as an institution option (this was discussed later in the meeting)

The detailed site report is included below, followed by additional information discussed during today's meeting.

3.5-m Telescope and Instruments Highlights, 12/02/25 – 01/13/26

1) Overview

The past month was a mix of excellent skies and very unsettled weather. About a third of the time was lost due to clouds, wind, and a few days when we could not open the enclosure due to an ice and snow storm.

We had one student training visit during this time period.

2) Operations

3.5m Telescope: The telescope is working quite well. We experienced one instance of collimation failure when one of the actuators for the secondary mirror did not reach or fail to move to the commanded position. The problem was corrected by homing the mirror. A moderate correction to collimation was applied during engineering last week.

0.5m Telescope: Telescope is working as expected. Dcam-spare was having issues with the CCD chamber frosting up. Even after multiple attempts, it could not be cleaned or made reliable. We pulled Dcam off of DIS and put it into service. Dcam and UVAcam are both available options for users.

KOSMOS: System is cold and stable. The instrument dispenser wheel had one instance of not moving when commanded. It was corrected by manually repositioning it. There was one night when the camera focus was not properly set, and some confusion resulted when the telescope focus seemed to be running low. It was corrected the next night. Some of the silicon wafers manufactured with slits have been damaged in their holders. They appear to have developed stress cracks from the differential thermal expansion of the aluminum holders. We are currently seeking companies that may be able to manufacture spares.

ARCTIC: The diffuser rotation mechanism is still unreliable even after a full servicing. The mechanism that moves the diffuser in and out of the optical path is still functional in the meantime. There was another instance this month of the filter wheel controller losing connection and requiring a reset. On another night, the instrument refused to take an exposure because another one was in progress that would not clear. An init resulted in a dropped upper left quad. The observer got back on the sky, using single amplifier mode. During the holiday break, a failure of the ion pump resulted in a loss of vacuum. The repair required a swap of the ion pump, putting the instrument back on a vacuum pump, a different ion pump controller, and a cable. Special thanks to Cary Smith, who came in over his time off to restore the instrument.

SoonerCam: The agile instrument rotator is still not performing nominally, and we will be troubleshooting it further to prepare for SoonerCam. SoonerCam parts are on hand, assembled, and software development has begun. The first biases using low-level commands were taken this past week.

ARCES: There was an intermittent failure of the thorium argon lamp power supply, resulting in lamp dropouts or weak lines. Cleaning the rheostat that controls the adjustment of amperage was a temporary fix until we can possibly repair it with a new part.

DIS: System is powered down and warm. Decommissioning plans have begun. The instrument has been decabled, and we will be lowering it off the observing floor soon.

NICFPS: System is cooled and usable.

TripleSpec: System is cooled and usable.

APOLLO: The instrument is usable for laser ranging. The team has experienced a streak of poor weather and software connection issues over the past month.

There was one student group that visited last month. We've had some winter weather and were closed for two nights due to ice on the enclosure and lightning arrestors.

As for instruments, both KOSMOS and ARCTIC each had some problems last month. There was an instance when the KOSMOS disperser wheel was commanded and then got stuck; this was remedied by the operator going up to the observing level and moving it by hand. We had one instance where the camera focus wasn't set properly and caused confusion regarding the telescope focus; this was corrected. We've seen some stress cracks in the UW-manufactured silicon wafer slits, likely due to differential expansion of the aluminum holders. We are working to identify a manufacturer for new spares/replacements.

ARCTIC had a failure of the filter wheel; this was discovered by an observing specialist in the afternoon and fixed by the engineering staff that same day so it did not impact observing. We had another instance where the instrument exposure was stuck and could not be commanded to take another. An init was done, which caused a dropped quad, and the user elected to observe in the single amplifier mode for the rest of night. We had a more serious issue with an ion pump that resulted in loss of vacuum to the instrument. This was due to a failure of a high voltage cable that became welded to the pump; both needed to be replaced. Special thanks for Cary Smith who came in during his time off during the winter break to restore the instrument.

First biases were achieved with the new SoonerCam setup. We are starting to develop the new ICC software and trying to learn the lower level commands for triggering the camera.

We've seen an intermittent failure of the ARCES ThAr lamp power supply, We cleaned the connectors and this seemed to help but the issue still sporadically appears. Users should check their ThAr exposures to make sure they have enough lines with appropriate S/N levels.

DIS decommissioning continues; the instrument is decabled but still on the observing floor for now.

There have been no issues with TripleSpec or NICFPS, and APOLLO has been working well.

We have completed hiring process for a new Support Astronomer (observing specialist). Harrison (Harry) Abbot will begin his work and training in a few days.

There will be a site infrastructure review held at APO next month. The goal is to review overall site and telescope infrastructure items (excluding instruments) in order to develop priorities for resource allocation in order to ensure that we maintain a highly functioning observatory.

The Q1 schedule is posted. There is some OPEN time available on Jan 18A and also in February and March. Users interested in requesting any of that time should send the request (including a program ID if the project already has been assigned time this quarter, otherwise a short proposal) to Ben, Nancy, Russet, Amanda, and their institutional schedulers.

The ARCSAT Q1 schedule is posted; there is one open week remaining at the end of the quarter.

• SoonerCam: See above site report and subsequent discussion.
• KOSMOS grisms: our proposal requesting funds to support the purchase of three new grisms for KOSMOS was not selected. We will move forward with the purchase of one or two grisms using ARC funds. If users have an opinion about this they should complete the following form no later than Friday January 30: https://forms.gle/Fr6LVu9o8Up782hW7
• ASPEN: we have an NSF proposal pending but learned that we did receive some funds from the State of New Mexico to further advance the design. We will move forward with building the atmospheric dispersion corrector first since it can be used with ARCES. We presented a poster about ASPEN at the Jan. 2026 AAS meeting and received some positive feedback, and we submitted several papers about the instrument for the SPIE conference this summer. We still plan to hold the Preliminary Design Review in the first half of 2026, likely now in Q2.
• Ocotillo: Things are progressing; we have the camera in hand and someone is working on camera characterization and software. We have the grating, some optics, and the fiber in hand. The TEG will be hiring an engineer/technician at the BS/MS level very soon; interested parties should look for the job ad on the UW hiring site shortly.

We flipped the switch this morning so that https://mainapo.apo.nmsu.edu became https://www.apo.nmsu.edu. This resulted in some preliminary problems as some certificates needed to be updated in order to preserve the old web site (which will be located at legacy.apo.nmsu.edu). We hope to have these sorted out by the end of the day. The links to the observing schedules are currently not working, but one can still access the Q1 schedule using this direct link: http://35m-schedule.apo.nmsu.edu/2026Q1.shtml. Shane will fix Aleksandr's reported issue ASAP. If users have difficulties finding what they need on the new site please let us know ASAP!

We had a booth alongside SDSS and SciServer so there was a strong APO presence at the Jan. 2026 AAS meeting in Phoenix. We were well positioned in the exhibit hall and received a lot of foot traffic; we gave away a lot of swag.

Open action items from previous meetings:

• Joanne Hughes: put the KOSMOS MOS data reduction software on the wiki. Joanne is waiting for an update from her colleagues at the College of Idaho. Update: Nancy connected with the folks working on this while at the AAS meeting so we are getting closer to have this software in hand and made available to the user community through the wiki. Status: OPEN.
• UC reps: please discuss the idea of archiving our 3.5m data with your users and report back with any reactions, either positive (e.g., advantages that they see to having a data archive) or negative (e.g., potential issues or concerns). No UC reps reported any feedback from users so we will be moving forward with piloting this as a next step. Status: CLOSED.

New action items from this meeting:

• UC reps: remind your users to review the Q1 schedule and request OPEN/DD01 time if they can make a good case for it.

None.

The next Users Committee meeting will be on Tuesday February 03 at 10:30 MST.