apo-wiki

Attending: Nancy Chanover (NMSU), Bill Ketzeback (APO),

Misty Bentz (GSU), Aleksandr Mosenkov (BYU), Anne Verbiscer (UVa), Michael Hayden (OU), Kevin Schlaufman (JHU), Joanne Hughes (Seattle U), Moire Prescott (NMSU), Chip Kobulnicky (UWy), Ben Williams (UW)

• Colorado - nothing to report
• BYU - nothing to report
• Oklahoma - nothing to report
• UVa - nothing to report
• Wyoming - nothing to report
• Georgia State - nothing to report
• NMSU - nothing to report
• Washington - nothing to report
• JHU - nothing to report (by email)
• Seattle - nothing to report
• NAPG - no report

The detailed site report is included below, followed by additional information discussed during today's meeting.

3.5-m Telescope and Instruments Highlights, 8/06/25 – 09/02/25

1) Overview

The month of August experienced a mix of late afternoon monsoon storms that either delayed opening or affected many of the A half observing programs. We had two visiting instrument teams from U. Virginia this month. There was a failure of one of the main site network subnets that prevented observing and took down site phones as well as communications via TUI. Email was able to be received and sent from off site but not on site. We tracked the problem to a failure of a managed network switch and we were able to replace it with a spare. There was some difficulty loading the management given the switches were different models. The root cause is still not well understood as it looked like a cascading failure including the site DHCP server that also resulted in loss of communications with the site DNS servers. The problem took a little over 24 hours to resolve.

The hiring committees for the Telescope Engineer and Night Operation positions are in the process of reviewing candidates. There were a large number of applications for each opening.

2) Operations

3.5m Telescope: Telescope is working as expected. Seasonal motion errors have been infrequent. A full remap of the tertiary rotation was redone and this seems to have corrected the reported position errors from last month's report.

0.5m Telescope: Telescope is working as expected. ACP Library support errors are still occurring even with the dcam-spare camera swap but with much lower frequency. Dcam-spare had a loss of CCD chamber integrity and was forming ice on the detector while cold. The camera was serviced twice but now appears to have water spots on or near the detector surface. Cary Smith will attempt to clean but it is not without risk to the bonding wires of the detector. University of Virginia tested another surplus Apogee camera during their coordinated observing run with DSSI on the 3.5m. The off-axis guider has not been very reliable, requiring power cycles.

KOSMOS: System is cooled and stable. Increased dark current for long exposures has been confirmed. We attempted a vacuum servicing of the cryostat but without much luck. We anticipate having to open the vacuum vessel in the clean room sometime during Quarter 4 to investigate possible thermal shorts or opens. kcamera-ICC was backed up over shutdown.

ARCTIC: The diffuser rotation mechanism is still unreliable even after a full servicing. Troubleshooting is continuing. The mechanism that moves the diffuser in and out of the optical path is still functional in the meantime. The rest of the instrument is cooled.

Agile: The camera is non-operational, the thermoelectric cooler failed again. The camera is warm and we do not plan further repair work with this camera. We are planning on decommissioning the camera. The agile instrument rotator is still not performing nominally and we are troubleshooting it further to prepare for SoonerCam.

ARCES: The CCD reservoir for the cooling system was brought back to temperature. IOL levels are quite good but slowly worsening. We hope that it will stabilize and reverse direction with cooler ambient temperatures as we have seen in previous fall time frames. Work continues on a replacement and modern ICC, so far testing is going well. A commissioning report is being worked on.

DIS: System is cooled but in an unknown state for science. Decommissioning plans have begun.

NICFPS: System is cooled and usable. The ICS software has needed multiple restarts over the past month. The cause is unclear at this time.

TripleSpec: System is cooled and usable.

APOLLO: The instrument is usable for laser ranging.

There was a recent failure of the main site network's subnet switch, which took out the site phones and 1075 network. This switch handles the common communications network for all site communications, on-site computers, and non-telescope-specific instruments and computers. The switch was replaced and it took about 24 hours to get everything back up and running correctly. It was an old switch so we had no direct spare; it was replaced with another managed switch. These need to be managed switches because they handle communications that pass through some other specific configurations for wifi security and other security protocols. Because it was a managed switch it was not a simple plug-and-play quick swap; the major part of the recovery was to get the network back up and running. Although this was all accomplished by 4 pm the next day, it did unfortunately result in one night of down time. It would be cost prohibitive to reengineer the entire network so there are no plans to overhaul the site network configuration. We are trying to source a replacement for the switch that we used as a spare and will evaluate what other steps we can take to ensure that we are better prepared for this kind of situation if it arises in the future.

Regarding the open positions, the hiring process is moving forward. Also, we welcome Tim McQuaid, a third-year graduate student at NMSU, who will be replacing Mark Croom as the emergency fill-in observing specialist. Tim has already begun his training so you will likely encounter him soon.

We continue to have challenges with the camera(s) on the 0.5m. It is possible that the DIS camera can be used once the decommissioning is further along. University of Virginia brought a spare Apogee camera for use on ARCSAT; it is still not known whether that can be used by the broader ARCSAT user community.

The KOSMOS vacuum servicing didn't resolve the issue of high dark current so we will have to open up the cryostat to see if we can figure out what is causing a decreased hold time or increase in dark current. We are currently planning to do this during Q4; the science schedule indicates that it may not be possible until early December unless the issue becomes critical and we need to take the instrument out of service. Chip reported that a recent UWy user suggested that the dark current is currently so high that the instrument is unusable for faint targets without doubling or tripling the exposure time. Nancy will follow up directly with that user to ascertain if the problem has worsened. This would be an argument for servicing KOSMOS sooner, but we need to balance that with the numerous class visits that will be scheduled in the first two months of Q4, most of which have requested to use KOSMOS.

ARCES had been displaying increased inter-order light that would indicate some form of contamination in the light path of the CCD. We warmed up half of the cryostat while keeping the shield cold so that the water vapor would migrate to shield. That did improve things significantly but subsequently things degraded somewhat; we are now hoping that it will stabilize or reverse direction. The upgrade to a modern computer for the ARCES ICC is underway.

NICFPS and TripleSpec are operating as expected. DIS is powered down. APOLLO has been laser ranging and we were recently able to get all 5 retroreflectors but it is still not at peak performance.

There is some OPEN time remaining in September (last month of Q3). We have received requests for some of it, but not all. To request this time please follow the standard procedure by emailing your request to Ben Williams, Russet McMillan, Amanda Townsend, Nancy Chanover, and your institutional scheduler. Be sure to include the specific slot you are requesting (or specify that any time slot will do) and a short justification.

The Q4 schedule is in the works; we expect to have a first draft in about a week.

September 8-17 (next week!) is the only open time remaining in the ARCSAT Q3 schedule.

• SoonerCam: we are ordering parts!
• KOSMOS grisms (no change from last month): we are still awaiting a decision on the proposal we submitted for funds to support the purchase of three new grisms for KOSMOS.
• ASPEN (no change from last month): we are still awaiting a decision on the proposal we submitted for funds to support the development of the new echelle spectrograph. Work on the design of the new optical echelle spectrograph continues with the goal of completing the Preliminary Design Review in early 2026.
• Ocotillo (no change from last month): a number of purchase orders were executed and we expect the first batch of hardware to come in within the next month or so.

We are close to having something operational. All of the requested pieces and updates were implemented. Although it is not yet finalized, users are encouraged to start using it: https://mainapo.apo.nmsu.edu UC reps: please notify your users that they should start using the new website and report any bugs to you, to be forwarded on to Ben Williams.

There are a number of events taking place over the next several months:

• We have 4 class groups coming APO in Q4. A fifth one will be moved to Q1. Please be patient with the site staff, who will be shorthanded during this time!
• There will be an in-person ASPEN team meeting at NMSU/APO in early October.
• We are holding a public-facing Open House at APO on Saturday October 11. This will include site and telescope tours, activities, and talks at the Sunspot Astronomy and Visitor's Center. We look forward to welcoming the public back to the site; our last Open House was in March 2020.
• The ARC Board of Governors meeting is on November 4; we will request science highlights from everyone at next month's meeting, with a due date in late October.
• The January 2026 AAS meeting will be in Phoenix, and we plan to have a booth. October 6 is the abstract deadline. The deadline for Splinter Session proposals is September 25; does anyone have an interest in an APO science-fest sort of session? UC reps: please poll your users and email Nancy with opinions.

Open action items from previous meetings:

• UC reps: ask your users to complete the survey from FAWG (per Nancy’s email sent on 7/23/25). Status: CLOSED.

New action items from this meeting:

• Nancy: follow up with UWy KOSMOS user to determine severity of dark current issue.
• UC reps: remind your users to review the Q3 schedule and request OPEN time in September if they can make a good case for it.
• UC reps: poll your users about a Jan. 2026 splinter session at the AAS for APO science highlights and networking (or other ideas!).

None.

The next meeting will be on October 7, 2025 at 10:30 MDT.