Differences

This shows you the differences between two versions of the page.

--- uc:uc20250902_september_2_2025 [2025/08/26 22:01]
nchanove created
+++ uc:uc20250902_september_2_2025 [2025/09/02 18:49] (current)
nchanove
@@ Line 3: / Line 3: @@
 ----
-UPDATE EVERYTHING BELOW THIS:
+Attending: Nancy Chanover (NMSU), Bill Ketzeback (APO), Misty Bentz (GSU), Aleksandr Mosenkov (BYU), Anne Verbiscer (UVa), Michael Hayden (OU), Kevin Schlaufman (JHU), Joanne Hughes (Seattle U), Moire Prescott (NMSU), Chip Kobulnicky (UWy), Ben Williams (UW)
-Attending: Nancy Chanover (NMSU), Bill Ketzeback (APO), Sarah Tuttle (UW), Chip Kobulnicky (UWy), Ben Williams (UW), Michael Hayden (OU), Misty Bentz (GSU), Eric Bellm (UW), Moire Prescott (NMSU)
 === User feedback and comments from institutional representatives ===
-  * NAPG - no report
-  * UVa - nothing to report (by email)
-  * Oklahoma - nothing to report
-  * Washington - nothing to report
-  * Seattle - nothing to report (by email)
-  * NMSU - There is still an interest in trying to understand/mitigate the abnormally high background seen when taking long (20-30 min) exposures with KOSMOS. It manifests as a snow pattern in the background pixels. The engineering staff is still working to address this issue but if other users have seen this in their data it would be really helpful to know.
-  * JHU - nothing to report (by email)
-  * Colorado - nothing to report (by email)
   * Georgia State - nothing to report
   * Wyoming - nothing to report
-  * BYU - nothing to report (by email)
+  * NMSU - nothing to report
+  * Washington - nothing to report (by email)
+  * Oklahoma - nothing to report
+  * UVa - Anne asked about whether there are any backup systems in place to avoid another occurrence of the recent network issue. This was discussed in more detail below.
+  * JHU - nothing to report
+  * BYU - nothing to report
+  * Seattle - Joanne reported that the dark current on KOSMOS has noticeably increased from April to now. See KOSMOS discussion in the below site report for more details.
+  * Colorado - no report
+  * NAPG - no report
 ----
@@ Line 28: / Line 26: @@
 The detailed site report is included below, followed by additional information discussed during today's meeting.
-// 3.5-m Telescope and Instruments Highlights, 7/02/25 – 08/05/25 //
+// 3.5-m Telescope and Instruments Highlights, 8/06/25 – 09/02/25 //
 ) Overview
-Summer shutdown activities occupied the first half of the month. Surprisingly in the second half of July we were able to get several nights of on-sky science despite the monsoon weather.
+The month of August experienced a mix of late afternoon monsoon storms that either delayed opening or affected many of the A half observing programs. We had two visiting instrument teams from U. Virginia this month. There was a failure of one of the main site network subnets that prevented observing and took down site phones as well as communications via TUI. Email was able to be received and sent from off site but not on site. We tracked the problem to a failure of a managed network switch and we were able to replace it with a spare. There was some difficulty loading the management given the switches were different models. The root cause is still not well understood as it looked like a cascading failure including the site DHCP server that also resulted in loss of communications with the site DNS servers. The problem took a little over 24 hours to resolve.
-Reminder: We have position openings for 3.5m Chief Telescope Technologist (Engineer) and Mechanical Engineer, Lead. These positions close on August 15 so please spread the word.
+The hiring committees for the Telescope Engineer and Night Operation positions are in the process of reviewing candidates. There were a large number of applications for each opening.
-Mechanical Engineer: [[https://careers.nmsu.edu/jobs/mechanical-engineer-ld-remote-locations-new-mexico-united-states]]
+) Operations
-Chief Telescope Technologist: [[https://careers.nmsu.edu/jobs/chief-telescope-technologist-remote-locations-new-mexico-united-states]]
+.5m Telescope: Telescope is working as expected. Seasonal motion errors have been infrequent. A full remap of the tertiary rotation was redone and this seems to have corrected the reported position errors from last month's report.
-We will be hiring only one person but have two postings to reflect different academic degrees and experience levels; interested candidates should apply for the one that best matches their skill set and preparation.
+.5m Telescope: Telescope is working as expected.  ACP Library support errors are still occurring even with the dcam-spare camera swap but with much lower frequency. Dcam-spare had a loss of CCD chamber integrity and was forming ice on the detector while cold. The camera was serviced twice but now appears to have water spots on or near the detector surface. Cary Smith will attempt to clean but it is not without risk to the bonding wires of the detector. University of Virginia tested another surplus Apogee camera during their coordinated observing run with DSSI on the 3.5m. The off-axis guider has not been very reliable, requiring power cycles.
-) Operations
+KOSMOS: System is cooled and stable. Increased dark current for long exposures has been confirmed. We attempted a vacuum servicing of the cryostat but without much luck. We anticipate having to open the vacuum vessel in the clean room sometime during Quarter 4 to investigate possible thermal shorts or opens. kcamera-ICC was backed up over shutdown.
-.5m Telescope: Telescope is working as expected. Seasonal motion errors have been more frequent. The tertiary rotation mechanism has experienced some issues after a software remapping of port positions. Further position refinements may be required.
+ARCTIC: The diffuser rotation mechanism is still unreliable even after a full servicing. Troubleshooting is continuing. The mechanism that moves the diffuser in and out of the optical path is still functional in the meantime. The rest of the instrument is cooled.
-.5m Telescope: Telescope is working as expected. Users have experienced several camera problems with both FlareCam and dcamera-spare related to frosting/icing or inability to cool properly. Cary serviced dcamera-spare by replacing both cooling fans and recharging the CCD vessel with gas. The off-axis guider has not been very reliable, requiring power cycles.
+Agile: The camera is non-operational, the thermoelectric cooler failed again. The camera is warm and we do not plan further repair work with this camera. We are planning on decommissioning the camera. The agile instrument rotator is still not performing nominally and we are troubleshooting it further to prepare for SoonerCam.
-KOSMOS: System is cooled and stable. Increased dark current for long exposures has been confirmed. We attempted a vacuum servicing of the cryostat but without much luck. We anticipate having to open the vacuum vessel in the clean room sometime during Quarter 4 2025 to investigate possible thermal shorts or opens. kcamera-ICC was backed up over shutdown.
+ARCES: The CCD reservoir for the cooling system was brought back to temperature. IOL levels are quite good but slowly worsening. We hope that it will stabilize and reverse direction with cooler ambient temperatures as we have seen in previous fall time frames. Work continues on a replacement and modern ICC, so far testing is going well. A commissioning report is being worked on.
-ARCTIC: The diffuser rotation mechanism is still unreliable even after a full servicing. Troubleshooting is continuing. The mechanism that moves the diffuser in and out of the optical path is still functional in the meantime. The rest of the instrument is cooled. We replaced the ion pump cable and moved the controller back to the intermediate level.
+DIS: System is cooled but in an unknown state for science. Decommissioning plans have begun.
-Agile: The camera is non-operational; the thermoelectric cooler failed again. The camera is warm and we do not plan further repair work with this camera. We are planning on decommissioning the camera. The Agile instrument rotator is still not performing nominally and we are troubleshooting it further to prepare for SoonerCam.
+NICFPS: System is cooled and usable. The ICS software has needed multiple restarts over the past month. The cause is unclear at this time.
-ARCES: The CCD reservoir for the cooling system is currently warm and not usable for science. We are keeping the shield reservoir cold. During shutdown we did a vacuum servicing but shortly after cooling the inter-order light showed elevated contamination. We have taken it out of service to address this problem. Work continues on a replacement and modern ICC, and so far testing is going well. A commissioning report for the new ICC is being worked on.
+TripleSpec: System is cooled and usable.
-DIS: System is warm and out of service. Decommissioning has begun.
+APOLLO: The instrument is usable for laser ranging.
-NICFPS: System is cooled and usable. The ICC was backed up fully during shutdown.
+----
-TripleSpec: System is cooled and usable. The ICCs (tspec and tcamera) were backed up during shutdown. Tcamera-ICC experienced another failure of one of its RAID hard drives and it needed to be rebuilt.
+=== Additional site, telescope and instruments discussion ===
-APOLLO: A team from NASA returned to site this past month to address issues with optical alignment and laser power issues. The instrument is again available for laser ranging but additional work will need to be done upgrading the power measurement equipment and working on some of the thermal metrology.
+There was a recent failure of the main site network's subnet switch, which took out the site phones and 1075 network. This switch handles the common communications network for all site communications, on-site computers, and non-telescope-specific instruments and computers. The switch was replaced and it took about 24 hours to get everything back up and running correctly. It was an old switch so we had no direct spare; it was replaced with another managed switch. These need to be managed switches because they handle communications that pass through some other specific configurations for wifi security and other security protocols. Because it was a managed switch it was not a simple plug-and-play quick swap; the major part of the recovery was to get the network back up and running. Although this was all accomplished by 4 pm the next day, it did unfortunately result in one night of down time. It would be cost prohibitive to reengineer the entire network so there are no plans to overhaul the site network configuration. We are trying to source a replacement for the switch that we used as a spare and will evaluate what other steps we can take to ensure that we are better prepared for this kind of situation if it arises in the future.
-All-Sky camera: 3.5m engineering staff has been working with the ALCOR manufacturer to remedy camera and software problems. Troubleshooting continues.
+Regarding the open positions, the hiring process is moving forward. Also, we welcome Tim McQuaid, a third-year graduate student at NMSU, who will be replacing Mark Croom as the emergency fill-in observing specialist. Tim has already begun his training so you will likely encounter him soon.
-----
+We continue to have challenges with the camera(s) on the 0.5m. It is possible that the DIS camera can be used once the decommissioning is further along. University of Virginia brought a spare Apogee camera for use on ARCSAT; it is still not known whether that can be used by the broader ARCSAT user community.
-=== Additional site, telescope and instruments discussion ===
+The KOSMOS vacuum servicing didn't resolve the issue of high dark current so we will have to open up the cryostat to see if we can figure out what is causing a decreased hold time or increase in dark current. We are currently planning to do this during Q4; the science schedule indicates that it may not be possible until early December unless the issue becomes critical and we need to take the instrument out of service. Chip reported that a recent UWy user suggested that the dark current is currently so high that the instrument is unusable for faint targets without doubling or tripling the exposure time. Nancy will follow up directly with that user to ascertain if the problem has worsened. This would be an argument for servicing KOSMOS sooner, but we need to balance that with the numerous class visits that will be scheduled in the first two months of Q4, most of which have requested to use KOSMOS.
+ARCES had been displaying increased inter-order light that would indicate some form of contamination in the light path of the CCD. We warmed up half of the cryostat while keeping the shield cold so that the water vapor would migrate to shield. That did improve things significantly but subsequently things degraded somewhat; we are now hoping that it will stabilize or reverse direction. The upgrade to a modern computer for the ARCES ICC is underway.
-xxx add stuff here xxx
+NICFPS and TripleSpec are operating as expected. DIS is powered down. APOLLO has been laser ranging and we were recently able to get all 5 retroreflectors but it is still not at peak performance.
 ----
@@ Line 76: / Line 76: @@
 === 2025 Q3 3.5m scheduling ===
-The 3.5m schedule for Q3 has been in flux given the recent echelle engineering. Users who lost observing time due to the echelle servicing or weather are especially encouraged to request OPEN/DD time in Q3. To request this time please follow the standard procedure by emailing your request to Ben Williams, Russet McMillan, Amanda Townsend, Nancy Chanover, and your institutional scheduler.
+There is some OPEN time remaining in September (last month of Q3). We have received requests for some of it, but not all. To request this time please follow the standard procedure by emailing your request to Ben Williams, Russet McMillan, Amanda Townsend, Nancy Chanover, and your institutional scheduler. Be sure to include the specific slot you are requesting (or specify that any time slot will do) and a short justification.
 ----
@@ Line 82: / Line 82: @@
 === 2025 Q4 3.5m scheduling ===
-All allocations for Q4 have been sent to the institutional schedulers; proposals are due on August 25. Note that the allocation emails stated that our ability to offer a virtual training in Q4 is uncertain due to staffing limitations; we indicated that we MAY offer it in Q4.
+The Q4 schedule is in the works; we expect to have a first draft in about a week.
 ----
@@ Line 88: / Line 88: @@
 === 2025 Q3 0.5 scheduling ===
-Next week is the only open week remaining in the ARCSAT Q3 schedule.
+September 8-17 (next week!) is the only open time remaining in the ARCSAT Q3 schedule.
 ----
@@ Line 94: / Line 94: @@
 === Status of new instrument initiatives ===
-  * SoonerCam: we received a modest number of survey responses and will proceed using the information we have. The camera was purchased and a meeting is scheduled in two weeks to discuss the next steps.
+  * SoonerCam: we are ordering parts!
-  * KOSMOS grisms: we are still awaiting a decision on the proposal we submitted for funds to support the purchase of three new grisms for KOSMOS.
+  * KOSMOS grisms (no change from last month): we are still awaiting a decision on the proposal we submitted for funds to support the purchase of three new grisms for KOSMOS.
-  * ASPEN: we are still awaiting a decision on the proposal we submitted for funds to support the development of the new echelle spectrograph. Work on the design of the new optical echelle spectrograph continues with the goal of completing the Preliminary Design Review in early 2026.
+  * ASPEN (no change from last month): we are still awaiting a decision on the proposal we submitted for funds to support the development of the new echelle spectrograph. Work on the design of the new optical echelle spectrograph continues with the goal of completing the Preliminary Design Review in early 2026.
-  * Ocotillo: a number of purchase orders were executed and we expect the first batch of hardware to come in within the next month or so.
+  * Ocotillo (no change from last month): a number of purchase orders were executed and we expect the first batch of hardware to come in within the next month or so.
+----
+=== Web site updates ===
+We are close to having something operational. All of the requested pieces and updates were implemented. Although it is not yet finalized, users are encouraged to start using it: [[https://mainapo.apo.nmsu.edu]] UC reps: please notify your users that they should start using the new website and report any bugs to you, to be forwarded on to Ben Williams.
+----
+=== Upcoming events ===
+There are a number of events taking place over the next several months:
+  * We have 4 class groups coming APO in Q4. A fifth one will be moved to Q1. Please be patient with the site staff, who will be shorthanded during this time!
+  * There will be an in-person ASPEN team meeting at NMSU/APO in early October.
+  * We are holding a public-facing Open House at APO on Saturday October 11. This will include site and telescope tours, activities, and talks at the Sunspot Astronomy and Visitor's Center. We look forward to welcoming the public back to the site; our last Open House was in March 2020.
+  * The ARC Board of Governors meeting is on November 4; we will request science highlights from everyone at next month's meeting, with a due date in late October.
+  * The January 2026 AAS meeting will be in Phoenix, and we plan to have a booth. October 6 is the abstract deadline. The deadline for Splinter Session proposals is September 25; does anyone have an interest in an APO science-fest sort of session? UC reps: please poll your users and email Nancy with opinions.
 ----
@@ Line 108: / Line 125: @@
 New action items from this meeting:
-  * None.
+  * Nancy: follow up with UWy KOSMOS user to determine severity of dark current issue.
+  * UC reps: remind your users to review the Q3 schedule and request OPEN time in September if they can make a good case for it.
+  * UC reps: poll your users about a Jan. 2026 splinter session at the AAS for APO science highlights and networking (or other ideas!).
 ----

apo-wiki

User Tools

Site Tools

Differences

Page Tools