Differences

This shows you the differences between two versions of the page.

--- uc:uc2026 [2026/08/06 22:42] – nchanove
+++ uc:uc2026 [2026/10/06 19:28] (current) – nchanove
@@ Line 1: / Line 1: @@
+===== APO 3.5-m Users Committee Meeting, 10/06/2026 =====
+Attending: Nancy Chanover (NMSU), Misty Bentz (GSU), Kevin Schlaufman (JHU), Aleksandr Mosenkov (BYU), Anne Verbiscer (UVa), Moire Prescott (NMSU), Ben Williams (UW), Eric Bellm (UW), Bill Ketzeback (APO), Adam Kowalski (CU), Michael Hayden (OU), Russet McMillan (APO)
+=== User feedback and comments from institutional representatives ===
+  * Washington - nothing to report
+  * Williams - no report
+  * Colgate - no report
+  * Wyoming - no report
+  * BYU - nothing to report
+  * JHU - nothing to report
+  * Colorado - nothing to report
+  * UVa - nothing to report other than a user can't access night logs on web page after 9/20/2026; Nancy will look into it and report back
+  * Oklahoma - nothing to report
+  * Georgia State - nothing to report
+  * NMSU - nothing to report
+----
+=== Telescope and Instruments Report ===
+The detailed site report is included below, followed by additional information discussed during today's meeting.
+// 3.5-m Telescope and Instruments Highlights, 09/02/26 – 10/02/26 //
+) Overview
+Southern New Mexico has experienced a strong monsoon season this year with above-average precipitation. As described in last month's report, the NWS historically defines the end of the monsoon season as September 30; however, recent forecasts indicate that late September into early October, tropical and monsoon moisture are expected to produce heavier-than-average rainfall at the end of the official season.
+The 3.5m telescope performance this cycle was nominal, with few minor reported problems. A single motion error in altitude during afternoon checkout was encountered this reporting cycle. A problem report with the NA2 rotator was tracked down to a software version control issue and user error. No science was lost to these events.
+Instruments experiencing issues this month were primarily KOSMOS, ARCTIC, and ECHELLE. Details for each are below.
+There were no visiting classes this reporting cycle. A team from NASA was on site for their annual instrument servicing of APOLLO. One PI driven instrument program this cycle, but none of the instrument team traveled to APO, operating their instrument entirely remotely.
+) Operations
+.5m Telescope: The telescope has been performing nominally, with a few reported minor issues. There was a single reported altitude motion error, indicating that we may be past the seasonal ambient temperature when most of these occur.  One power brownout and transfer to the backup generator during the day resulted in automatic mirror covers needing to be reset. A reported problem zeroing the NA2 rotator, at the end of DSSI time, appeared to be due to an older version of the low-level engineering application used to communicate with the axis controller being out of date on the dome computer. It worked from other computers, however. Combined with a user error led to confusion as to the true failure mode.
+.5m Telescope: The telescope is currently offline while we attempt to obtain spare parts and time for the technical crew to troubleshoot the problem further. The telescope is intermittently experiencing runaway slews into the horizon limits. The direction and axis affected appear random and lead us to suspect the DFM control electronics.
+KOSMOS: The system is cold and stable. There have been a few reported problems with the disperser wheel not moving when commanded. Each report was from the red-to-blue grism. Once moved by hand to blue and a status issued by the Operator, it typically works the remainder of the night. We will be testing if power-cycling the motor controller during the next monthly engineering time will improve the situation at all.
+ARCTIC: The diffuser rotation mechanism is still unreliable. The mechanism that moves the diffuser in and out of the optical path is still functional in the meantime. The lower-left quadrant of the CCD has shown intermittent elevated noise to complete saturation in frames. Additionally, the problem has progressed to the point that single-amplifier readout is also not available for users. In QUAD readout mode, the lower-left quadrant is unusable, but the other three can be used for science. This problem has progressively gotten worse over the past several months. In the past day, it did return to normal (no noise) without intervention. We do not know what to make of that or if it is related to ambient temperatures. We will continue to monitor, as we had suspected we had isolated the problem to the inside of the cryostat and now we are uncertain. With no support from Astronomical Research Cameras (Bob Leech has sunsetted the company), this may be a difficult problem to solve and may require taking the instrument offline for an extended time. Users are cautioned to carefully inspect their data, especially their calibration frames, and report additional unusual behavior suggesting further degradation.
+ARCES: Vacuum servicing was performed on the instrument in July. The instrument is cold and usable; however, much of the gain in improvements to the interorder light ratios (IOL), a measurement of contamination along the light path, was lost.  The IOL has remained level for the past two weeks. It is possible it will improve with cooling ambient temperatures well into October and November. A new problem is being referred to as salt-and-pepper noise in images, where many pixels, randomly distributed from frame to frame, with levels from 1024 to saturation, occur with some regularity every 7 to 8 days. We are still attempting to confirm that it is associated with the new ICC or its internal hardware. Given that it is difficult to reproduce, we have had to wait for it to recur before further troubleshooting. We have had some success clearing the problem with a power cycle of the computer itself and restarting all its services.
+NICFPS: System is cooled and usable.
+TripleSpec: System is cooled and usable.
+APOLLO: The instrument is usable for laser ranging. Maintaining the instrument's temperature during extreme environmental conditions remains challenging. A team from NASA was on site for their annual servicing of the instrument. They were unable to complete their entire task list due to time constraints.
+SoonerCam: Agile has been decommissioned. The instrument rotator work was delayed over the summer shutdown and will continue soon to prepare for SoonerCam. Parts to create a dummy weight for testing and modifications to the instrument filter wheel are ready to be implemented. A working group to discuss instrument and software requirements of SoonerCam-ICC development has begun. Gordon continues to report good progress in understanding the GPS timing card software API but was looking for some direction in the future use of the instrument to make some critical development decisions.
+DIS: The instrument has been decommissioned. It has been removed from the observing floor and will eventually have expensive and critical parts removed before the rest is scrapped. The removal of the instrument from the observing floor frees floor space for the future delivery of Ocotillo.
+----
+=== Additional site, telescope and instruments discussion ===
+The 3.5m telescope is behaving nominally. The 0.5m is still inoperable; Bill recovered some spare parts from another site but the telescope still needs to be repaired. We anticipate scheduling Q4 ARCSAT observing programs to start no earlier than November.
+Through most of September ARCTIC had issues with the lower left quadrant of the chip being saturated. With the cooler weather that quadrant seems to have recovered itself. This recovery is not well understood; during some troubleshooting we swapped out a bunch of components but we still saw the issue. It was originally believed to be a problem isolated inside the cryostat but that is now less certain. For now we are in a "watch and wait" mode. This past weekend we saw evidence that there was water vapor stuck between ARCTIC's L2 and L3, which is a new problem to address. The purge air was on so it is unclear what caused condensation to form there.
+The inter-order light in ARCES had been slowly improving over the last few weeks but last weekend there was an inadvertent warm-up of the instrument. It is now offline but we hope that it will be back for tomorrow's scheduled observations. If we are unsuccessful in bringing it back, or if the vacuum has softened too much to cool it back down, we will need to take it offline for more extensive servicing. The salt and pepper noise that has been reported in ARCES frames is likely due to the new ICC, possibly the digitization card, a problem with the kernel, or the CPU board itself. This issue can appear any time, although recovery is quick. We are currently developing a procedure so the observing specialists can clear it up if seen, while Gordon continues to search for the root cause of the issue.
+The annual APOLLO servicing visit took place recently. The NASA team was on site, but they did not complete their entire [ambitious!] task list. We expect they will return to complete the remaining tasks some time in 2027.
+----
+=== 2026 Q4 3.5m scheduling ===
+The Q4 schedule is posted. There is a limited amount of unscheduled time available: one DD01 dark half on Nov 10A. Users should submit requests for this time to the scheduling team (Amanda Townsend, Russet McMillan, Ben Williams, and Nancy Chanover), and cc their institutional scheduler.
+=== 2026 Q4 0.5 scheduling ===
+We have Q4 proposals in hand, and will not schedule them to start until November.
+----
+=== New instrument initiatives ===
+  * KOSMOS new grism - We have a revised quote from Wasatch; Sarah needs to complete her review of it and then we can order it.
+  * Ocotillo - Lab work is continuing, and we are planning for a pre-ship review by end of this quarter. We had a check-in meeting on 9/25 and discussed progress and expectations for the next review. Moire asked if there is a spec sheet available so that users could view the expected performance and begin thinking about how they will want to use it. Nancy reported that an Ocotillo report is being prepared for the ARC Board meeting, and a spec sheet will likely be included in that document. If it is, she will share it with this group.
+  * ASPEN - Work on the ADC is moving forward. A request for funding was submitted for the ADC, and the ASPEN team is gearing up for another proposal submission next month.
+  * SoonerCam - a focus group of prospective SoonerCam users recently met with Gordon to discuss modes of operation for the instrument. This will help him develop the instrument control software.
+----
+=== Request for Science Highlights ===
+Nancy issued her annual request for science highlights (at least one from every ARC member institution and leasing partner!) for the ARC Board Meeting on October 27. Any file format is fine, but please make sure that each slide contains: researcher’s name and institution, a few bullets describing the science result and instrument used, and a graphic to complement the text. Users can send Nancy the slide directly or (preferably) UC reps can collect them and send them to her in a single packet. The deadline for submitting slides is **Friday October 16**. These are important for conveying to the Board members that their institution's investment in ARC and the 3.5m is worthwhile!
+----
+=== Plans for January AAS Meeting ===
+APO plans to have a booth at the next AAS meeting (sharing a double booth with SDSS). If any 3.5m users (especially students!) plan to present results that were enabled through 3.5m observations we would love to know about it so that we can highlight the talks/posters at the meeting.
+----
+=== ACTION ITEMS ===
+Open action items from previous meetings:
+  * Nancy: revisit idea of time trades with WIYN. Status: CLOSED. Indiana is not interested in revisiting our previous arrangement.
+New action items from this meeting:
+  * UC reps: notify your users about OPEN time available in Q4. Status: OPEN.
+  * UC reps: solicit science highlights slides from users, send to Nancy by Oct 16. Status: OPEN.
+----
+=== All Other Business ===
+None.
+----
+=== Next Meeting ===
+The next Users Committee meeting will be on Tuesday November 3 at 10:30 MST.
+----
+===== APO 3.5-m Users Committee Meeting, 9/01/2026 =====
+Attending: Nancy Chanover (NMSU), Misty Bentz (GSU), Kevin Schlaufman (JHU), Aleksandr Mosenkov (BYU), Amanda Townsend (APO), Moire Prescott (NMSU), Michael Hayden (OU), Bill Ketzeback (APO), Ben Williams (UW), Adam Kowalski (CU), Anne Verbiscer (UVa)
+=== User feedback and comments from institutional representatives ===
+  * Georgia State - nothing to report
+  * Washington - no report
+  * Oklahoma - nothing to report
+  * Wyoming - no report
+  * Colorado - nothing to report
+  * Williams - no report
+  * Colgate - no report
+  * NMSU - nothing to report
+  * UVa - nothing to report
+  * JHU - nothing to report
+  * BYU - nothing to report
+----
+=== Telescope and Instruments Report ===
+The detailed site report is included below, followed by additional information discussed during today's meeting.
+// 3.5-m Telescope and Instruments Highlights, 08/05/26 – 09/01/26 //
+) Overview
+Monsoon rains and unsettled weather continued during the month of August. A nearby lightning strike, near the back entrance to Sunspot,  skipped down the trunk of a tree, travelled along the root system, throwing up bark and dirt as it travelled 30-some feet to hit the observatory's buried water line. It caused extensive damage to a two-foot section of water line, resulting in the drainage of the entire contents of the Sunspot water tower. The leak was quickly located, isolated, and repaired in two days' time. In the interim, we asked only essential staff to be on site; we had to haul water for facilities use and provide bottled water for drinking and cooking. Also damaged were our hard-wired phone and fax lines; our phone carrier was able to repair these lines without too much difficulty.
+During this reporting cycle, there were a few telescope-reported troubles. The occasional motion errors for the altitude axis continue to occur during afternoon checkout. There was a report of the altitude axis having a runaway that drove it to the upper limit switch but not the hard stop. The operator was able to recover quickly, and the telescope position was not lost after the restarting of the axis controller for altitude. A loss of communication with the rotator controller was cleared by rebooting the controller. There were also a few instances where the tertiary actuators had motor faults. We have a workaround in place until the problem can be properly evaluated.
+A few notable instrument issues were recurring during this reporting cycle. Instruments experiencing issues again this month were primarily ARCTIC and ECHELLE. Details for each are below.
+There were no on-site training classes or visiting instruments this reporting cycle.
+The APO house trailer is no longer available for occupancy. NMSU and APO are discussing its future.
+) Operations
+.5m Telescope: The telescope has been performing nominally; however, with the following problems. The altitude axis had a very rare runaway which we believe was command and reply confusion in the TCC software.  There were a small number of altitude motion errors this month, but all on a single afternoon, while performing test slews. The rotator had a loss of communication issue and required a reboot at the start of the night. Multiple actuator motor faults on the tertiary mirror when switching between instrument ports were troublesome during August. We have an operational workaround in place until the problem can be properly evaluated.
+.5m Telescope: Dcam and UVAcam are both available options for users. The software (and possible computer hardware) still seem unstable, requiring multiple reboots and application restarts. The telescope is experiencing occasional runaways in right asscension, and this past weekend the main PC power supply seems to have failed (again) for the second or third time in the past two years.
+KOSMOS: The system is cold and stable. The instrument is performing nominally.
+ARCTIC: The diffuser rotation mechanism is still unreliable. The mechanism that moves the diffuser in and out of the optical path is still functional. The problem described last month, where the lower-left quadrant of the CCD has shown intermittent elevated noise or complete saturation in bias frames, is now seen most of the time. This problem has progressively gotten worse over the past three months. We continue to experiment with swapping various cards in the controller with spares, also with no improvement. Our current goal is to isolate the problem to inside or outside the cryostat, to electronics, CCD, or wiring. With no support from Astronomical Research Cameras (Bob Leech has sunsetted the company), this may be a difficult problem to solve. Users are cautioned to inspect their data, especially calibration frames. We may be forced to switch to single amplifier readout for science observations while we continue to troubleshoot. Spares are nearly non-existent.
+ARCES: Vacuum servicing was performed on the instrument in July. The instrument is cold and usable, but our expected improvements to the interorder light ratios, a measurement of contamination along the light path, seem short-lived. The amount of contamination causing scattered light seems to be steadily degrading science exposures.
+NICFPS: System is cooled and usable.
+TripleSpec: System is cooled and usable. A new instrument block was performed on sky as part of shutdown recovery to improve pointing and offsets.
+APOLLO: The instrument is usable for laser ranging. Maintaining the instrument's temperature during extreme environmental conditions remains challenging. Laser power adjustments were done as part of summer shutdown recovery. A NASA servicing trip is scheduled for the third week of September.
+Agile/ SoonerCam: Agile has been decommissioned. The instrument rotator is still not performing nominally, and we will continue troubleshooting to prepare for SoonerCam. Parts to create a dummy weight for testing and modifications to the instrument filter wheel are in the planning stage. This was one of the planned shutdown tasks that were deferred due to unexpected summer machining delays and to other instrument work. SoonerCam-ICC software development has begun. Gordon continues to report good progress and has written the bare bones of an actor for the hub.
+DIS: The instrument has been decommissioned. It has been removed from the observing floor and will eventually have expensive and critical parts removed before the rest is scrapped. The removal of the instrument from the observing floor frees floor space for the future delivery of Ocotillo.
+----
+=== Additional site, telescope and instruments discussion ===
+We are continuing to see minor issues with the telescope altitude motion errors; they usually occur during test slews in the afternoon or after the telescope has been pointed in one spot for a long time. We experienced a loss of communication with the rotator over the weekend but recovered from it quickly. There is a nagging issue with the tertiary motor actuators faulting out; we still have not identified the root cause but are working on it. In the meantime we found an operational workaround to prevent this from happening. ARCSAT is still having numerous problems (runaways), and it had an issue with the computer shutting down over the weekend. We are uncertain what triggered it to shut down, but last year when this happened it was a precursor to a failing power supply. The computer is very old so finding a replacement power supply will be challenging (the one in use now was a refurbished used one, and expensive). We are still running diagnostics to better understand both the computer issue and the telescope runaway problem.
+Regarding ARCTIC, we had previously reported that the LL quad of the chip had a problem where the edge would start to saturate and as more exposures were taken the saturation would migrate across the chip, resulting in a saturated LL quadrant. Previously this would clear up after being in use for a few hours, but it is persistent now. There are no spares with the sunsetting of Leach controllers so it will be challenging to solve this. In the near future we hope to isolate the problem further by determining if it exists inside or outside of the cryostat.
+We did a vacuum servicing of ARCES during the summer shutdown to mitigate the inter-order light (IOL) that is a manifestation of contamination. We saw improvement during shutdown but then it quickly degraded again. Last month we did a partial warm up for about a week, which improved things. Now it is degrading again, but at a slower rate. The instrument likely will need to come offline again for another servicing. We also had other problem whereby some ARCES exposures had salt-and-pepper noise, with pixel values fluctuating between below the bias level to saturation. This can be seen in a histogram of the data files, evidenced by spikes outside of the normal range of signal counts. We believe the problem is solved but users should keep an eye out.
+For both ARCTIC and ARCES, we strongly encourage users to look at their data carefully as they are being acquired! This will help to identify problems such as the aforementioned issues quickly, which hopefully will lead to a more timely resolution.
+Moire asked for an update about KOSMOS and a recent issue related to instrument drift. Amanda reported that we did a new instrument block and some additional testing during some recent engineering time. The problem seems to be fixed with the new instrument block; it may have been due to a guiding issue but we're not sure why it would affect MOS mode specifically. APO is eager to receive feedback about how the next MOS run goes.
+----
+=== 2026 Q3 3.5m scheduling ===
+The Q3 schedule is posted. There is one slot of DD time remaining (Sep 13A, dark) and one [short] slot of OPEN time remaining (Sep 14B, dark). Users should submit requests for this time to the scheduling team (Amanda Townsend, Russet McMillan, Ben Williams, and Nancy Chanover), and cc their institutional scheduler.
+=== 2026 Q4 3.5m scheduling ===
+The Q4 proposals have all been received and the schedule is in work, with an estimated publication date of late next week. As far as we understand it there will be no class visits to the site in Q4. If that is not the case please let us know ASAP!
+=== 2026 Q3 0.5 scheduling ===
+The ARCSAT schedule is posted and ARCSAT is fully booked for Q3. The call for Q4 proposals will be issued once the 3.5m schedule for Q4 is published.
+----
+=== New instrument initiatives ===
+  * KOSMOS new grism - no update.
+  * Ocotillo - Lab work is continuing, hoping for a pre-ship review next quarter.
+  * ASPEN - No major updates; moving forward with ADC and gearing up for another funding proposal.
+  * SoonerCam - no update.
+----
+=== Pilot data archive with NOIRLab ===
+We are moving forward with an arrangement to have ARC 3.5m data ingested into the NOIRLab data archive. We will pilot this first with a single instrument (likely ARCES), in an OPT-OUT mode. The proprietary period is expected to be 18 months. We will update users once we get closer to being ready to test this out.
+----
+=== Request for Science Highlights ===
+Nancy issued her annual request for science highlights (at least one from every ARC member institution and leasing partner!) for the ARC Board Meeting on October 27. Any file format is fine, but please make sure that each slide contains: researcher’s name and institution, a few bullets describing the science result and instrument used, and a graphic to complement the text. Users can send Nancy the slide directly or (preferably) UC reps can collect them and send them to her in a single packet. The deadline for submitting slides is **Friday October 16**. These are important for conveying to the Board members that their institution's investment in ARC and the 3.5m is worthwhile!
+----
+=== Plans for January AAS Meeting ===
+APO plans to have a booth at the next AAS meeting (sharing a double booth with SDSS). Nancy polled the group to see if there is any interest in a special session or a splinter meeting related to APO-specific topics. There was no interest so we will not pursue this further.
+----
+=== ACTION ITEMS ===
+Open action items from previous meetings:
+  * Nancy will ensure that the final report from the Future of ARCSAT Working Group is made available to the Users Committee and put on the wiki. Status: CLOSED. The file is available for download from the [[:fawg|FAWG page of the wiki]].
+  * APO: update text on TUI downloads page to reflect new procedure for downloading TUI. Status: CLOSED.
+  * Nancy: follow-up on PypeIt implementation and how it handles KOSMOS reduction complexities. Status: CLOSED. Adam K. reports positive experience using PypeIt for APO spectroscopic data reduction. He has used it with KOSMOS data and he plans to test it with TripleSpec data next.
+  * Nancy: revisit idea of time trades with WIYN. Status: CLOSED. There does not seem to be interest from Indiana; Adam will discuss this with his user and follow up with Nancy if needed.
+  * UC reps: identify KOSMOS and/or TripleSpec users in your department who would be willing to help test out the new PypeIt modules for those instruments. Have those users report their engagement with Jon Holtzman. Status: CLOSED. No interest or reported usage other than 2 CU users.
+  * UC reps: identify time domain / former Agile users who are willing to serve on SoonerCam focus group. Status: CLOSED. No volunteers. Nancy will ask individuals directly.
+New action items from this meeting:
+  * UC reps: notify your users about OPEN time available in August-September.
+----
+=== All Other Business ===
+None.
+----
+=== Next Meeting ===
+The next Users Committee meeting will be on Tuesday October 6 at 10:30 MDT.
+----
 ===== APO 3.5-m Users Committee Meeting, 8/04/2026 =====