Differences

This shows you the differences between two versions of the page.

--- uc:uc2026 [2026/09/01 16:22] – nchanove
+++ uc:uc2026 [2026/10/06 19:28] (current) – nchanove
@@ Line 1: / Line 1: @@
-===== APO 3.5-m Users Committee Meeting, 9/01/2026 =====
+===== APO 3.5-m Users Committee Meeting, 10/06/2026 =====
-Attending: Nancy Chanover (NMSU),
+Attending: Nancy Chanover (NMSU), Misty Bentz (GSU), Kevin Schlaufman (JHU), Aleksandr Mosenkov (BYU), Anne Verbiscer (UVa), Moire Prescott (NMSU), Ben Williams (UW), Eric Bellm (UW), Bill Ketzeback (APO), Adam Kowalski (CU), Michael Hayden (OU), Russet McMillan (APO)
-Aleksandr Mosenkov (BYU), Anne Verbiscer (UVa), Moire Prescott (NMSU), Michael Hayden (OU), Sarah Tuttle (UW), Bill Ketzeback (APO), Adam Kowalski (CU), Eric Bellm (UW), Ben Williams (UW)
+=== User feedback and comments from institutional representatives ===
+  * Washington - nothing to report
+  * Williams - no report
+  * Colgate - no report
+  * Wyoming - no report
+  * BYU - nothing to report
+  * JHU - nothing to report
+  * Colorado - nothing to report
+  * UVa - nothing to report other than a user can't access night logs on web page after 9/20/2026; Nancy will look into it and report back
+  * Oklahoma - nothing to report
+  * Georgia State - nothing to report
+  * NMSU - nothing to report
+----
+=== Telescope and Instruments Report ===
+The detailed site report is included below, followed by additional information discussed during today's meeting.
+// 3.5-m Telescope and Instruments Highlights, 09/02/26 – 10/02/26 //
+) Overview
+Southern New Mexico has experienced a strong monsoon season this year with above-average precipitation. As described in last month's report, the NWS historically defines the end of the monsoon season as September 30; however, recent forecasts indicate that late September into early October, tropical and monsoon moisture are expected to produce heavier-than-average rainfall at the end of the official season.
+The 3.5m telescope performance this cycle was nominal, with few minor reported problems. A single motion error in altitude during afternoon checkout was encountered this reporting cycle. A problem report with the NA2 rotator was tracked down to a software version control issue and user error. No science was lost to these events.
+Instruments experiencing issues this month were primarily KOSMOS, ARCTIC, and ECHELLE. Details for each are below.
+There were no visiting classes this reporting cycle. A team from NASA was on site for their annual instrument servicing of APOLLO. One PI driven instrument program this cycle, but none of the instrument team traveled to APO, operating their instrument entirely remotely.
+) Operations
+.5m Telescope: The telescope has been performing nominally, with a few reported minor issues. There was a single reported altitude motion error, indicating that we may be past the seasonal ambient temperature when most of these occur.  One power brownout and transfer to the backup generator during the day resulted in automatic mirror covers needing to be reset. A reported problem zeroing the NA2 rotator, at the end of DSSI time, appeared to be due to an older version of the low-level engineering application used to communicate with the axis controller being out of date on the dome computer. It worked from other computers, however. Combined with a user error led to confusion as to the true failure mode.
+.5m Telescope: The telescope is currently offline while we attempt to obtain spare parts and time for the technical crew to troubleshoot the problem further. The telescope is intermittently experiencing runaway slews into the horizon limits. The direction and axis affected appear random and lead us to suspect the DFM control electronics.
+KOSMOS: The system is cold and stable. There have been a few reported problems with the disperser wheel not moving when commanded. Each report was from the red-to-blue grism. Once moved by hand to blue and a status issued by the Operator, it typically works the remainder of the night. We will be testing if power-cycling the motor controller during the next monthly engineering time will improve the situation at all.
+ARCTIC: The diffuser rotation mechanism is still unreliable. The mechanism that moves the diffuser in and out of the optical path is still functional in the meantime. The lower-left quadrant of the CCD has shown intermittent elevated noise to complete saturation in frames. Additionally, the problem has progressed to the point that single-amplifier readout is also not available for users. In QUAD readout mode, the lower-left quadrant is unusable, but the other three can be used for science. This problem has progressively gotten worse over the past several months. In the past day, it did return to normal (no noise) without intervention. We do not know what to make of that or if it is related to ambient temperatures. We will continue to monitor, as we had suspected we had isolated the problem to the inside of the cryostat and now we are uncertain. With no support from Astronomical Research Cameras (Bob Leech has sunsetted the company), this may be a difficult problem to solve and may require taking the instrument offline for an extended time. Users are cautioned to carefully inspect their data, especially their calibration frames, and report additional unusual behavior suggesting further degradation.
+ARCES: Vacuum servicing was performed on the instrument in July. The instrument is cold and usable; however, much of the gain in improvements to the interorder light ratios (IOL), a measurement of contamination along the light path, was lost.  The IOL has remained level for the past two weeks. It is possible it will improve with cooling ambient temperatures well into October and November. A new problem is being referred to as salt-and-pepper noise in images, where many pixels, randomly distributed from frame to frame, with levels from 1024 to saturation, occur with some regularity every 7 to 8 days. We are still attempting to confirm that it is associated with the new ICC or its internal hardware. Given that it is difficult to reproduce, we have had to wait for it to recur before further troubleshooting. We have had some success clearing the problem with a power cycle of the computer itself and restarting all its services.
+NICFPS: System is cooled and usable.
+TripleSpec: System is cooled and usable.
+APOLLO: The instrument is usable for laser ranging. Maintaining the instrument's temperature during extreme environmental conditions remains challenging. A team from NASA was on site for their annual servicing of the instrument. They were unable to complete their entire task list due to time constraints.
+SoonerCam: Agile has been decommissioned. The instrument rotator work was delayed over the summer shutdown and will continue soon to prepare for SoonerCam. Parts to create a dummy weight for testing and modifications to the instrument filter wheel are ready to be implemented. A working group to discuss instrument and software requirements of SoonerCam-ICC development has begun. Gordon continues to report good progress in understanding the GPS timing card software API but was looking for some direction in the future use of the instrument to make some critical development decisions.
+DIS: The instrument has been decommissioned. It has been removed from the observing floor and will eventually have expensive and critical parts removed before the rest is scrapped. The removal of the instrument from the observing floor frees floor space for the future delivery of Ocotillo.
+----
+=== Additional site, telescope and instruments discussion ===
+The 3.5m telescope is behaving nominally. The 0.5m is still inoperable; Bill recovered some spare parts from another site but the telescope still needs to be repaired. We anticipate scheduling Q4 ARCSAT observing programs to start no earlier than November.
+Through most of September ARCTIC had issues with the lower left quadrant of the chip being saturated. With the cooler weather that quadrant seems to have recovered itself. This recovery is not well understood; during some troubleshooting we swapped out a bunch of components but we still saw the issue. It was originally believed to be a problem isolated inside the cryostat but that is now less certain. For now we are in a "watch and wait" mode. This past weekend we saw evidence that there was water vapor stuck between ARCTIC's L2 and L3, which is a new problem to address. The purge air was on so it is unclear what caused condensation to form there.
+The inter-order light in ARCES had been slowly improving over the last few weeks but last weekend there was an inadvertent warm-up of the instrument. It is now offline but we hope that it will be back for tomorrow's scheduled observations. If we are unsuccessful in bringing it back, or if the vacuum has softened too much to cool it back down, we will need to take it offline for more extensive servicing. The salt and pepper noise that has been reported in ARCES frames is likely due to the new ICC, possibly the digitization card, a problem with the kernel, or the CPU board itself. This issue can appear any time, although recovery is quick. We are currently developing a procedure so the observing specialists can clear it up if seen, while Gordon continues to search for the root cause of the issue.
+The annual APOLLO servicing visit took place recently. The NASA team was on site, but they did not complete their entire [ambitious!] task list. We expect they will return to complete the remaining tasks some time in 2027.
+----
+=== 2026 Q4 3.5m scheduling ===
+The Q4 schedule is posted. There is a limited amount of unscheduled time available: one DD01 dark half on Nov 10A. Users should submit requests for this time to the scheduling team (Amanda Townsend, Russet McMillan, Ben Williams, and Nancy Chanover), and cc their institutional scheduler.
+=== 2026 Q4 0.5 scheduling ===
+We have Q4 proposals in hand, and will not schedule them to start until November.
+----
+=== New instrument initiatives ===
+  * KOSMOS new grism - We have a revised quote from Wasatch; Sarah needs to complete her review of it and then we can order it.
+  * Ocotillo - Lab work is continuing, and we are planning for a pre-ship review by end of this quarter. We had a check-in meeting on 9/25 and discussed progress and expectations for the next review. Moire asked if there is a spec sheet available so that users could view the expected performance and begin thinking about how they will want to use it. Nancy reported that an Ocotillo report is being prepared for the ARC Board meeting, and a spec sheet will likely be included in that document. If it is, she will share it with this group.
+  * ASPEN - Work on the ADC is moving forward. A request for funding was submitted for the ADC, and the ASPEN team is gearing up for another proposal submission next month.
+  * SoonerCam - a focus group of prospective SoonerCam users recently met with Gordon to discuss modes of operation for the instrument. This will help him develop the instrument control software.
+----
+=== Request for Science Highlights ===
+Nancy issued her annual request for science highlights (at least one from every ARC member institution and leasing partner!) for the ARC Board Meeting on October 27. Any file format is fine, but please make sure that each slide contains: researcher’s name and institution, a few bullets describing the science result and instrument used, and a graphic to complement the text. Users can send Nancy the slide directly or (preferably) UC reps can collect them and send them to her in a single packet. The deadline for submitting slides is **Friday October 16**. These are important for conveying to the Board members that their institution's investment in ARC and the 3.5m is worthwhile!
+----
+=== Plans for January AAS Meeting ===
+APO plans to have a booth at the next AAS meeting (sharing a double booth with SDSS). If any 3.5m users (especially students!) plan to present results that were enabled through 3.5m observations we would love to know about it so that we can highlight the talks/posters at the meeting.
+----
+=== ACTION ITEMS ===
+Open action items from previous meetings:
+  * Nancy: revisit idea of time trades with WIYN. Status: CLOSED. Indiana is not interested in revisiting our previous arrangement.
+New action items from this meeting:
+  * UC reps: notify your users about OPEN time available in Q4. Status: OPEN.
+  * UC reps: solicit science highlights slides from users, send to Nancy by Oct 16. Status: OPEN.
+----
+=== All Other Business ===
+None.
+----
+=== Next Meeting ===
+The next Users Committee meeting will be on Tuesday November 3 at 10:30 MST.
+----
+===== APO 3.5-m Users Committee Meeting, 9/01/2026 =====
+Attending: Nancy Chanover (NMSU), Misty Bentz (GSU), Kevin Schlaufman (JHU), Aleksandr Mosenkov (BYU), Amanda Townsend (APO), Moire Prescott (NMSU), Michael Hayden (OU), Bill Ketzeback (APO), Ben Williams (UW), Adam Kowalski (CU), Anne Verbiscer (UVa)
 === User feedback and comments from institutional representatives ===
   * Georgia State - nothing to report
-  * Washington - nothing to report
+  * Washington - no report
   * Oklahoma - nothing to report
-  * Wyoming - nothing to report
+  * Wyoming - no report
-  * Colorado - a user inquired about the possibility of time trades with WIYN. Adam will put the user in touch with Nancy to discuss it further.
+  * Colorado - nothing to report
   * Williams - no report
   * Colgate - no report
@@ Line 65: / Line 189: @@
 === Additional site, telescope and instruments discussion ===
-xxx
+We are continuing to see minor issues with the telescope altitude motion errors; they usually occur during test slews in the afternoon or after the telescope has been pointed in one spot for a long time. We experienced a loss of communication with the rotator over the weekend but recovered from it quickly. There is a nagging issue with the tertiary motor actuators faulting out; we still have not identified the root cause but are working on it. In the meantime we found an operational workaround to prevent this from happening. ARCSAT is still having numerous problems (runaways), and it had an issue with the computer shutting down over the weekend. We are uncertain what triggered it to shut down, but last year when this happened it was a precursor to a failing power supply. The computer is very old so finding a replacement power supply will be challenging (the one in use now was a refurbished used one, and expensive). We are still running diagnostics to better understand both the computer issue and the telescope runaway problem.
+Regarding ARCTIC, we had previously reported that the LL quad of the chip had a problem where the edge would start to saturate and as more exposures were taken the saturation would migrate across the chip, resulting in a saturated LL quadrant. Previously this would clear up after being in use for a few hours, but it is persistent now. There are no spares with the sunsetting of Leach controllers so it will be challenging to solve this. In the near future we hope to isolate the problem further by determining if it exists inside or outside of the cryostat.
+We did a vacuum servicing of ARCES during the summer shutdown to mitigate the inter-order light (IOL) that is a manifestation of contamination. We saw improvement during shutdown but then it quickly degraded again. Last month we did a partial warm up for about a week, which improved things. Now it is degrading again, but at a slower rate. The instrument likely will need to come offline again for another servicing. We also had other problem whereby some ARCES exposures had salt-and-pepper noise, with pixel values fluctuating between below the bias level to saturation. This can be seen in a histogram of the data files, evidenced by spikes outside of the normal range of signal counts. We believe the problem is solved but users should keep an eye out.
+For both ARCTIC and ARCES, we strongly encourage users to look at their data carefully as they are being acquired! This will help to identify problems such as the aforementioned issues quickly, which hopefully will lead to a more timely resolution.
+Moire asked for an update about KOSMOS and a recent issue related to instrument drift. Amanda reported that we did a new instrument block and some additional testing during some recent engineering time. The problem seems to be fixed with the new instrument block; it may have been due to a guiding issue but we're not sure why it would affect MOS mode specifically. APO is eager to receive feedback about how the next MOS run goes.
 ----
@@ Line 75: / Line 207: @@
 === 2026 Q4 3.5m scheduling ===
-The Q4 proposals have all been received and the schedule is in work, with an estimated publication date of xxx. As far as we understand it there will be no class visits to the site in Q4. If that is not the case please let us know ASAP!
+The Q4 proposals have all been received and the schedule is in work, with an estimated publication date of late next week. As far as we understand it there will be no class visits to the site in Q4. If that is not the case please let us know ASAP!
 === 2026 Q3 0.5 scheduling ===
@@ Line 83: / Line 215: @@
 ----
-=== New instrument initiatives (update all) ===
+=== New instrument initiatives ===
-  * KOSMOS new grism - the new quote arrived and we will be placing the order shortly.
+  * KOSMOS new grism - no update.
-  * Ocotillo - We now have all of the optics in hand and just ordered the bulk of mechanical parts. They will arrive in approximately 3 weeks and then assembly will begin. The TEG team is now working on software, specifically to ensure that there is infrastructure in place to ensure communication between the 3 channels even if only one is functional first.
+  * Ocotillo - Lab work is continuing, hoping for a pre-ship review next quarter.
   * ASPEN - No major updates; moving forward with ADC and gearing up for another funding proposal.
-  * SoonerCam - Gordon reports that he is able to control the timer card, he has become familiarized with the API, he can control the camera and read out FITS files. He is working on the time match and external triggering interface; it works using a unique feature of a timing card or with similar Agile heartbeat signal, one implementation is less complex than the other. He would like to assemble a focus group of potential SoonerCam users to better understand the envisioned use cases (e.g. what calibration frames would be needed, other design requirements) so he can finalize C++ library for imaging. UC reps: please notify your users that we are looking for some time domain / occultation / former Agile users to take part in this focus group. Any interested parties should let Nancy know so she can assemble this group.
+  * SoonerCam - no update.
 ----
@@ Line 106: / Line 238: @@
 === Plans for January AAS Meeting ===
-xxx special or splinter session? xxx
+APO plans to have a booth at the next AAS meeting (sharing a double booth with SDSS). Nancy polled the group to see if there is any interest in a special session or a splinter meeting related to APO-specific topics. There was no interest so we will not pursue this further.
 ----
@@ Line 113: / Line 245: @@
 Open action items from previous meetings:
-  * Nancy will ensure that the final report from the Future of ARCSAT Working Group is made available to the Users Committee and put on the wiki. Status: OPEN (pending receipt of final version of report; still awaiting one appendix. Nancy said she will distribute the remainder of the report now even without the appendix).
+  * Nancy will ensure that the final report from the Future of ARCSAT Working Group is made available to the Users Committee and put on the wiki. Status: CLOSED. The file is available for download from the [[:fawg|FAWG page of the wiki]].
-  *  APO: update text on TUI downloads page to reflect new procedure for downloading TUI. Status: PENDING; needs verification
+  * APO: update text on TUI downloads page to reflect new procedure for downloading TUI. Status: CLOSED.
-  * UC reps: notify your users about OPEN time available in June. Status: CLOSED.
+  * Nancy: follow-up on PypeIt implementation and how it handles KOSMOS reduction complexities. Status: CLOSED. Adam K. reports positive experience using PypeIt for APO spectroscopic data reduction. He has used it with KOSMOS data and he plans to test it with TripleSpec data next.
-  * Nancy: follow-up on PypeIt implementation and how it handles KOSMOS reduction complexities. Status: OPEN.
+  * Nancy: revisit idea of time trades with WIYN. Status: CLOSED. There does not seem to be interest from Indiana; Adam will discuss this with his user and follow up with Nancy if needed.
-  * Nancy: Post job ad links in July meeting minutes. Status: CLOSED.
+  * UC reps: identify KOSMOS and/or TripleSpec users in your department who would be willing to help test out the new PypeIt modules for those instruments. Have those users report their engagement with Jon Holtzman. Status: CLOSED. No interest or reported usage other than 2 CU users.
+  * UC reps: identify time domain / former Agile users who are willing to serve on SoonerCam focus group. Status: CLOSED. No volunteers. Nancy will ask individuals directly.
 New action items from this meeting:
-  * Nancy: revisit idea of time trades with WIYN.
-  * UC reps: identify KOSMOS and/or TripleSpec users in your department who would be willing to help test out the new PypeIt modules for those instruments. Have those users report their engagement with Jon Holtzman.
-  * UC reps: identify time domain / former Agile users who are willing to serve on SoonerCam focus group.
   * UC reps: notify your users about OPEN time available in August-September.
@@ Line 136: / Line 265: @@
 === Next Meeting ===
-The next Users Committee meeting will be on Tuesday September 1 at 10:30 MDT.
+The next Users Committee meeting will be on Tuesday October 6 at 10:30 MDT.
 ----