|
Next revision
|
Previous revision
|
uc:uc20251007_october_7_2025 [2025/10/07 16:04]
nchanove created |
uc:uc20251007_october_7_2025 [2025/10/14 20:55] (current)
nchanove |
| ---- | ---- |
| |
| Attending: Nancy Chanover (NMSU), Bill Ketzeback (APO), | Attending: Nancy Chanover (NMSU), Aleksandr Mosenkov (BYU), Russet McMillan (APO), Anne Verbiscer (UVa), Adam Kowalski (CU), Misty Bentz (GSU), Ben Williams (UW), Moire Prescott (NMSU), Joanne Hughes (Seattle U), Chip Kobulnicky (UWy), Michael Hayden (OU), Eric Bellm (UW), Sarah Tuttle (UW), Bill Ketzeback (APO), Gordon MacDonald (APO) |
| |
| Misty Bentz (GSU), Aleksandr Mosenkov (BYU), Anne Verbiscer (UVa), Michael Hayden (OU), Kevin Schlaufman (JHU), Joanne Hughes (Seattle U), Moire Prescott (NMSU), Chip Kobulnicky (UWy), Ben Williams (UW) | |
| |
| |
| === User feedback and comments from institutional representatives === | === User feedback and comments from institutional representatives === |
| |
| * Colorado - nothing to report | * Colorado - Several CU users were wondering about the status of data reduction software for KOSMOS, and whether PyKOSMOS is considered finished and ready for widespread use or whether people should still be using standard IRAF routines. A user found that the spectra are tilted so much that they need to use the Marsh algorithm rather than the Horn algorithm in the reduction, and the PyKOSMOS documentation currently does not address this. Nancy and Sarah Tuttle took the action item to follow up with Jim Davenport on this and report back. |
| * BYU - nothing to report | * BYU - nothing to report |
| * Oklahoma - nothing to report | * Oklahoma - nothing to report |
| * UVa - nothing to report | * UVa - nothing to report |
| * Wyoming - nothing to report | * Wyoming - nothing to report other than general interest in the timetable for the KOSMOS repair |
| * Georgia State - nothing to report | * Georgia State - nothing to report |
| * NMSU - nothing to report | * NMSU - nothing to report |
| * Washington - nothing to report | * Washington - nothing to report other than general interest in the timetable for the KOSMOS repair |
| * JHU - nothing to report (by email) | * JHU - nothing to report (by email) |
| * Seattle - nothing to report | * Seattle - nothing to report regarding recent observations, but Joanne had a comment on the topic of KOSMOS data reduction. She and her collaborators had a College of Idaho student working with Jon Holtzman on software for slit mask/MOS data reduction. The software is pretty stable and is publicly available but it is not on the wiki yet. Joanne said she would be happy to share it and took the action item to put it on the wiki. |
| * NAPG - no report | * NAPG - no report |
| |
| The detailed site report is included below, followed by additional information discussed during today's meeting. | The detailed site report is included below, followed by additional information discussed during today's meeting. |
| |
| // 3.5-m Telescope and Instruments Highlights, 8/06/25 – 09/02/25 // | // 3.5-m Telescope and Instruments Highlights, 9/03/25 – 10/07/25 // |
| |
| 1) Overview | 1) Overview |
| |
| The month of August experienced a mix of late afternoon monsoon storms that either delayed opening or affected many of the A half observing programs. We had two visiting instrument teams from U. Virginia this month. There was a failure of one of the main site network subnets that prevented observing and took down site phones as well as communications via TUI. Email was able to be received and sent from off site but not on site. We tracked the problem to a failure of a managed network switch and we were able to replace it with a spare. There was some difficulty loading the management given the switches were different models. The root cause is still not well understood as it looked like a cascading failure including the site DHCP server that also resulted in loss of communications with the site DNS servers. The problem took a little over 24 hours to resolve. | Monsoonal patterns continued for much of September, with unsettled weather affecting all but a small handful of nights this month. There were no visiting classes or PI instruments staying on site during the month. However, we did host daytime tours and had remote training sessions this past month. The 3.5m Night Operations team saw the departure of both Candace Gray and Mark Croom. We wish them both in their future adventures! |
| |
| The hiring committees for the Telescope Engineer and Night Operation positions are in the process of reviewing candidates. There were a large number of applications for each opening. | The site will be holding an open house event for the public (daytime only) on October 11. Science demonstrations, public talks, and tours of the telescopes will be the highlights of the four-hour event. |
| |
| 2) Operations | 2) Operations |
| |
| 3.5m Telescope: Telescope is working as expected. Seasonal motion errors have been infrequent. A full remap of the tertiary rotation was redone and this seems to have corrected the reported position errors from last month's report. | 3.5m Telescope: Telescope is working as expected. Seasonal motion errors continue to occur infrequently. We anticipate that there will need to be some tuning of the axis drives by mid-November. This past Saturday, there was an occurrence where one of the three axial actuators for the secondary mirror did not move after an instrument/ port change. The operator corrected the problem by homing the secondary. The weather at the time was poor, so no time was lost. |
| |
| 0.5m Telescope: Telescope is working as expected. ACP Library support errors are still occurring even with the dcam-spare camera swap but with much lower frequency. Dcam-spare had a loss of CCD chamber integrity and was forming ice on the detector while cold. The camera was serviced twice but now appears to have water spots on or near the detector surface. Cary Smith will attempt to clean but it is not without risk to the bonding wires of the detector. University of Virginia tested another surplus Apogee camera during their coordinated observing run with DSSI on the 3.5m. The off-axis guider has not been very reliable, requiring power cycles. | 0.5m Telescope: Telescope is working as expected. Dcam-spare and UVAcam are both available options for users. |
| |
| KOSMOS: System is cooled and stable. Increased dark current for long exposures has been confirmed. We attempted a vacuum servicing of the cryostat but without much luck. We anticipate having to open the vacuum vessel in the clean room sometime during Quarter 4 to investigate possible thermal shorts or opens. kcamera-ICC was backed up over shutdown. | KOSMOS: As discussed last month, the increase in dark current warranted opening the cryostat for inspection. Three of the four thermal connections from the liquid nitrogen can to the CCD cold block had failed, leaving only a single thin connection. Repairs were delayed by the shipping of material from Europe. We are currently searching for a vendor to assemble the parts, as we have not had success ourselves. |
| |
| ARCTIC: The diffuser rotation mechanism is still unreliable even after a full servicing. Troubleshooting is continuing. The mechanism that moves the diffuser in and out of the optical path is still functional in the meantime. The rest of the instrument is cooled. | ARCTIC: The diffuser rotation mechanism is still unreliable even after a full servicing. The mechanism that moves the diffuser in and out of the optical path is still functional in the meantime. The rest of the instrument is cooled. |
| |
| Agile: The camera is non-operational, the thermoelectric cooler failed again. The camera is warm and we do not plan further repair work with this camera. We are planning on decommissioning the camera. The agile instrument rotator is still not performing nominally and we are troubleshooting it further to prepare for SoonerCam. | Agile: The camera is non-operational; the thermoelectric cooler failed again. The camera is warm, and we do not plan further repair work with this camera. We are planning on decommissioning the camera. The agile instrument rotator is still not performing nominally, and we are troubleshooting it further to prepare for SoonerCam. |
| |
| ARCES: The CCD reservoir for the cooling system was brought back to temperature. IOL levels are quite good but slowly worsening. We hope that it will stabilize and reverse direction with cooler ambient temperatures as we have seen in previous fall time frames. Work continues on a replacement and modern ICC, so far testing is going well. A commissioning report is being worked on. | ARCES: While working on automating the interorder light reductions and archiving the data in a new database, we discovered that the FITS header KEYWORD for DATE-OBS was wrong by approximately 45 minutes in the future of when an image was written. Further data mining revealed this occurred during the summer/fall of 2024 when the ICC experienced power supply failure and hard drive troubles. It was not caught that the operating system service that syncs time to the site timeserver was not performing. The problem would have started on or after August 27, 2024. If time stamps are important for your echelle observations users should take note. A report from a user that the new ThAr bulb, replaced on July 22nd, had very weak lines or missing lines in the mid and blue orders prompted us to replace the bulb again on September 29. Wednesday, September 30th, observers then reported that the ThAr counts were around 3250 DN last year, but only 3100 DN counts tonight. This was the last new HCL we had on the shelf. It may be these two bulbs had lower pressure argon gas fill and therefore those are the lines missing. Comparison with line lists would need to be done to confirm. More bulbs are on order; however, Thorium-Argon lamps of the same type and manufacture as we were purchasing years ago are no longer available. We are buying up one European vendors back stock but we do not know if these will be of the same lamp strength as the old bulbs. |
| |
| DIS: System is cooled but in an unknown state for science. Decommissioning plans have begun. | DIS: System is powered down and warm. Decommissioning plans have begun. |
| |
| NICFPS: System is cooled and usable. The ICS software has needed multiple restarts over the past month. The cause is unclear at this time. | NICFPS: System is cooled and usable. |
| |
| TripleSpec: System is cooled and usable. | TripleSpec: System is cooled and usable. |
| === Additional site, telescope and instruments discussion === | === Additional site, telescope and instruments discussion === |
| |
| There was a recent failure of the main site network's subnet switch, which took out the site phones and 1075 network. This switch handles the common communications network for all site communications, on-site computers, and non-telescope-specific instruments and computers. The switch was replaced and it took about 24 hours to get everything back up and running correctly. It was an old switch so we had no direct spare; it was replaced with another managed switch. These need to be managed switches because they handle communications that pass through some other specific configurations for wifi security and other security protocols. Because it was a managed switch it was not a simple plug-and-play quick swap; the major part of the recovery was to get the network back up and running. Although this was all accomplished by 4 pm the next day, it did unfortunately result in one night of down time. It would be cost prohibitive to reengineer the entire network so there are no plans to overhaul the site network configuration. We are trying to source a replacement for the switch that we used as a spare and will evaluate what other steps we can take to ensure that we are better prepared for this kind of situation if it arises in the future. | APO will be hosting an Open House this coming weekend, during which we will be giving public demos, tours, and talks over a 4-hour event. |
| |
| Regarding the open positions, the hiring process is moving forward. Also, we welcome Tim McQuaid, a third-year graduate student at NMSU, who will be replacing Mark Croom as the emergency fill-in observing specialist. Tim has already begun his training so you will likely encounter him soon. | The 3.5m telescope is performing as expected for this time of year. We had a failure of one of the actuators on the secondary mirror, which froze when changing between two ports. Homing of the mirror corrected the problem with no further issues; this was caught during poor weather so no science time was lost. |
| |
| We continue to have challenges with the camera(s) on the 0.5m. It is possible that the DIS camera can be used once the decommissioning is further along. University of Virginia brought a spare Apogee camera for use on ARCSAT; it is still not known whether that can be used by the broader ARCSAT user community. | The 0.5m telescope is working as expected; Dcam-spare and UVAcam are available to users for Q4. |
| |
| The KOSMOS vacuum servicing didn't resolve the issue of high dark current so we will have to open up the cryostat to see if we can figure out what is causing a decreased hold time or increase in dark current. We are currently planning to do this during Q4; the science schedule indicates that it may not be possible until early December unless the issue becomes critical and we need to take the instrument out of service. Chip reported that a recent UWy user suggested that the dark current is currently so high that the instrument is unusable for faint targets without doubling or tripling the exposure time. Nancy will follow up directly with that user to ascertain if the problem has worsened. This would be an argument for servicing KOSMOS sooner, but we need to balance that with the numerous class visits that will be scheduled in the first two months of Q4, most of which have requested to use KOSMOS. | KOSMOS is warm and taken apart as a result of increased dark current reported by the users. At last month's Users Committee meeting we concluded that it was becoming unusable for science and the consensus was that we should not wait until December to do an inspection. Upon taking it offline it was discovered that of the four thermal connections between the LN2 cold strapping and the cold block that cools the CCD, three of them were broken; only one was providing cooling. We subsequently ordered new strapping material to do the repair; the material was delayed because it came from Europe. Once it arrived it became apparent that the repair would be more difficult than what we were comfortable doing on site. We contacted several vendors and got quotes, and ultimately the cold head block and various pieces were sent out to a shop in Albuquerque. They need to procure some materials to do the repair, but we are hoping that by the week of Oct 13-17 the repairs will be completed. Then we will pick it up, return it to APO, and do the vacuum baking out to prepare the dewar for service. Ideally we are hoping that it could return to service late in the week of Oct 13-17 but it could be later, depending on how quickly the shop can procure their needed materials and complete the repair. We will continue to update the scheduled KOSMOS users as we learn more about the timetable. |
| |
| ARCES had been displaying increased inter-order light that would indicate some form of contamination in the light path of the CCD. We warmed up half of the cryostat while keeping the shield cold so that the water vapor would migrate to shield. That did improve things significantly but subsequently things degraded somewhat; we are now hoping that it will stabilize or reverse direction. The upgrade to a modern computer for the ARCES ICC is underway. | ARCES: While working on streamlining the software for calculating IOL it was discovered that the DATE-OBS keyword in the FITS headers was wrong (45 min in the future from when the data file was written). After some data mining we found that this occurred in late August 2024 but it was not caught until now. **If time stamps are important for your echelle observations ARCES observers should take note and we can provide the detailed time offset for anyone for whom this is critical.** There was a report from a user a few weeks ago that the new ThAr bulb that was replaced in July 2025 had some weak/missing Ar lines in the blue orders. We replaced it with a different spare on September 29 and the next day we learned that the new bulb was slightly fainter than the bulb from a year ago so ARCES users should also note this. These hollow cathode lamps are no longer being made with Th metal filaments and it is increasingly hard to find spare lamps on the shelf. We just purchased a supply so that we have some spares and we hope that each one will last a few years. Given that it is becoming harder to find the old style ThAr lamps an alternative source may need to be identified. |
| |
| NICFPS and TripleSpec are operating as expected. DIS is powered down. APOLLO has been laser ranging and we were recently able to get all 5 retroreflectors but it is still not at peak performance. | ARCTIC, NICFPS, TripleSpec and APOLLO are all functioning nominally. DIS and Agile are still slated for decommissioning. |
| | |
| ---- | |
| | |
| === 2025 Q3 3.5m scheduling === | |
| | |
| There is some OPEN time remaining in September (last month of Q3). We have received requests for some of it, but not all. To request this time please follow the standard procedure by emailing your request to Ben Williams, Russet McMillan, Amanda Townsend, Nancy Chanover, and your institutional scheduler. Be sure to include the specific slot you are requesting (or specify that any time slot will do) and a short justification. | |
| |
| ---- | ---- |
| === 2025 Q4 3.5m scheduling === | === 2025 Q4 3.5m scheduling === |
| |
| The Q4 schedule is in the works; we expect to have a first draft in about a week. | The Q4 schedule is posted. There is a fair amount of OPEN/DD01 time in late October (dark time, all B halves). We hope to use some of it to pay back KOSMOS users who had to give up their time but we're not committing to that until we know when KOSMOS will be returning. Some of the scheduled KOSMOS time may need to be allocated to other users, ideally from the same institution. There is additional OPEN/DD01 time later in quarter, including a limited number of A halves or partial A halves. A huge thank you to Russet for doing such a great job with the challenging schedule for this quarter. |
| |
| ---- | ---- |
| |
| === 2025 Q3 0.5 scheduling === | === 2025 Q4 0.5 scheduling === |
| |
| September 8-17 (next week!) is the only open time remaining in the ARCSAT Q3 schedule. | The ARCSAT Q4 schedule is posted. It is pretty full, but there is some time available later in the quarter. The two available instruments are UVa-cam (larger FOV) and Dcam-spare (smaller FOV and less oversampled). Dcam-spare did have some frost on the window after shutdown; it is gone now but we can still see signatures of it on the detector. They flat-field out but the frames look ugly when taking data. |
| |
| ---- | ---- |
| === Status of new instrument initiatives === | === Status of new instrument initiatives === |
| |
| * SoonerCam: we are ordering parts! | * SoonerCam: Parts have been trickling in. So far we received the fiber and software developer's kit but we're still waiting on the fiber to coaxpress converters. We will dig into this after early November. |
| * KOSMOS grisms (no change from last month): we are still awaiting a decision on the proposal we submitted for funds to support the purchase of three new grisms for KOSMOS. | * KOSMOS grisms (no change from last month): we are still awaiting a decision on the proposal we submitted for funds to support the purchase of three new grisms for KOSMOS. |
| * ASPEN (no change from last month): we are still awaiting a decision on the proposal we submitted for funds to support the development of the new echelle spectrograph. Work on the design of the new optical echelle spectrograph continues with the goal of completing the Preliminary Design Review in early 2026. | * ASPEN: We recently learned that a proposal we submitted for funds to support the development of the new echelle spectrograph was not selected for funding. Work on the design of the new optical echelle spectrograph continues with the goal of completing the Preliminary Design Review in early spring 2026. We are hosting an ASPEN team meeting at APO and NMSU this week. |
| * Ocotillo (no change from last month): a number of purchase orders were executed and we expect the first batch of hardware to come in within the next month or so. | * Ocotillo: The vibration stabilization bench and optical table have been delivered, along with the enclosure. We are starting to get optical mounting parts and a student is working on the software and data side. Late this week the fibers should arrive, and the optics will be coming in next few weeks, including the grating. Sarah plans to submit an NSF ATI proposal for the robot side of the third channel. |
| | |
| ---- | |
| | |
| === Web site updates === | |
| | |
| We are close to having something operational. All of the requested pieces and updates were implemented. Although it is not yet finalized, users are encouraged to start using it: [[https://mainapo.apo.nmsu.edu]] UC reps: please notify your users that they should start using the new website and report any bugs to you, to be forwarded on to Ben Williams. | |
| |
| ---- | ---- |
| There are a number of events taking place over the next several months: | There are a number of events taking place over the next several months: |
| * We have 4 class groups coming APO in Q4. A fifth one will be moved to Q1. Please be patient with the site staff, who will be shorthanded during this time! | * We have 4 class groups coming APO in Q4. A fifth one will be moved to Q1. Please be patient with the site staff, who will be shorthanded during this time! |
| * There will be an in-person ASPEN team meeting at NMSU/APO in early October. | * We are in the midst of an in-person ASPEN team meeting at NMSU/APO this week. |
| * We are holding a public-facing Open House at APO on Saturday October 11. This will include site and telescope tours, activities, and talks at the Sunspot Astronomy and Visitor's Center. We look forward to welcoming the public back to the site; our last Open House was in March 2020. | * We are holding a public-facing Open House at APO on Saturday October 11. This will include site and telescope tours, activities, and talks at the Sunspot Astronomy and Visitor's Center. We look forward to welcoming the public back to the site; our last Open House was in March 2020. |
| * The ARC Board of Governors meeting is on November 4; we will request science highlights from everyone at next month's meeting, with a due date in late October. | * The ARC Board of Governors meeting is on November 4. Please submit all science highlights to Nancy no later than Monday October 27. |
| * The January 2026 AAS meeting will be in Phoenix, and we plan to have a booth. October 6 is the abstract deadline. The deadline for Splinter Session proposals is September 25; does anyone have an interest in an APO science-fest sort of session? UC reps: please poll your users and email Nancy with opinions. | * The January 2026 AAS meeting will be in Phoenix, and we plan to have a booth. There was no motivation for a special session so we did not submit a proposal. |
| |
| ---- | ---- |
| |
| Open action items from previous meetings: | Open action items from previous meetings: |
| * UC reps: ask your users to complete the survey from FAWG (per Nancy’s email sent on 7/23/25). Status: CLOSED. | * Nancy: follow up with UWy KOSMOS user to determine severity of dark current issue. Status: CLOSED. |
| | * UC reps: remind your users to review the Q3 schedule and request OPEN time in September if they can make a good case for it. Status: CLOSED. |
| | * UC reps: poll your users about a Jan. 2026 splinter session at the AAS for APO science highlights and networking (or other ideas!). Status: CLOSED. |
| |
| New action items from this meeting: | New action items from this meeting: |
| * Nancy: follow up with UWy KOSMOS user to determine severity of dark current issue. | * Nancy/Sarah: follow up with Jim Davenport about the status of PyKOSMOS and report back. |
| * UC reps: remind your users to review the Q3 schedule and request OPEN time in September if they can make a good case for it. | * Joanne Hughes: put the KOSMOS MOS data reduction software on the wiki. |
| * UC reps: poll your users about a Jan. 2026 splinter session at the AAS for APO science highlights and networking (or other ideas!). | * UC reps: remind your users to review the Q4 schedule and request OPEN/DD01 time if they can make a good case for it. |
| | * UC reps: remind your users to ** provide Nancy with science highlights from 2025; she needs to receive them no later than Monday October 27.** |
| |
| ---- | ---- |
| === All Other Business === | === All Other Business === |
| |
| None. | We discussed the timeline of events that led to the temperature issues for KOSMOS. In January 2025 the chip temperature increased. Data taken in late winter and early spring were probably not affected, but users started to notice the issue in the mid/late spring. |
| | |
| | Gordon is working on a new release for TUI (v3.2), which will be coming out soon. The new version cleaned out some old scripts and removed the option of selecting instruments that are being decommissioned. The most exciting aspect is that he is working on packaging to make the software installation process more simple for users, especially students. Although we encourage users to stay current with their TUI versions, older versions of the software will still work. |
| |
| ---- | ---- |
| === Next meeting === | === Next meeting === |
| |
| The next meeting will be on October 7, 2025 at 10:30 MDT. | The next Users Committee meeting is currently scheduled for Tuesday November 4, 2025, which directly conflicts with the ARC Board of Governors meeting. Therefore the November UC meeting is canceled, and we will meet next on Tuesday December 2, 2025, at 10:30 MST. |
| |
| ---- | ---- |
| |
