|
Next revision
|
Previous revision
|
uc:uc20250204_february_4_2025 [2025/02/04 16:46]
nchanove created |
uc:uc20250204_february_4_2025 [2025/02/04 18:50] (current)
nchanove |
| ---- | ---- |
| |
| Attending: Nancy Chanover (NMSU), | Attending: Nancy Chanover (NMSU), Kevin Schlaufman (JHU), Zach Berta-Thompson (CU), Aleksandr Mosenkov (BYU), Bill Ketzeback (APO), Eric Bellm (UW), Misty Bentz (GSU), Russet McMillan (APO), Anne Verbiscer (UVa), Moire Prescott (NMSU), Joanne Hughes (Seattle U), Michael Hayden (OU) |
| | |
| Aleksandr Mosenkov (BYU), Bill Ketzeback (APO), Eric Bellm (UW), Anne Verbiscer (UVa), Moire Prescott (NMSU), Moire Prescott (NMSU), Russet McMillan (APO), Misty Bentz (GSU), Kevin Schlaufman (JHU), Zach Berta-Thompson (CU), Joanne Hughes (Seattle U), Ben Williams (UW), Chip Kobulnicky (UWy) | |
| |
| === User feedback and comments from institutional representatives === | === User feedback and comments from institutional representatives === |
| * Wyoming - nothing to report | * Wyoming - nothing to report |
| * BYU - nothing to report | * BYU - nothing to report |
| * Oklahoma - no report | * Colorado - nothing to report |
| * JHU - nothing to report | * JHU - nothing to report |
| * NMSU - nothing to report | * NMSU - Moire has not been receiving the quarterly institutional passwords by postal mail - Bill will follow up |
| | * Oklahoma - nothing to report |
| * Georgia State - nothing to report | * Georgia State - nothing to report |
| * Colorado - the ARCTIC diffuser rotator stopped working and users are asking about the timeline for getting it repaired; one user had a weird issue with KOSMOS when it stopped taking exposures | |
| * UVa - nothing to report | * UVa - nothing to report |
| * NAPG - no report | * NAPG - no report |
| The detailed site report is included below, followed by additional information discussed during today's meeting. | The detailed site report is included below, followed by additional information discussed during today's meeting. |
| |
| // 3.5-m Telescope and Instruments Highlights, 12/03/24 – 01/07/25 // | // 3.5-m Telescope and Instruments Highlights, 1/08/25 – 02/04/25 // |
| |
| 1) Overview | 1) Overview |
| |
| Weather wise, December was very mild in terms of both temperatures and precipitation. There were no astronomers, instrument teams or classes that visited APO in the month of December. | Weather wise, we had a very cold January with a mix of clear and cloudy nights. Numerous cold related problems were present in January, detailed in the following report. There were no astronomers, instrument teams or classes that visited APO in the month of January. |
| |
| 2) Operations | 2) Operations |
| |
| 3.5m Telescope: Telescope is working as expected. | The main site generator has been experiencing a few problems since the start of the new year. Fuel contamination is suspected and we obtained some test kits to determine the cause. We are waiting for the results to see if it is bacterial or water in the diesel. A repair of some wiring was also required as the main electrical transfer switch was not sensing enough voltage from the generator to switch over on a drop of utility power. In the meantime the backup generator has been on standby. |
| | |
| 0.5m Telescope: Telescope is working as expected. | An issue with the computer UPS occurred after a planned replacement of three quarters of the backup batteries. Batteries were only charging to a fraction of their full load and runtime was quite short. The problem has been resolved and the unit is now 100% charged with the expected hour of backup runtime should we lose all utility and backup power. This would be enough time to gracefully shutdown servers. |
| | |
| KOSMOS: nothing to report. | 3.5m Telescope: Telescope is working as expected. Following a planned power outage for line maintenance by the local power company, there was a failure of a circuit breaker to the enclosure motor amplifier cabinet that prevented moving the building one night. Disassembly and cleaning the breaker the next day allowed us operate until a replacement could be obtained and installed. |
| | |
| ARCTIC: We performed another emergency vacuum service due to the vacuum softening. The current hypothesis for the cause of this issue is that we may have gotten/installed an ion pump that had a defective/reduced lifetime; it is still struggling to maintain vacuum. We plan to replace it ASAP with minimal or no impact to the science schedule. In the meantime, we have restricted instrument rotation to prevent the ion pump from shorting out when upside down, which seems to be a pattern. Additional maintenance on the diffuser mechanism will be scheduled as well given the reports of intermittent behavior at times. | 0.5m Telescope: Telescope is working as expected. |
| | |
| Agile: It is being worked on at the repair house to investigate the hardware that was changed and how that impacts observing modes/gain settings. APO software control was not able to configure the camera as expected when it was initially tested at APO after the repairs. | KOSMOS: Experienced multiple failures of the slit viewer shutter when temperatures dipped near and below the -10C rating. We replaced the shutter with a spare. Some adjustment to guide optics was also needed as the image scale was changed during the shutter repair. Additional observing overhead was required in the meantime to center up targets using the science detector in imaging mode. The disperser and mask wheels also have been struggling with colder ambient temperatures during this time, making use of the instrument challenging during part of the colder nights. |
| | |
| ARCES: The system is up, however, over the holiday break there was an error indicating a failure to read images, which led to the instrument control software being restarted. That restart led to additional errors when the system recovered in an unexpected way. We are investigating this further. Science was/is not affected, this only impacted daily engineering IOL data. | ARCTIC: Maintenance on the diffuser mechanism will be scheduled as well given the reports of intermittent behavior at times. |
| | |
| | Agile: The camera is back from the repair house for local bench testing with APO software for failure to software select gain settings as well as a demo camera for testing. The earlier problems with the TEC and readout of the CCD have been resolved. We are waiting to hear back from the repair house if they learned enough from the testing we did to try again to repair the camera or if we will reinstall with manual gain settings. |
| | |
| | ARCES: One reported failure of the sky/cal mirror was reported. This certainly is another cold temperature related problem as the motor driver is at the limit of its current output. The system is currently usable. |
| DIS: System is currently up and usable. | DIS: System is currently up and usable. |
| | |
| NICFPS: System is cooled and usable. | NICFPS: System is cooled and usable. |
| | |
| TripleSpec: System is up and usable. | TripleSpec: System is up and usable. There was one report of the main detector reading all zeros. A reset of the camera controller corrected the problem. |
| | |
| APOLLO: system leaks were repaired and the system is in operation. | APOLLO: Installed a new chiller for the laser. We have ranged a few times since installation without seeing abnormal operating temperatures. The system is in operation. |
| |
| ---- | ---- |
| === Additional site, telescope and instrument discussion === | === Additional site, telescope and instrument discussion === |
| |
| Nothing to add other than what was in agenda. There is a multi-day weather system coming in tonight, which could include snow. The KOSMOS weirdness that Zach reported above has been resolved. There was an emergency vacuum servicing done for ARCTIC; the ion pump was replaced this past weekend. ARCTIC was used B half last night without issue. Russet suggested that we remove the rotation restriction for ARCTIC. The diffuser rotator repair is still to be scheduled; it was tensioned but the rotation sensor is not working properly. Agile is still at the repair shop. ARCES is back and usable after the issue with daytime metrology data described in the site report above. Russet will check with Amanda about headers not being written properly for KOSMOS; this anomaly affected Zach BT a few weeks ago. | The site had a few power related issues recently. There was fuel contamination in the backup generator; a wiring issue in the generator was also repaired. A full servicing of the main generator is complete and it is ready to be put into service for full back-up utility power (as of this morning). There was an issue with the computer room UPS that led to servicing in January; the batteries weren't fully charging but it is fixed now. We are investigating whether we can get that unit back on a service contract so if components fail in the future it's not as expensive for us to repair. |
| | |
| | During a recent routine, scheduled power outage, after coming back to main utility power a breaker that controls the building enclosure controller amplifier failed. We didn't have a spare so the telescope was down for a night because we couldn't move the enclosure building. It was brought back up the next day (the technical staff worked out a temporary solution until the spare was received and put into place). Hopefully we will not be switching back and forth between generator and utility power for testing/burning off fuel too much more. |
| | |
| | Both the 3.5m and 0.5m telescopes are working as expected. The cold weather in January led to some issues with some of the instruments (primarily with shutters). The KOSMOS slit viewer shutter was sticky when temperatures fell below its temperature rating. We replaced it with a spare and are looking to get more spares on hand. The disperser and mask wheels also were struggling in cold weather. The ARCTIC diffuser mechanism is also something that we're hoping to schedule a repair for during next engineering time. Agile is back and we're working with the repair shop on a path forward (reinstall as is or have them try to repair it further; the programmable gain still not functioning). The rest of the instruments have all been behaving nominally. TripleSpec had an issue with the detector reading zeros but a reset of the camera controller did the trick. |
| | |
| | Misty asked if we have discussed any long-term solutions to avoiding these instrument cold issues in light of the extreme weather we have been (and likely will continue to be) experiencing. Bill answered that when we install instrumentation or spec new instruments, we try to spec them for an operating temperature of -20C. If we get down to -15C that's our //de facto// closure limit (because that's where the telescope itself starts having issues). However, there are some parts for which we cannot get manufacturers to give us an extended cold temperature range. But this is not something that we've spent a lot of additional time trying to find other solutions for. |
| |
| ---- | ---- |
| === 2025 Q1 3.5m scheduling === | === 2025 Q1 3.5m scheduling === |
| |
| We have some unassigned time in Q1 (mostly B halves). This time is listed as OPEN on the Q1 schedule. Users should look at the schedule and follow the usual channels for requesting this time (i.e. email Russet, Amanda, Ben, Nancy and their institutional scheduler when submitting requests, and provide a proposal cover page if you don't already have a program scheduled for the current quarter). | We have some unassigned time in Q1 (mostly B halves) - most urgently Feb 08B, but there are a few others in February and March as well. We have received some requests for this time already but if there are any urgent requests or high priority (e.g. students) programs please let us know ASAP, otherwise we will allocate time (at least the slots in the first few weeks of February) to the people who already made requests. This time is listed as OPEN on the Q1 schedule. Users should look at the schedule and follow the usual channels for requesting this time (i.e. email Russet, Amanda, Ben, Nancy and their institutional scheduler when submitting requests, and provide a proposal cover page if you don't already have a program scheduled for the current quarter). |
| |
| ---- | ---- |
| |
| === 2025 Q1 0.5 scheduling === | === 2025 Q2 3.5m scheduling === |
| |
| The 2025 Q1 ARCSAT scheduled is posted. We only had 2 requests this quarter (note added in proof: we received a third request, so there are now 3 programs scheduled to use ARCSAT in Q1). We are continuing to struggle with frequent failures in communication with the camera (which manifests as a "support library error"); FlareCam is getting old. The temporary fix is fairly quick so if observers can keep an eye on it and report errors it can be recovered pretty quickly. In early March we will have an onsite class and other stuff going on so ARCSAT is not available during weeks when the schedule is marked BUSY. | The allocations were sent to all institutions yesterday; proposals are due on February 24. A few items to note: (1) any users who request to use Agile in Q2 should specify a backup instrument, (2) DIS is offered in shared risk mode only and has not been serviced for several quarters, (3) the allocation emails also contain a breakdown of the split between A halves and B halves. Institutions should try to avoid requesting all of their time in A's (or B's) since we will not be able to accommodate everyone's requests if that happens. |
| |
| ---- | ---- |
| |
| === Website Modernization Effort === | === 2025 Q1 0.5 scheduling === |
| | |
| | The 2025 Q1 ARCSAT scheduled is posted. There is available time that people can request in the next week and a half, and then again in the last week of March. In early March we will have an onsite class and other stuff going on so ARCSAT is not available during weeks when the schedule is marked BUSY. |
| |
| There is a plan to run a test tomorrow for converting the site to new servers (but this may be postponed due to illness). | |
| |
| ---- | ---- |
| | |
| |
| === IT Security === | === IT Security === |
| |
| The IT team at APO is still researching applications to improve password handling; all we know right now is that the switching of the TUI password is coming. | No new announcements. We are trying to resolve some recent issues related to TUI logins; lots of work is going on behind the scenes but there is no news or action items for the Users Committee this month. |
| |
| ---- | ---- |
| === Status of new instrument initiatives === | === Status of new instrument initiatives === |
| |
| We have heard no news from NSF regarding the pre-proposal that was submitted for ASPEN; we hope to hear soon. Sarah reported that since having received the Ocotillo CDR report on December 2 her team has been evaluating the comments and preparing a response. They also applied for some funding and are waiting to hear back. It is clear that the instrument will come together in a phased approach, bringing the three channels online one at a time. They will start with the IFU and follow up with MOS component, which is more costly and mechanically complicated. The hope is that this would enable supported data reduction development along the way and troubleshooting as we go. It also spaces out the cost, which reduces impact on the overall observatory budget. Their goal is to have their response report back to the CDR review panel by the end of the month. | We still have heard no news from NSF regarding the pre-proposal that was submitted for ASPEN. The UW Telescope Engineering Group has been working on their response to the Ocotillo Critical Design Review report; they expect to submit it to Nancy next week. There have been some preliminary discussions about the possibility of replacing Agile with a more modern and capable high-speed imaging camera. This is in the very early stages of discussion but if there are users who would want to take advantage of this capability they should contact Nancy so we can make sure that we can advocate for those science cases. |
| |
| ---- | ---- |
| === DIS Decommissioning Plans === | === DIS Decommissioning Plans === |
| |
| Nancy is working on developing a document that lays out the decommissioning plan for DIS. The instrument has not been serviced and will only be available in Q1 (and Q2) in a shared risk mode. | Nancy has drafted an instrument decommissioning document that she and Bill are working on. The hope is to have this general document finalized before the end of the month so that it can be applied to DIS. The first order of business will be to issue a formal notification to all stakeholders about our plans to decommission DIS. |
| |
| ---- | ---- |
| |
| Open action items from previous meetings: | Open action items from previous meetings: |
| * Nancy: get ROM costs for new KOSMOS grisms. Sarah reported that she has a finalized quote from one vendor. The grism design turns out to be a little complicated, but also, costs have gone up, so these are more expensive than we expected (ballpark $20k apiece). Therefore, we need to explore potential funding sources. We have reached an impasse in terms of prioritizing the desired grism; Nancy will amass all previous emails with grism requests and send to UC reps to query users about whether there is broad interest in a particular grism. This could help identify funding sources, either individually or collaboratively. STATUS: OPEN. Nancy has started a proposal to the Mt. Cuba Foundation for several grisms and hopes to have it submitted before the next Users Committee meeting. | * Nancy: get Mt. Cuba Foundation proposal for new KOSMOS grisms submitted. STATUS: OPEN. The proposal is done; Nancy needs to distribute it to the users who requested the grisms to ensure that their science cases are adequately described. She plans to submit it before the next Users Committee meeting. |
| * Nancy: Ensure that a short description of the impact of satellites be posted on the APO website somewhere and updated every 6 months or so. Nancy received examples of this issue from Anne Verbiscer and will get something added to the current (and new) web site. STATUS: OPEN. | * Nancy: Ensure that a short description of the impact of satellites be posted on the APO website somewhere and updated every 6 months or so. Nancy received examples of this issue from Anne Verbiscer and will get something added to the current (and new) web site. STATUS: OPEN. |
| * UC reps: remind your users to review the Q1 schedule and request OPEN/DD time if they can make a good case for it. STATUS: OPEN. | * UC reps: remind your users to review the Q1 schedule and request OPEN/DD time if they can make a good case for it. STATUS: OPEN. |
| === All Other Business === | === All Other Business === |
| |
| None | There was a question about the development of Ocotillo (and other instruments) and whether it is contingent on getting federal funds (e.g. from NSF). This led to the question of how reliant APO is (more broadly) on federal funds. These items were discussed in light of the recent chaos resulting from the suspension of federal grant payments until they can be confirmed to be compliant with the recent Executive Orders. |
| |
| ---- | ---- |
| === Next meeting === | === Next meeting === |
| |
| The next meeting will be on February 4, 2025 at 10:30 MST. | The next meeting will be on March 4, 2025 at 10:30 MST. |
| |
| ---- | ---- |
| |
