Differences

This shows you the differences between two versions of the page.

--- uc:uc20251007_october_7_2025 [2025/10/07 16:04]
nchanove created
+++ uc:uc20251007_october_7_2025 [2025/10/07 17:20] (current)
nchanove
@@ Line 3: / Line 3: @@
 ----
-Attending: Nancy Chanover (NMSU), Bill Ketzeback (APO),
+Attending: Nancy Chanover (NMSU), Aleksandr Mosenkov (BYU), Russet McMillan (APO), Anne Verbiscer (UVa), Adam Kowalski (CU), Misty Bentz (GSU), Ben Williams (UW), Moire Prescott (NMSU), Joanne Hughes (Seattle U), Chip Kobulnicky (UWy), Michael Hayden (OU),  Eric Bellm (UW), Sarah Tuttle (UW), Bill Ketzeback (APO), Gordon MacDonald (APO)
-Misty Bentz (GSU), Aleksandr Mosenkov (BYU), Anne Verbiscer (UVa), Michael Hayden (OU), Kevin Schlaufman (JHU), Joanne Hughes (Seattle U), Moire Prescott (NMSU), Chip Kobulnicky (UWy), Ben Williams (UW)
 === User feedback and comments from institutional representatives ===
-  * Colorado - nothing to report
+  * Colorado - user or two wondering about status about data reduction for KOSMOS - is PyKOSMOS finished and ready for widespead use or using standard IRAF - spectra are tilted so much that need to use Marsh algorithm rather than Horn algorithm. Documentation currently lacking. NC takes action to chase this in collaboration with Sarah
   * BYU - nothing to report
   * Oklahoma - nothing to report
   * UVa - nothing to report
-  * Wyoming - nothing to report
+  * Wyoming - nothing to report - interest in repair of KOSMOS
   * Georgia State - nothing to report
   * NMSU - nothing to report
-  * Washington - nothing to report
+  * Washington - nothing to report - echo Chip on KOSMOS repair
   * JHU - nothing to report (by email)
-  * Seattle - nothing to report
+  * Seattle - nothing to report - comment on the KOSMOS reductions - we had a college of idaho student working with Holtz on mask MOS reduction - pretty stable/student resistant, publicly available but not on the wiki yet, she would be happy to share it - will put on wiki
   * NAPG - no report
@@ Line 28: / Line 27: @@
 The detailed site report is included below, followed by additional information discussed during today's meeting.
-// 3.5-m Telescope and Instruments Highlights, 8/06/25 – 09/02/25 //
+// 3.5-m Telescope and Instruments Highlights, 9/03/25 – 10/07/25 //
 ) Overview
-The month of August experienced a mix of late afternoon monsoon storms that either delayed opening or affected many of the A half observing programs. We had two visiting instrument teams from U. Virginia this month. There was a failure of one of the main site network subnets that prevented observing and took down site phones as well as communications via TUI. Email was able to be received and sent from off site but not on site. We tracked the problem to a failure of a managed network switch and we were able to replace it with a spare. There was some difficulty loading the management given the switches were different models. The root cause is still not well understood as it looked like a cascading failure including the site DHCP server that also resulted in loss of communications with the site DNS servers. The problem took a little over 24 hours to resolve.
+Monsoonal patterns continued for much of September, with unsettled weather affecting all but a small handful of nights this month. There were no visiting classes or PI instruments staying on site during the month. However, we did host daytime tours and had remote training sessions this past month. The 3.5m Night Operations team saw the departure of both Candace Gray and Mark Croom. We wish them both in their future adventures!
-The hiring committees for the Telescope Engineer and Night Operation positions are in the process of reviewing candidates. There were a large number of applications for each opening.
+The site will be holding an open house event for the public (daytime only) on October 11. Science demonstrations, public talks, and tours of the telescopes will be the highlights of the four-hour event.
 ) Operations
-.5m Telescope: Telescope is working as expected. Seasonal motion errors have been infrequent. A full remap of the tertiary rotation was redone and this seems to have corrected the reported position errors from last month's report.
+.5m Telescope: Telescope is working as expected. Seasonal motion errors continue to occur infrequently. We anticipate that there will need to be some tuning of the axis drives by mid-November. This past Saturday, there was an occurrence where one of the three axial actuators for the secondary mirror did not move after an instrument/ port change. The operator corrected the problem by homing the secondary. The weather at the time was poor, so no time was lost.
-.5m Telescope: Telescope is working as expected.  ACP Library support errors are still occurring even with the dcam-spare camera swap but with much lower frequency. Dcam-spare had a loss of CCD chamber integrity and was forming ice on the detector while cold. The camera was serviced twice but now appears to have water spots on or near the detector surface. Cary Smith will attempt to clean but it is not without risk to the bonding wires of the detector. University of Virginia tested another surplus Apogee camera during their coordinated observing run with DSSI on the 3.5m. The off-axis guider has not been very reliable, requiring power cycles.
+.5m Telescope: Telescope is working as expected.  Dcam-spare and UVAcam are both available options for users.
-KOSMOS: System is cooled and stable. Increased dark current for long exposures has been confirmed. We attempted a vacuum servicing of the cryostat but without much luck. We anticipate having to open the vacuum vessel in the clean room sometime during Quarter 4 to investigate possible thermal shorts or opens. kcamera-ICC was backed up over shutdown.
+KOSMOS: System is cold and stable. Increased dark current for long exposures has been confirmed. As discussed last month, the increase in dark current warranted opening the cryostat for inspection. Three of the four thermal connections from the liquid nitrogen can to the CCD cold block had failed, leaving only a single thin connection. Repairs were delayed by the shipping of material from Europe. We are currently searching for a vendor to assemble the parts, as we have not had success ourselves.
-ARCTIC: The diffuser rotation mechanism is still unreliable even after a full servicing. Troubleshooting is continuing. The mechanism that moves the diffuser in and out of the optical path is still functional in the meantime. The rest of the instrument is cooled.
+ARCTIC: The diffuser rotation mechanism is still unreliable even after a full servicing. The mechanism that moves the diffuser in and out of the optical path is still functional in the meantime. The rest of the instrument is cooled.
-Agile: The camera is non-operational, the thermoelectric cooler failed again. The camera is warm and we do not plan further repair work with this camera. We are planning on decommissioning the camera. The agile instrument rotator is still not performing nominally and we are troubleshooting it further to prepare for SoonerCam.
+Agile: The camera is non-operational; the thermoelectric cooler failed again. The camera is warm, and we do not plan further repair work with this camera. We are planning on decommissioning the camera. The agile instrument rotator is still not performing nominally, and we are troubleshooting it further to prepare for SoonerCam.
-ARCES: The CCD reservoir for the cooling system was brought back to temperature. IOL levels are quite good but slowly worsening. We hope that it will stabilize and reverse direction with cooler ambient temperatures as we have seen in previous fall time frames. Work continues on a replacement and modern ICC, so far testing is going well. A commissioning report is being worked on.
+ARCES: While working on automating the interorder light reductions and archiving the data in a new database, we discovered that the FITS header KEYWORD for DATE-OBS was wrong by approximately 45 minutes in the future of when an image was written. Further data mining revealed this occurred during the summer/fall of 2024 when the ICC experienced power supply failure and hard drive troubles. It was not caught that the operating system service that syncs time to the site timeserver was not performing. The problem would have started on or after August 27, 2024. If time stamps are important for your echelle observations users should take note.  A report from a user that the new ThAr bulb, replaced on July 22nd, had very weak lines or missing lines in the mid and blue orders prompted us to replace the bulb again on September 29. Wednesday, September 30th, observers then reported that the ThAr counts were around 3250 DN last year, but only 3100 DN counts tonight. This was the last new HCL we had on the shelf. It may be these two bulbs had lower pressure argon gas fill and therefore those are the lines missing. Comparison with line lists would need to be done to confirm. More bulbs are on order; however, Thorium-Argon lamps of the same type and manufacture as we were purchasing years ago are no longer available. We are buying up one European vendors back stock but we do not know if these will be of the same lamp strength as the old bulbs.
-DIS: System is cooled but in an unknown state for science. Decommissioning plans have begun.
+DIS: System is powered down and warm. Decommissioning plans have begun.
-NICFPS: System is cooled and usable. The ICS software has needed multiple restarts over the past month. The cause is unclear at this time.
+NICFPS: System is cooled and usable.
 TripleSpec: System is cooled and usable.
@@ Line 62: / Line 61: @@
 === Additional site, telescope and instruments discussion ===
-There was a recent failure of the main site network's subnet switch, which took out the site phones and 1075 network. This switch handles the common communications network for all site communications, on-site computers, and non-telescope-specific instruments and computers. The switch was replaced and it took about 24 hours to get everything back up and running correctly. It was an old switch so we had no direct spare; it was replaced with another managed switch. These need to be managed switches because they handle communications that pass through some other specific configurations for wifi security and other security protocols. Because it was a managed switch it was not a simple plug-and-play quick swap; the major part of the recovery was to get the network back up and running. Although this was all accomplished by 4 pm the next day, it did unfortunately result in one night of down time. It would be cost prohibitive to reengineer the entire network so there are no plans to overhaul the site network configuration. We are trying to source a replacement for the switch that we used as a spare and will evaluate what other steps we can take to ensure that we are better prepared for this kind of situation if it arises in the future.
+Highlight some of the items - APO hosting Open House this coming weekend. Giving public demos, tours, talks over 4 hour event. 3.5m telescope is performing as expected for this time of year; had a significant failure one of actuators on secondary froze when changing between 2 ports. Homing of mirror correct problem with no further issues; caught during poor weather so science no time lost. 0.5m working as expected; Dcam spare and Uvacam are available to users for Q4. KOSMOS. Report incorrectly stats that cold and stable; it is not it is warm and in parts. Increased dark current reported by users; last month concluded that becoming unusable for science. Didn't want to wait for December to do inspection, Upon taking it down, 4 thermal connections between LN2 cold strapping and cold block that cools CCD; 3 of them were broken; only 1 was providing cooling. Ordered new strapping material to do a repair. Delayed because had to come from Europe. Once arrived we did not have correct knowledge to repair; much more difficult than what we can do on site. Reached out to vendors, got quotes, cold head block and various pieces sent out to shop in Albq. Need to procure materials. Hoping by early next week the repairs will be completed and picked up, then vacuum cleaning baking out, preparing dewar for service. Hoping for end of next week for return to service.
-Regarding the open positions, the hiring process is moving forward. Also, we welcome Tim McQuaid, a third-year graduate student at NMSU, who will be replacing Mark Croom as the emergency fill-in observing specialist. Tim has already begun his training so you will likely encounter him soon.
+Nothing new on ARCTIC. Echelle - working on steamlining reaction software for calculating IOL; discovered that DATE-OBS keyword was wrong (45 min in the future from when image was written) - then did some data mining; this occurred in late August 2024 but not caught until now. If time steps are important for your echelle observations - ARCES observers should take note and we can provide the detailed time offset for anyone for whom this is critical. Report by a user a few weeks ago that new Th Ar bulb replaced in July 2025 had some weak/missing Ar lines in blue orders. We replaced with a different spare - did this on Sep 29. On Sep 30 reports that the new bulb was slightly fainter than bulb a year ago. Reduction process is more efficient now. Users should also note this. Hollow cathode lamps - can no longer obtain Th metal filaments; increasingly hard to find on the shelf. We just purchased some supply so that we have spares - hope that each one will last a few years each. Becoming harder to find old style THAr lamps. Alternative source may need to be identified.
-We continue to have challenges with the camera(s) on the 0.5m. It is possible that the DIS camera can be used once the decommissioning is further along. University of Virginia brought a spare Apogee camera for use on ARCSAT; it is still not known whether that can be used by the broader ARCSAT user community.
+Agile/DIS decommissioning.
-The KOSMOS vacuum servicing didn't resolve the issue of high dark current so we will have to open up the cryostat to see if we can figure out what is causing a decreased hold time or increase in dark current. We are currently planning to do this during Q4; the science schedule indicates that it may not be possible until early December unless the issue becomes critical and we need to take the instrument out of service. Chip reported that a recent UWy user suggested that the dark current is currently so high that the instrument is unusable for faint targets without doubling or tripling the exposure time. Nancy will follow up directly with that user to ascertain if the problem has worsened. This would be an argument for servicing KOSMOS sooner, but we need to balance that with the numerous class visits that will be scheduled in the first two months of Q4, most of which have requested to use KOSMOS.
+NICFPS and Tspec working nominally as is APOLLO.
-ARCES had been displaying increased inter-order light that would indicate some form of contamination in the light path of the CCD. We warmed up half of the cryostat while keeping the shield cold so that the water vapor would migrate to shield. That did improve things significantly but subsequently things degraded somewhat; we are now hoping that it will stabilize or reverse direction. The upgrade to a modern computer for the ARCES ICC is underway.
-NICFPS and TripleSpec are operating as expected. DIS is powered down. APOLLO has been laser ranging and we were recently able to get all 5 retroreflectors but it is still not at peak performance.
-----
-=== 2025 Q3 3.5m scheduling ===
-There is some OPEN time remaining in September (last month of Q3). We have received requests for some of it, but not all. To request this time please follow the standard procedure by emailing your request to Ben Williams, Russet McMillan, Amanda Townsend, Nancy Chanover, and your institutional scheduler. Be sure to include the specific slot you are requesting (or specify that any time slot will do) and a short justification.
 ----
@@ Line 84: / Line 73: @@
 === 2025 Q4 3.5m scheduling ===
-The Q4 schedule is in the works; we expect to have a first draft in about a week.
+The Q4 schedule is posted. Fair amount of OPEN/DD01 time in late October (dark). All B halves. Hoping to use it to pay back some of the KOSMOS users but not committing until know when KOSMOS returning. Some of KOSMOS time may need to be allocated to other users. Ideally from same institution. Additional OPEN/DD01 time later in quarter, including limited number of A halves or partial A halves. Thank you Russet for doing such a good job with the challenges for this quarter.
 ----
-=== 2025 Q3 0.5 scheduling ===
+=== 2025 Q4 0.5 scheduling ===
-September 8-17 (next week!) is the only open time remaining in the ARCSAT Q3 schedule.
+The ARCSAT Q4 schedule is posted. Pretty full, but this week is open (rest of this week). Some other time available later in the quarter. Two instruments are UVa-cam (larger FOV), Dcam-spare, smaller FOV and less oversampled but Dcam-spare did have some frost after shutdown; gone now but see signatures on detector; they flat field out but looks ugly when takking data.
 ----
@@ Line 96: / Line 85: @@
 === Status of new instrument initiatives ===
-  * SoonerCam: we are ordering parts!
+  * SoonerCam: parts have been trickling in. Received fiber, software dev kit. Waiting on fiber to coax press converters, arriving today? Not computer yet. dig in after november
   * KOSMOS grisms (no change from last month): we are still awaiting a decision on the proposal we submitted for funds to support the purchase of three new grisms for KOSMOS.
-  * ASPEN (no change from last month): we are still awaiting a decision on the proposal we submitted for funds to support the development of the new echelle spectrograph. Work on the design of the new optical echelle spectrograph continues with the goal of completing the Preliminary Design Review in early 2026.
+  * ASPEN: We recently learned that a proposal we submitted for funds to support the development of the new echelle spectrograph was not selected for funding. Work on the design of the new optical echelle spectrograph continues with the goal of completing the Preliminary Design Review in early spring 2026.
-  * Ocotillo (no change from last month): a number of purchase orders were executed and we expect the first batch of hardware to come in within the next month or so.
+  * Ocotillo: Now have vibration stabilization and optical table are in house, enclosure too. Started to get optical mounting parts. Student working on software and data side. late this week fibers are coming. Optics are coming in next few weeks (end of the month) including the grating by mid-November (ahead of schedule?). Submitting an ATI for robot side of channel.
-----
-=== Web site updates ===
-We are close to having something operational. All of the requested pieces and updates were implemented. Although it is not yet finalized, users are encouraged to start using it: [[https://mainapo.apo.nmsu.edu]] UC reps: please notify your users that they should start using the new website and report any bugs to you, to be forwarded on to Ben Williams.
 ----
@@ Line 113: / Line 96: @@
 There are a number of events taking place over the next several months:
   * We have 4 class groups coming APO in Q4. A fifth one will be moved to Q1. Please be patient with the site staff, who will be shorthanded during this time!
-  * There will be an in-person ASPEN team meeting at NMSU/APO in early October.
+  * We are in the midst of an in-person ASPEN team meeting at NMSU/APO this week.
   * We are holding a public-facing Open House at APO on Saturday October 11. This will include site and telescope tours, activities, and talks at the Sunspot Astronomy and Visitor's Center. We look forward to welcoming the public back to the site; our last Open House was in March 2020.
-  * The ARC Board of Governors meeting is on November 4; we will request science highlights from everyone at next month's meeting, with a due date in late October.
+  * The ARC Board of Governors meeting is on November 4. Please submit all science highlights to Nancy no later than Monday October 27.
-  * The January 2026 AAS meeting will be in Phoenix, and we plan to have a booth. October 6 is the abstract deadline. The deadline for Splinter Session proposals is September 25; does anyone have an interest in an APO science-fest sort of session? UC reps: please poll your users and email Nancy with opinions.
+  * The January 2026 AAS meeting will be in Phoenix, and we plan to have a booth. There was no motivation for a special session so we did not submit a proposal.
 ----
@@ Line 123: / Line 106: @@
 Open action items from previous meetings:
-  * UC reps: ask your users to complete the survey from FAWG (per Nancy’s email sent on 7/23/25). Status: CLOSED.
-New action items from this meeting:
   * Nancy: follow up with UWy KOSMOS user to determine severity of dark current issue.
   * UC reps: remind your users to review the Q3 schedule and request OPEN time in September if they can make a good case for it.
   * UC reps: poll your users about a Jan. 2026 splinter session at the AAS for APO science highlights and networking (or other ideas!).
+New action items from this meeting:
+  * xxx
 ----
@@ Line 135: / Line 117: @@
 === All Other Business ===
-None.
+Timeline for KOSMOS? Jan. chip temp increase. Late winder/early spring prob not affected. Mark Croom
 ----
@@ Line 141: / Line 123: @@
 === Next meeting ===
-The next meeting will be on October 7, 2025 at 10:30 MDT.
+The next meeting is currently scheduled for Tuesday November 4, 2025. This directly conflicts with the ARC Board meeting; shall we reschedule for one week later or cancel the November meeting and meet again in December?
+Gordon - working on new release for TUI - v3.2 - plans to get it out soon. Had some instruments decommissioned, cleaned out old scripts. Most exciting aspect is working on packaging to make install process more simple for users, especially students. Get the changes automated to be download to Git Hub. Prev versions will still work but encourage uses to keep updated.
 ----

apo-wiki

User Tools

Site Tools

Differences

Page Tools