Problems Taking an Exposure

The data acquisition system for the Levy Spectrometer uses a UCAM controller to control the CCD inside the instrument. The UCAM is an in-house system developed by the UCO detector lab, and has been in regular use at Lick Observatory since the late 1990’s.

The hardware and software for the UCAM are both engineering-grade, and were never brought forward to what might be considered a production-grade design, despite widespread use at Lick. The software in particular has a number of quirks, and staff time is not available to effect a remedy.

The scripted observing software that controls the data taking procedure will warn if there is a problem.

The UCAM Launcher

A running ucam_launcher.

There is a graphical user interface that is used to in turn start the UCAM user interface. The rationale for the existence of this meta-GUI is that it allows keyword-based controls over whether the UCAM data taker is running, eliminating the need for ssh’ing over to the data taking host to stop or restart the UCAM user interface. Because this launcher tool is itself a GUI, by construction the UCAM user interface will always come up on the correct display.

The UCAM launcher is also set up as an APF task, the UCAMLAUNCHER task. The keywords used to monitor and control the running state of the UCAM data taker are:

apftask.UCAMLAUNCHER_UCAM_STATUS
apftask.UCAMLAUNCHER_UCAM_COMMAND

The UCAM launcher shows the current status of the UCAM software, and has a single button on the right-hand-side that can be used to stop or start the UCAM software, with the action performed depending on whether the UCAM software is already running.

The UCAM launcher should only be run on warsaw, and even then only from the APF Instrument 2 VNC session, and is invoked by running:

% ucam_launcher

The command keyword offers three functions: starting the UCAM software, stopping the UCAM software, and rebooting the UCAM host. Examples:

modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Run
modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Stop
modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Reboot

Where the UCAM software runs

There is a dedicated VNC session on warsaw that is started at boot time. The ucam_launcher tool runs inside this dedicated VNC session; by extension, the UCAM data taker also runs inside the dedicated VNC session.

To view the VNC session, you must first ssh to warsaw, and invoke a VNC viewer; one could also use ssh tunnels to accomplish the same result but with improved performance. There is a helper script installed on warsaw that doesn’t do anything except invoke vncviewer, and tell you to re-invoke vncucam if/when the existing session exits. Example commands:

% ssh warsaw
% vncucam

This dedicated VNC session is typically open within the APF Instrument 2 (also known as :5) VNC session on apf-vnc.ucolick.org.

Problems with the UCAM software on warsaw

If the scripted observing tool reports:

APFUCAM keyword combo_ps is zombieprocesses, should be ok

...the UCAM software needs to be shut down. Either use the “Stop UCAM” button on ucam_launcher, or issue the following keyword modify:

% modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Stop

After waiting a few seconds, type the following command:

% gshow -s apfucam COMBO_PS PS%

Which should show the following:

COMBO_PS = MissingProcesses
PS_DESCRAM =
PS_DISPLAY =
PS_FITS =
PS_NEWCAM =
PS_READIMG =
PS_RECORD =
PS_TCLTALK =

To restart the UCAM softwareI, press the “Start UCAM” button on the UCAM launcher, or issue the following keyword modify:

% modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Run

If, and only if restarting does not work or the gshow command does not yield the output above, the UCAM host will require a reboot. Using ucam_launcher to perform the reboot ensures that the UCAM software is cleanly shut down before the computer shuts down:

% modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Reboot

This will end your session on warsaw. After waiting for warsaw to finish rebooting, which could take up to 2 minutes. The command:

% ping warsaw

...will tell you when warsaw is back up and running. After warsaw is back up, reconnect to the dedicated VNC session on warsaw as above, and restart the UCAM sofware via ucam_launcher.

UCAM GUI not responding

If the UCAM GUI stops responding or the robot reports:

APFUCAM keyword ctalkto is ????, should be okay

...then the UCAM controller is in a state that requires it to be power cycled. First shut down the running UCAM software, and then power cycle the UCAM controller:

% modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Stop
% /usr/local/lick/bin/robot_power_cycle_ucam
% modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Run

The use of ucam_launcher to stop and restart the UCAM data taker may be incorporated into future revisions of the power_cycle_ucam script.