Problems Taking an Exposure
===========================

The data acquisition system for the Levy Spectrometer uses a UCAM controller
to control the CCD inside the instrument. The UCAM is an in-house system
developed by the UCO detector lab, and has been in regular use at Lick
Observatory since the late 1990's.

The hardware and software for the UCAM are both engineering-grade, and were
never brought forward to what might be considered a production-grade design,
despite widespread use at Lick. The software in particular has a number of
quirks, and staff time is not available to effect a remedy.

The scripted observing software that controls the data taking procedure will
warn if there is a problem.


The UCAM Launcher
-----------------

.. figure:: ucam_gui.png
   :align:  right

   *A running ucam_launcher.*

There is a graphical user interface that is used to in turn start the UCAM
user interface. The rationale for the existence of this meta-GUI is that it
allows keyword-based controls over whether the UCAM data taker is running,
eliminating the need for ssh'ing over to the data taking host to stop or
restart the UCAM user interface. Because this launcher tool is itself a GUI,
by construction the UCAM user interface will always come up on the correct
display.

The UCAM launcher is also set up as an APF task, the UCAMLAUNCHER task. The
keywords used to monitor and control the running state of the UCAM data taker
are::

        apftask.UCAMLAUNCHER_UCAM_STATUS
        apftask.UCAMLAUNCHER_UCAM_COMMAND

The UCAM launcher shows the current status of the UCAM software, and has a
single button on the right-hand-side that can be used to stop or start the
UCAM software, with the action performed depending on whether the UCAM software
is already running.

The UCAM launcher should only be run on warsaw, and even then only from the
APF Instrument 2 VNC session, and is invoked by running::

        % ucam_launcher

The command keyword offers three functions: starting the UCAM software,
stopping the UCAM software, and rebooting the UCAM host. Examples::

        modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Run
        modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Stop
        modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Reboot



Where the UCAM software runs
----------------------------

There is a dedicated VNC session on warsaw that is started at boot time.
The ``ucam_launcher`` tool runs inside this dedicated VNC session; by
extension, the UCAM data taker also runs inside the dedicated VNC session.

To view the VNC session, you must first ssh to warsaw, and invoke a VNC
viewer; one could also use ssh tunnels to accomplish the same result but
with improved performance. There is a helper script installed on warsaw
that doesn't do anything except invoke ``vncviewer``, and tell you to
re-invoke ``vncucam`` if/when the existing session exits. Example commands::

        % ssh warsaw
        % vncucam

This dedicated VNC session is typically open within the APF Instrument 2
(also known as :5) VNC session on apf-vnc.ucolick.org.


Problems with the UCAM software on warsaw
-----------------------------------------

If the scripted observing tool reports::

        APFUCAM keyword combo_ps is zombieprocesses, should be ok

...the UCAM software needs to be shut down. Either use the "Stop UCAM"
button on ``ucam_launcher``, or issue the following keyword modify::

        % modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Stop

After waiting a few seconds, type the following command::

        % gshow -s apfucam COMBO_PS PS%

Which should show the following::

      COMBO_PS = MissingProcesses
      PS_DESCRAM =
      PS_DISPLAY =
      PS_FITS =
      PS_NEWCAM =
      PS_READIMG =
      PS_RECORD =
      PS_TCLTALK =

To restart the UCAM softwareI, press the "Start UCAM" button on the UCAM
launcher, or issue the following keyword modify::

        % modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Run

If, and only if restarting does not work or the ``gshow`` command does
not yield the output above, the UCAM host will require a reboot. Using
``ucam_launcher`` to perform the reboot ensures that the UCAM software
is cleanly shut down before the computer shuts down::

        % modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Reboot

This will end your session on warsaw. After waiting for warsaw to finish
rebooting, which could take up to 2 minutes. The command::

        % ping warsaw

...will tell you when warsaw is back up and running. After warsaw is back
up, reconnect to the dedicated VNC session on warsaw as above, and restart
the UCAM sofware via ``ucam_launcher``.



UCAM GUI not responding
-----------------------

If the UCAM GUI stops responding or the robot reports::

        APFUCAM keyword ctalkto is ????, should be okay

...then the UCAM controller is in a state that requires it to be power
cycled. First shut down the running UCAM software, and then power cycle
the UCAM controller::

        % modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Stop
        % /usr/local/lick/bin/robot_power_cycle_ucam
        % modify -s apftask UCAMLAUNCHER_UCAM_COMMAND=Run

The use of ``ucam_launcher`` to stop and restart the UCAM data taker may
be incorporated into future revisions of the power_cycle_ucam script.