Secondary Tip/Tilt Fail Safe¶
The actuators for the secondary mirror (M2) can trigger a tip/tilt failsafe.
DRAFT Procedure for Reseting the Telescope DRAFT¶
Engage on a physical emergency stop .
Carefully lower the telescope and pin it using the physical pin, at the lowest position possible.
Set up the ladder to access the secondary mirror structure.
The tip/tilt failsafes are a set of three rocker switches arranged around an off-axis metal rod that is attached to the mounting plate for the secondary, see picture below. The switches are normally in contact with the rod at all times. This rod moves with the tip and tilt of the secondary, while the rocker switches are mounted to a plate on the stationary M2 support structure.
Short the switch, either with a large IC jumper lead or a DMM. Voltage across the switch that is tripped should be just over 24VDC. Extra care is needed not to short the connections to the very nearby frame.
At this point, the the eostdio.FSFLTSBT keyword, which had been reporting True, should transition back to False.
One of them should be at a location where they can both watch M2 AND promptly engage a physical E-stop if needed.
Unset the physical estop.
Disengage the software e-stop.
Enable azimuth power.
Shut down the telescope server via the EOS system management interface.
Via rdesktop to tcc, rdesktop -g 1100x900 tcc; username: eost passwd: the usual
run the pewinpro32 (or whatever it’s called, some combination of those letters) that is a shortcut on the desktop.
tcc may reboot at this point. If it does, wait, and start over. In some cases, if TCC reboots, it may be necessary for someone in level one of the dome to log into TCC using its local, pull-out/fold-up console and answer “No” the Microsoft pop-up questions about “A serious error has occurred. Do you want to send a report?”. The TCC and its console are located in the tall blue “Telescope Control” rack on level one of the dome, next to the long table with the 3 LCD monitors.
- In pewin:
^D ; That’s a ctrl-D; shut down on-board programs
M0214=1 ; Turn on servo amplifier for motor #2
#2o0 ; Zero output on motor #2
- #2J^-5000 ; Move motor #2 -5,000 encoder counts, verify
that the encoder is moving in the correct direction
#2J^-5000 ; Do it again, for sanity’s sake
#2J^-10000 ; Start ramping up the distance
#2J^-50000
#2J^-100000 ; At this point, the limit disengaged; jumper removed
#2J^-130000
#2J^-100000
#2J^-200000
#2J^-200000 ; Back near the ‘nominal’ position
$$$ ; Reset the PMAC controller
pewin must be shut down before starting the telescope server back up again. This may also cause the tcc computer to reboot. See above.
Restart the tcc services are up and running, telescope DIO, i8000, GPS
Safe the telescope again by removing power and setting the e-stop.
Remove short in M2 and confirm that the eostdio.FSFLTSBT keyword continues to report “False”. If so, then remove ladder and proceed.
Reenage the e-stops.
Slowly and safely move the telescope “by hand” back to zenith after removing pin. If the mirror cover is closed, the telescope is quite bottom heavy and will want to race to the zenith if not restrained.
Have the staff in the dome move to a safe place.
Clear the e-stops, power up the telescope.
Open the mirror covers.
- If this is during the day, disable sun avoidance:
- use device browser
- select telescope server
- on the command panel (lower right) find Telescope
- Execute Disable Sun Avoidance
Disable the telescope and then home.
We discuss our first encounter with this problem below.
Aug. 22 Event¶
The homing sequence failed, as the secondary drove itself into a tip/tilt failsafe limit. In the homing sequence, all three actuators for the secondary home at the same time; they reached the same “nominal” position, but then the B actuator (PMAC motor axis #2) took off in the positive direction, while the other two actuators remained stationaryThis triggered the tip/tilt failsafe, without the B actuator running into one of its end-of-travel limits. There may be a problem with the index mark for the B actuator, or otherwise some problem with the Heidenhain linear encoder for this actuator. Naturally, we have no spares.
The tip/tilt failsafes are a set of three rocker switches arranged around an off-axis metal rod that is attached to the mounting plate for the secondary. The switches are normally in contact with the rod at all times. This rod moves with the tip and tilt of the secondary, while the rocker switches are mounted to a plate on the stationary M2 support structure. Because this apparatus is off-axis, these switches will not be engaged at the same tip/tilt for all three actuators.
If one of these tip/tilt failsafe switches is activated, it inhibits (via software, not a hardware interlock) any/all motor driven control for the telescope, making it impossible to recover without on-site intervention. If you do try to move something, an asynchronous check will shut down the offending motion in less than half a second. With patience, this is sufficient time to close the mirror cover, if you’re willing to repeatedly initiate mirror close operations.
There are two ways to recover from the limit switch failure.
The first is to back out the stage by hand: this necessitates disassembling the motor housing to gain access to the lead screw. This is accomplished by removing two screws at the M2-end of the motor housing, and sliding back the metal cover over that section of the assembly.
The second is to jumper the pins on the activated limit switch, which nicely bypasses the limit condition (the switch opens the circuit when activated). One can then move the actuator via software control, but _not_ via the full telescope control interface: one must use the low-level PMAC control software (pewin) to carefully issue the necessary commands.
Via the keyword history database, we could see that the B actuator had run itself in the positive direction by some 800,000 encoder counts (1 millimeter of physical travel). We thus knew that the actuator needed to move the same distance in the negative direction to return to nominal, and that the tip/tilt failsafe should disengage well before moving that amount.
After Donnie jumpered the pins on the switch, the eosdio.FSFLTSBT keyword, which had been reporting True, immediately transitioned back to False. I disengaged the software e-stop, enabled azimuth power, and in pewin issued the following low-level commands:
^D ; That's a ctrl-D; shut down on-board programs
M0214=1 ; Turn on servo amplifier for motor #2
#2o0 ; Zero output on motor #2
#2J^-5000 ; Move motor #2 -5,000 encoder counts
#2J^-5000 ; Do it again, for sanity's sake
#2J^-10000 ; Start ramping up the distance
#2J^-50000
#2J^-100000 ; At this point, the limit disengaged; jumper removed
#2J^-130000
#2J^-100000
#2J^-200000
#2J^-200000 ; Back near the 'nominal' position
$$$ ; Reset the PMAC controller
After Donnie and Paul gave the all-clear, we then performed another full homing sequence, which succeeded. Many thanks to Paul for his patience through the extended troubleshooting session, and for being a good sport about discovering that the cap at the far end of the M2 assembly is not at all hollow, and easily weighs more than fifty pounds.
Reseting the switch¶
The limit switch that tripped last time was the most difficult to reach electrically. You may be able to short it with a large IC jumper lead. I used the DMM in the box on the desk set to DC current in order to short the switch connections (it’s a normaly closed switch) while Kyle sent commands to move the mirror out of a limit. Extra care is needed not to short the connections to the very nearby frame. Voltage across the switch that is tripped should be just over 24VDC. The ladder up there will likely be needed as well.