Prison workers have called for a vote of no confidence in the Territory’s Corrections Commissioner Matthew Varley.