Zero-Loss Spot GPU Training: Automated S3 Checkpointing and SIGTERM Handlers

Spot GPU preemption follows a predictable lifecycle: (1) Provider signals preemption (30-120 seconds notice depending on provider). (2) SIGTERM sent to running process. (3) GPU compute halted. (4) Instance terminated. The critical window is between SIGTERM and termination — you have 30 seconds to save state. For training jobs, this means: checkpoint every 30 minutes minimum, and implement SIGTERM trapping to trigger emergency checkpoint. Vast.ai gives 30s, RunPod gives 30s, Lambda gives 30s. All providers treat SIGTERM as 'save and exit'.

Standard PyTorch training loops do not handle SIGTERM. You must register a signal handler: import signal, torch, os def emergency_checkpoint(signum, frame): print(f"Received SIGTERM, saving emergency checkpoint...") torch.save({ 'epoch': epoch, 'model_state_dict': model.state_dict(), 'optimizer_state_dict': optimizer.state_dict(), 'loss': loss, }, f"/checkpoint/emergency_{os.getpid()}.pt") # Upload to S3 asynchronously os.system(f"aws s3 cp /checkpoint/emergency_{os.getpid()}.pt s3://bucket/checkpoints/") exit(0) signal.signal(signal.SIGTERM, emergency_checkpoint) The handler must be non-blocking — upload to S3 in a subprocess or background thread to avoid exceeding the 30s window.

Synchronous S3 uploads block the training loop — a 2GB checkpoint takes ~10 seconds to upload at 200 MB/s network speed. During those 10 seconds, you cannot compute gradients. Solution: asynchronous S3 streaming: import boto3, threading def async_upload(local_path, s3_path): s3 = boto3.client('s3') threading.Thread( target=s3.upload_file, args=(local_path, 'bucket', s3_path) ).start() # In training loop: torch.save(model.state_dict(), '/tmp/checkpoint.pt') async_upload('/tmp/checkpoint.pt', f'checkpoints/step_{step}.pt') This uploads in the background while training continues. The 30s SIGTERM window is now sufficient for both checkpoint and upload.

After spot preemption, the new instance must automatically resume training. Implementation: (1) On startup, check S3 for latest checkpoint. (2) Download checkpoint to local storage. (3) Load model weights, optimizer state, and scheduler state. (4) Resume from the saved step. (5) Log resumption event for monitoring. The full resumption script should be in the instance's user-data or Docker entrypoint. Target: <5 minutes from new instance creation to training resumption. For a 70B model, checkpoint download from S3 takes ~60 seconds (2GB at 200 MB/s), model loading takes ~90 seconds, and warmup takes ~30 seconds. Total: ~3 minutes.