Data Safety Best Practices

Understand Ephemeral Storage

By default, the Container Disk and Volume Disk on instances are ephemeral. If an instance is terminated — whether manually by you, or automatically if your account balance drops below the backstop threshold — all data on these disks is permanently deleted.

We highly recommend attaching a Network Volume for any data that needs to survive instance lifecycle changes. Anything saved to a Network Volume remains intact even after the instance is terminated.

Checkpointing for Long-Running Jobs

When running extensive training or data processing jobs, do not wait until the job completes to save your progress. Implement regular checkpointing to save your model weights and optimizer states periodically.

Always configure your scripts to write checkpoints to your /workspace directory (if a Network Volume is attached).

For framework-specific guidance on implementing checkpoints, refer to their official documentation:

• PyTorch: Saving and Loading a General Checkpoint
• Hugging Face Transformers: Trainer Checkpointing
• PyTorch Lightning: Checkpointing

The 3-2-1 Backup Strategy

Network Volumes provide robust persistent storage, but adhering to the industry-standard 3-2-1 backup rule is the best defense against data loss:

• 3
  Maintain three copies of your data (your primary active data, plus two backups).
• 2
  Store the backups on two different types of storage media or environments.
• 1
  Keep at least one backup offsite (e.g., downloaded to your local infrastructure or an external cold storage service).

Use SSH file transfer methods like rsync to routinely pull data from your Network Volumes to a local or external location.