What Is Checkpointing?

Checkpointing is a fault-tolerance technique in which the state of a running computation is periodically saved to stable storage so that, if a failure occurs, execution can be restarted from the most recent saved state rather than from the beginning. The saved state, called a checkpoint, captures all information needed to resume the computation: process memory, register contents, file descriptors, and, in distributed systems, the state of inter-process communication channels. Checkpointing is the dominant resilience strategy in high-performance computing (HPC) and is also applied in parallel databases, deep learning training, and long-running scientific simulations.

The technique draws on operating systems theory, parallel and distributed systems research, and storage architecture. Its practical design involves balancing the overhead of writing checkpoints, the storage capacity consumed, and the amount of work that must be replicated after a failure, a set of trade-offs formalized in the Daly model and related checkpoint interval optimization frameworks.

Checkpoint-Restart Mechanisms

A checkpoint-restart system must capture a consistent global state of the computation and write it to a location that survives the failure being protected against, typically a parallel file system or a burst buffer. System-level checkpointing, as implemented by tools such as BLCR (Berkeley Lab Checkpoint/Restart) and DMTCP (Distributed MultiThreaded CheckPointing), intercepts system calls to capture process state transparently, without modification to the application. Application-level checkpointing requires the programmer to insert save and restore routines at logical boundaries in the code, typically between iterations of a time-stepping loop, which reduces the checkpoint size by saving only semantically meaningful state variables. The PNNL Scalable Fault Tolerance research program investigates multi-level checkpointing strategies that combine fast in-memory snapshots with periodic flushes to persistent storage, reducing the I/O bottleneck that limits checkpointing frequency on large clusters.

Coordinated and Uncoordinated Protocols

In distributed systems with multiple communicating processes, all processes must save checkpoints that together form a consistent global snapshot: no checkpoint can be taken while in-flight messages exist that the receiving process has not yet recorded. Coordinated checkpointing achieves consistency by synchronizing all processes before any writes occur, simplifying recovery at the cost of synchronization overhead that scales with the process count. Uncoordinated protocols allow each process to checkpoint independently and rely on message logging to track which messages were sent since the last checkpoint, enabling recovery without re-running all processes, at the cost of maintaining a message log and potential cascading rollback if logs are lost. The survey of fault tolerance mechanisms and checkpoint/restart implementations for HPC systems in the Journal of Supercomputing documents over twenty checkpoint/restart implementations and analyzes the applicability of each protocol class across cluster, grid, and cloud environments.

Storage and Performance Trade-offs

The optimal checkpoint interval T balances the cost of writing checkpoints against the expected re-execution cost after a failure. For a system with mean time between failures of mu and checkpoint write time C, the optimal interval is approximately the square root of 2·mu·C, a result known as the Daly formula. On modern petascale systems, checkpoint write times of minutes and failure rates rising with scale make this balance challenging. Burst buffers, which are fast SSD-based storage tiers interposed between compute nodes and the parallel file system, reduce effective checkpoint write times by an order of magnitude and are now standard in pre-exascale and exascale systems. Deep learning frameworks including PyTorch and TensorFlow implement application-level checkpointing natively, saving model weights, optimizer state, and training step counts to support both fault recovery and optimization of checkpointing I/O in parallel scientific computing.

Applications

Checkpointing has applications in a wide range of fields, including:

• Long-running climate and weather simulation on HPC clusters
• Deep learning model training spanning days or weeks
• Parallel database transaction recovery and crash consistency
• Virtual machine migration and live snapshot for cloud computing
• Real-time control systems requiring fail-safe state preservation