# graceful_interrupt

Python API: `luxonis_train.callbacks.graceful_interrupt`

Turns the first interrupt into a clean stop.

The callback saves a resume checkpoint and skips the remaining train-end callbacks. A second interrupt exits at once.

## Classes

### GracefulInterruptCallback

Callback that stops a fit cleanly on `SIGINT` or `SIGTERM`.

[LuxonisModel](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/core/core.md)
adds this callback to its main trainer and to the trainer of each tuning trial. While a fit runs, the callback replaces the
handlers of `SIGINT` and `SIGTERM`. The new handler acts as follows:

 * First signal: the handler logs a warning with the path and saves `resume.ckpt` in `save_dir`. When the callback has a tracker,
   the handler then uploads the checkpoint and finalizes the run with the status `"failed"`. The handler logs the error of a
   failed step and does not raise it. A failed save does not stop the upload. A failed upload skips the finalization. Then the
   handler sets `trainer.should_stop` to `True`.
 * Second signal: the handler logs a warning and calls `os._exit(1)`. The process ends at once, without cleanup.

After the first `SIGINT`, Lightning ends the fit. Then
[GracefulInterruptCallback.on_train_end](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/graceful_interrupt.md)
raises `SystemExit`, so no train-end hook of a later callback runs. When the fit starts to run, Lightning installs its own
`SIGTERM` handler, which also calls the handler of the callback. After the first `SIGTERM`, Lightning raises its
`SIGTERMException`, a `SystemExit`, at the end of the current batch or epoch. Then no train-end hook runs.

The handler ignores a signal in a process other than the one that created the callback, for example in a data loader worker.

#### Methods

##### init

```python
def __init__(save_dir: Path, tracker: LuxonisTrackerPL | None = None):
```

Initialize the callback.

Parameters

 * `save_dir` (`Path`): The directory for `resume.ckpt`. The callback converts the value to a `pathlib.Path`, so a `str` is also
   valid.
 * `tracker` (`LuxonisTrackerPL | None`): The tracker that receives `resume.ckpt` on the first interrupt. The first interrupt also
   finalizes its run with the status `"failed"`. `None` skips the upload and the finalization.

##### on_train_end

```python
def on_train_end(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule):
```

Stop the process after an interrupted fit.

Lightning calls this hook at the end of a fit. After a fit without an interrupt, the hook does nothing. After an interrupt, the
hook logs a warning and calls `sys.exit(0)`. After a `SIGTERM`, Lightning raises its own exception before this hook runs.

The `SystemExit` stops the train-end hooks of the callbacks that come after this one.
[LuxonisModel](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/core/core.md)
gives this callback to the trainer. Lightning adds the callbacks of
[LuxonisLightningModule.configure_callbacks](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/lightning/luxonis_lightning.md)
after it. Thus the callbacks of the config, for example
[TestOnTrainEnd](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/test_on_train_end.md)
and
[ExportOnTrainEnd](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/export_on_train_end.md),
come later. When the config lists this callback itself, Lightning drops the instance that
[LuxonisModel](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/core/core.md)
created. The instance of the config then runs at its position in the config. After the exception, Lightning calls the
`on_exception` hooks, but not the `teardown` hooks.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. Unused.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.

Raises

 * `SystemExit`: With the code `0`, when an interrupt stopped the fit.

##### setup

```python
def setup(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule, stage: str | None = None):
```

Install the signal handler when a fit starts.

Lightning calls this hook at the start of every stage. The hook stores `trainer` for every stage, so that the handler can save a
checkpoint and stop the trainer. For a stage other than `"fit"`, the hook does nothing else.

For `"fit"`, in the process that created the callback, the hook saves the current handlers of `SIGINT` and `SIGTERM` and installs
its own handler. In any process, the hook then logs `Added GracefulInterrupt callback` at the `INFO` level.

Parameters

 * `trainer` (`pl.Trainer`): The trainer to save and stop on an interrupt.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.
 * `stage` (`str | None`): The stage that starts, for example `"fit"`.

##### teardown

```python
def teardown(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule, stage: str | None = None):
```

Restore the signal handlers when a fit ends.

Lightning calls this hook at the end of every stage that finishes without an exception. For `"fit"`, in the process that created
the callback, the hook puts back the handlers that
[GracefulInterruptCallback.setup](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/graceful_interrupt.md)
saved. For other stages and in other processes, the hook does nothing. After an interrupt, Lightning does not call this hook, so
the handler of the callback stays installed.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. Unused.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.
 * `stage` (`str | None`): The stage that ends.