# training_progress_callback

Python API: `luxonis_train.callbacks.training_progress_callback`

Logs the progress and the timing of each batch and each epoch.

## Classes

### TrainingProgressCallback

Callback that logs the progress and the timing of each loop.

The callback sends these keys to `trainer.logger` with `log_metrics`, where `<mode>` is `train`, `val`, or `test`:

 * `<mode>/epoch_progress_percent`: The share of the batches of the epoch that are done, in percent.
 * `<mode>/epoch_duration_sec`: The seconds from the start of the epoch to the log entry.
 * `<mode>/batch_total_sec`: The seconds from the start hook to the end hook of the logged batch.
 * `<mode>/epoch_completion_sec`: The seconds from the start to the end of the epoch.

The constructor sets one batch counter for each loop to 0. A counter does not reset between epochs. The counter of a loop is the
step of each key of that loop. `val/epoch_completion_sec` and `test/epoch_completion_sec` are the exceptions. Their step is
`trainer.current_epoch`.

A batch-end hook logs only when the number of batches done in the epoch is a multiple of `log_every_n_batches`. The callback
ignores the validation sanity check. The batch-end and epoch-end hooks run on rank zero only. Without `trainer.logger`, the
callback sends no keys. The start of each train epoch then logs a warning.
[LuxonisLightningModule.get_mlflow_logging_keys](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/lightning/luxonis_lightning.md)
lists the twelve keys when the config lists this callback.

Add the callback to `trainer.callbacks` of the config:

```yaml
trainer:
  callbacks:
    - name: TrainingProgressCallback
      params:
        log_every_n_batches: 10
```

#### Methods

##### init

```python
def __init__(log_every_n_batches: int = 1):
```

Initialize the callback.

Parameters

 * `log_every_n_batches` (`int`): Log at the end of a batch once every this many batches of an epoch. `1` logs every batch. A
   higher value reduces the logging overhead. A value below `1` acts as `1`.

##### on_test_batch_end

```python
def on_test_batch_end(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule, outputs: STEP_OUTPUT, batch: Any, batch_idx: int, dataloader_idx: int = 0):
```

Log the progress and the timing of the test batch.

Lightning calls this hook at the end of every test batch. The hook runs on rank zero only. It adds 1 to the batch count of the
epoch and to the test batch counter. When the batch count is a multiple of `log_every_n_batches`, it logs these keys at the step
of the test batch counter:

 * `test/batch_total_sec`: The seconds since
   [on_test_batch_start](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/training_progress_callback.md).
 * `test/epoch_progress_percent`: 100c ⁄ n, where c is the batch count and n is the sum of the finite entries of
   `trainer.num_test_batches`. The value is `0.0` when n is `0`.
 * `test/epoch_duration_sec`: The seconds since
   [on_test_epoch_start](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/training_progress_callback.md).

The batch count covers the batches of every test data loader. Without `trainer.logger`, the hook logs nothing.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. The hook reads `num_test_batches` and logs to `logger`.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.
 * `outputs` (`STEP_OUTPUT`): The output of the test step. Unused.
 * `batch` (`Any`): The batch. Unused.
 * `batch_idx` (`int`): The index of the batch. Unused.
 * `dataloader_idx` (`int`): The index of the data loader. Unused.

##### on_test_batch_start

```python
def on_test_batch_start(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule, batch: Any, batch_idx: int, dataloader_idx: int = 0):
```

Start the timer of the test batch.

Lightning calls this hook before the test step of every batch. The hook stores the start time of `test/batch_total_sec`.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. Unused.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.
 * `batch` (`Any`): The batch. Unused.
 * `batch_idx` (`int`): The index of the batch. Unused.
 * `dataloader_idx` (`int`): The index of the data loader. Unused.

##### on_test_epoch_end

```python
def on_test_epoch_end(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule):
```

Log the duration of the finished test epoch.

Lightning calls this hook at the end of every test epoch. The hook runs on rank zero only. It does nothing without
`trainer.logger`.

When the batch count of the epoch is above 0 and not a multiple of `log_every_n_batches`, the last batch has no log entry. The
hook then logs `test/epoch_progress_percent` as `100.0` and `test/epoch_duration_sec` at the step of the test batch counter. In
every case, it logs `test/epoch_completion_sec`, the seconds since
[on_test_epoch_start](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/training_progress_callback.md),
at the step `trainer.current_epoch`.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. The hook reads `current_epoch` and logs to `logger`.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.

##### on_test_epoch_start

```python
def on_test_epoch_start(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule):
```

Start the timer of the test epoch and log zero progress.

Lightning calls this hook at the start of every test epoch. The hook stores the start time and sets the batch count of the epoch
to 0. It then logs `test/epoch_progress_percent` as `0.0` at the step of the test batch counter. Without `trainer.logger`, it logs
nothing.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. The hook logs to its `logger`.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.

##### on_train_batch_end

```python
def on_train_batch_end(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule, outputs: STEP_OUTPUT, batch: Any, batch_idx: int):
```

Log the progress and the timing of the train batch.

Lightning calls this hook at the end of every training batch. The hook runs on rank zero only. It adds 1 to the train batch
counter. When `batch_idx + 1` is a multiple of `log_every_n_batches`, it logs these keys at the step of that counter:

 * `train/epoch_progress_percent`: 100(i + 1) ⁄ n, where i is `batch_idx` and n is `trainer.num_training_batches`. The value is
   `0.0` when n is `0` or infinite.
 * `train/epoch_duration_sec`: The seconds since
   [on_train_epoch_start](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/training_progress_callback.md).
 * `train/batch_total_sec`: The seconds since
   [on_train_batch_start](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/training_progress_callback.md).

Without `trainer.logger`, the hook logs nothing.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. The hook reads `num_training_batches` and logs to `logger`.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.
 * `outputs` (`STEP_OUTPUT`): The output of the training step. Unused.
 * `batch` (`Any`): The batch. Unused.
 * `batch_idx` (`int`): The index of the batch in the epoch.

##### on_train_batch_start

```python
def on_train_batch_start(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule, batch: Any, batch_idx: int):
```

Start the timer of the train batch.

Lightning calls this hook before the training step of every batch. The hook stores the start time of `train/batch_total_sec`.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. Unused.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.
 * `batch` (`Any`): The batch. Unused.
 * `batch_idx` (`int`): The index of the batch in the epoch. Unused.

##### on_train_epoch_end

```python
def on_train_epoch_end(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule):
```

Log the duration of the finished train epoch.

Lightning calls this hook at the end of every training epoch. When a validation runs in the epoch, the call comes after it, so the
duration includes the validation time. The hook runs on rank zero only. It logs `train/epoch_completion_sec`, the seconds since
[on_train_epoch_start](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/training_progress_callback.md),
and `train/epoch_progress_percent` as `100.0`. Both keys use the step of the train batch counter. Without `trainer.logger`, the
hook logs nothing.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. The hook logs to its `logger`.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.

##### on_train_epoch_start

```python
def on_train_epoch_start(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule):
```

Start the timer of the train epoch and log zero progress.

Lightning calls this hook at the start of every training epoch. The hook stores the start time. It then logs
`train/epoch_progress_percent` as `0.0` at the step of the train batch counter. Without `trainer.logger`, it logs a warning
instead.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. The hook logs to its `logger`.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.

##### on_validation_batch_end

```python
def on_validation_batch_end(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule, outputs: STEP_OUTPUT, batch: Any, batch_idx: int, dataloader_idx: int = 0):
```

Log the progress and the timing of the validation batch.

Lightning calls this hook at the end of every validation batch. The hook runs on rank zero only and does nothing during the sanity
check. It adds 1 to the batch count of the epoch and to the validation batch counter. When the batch count is a multiple of
`log_every_n_batches`, it logs these keys at the step of the validation batch counter:

 * `val/batch_total_sec`: The seconds since
   [on_validation_batch_start](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/training_progress_callback.md).
 * `val/epoch_progress_percent`: 100c ⁄ n, where c is the batch count and n is the sum of the finite entries of
   `trainer.num_val_batches`. The value is `0.0` when n is `0`.
 * `val/epoch_duration_sec`: The seconds since
   [on_validation_epoch_start](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/training_progress_callback.md).

The batch count covers the batches of every validation data loader. Without `trainer.logger`, the hook logs nothing.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. The hook reads `sanity_checking` and `num_val_batches`, and logs to `logger`.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.
 * `outputs` (`STEP_OUTPUT`): The output of the validation step. Unused.
 * `batch` (`Any`): The batch. Unused.
 * `batch_idx` (`int`): The index of the batch. Unused.
 * `dataloader_idx` (`int`): The index of the data loader. Unused.

##### on_validation_batch_start

```python
def on_validation_batch_start(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule, batch: Any, batch_idx: int, dataloader_idx: int = 0):
```

Start the timer of the validation batch.

Lightning calls this hook before the validation step of every batch. Outside the sanity check, the hook stores the start time of
`val/batch_total_sec`.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. The hook reads `sanity_checking`.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.
 * `batch` (`Any`): The batch. Unused.
 * `batch_idx` (`int`): The index of the batch. Unused.
 * `dataloader_idx` (`int`): The index of the data loader. Unused.

##### on_validation_epoch_end

```python
def on_validation_epoch_end(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule):
```

Log the duration of the finished validation epoch.

Lightning calls this hook at the end of every validation epoch. The hook runs on rank zero only. It does nothing during the sanity
check or without `trainer.logger`.

When the batch count of the epoch is above 0 and not a multiple of `log_every_n_batches`, the last batch has no log entry. The
hook then logs `val/epoch_progress_percent` as `100.0` and `val/epoch_duration_sec` at the step of the validation batch counter.
In every case, it logs `val/epoch_completion_sec`, the seconds since
[on_validation_epoch_start](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/training_progress_callback.md),
at the step `trainer.current_epoch`.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. The hook reads `sanity_checking` and `current_epoch`, and logs to `logger`.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.

##### on_validation_epoch_start

```python
def on_validation_epoch_start(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule):
```

Start the validation epoch timer and log zero progress.

Lightning calls this hook at the start of every validation epoch, the sanity check included. The hook stores the start time and
sets the batch count of the epoch to 0. Outside the sanity check, it logs `val/epoch_progress_percent` as `0.0` at the step of the
validation batch counter. Without `trainer.logger`, it logs nothing.

Parameters

 * `trainer` (`pl.Trainer`): The trainer. The hook reads `sanity_checking` and logs to `logger`.
 * `pl_module` (`lxt.LuxonisLightningModule`): The model. Unused.