deepdow.callbacks module

Collection of different callbacks.

class BenchmarkCallback(lookbacks=None)[source]

Bases: deepdow.callbacks.Callback

Computation of benchmarks performance over different metrics and dataloaders.

Parameters

lookbacks (list or None) – If list then list of integers representing the different lookbacks. The benchmarks will be run for all of them. If None then just the default one implied by the dataloader.

run

Run instance that is using this callback.

Type

deepdow.experiments.Run

Notes

Very useful for establishing baselines for deep learning models.

on_train_begin(metadata)[source]

Compute performance of all benchmarks.

class Callback[source]

Bases: object

Parent class for all callbacks.

General construct that allows for taking different actions at different points of the training process. One can provide a list of callbacks to the deepdow.experiments.Run.

Notes

To implement new callbacks one needs to subclass this class.

on_batch_begin(metadata)[source]

Take actions at the beginning of a batch.

Parameters

metadata (dict) – Dictionary that is going to be populated with relevant data within Run.launch. Keys available are ‘asset_names’, ‘batch’, ‘epoch’, ‘timestamps’, ‘X_batch’, ‘y_batch’.

on_batch_end(metadata)[source]

Take actions at the beginning of a batch.

Parameters

metadata (dict) – Dictionary that is going to be populated with relevant data within Run.launch. Keys available are ‘asset_names’, ‘batch’, ‘batch_loss’, ‘epoch’, ‘timestamps’, ‘weights’, ‘X_batch’, ‘y_batch’.

on_epoch_begin(metadata)[source]

Take actions at the beginning of an epoch.

Parameters

metadata (dict) – Dictionary that is going to be populated with relevant data within Run.launch. Keys available are ‘epoch’.

on_epoch_end(metadata)[source]

Take actions at the beginning of an epoch.

Parameters

metadata (dict) – Dictionary that is going to be populated with relevant data within Run.launch. Keys available are epoch, n_epochs.

on_train_begin(metadata)[source]

Take actions at the beginning of the training.

Parameters

metadata (dict) – Dictionary that is going to be populated with relevant data within Run.launch.

on_train_end(metadata)[source]

Take actions at the end of the training.

Parameters

metadata (dict) – Dictionary that is going to be populated with relevant data within Run.launch.

on_train_interrupt(metadata)[source]

Take actions on training interruption.

Parameters

metadata (dict) – Dictionary that is going to be populated with relevant data within Run.launch. Keys available are ‘exception’, ‘locals`.

class EarlyStoppingCallback(dataloader_name, metric_name, patience=5)[source]

Bases: deepdow.callbacks.Callback

Early stopping callback.

In the background, we keep a running minimum of a metric of interest. If it does not change for more than patience epochs the training is stopped.

Parameters
  • dataloader_name (str) – Name of the dataloader, needs to correspond to a key in val_dataloaders in deepdow.experiments.Run.

  • metric_name (str) – Name of the metric to use (the lower the better), needs to correspond to a key in metrics in deepdow.experiments.Run.

  • patience (int) – Number of epochs without improvement before the training is stopped.

min

Running minimum of the metric.

Type

float

n_epochs_no_improvement

Number of epochs without improvement - not going below the previous minimum.

Type

int

on_epoch_end(metadata)[source]

Extract statistics and if necessary stop training.

on_train_begin(metadata)[source]

Check if dataloader name and metric name even exist.

on_train_interrupt(metadata)[source]

Handle EarlyStoppingException.

exception EarlyStoppingException[source]

Bases: Exception

Custom exception raised by EarlyStoppingCallback to stop the training.

class MLFlowCallback(run_name=None, mlflow_path=None, experiment_name=None, run_id=None, log_benchmarks=False)[source]

Bases: deepdow.callbacks.Callback

MLFlow logging callback.

Parameters
  • run_name (str or None) – If str then represents the name of a new run to be created. If None then the user eithers provides run_id of an existing run and everything will be logged into it or a new run with random name would be generated.

  • mlflow_path (str or pathlib.Path or None) – If str or pathlib.Path then represents the absolute path to a folder in which mlruns lie. If None then home folder used.

  • experiment_name (str or None) – Experiment to be use. If None using the default one.

  • run_id (str or None) – If provided and run_name is None then continuing an existing run. If None than a new run is created.

  • log_benchmarks (bool) – If True then all benchmarks will be logged under separate mlflow runs.

run

Run instance that is using this callback.

Type

deepdow.experiments.Run

on_epoch_end(metadata)[source]

Read relevant results and log into MLflow.

on_train_begin(metadata)[source]

Log hyperparameters and potentially benchmarks performance.

class ModelCheckpointCallback(folder_path, dataloader_name, metric_name, verbose=False, save_best_only=False)[source]

Bases: deepdow.callbacks.Callback

Model checkpointing callback.

In the background, we keep a running minimum of a metric of interest.

Parameters
  • folder_path (str or pathlib.Path) – Directory to which to save the checkpoints.

  • dataloader_name (str) – Name of the dataloader, needs to correspond to a key in val_dataloaders in deepdow.experiments.Run.

  • metric_name (str) – Name of the metric to use (the lower the better), needs to correspond to a key in metrics in deepdow.experiments.Run.

  • verbose (bool) – If True, each checkpointing triggers a message.

min

Running minimum of the metric.

Type

float

on_epoch_end(metadata)[source]

Store checkpoint if metric is in its all time low.

on_train_begin(metadata)[source]

Check if dataloader name and metric name even exist.

class ProgressBarCallback(output='stderr', n_decimals=3)[source]

Bases: deepdow.callbacks.Callback

Progress bar reporting remaining steps and relevant metrics.

bar

Bar object that is going to be instantiated at the beginning of each epoch.

Type

tqdm.tqdm

metrics

Keys are equal to self.run.metrics.keys() and the values are list that are appended on batch end with after gradient step metrics.

Type

dict

run

Run object that is running the main training loop. One can get access to multiple useful things like the network (run.network), train dataloader (run.train_dataloader) etc.

Type

Run

output

Where to output the progress bar.

Type

str, {‘stdout’, ‘stderr’}

static create_custom_postfix_str(metrics, n_decimals=5)[source]

Create a custom string with metrics.

Parameters
  • metrics (dict) – Keys represent names and the

  • n_decimals (int) – Number of decimals to display.

Returns

formatted – Nicely formatted string to be appended to the progress bar.

Return type

str

on_batch_end(metadata)[source]

Update progress bar with batch metrics.

on_epoch_begin(metadata)[source]

Initialize tqdm bar and metric lists.

on_epoch_end(metadata)[source]

Update finished progress bar with latest epoch metrics.

class TensorBoardCallback(log_dir=None, ts=None, log_benchmarks=False)[source]

Bases: deepdow.callbacks.Callback

Tensorboard logging interface.

Currently supports:
  • images (evolution of predicted weights over time)

  • histograms (activations of input and outputs of all layers)

  • hyperparamters

  • scalars (logged metrics)

Parameters
  • log_dir (None or str or pathlib.Path) – Represent the folder where to checkpoints will be saved. If None then using cwd/runs/CURRENT_DATETIME_HOSTNAME. Else the exact path.

  • ts (datetime.datetime or None) – If datetime.datetime, then only logging specific sample corresponding to provided timestamp. If None then logging every sample.

  • log_benchmarks (bool) – If True, then benchmark metrics are logged to scalars. The folder is log_dir / bm_name.

run

Run instance that is using this callback.

Type

deepdow.experiments.Run

on_batch_begin(metadata)[source]

Set up forward hooks.

on_batch_end(metadata)[source]

Log activations.

on_epoch_end(metadata)[source]

Log images, metrics and hyperparamters.

on_train_begin(metadata)[source]

Log benchmarks performance.

class ValidationCallback(freq=1, lookbacks=None)[source]

Bases: deepdow.callbacks.Callback

Logging of all metrics for all validation dataloaders.

Parameters
  • freq (int) – With what frequiency to compute metrics. If equal to 1 then every epoch. The higher the less frequent the logging will be.

  • lookbacks (list or None) – If list then list of integers representing the different lookbacks. The benchmarks will be run for all of them. If None then just the default one implied by the dataloader.

run

Run instance that is using this callback.

Type

deepdow.experiments.Run

on_epoch_end(metadata)[source]

Compute metrics and log them into the history object.