Experiments

This section focuses on putting all of the previous sections together and proposes a framework for training and evaluation of networks. The central object is the deepdow.experiments.Run class.

To instantiate run, we need to provide multiple parameters:

  • network - Network to be trained and evaluated. See Networks for details.

  • loss - Loss criterion. See Losses for details.

  • train_dataloader - Dataloader streaming training data. See Data Loading for details.

  • val_dataloaders - Dictionary where keys are names and values are instances of RigidDataloader. See Data Loading for details.

  • metrics - Additional metrics to be monitored. See Losses for details.

  • benchmarks - Additional baseline models to be used for comparison. See Benchmarks for details.

  • callbacks - Additional callbacks to be used (on top of the default ones). See Callbacks for details.

Once we construct the Run, we can start the training and evaluation loop via the launch method.

from deepdow.benchmarks import OneOverN
from deepdow.data import InRAMDataset, RigidDataLoader
from deepdow.experiments import Run
from deepdow.losses import MaximumDrawdown, SharpeRatio
from deepdow.nn import LinearNet

n_samples, n_channels, lookback, n_assets = 200, 2, 20, 6
horizon = 15

X = np.random.random((n_samples, n_channels, lookback, n_assets)) - 0.5
y = np.random.random((n_samples, n_channels, horizon, n_assets)) - 0.5

dataset = InRAMDataset(X, y)
train_dataloader = RigidDataLoader(dataset, indices=list(range(100)), batch_size=10)
val_dataloaders = {'val': RigidDataLoader(dataset, indices=list(range(130, 180)), batch_size=10)}

network = LinearNet(n_channels, lookback, n_assets)
loss = SharpeRatio(returns_channel=0)
benchmarks = {'1overN': OneOverN()}
metrics = {'drawdown': MaximumDrawdown(returns_channel=0)}

run = Run(network,
          loss,
          train_dataloader,
          val_dataloaders=val_dataloaders,
          metrics=metrics,
          benchmarks=benchmarks)

history = run.launch(n_epochs=1)
model   metric    epoch  dataloader
1overN  drawdown  -1     val           0.283
        loss      -1     val          -0.331

We get results on the benchmarks in the standard output (see above). Additionally, progress bar is sent to the standard error. It monitors progress of our network. To read more details on the Run class see deepdow.experiments module. Last but not least, we also get an instance of the History class. See below section for more information.

History

The launch method returns an instance of the History class. It captures all the useful information that was recorded during training. This information can be accessed via the metrics property that is a pd.DataFrame with the following columns

  • model - name of the model

  • metric - name of the loss

  • value - value of the loss

  • batch - batch

  • epoch - epoch

  • dataloader - name of the dataloader

  • lookback - lookback size, by default only using the one from the dataloader

  • timestamp - it can be used to unique identify a given sample

  • current_time - time when the entry logged

Callbacks

Callbacks are intended to be run at precise moments of the training loop. All callbacks have a shared interface deepdow.callbacks.Callback that provides the following methods

  • on_batch_begin - run at the beginning of each batch

  • on_batch_end - run at the end of each batch

  • on_epoch_begin - run at the beginning of each epoch

  • on_epoch_end - run at the end of each epoch

  • on_train_begin - run at the beginning of the training

  • on_train_end- run at the end of the training

  • on_train_interrupt - run in case training interrupted

Each of these methods inputs the metadata dictionary. It contains the most recent value of the most relevant variables.

Note that when constructing a Run there are three callbacks inserted by default

  • BenchmarkCallback

  • ValidationCallback

  • ProgressBarCallback

One can chose additional one by defining adding a list of callbacks as the callbacks variable.

Lastly, callback instances can access the Run instance within under the run attribute. It is always injected when the training is launched.

In what follows, we provide an overview of all available callbacks. For detailed usage instructions see deepdow.callbacks module.

BenchmarkCallback

Automatically added to Run instances. It computes all metrics for all provided benchmarks over all validation dataloaders.

EarlyStoppingCallback

This callback monitors a given metric and if there are no improvements over specific number of epochs it stops the training.

MLFlowCallback

Callback that logs relevant metrics to MLflow.

ModelCheckpointCallback

Saving a model each epoch it achieves lower than the previous lowest loss.

ProgressBarCallback

Automatically added to Run instances. Displays progress bar with all relevant metrics. One can choose where outputted with output parameter.

TensorBoardCallback

Callback that logs relevant metrics to MLflow together with images and histograms.

ValidationCallback

Automatically added to Run instances. It computes all metrics of the trained network over all validation dataloaders.