thelper.train package

Trainer package.

This package contains classes specialized for training models on various tasks.

Submodules

thelper.train.ae module

class thelper.train.ae.AutoEncoderTrainer(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Bases: thelper.train.base.Trainer

__init__(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Receives session parameters, parses image/label keys from task object, and sets up metrics.

eval_epoch(model, epoch, dev, loader, metrics, output_path)

Evaluates the model using the provided objects.

Parameters:

• model – the model to evaluate that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• dev – the target device that tensors should be uploaded to.
• loader – the data loader used to get transformed valid/test samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

supports_classification = True

supports_segmentation = True

train_epoch(model, epoch, dev, classif_loss, optimizer, loader, metrics, output_path)

Trains the model for a single epoch using the provided objects.

Parameters:

• model – the model to train that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• dev – the target device that tensors should be uploaded to.
• loss – the loss function used to evaluate model fidelity.
• optimizer – the optimizer used for back propagation.
• loader – the data loader used to get transformed training samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

thelper.train.base module

Model training/evaluation base interface module.

This module contains the interface required to train and/or evaluate a model based on different tasks. The trainers based on this interface are instantiated in launched sessions based on configuration dictionaries.

class thelper.train.base.Trainer(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Bases: thelper.session.base.SessionRunner

Abstract trainer interface that defines basic session i/o and setup operations.

This interface defines the general behavior of a training session which includes configuration parsing, tensorboard setup, metrics and goal setup, and loss/optimizer setup. It also provides utilities for uploading models and tensors on specific devices, and for saving the state of a session. This interface should be specialized for every task by implementing the train_epoch and eval_epoch functions in a derived class. For better support from visualization utilities, the derived class should also implement to_tensor. See thelper.train.classif.ImageClassifTrainer for a complete example.

Any trainer derived from this class will alternate between training and validation epochs. It will also support post-training (final) evaluation using a separate test set. If requested, visualizations can be computed after the validation epoch during training (e.g. sample activation maps, or t-SNE plots). See thelper.viz for more information on these.

The main parameters that will be parsed by this interface from a configuration dictionary are the following:

• epochs (mandatory if training): number of epochs to train for; one epoch is one iteration over all mini-batches.
• optimization (mandatory if training): sub-dictionary containing types and extra parameters required for instantiating the loss, optimizer, and scheduler objects. See the code of each related loading function for more information on special parameters.
• save_freq (optional, default=1): checkpoint save frequency (will save every epoch multiple of given number).
• save_raw (optional, default=True): specifies whether to save raw types or thelper objects in checkpoints.
• use_tbx (optional, default=False): defines whether to use tensorboardX writers for logging or not.
• device (optional): specifies which device to train/evaluate the model on (default=all available).
• metrics: list of metrics to instantiate and update during training/evaluation; see related loading function for more information.
• monitor: specifies the name of the metric that should be monitored on the validation set for model improvement.

Example configuration file:

# ...
"trainer": {
    # type of trainer to instantiate (linked to task type)
    "type": "thelper.train.ImageClassifTrainer",
    # train for 40 epochs
    "epochs": 40,
    # save every 5 epochs
    "save_freq": 5,
    # monitor validation accuracy and save best model based on that
    "monitor": "accuracy",
    # optimization parameters block
    "optimization": {
        # all types & params below provided by PyTorch
        "loss": {
            "type": "torch.nn.CrossEntropyLoss"
        },
        "optimizer": {
            "type": "torch.optim.SGD",
            "params": {
                "lr": 0.1,
                "momentum": 0.9,
                "weight_decay": 1e-06,
                "nesterov": true
            }
        },
        "scheduler": {
            "type": "torch.optim.lr_scheduler.StepLR",
            "params": {
                "step_size": 10,
                "step_size": 0.1
            }
        }
    },
    # visualization block (optional)
    "viz": {
        # multiple visualization techniques can be toggled by name
        "tsne": {
            # visualization parameters would be provided here
        }
    },
    # in this example, we use two consumers in total
    # (one metric for monitoring, and one for logging)
    "metrics": {
        "accuracy": {
            "type": "thelper.optim.Accuracy"
        },
        "fullreport": {
            "type": "thelper.train.ClassifReport"
        }
    }
}
# ...

Variables:

• config – session configuration dictionary holding all original settings, including trainer configuration.
• model – reference to the model being trained or used for evaluation/prediction.
• task – reference to the object used to specialize the model and that holds task meta-information.

TODO: move static utils to their related modules

See also

thelper.session.base.SessionRunner

thelper.train.classif.ImageClassifTrainer

thelper.train.segm.ImageSegmTrainer

thelper.train.detect.ObjDetectTrainer

thelper.train.regr.RegressionTrainer

thelper.train.utils.create_trainer()

__init__(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Receives the trainer configuration dictionary, parses it, and sets up the session.

eval()

Starts the evaluation process.

This function will evaluate the model using the test data (or the validation data, if no test data is available), and return the results. Note that the code related to the forwarding of samples inside the model itself is implemented in a derived class via thelper.train.base.Trainer.eval_epoch().

eval_epoch(model, epoch, device, loader, metrics, output_path)

Evaluates the model using the provided objects.

Parameters:

• model – the model to evaluate that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• device – the target device that tensors should be uploaded to.
• loader – the data loader used to get transformed valid/test samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

train()

Starts the training process.

This function will train the model until the required number of epochs is reached, and then evaluate it on the test data. The setup of loggers, tensorboard writers is done here, so is model improvement tracking via monitored metrics. However, the code related to loss computation and back propagation is implemented in a derived class via thelper.train.base.Trainer.train_epoch().

train_epoch(model, epoch, dev, loss, optimizer, loader, metrics, output_path)

Trains the model for a single epoch using the provided objects.

Parameters:

• model – the model to train that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• dev – the target device that tensors should be uploaded to.
• loss – the loss function used to evaluate model fidelity.
• optimizer – the optimizer used for back propagation.
• loader – the data loader used to get transformed training samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

thelper.train.classif module

Classification trainer/evaluator implementation module.

class thelper.train.classif.ImageClassifTrainer(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Bases: thelper.train.base.Trainer

Trainer interface specialized for image classification.

This class implements the abstract functions of thelper.train.base.Trainer required to train/evaluate a model for image classification or recognition. It also provides a utility function for fetching i/o packets (images, class labels) from a sample, and that converts those into tensors for forwarding and loss estimation.

See also

thelper.train.base.Trainer

__init__(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Receives session parameters, parses image/label keys from task object, and sets up metrics.

eval_epoch(model, epoch, dev, loader, metrics, output_path)

Evaluates the model using the provided objects.

Parameters:

• model – the model to evaluate that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• dev – the target device that tensors should be uploaded to.
• loader – the data loader used to get transformed valid/test samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

supports_classification = True

train_epoch(model, epoch, dev, loss, optimizer, loader, metrics, output_path)

Trains the model for a single epoch using the provided objects.

Parameters:

• model – the model to train that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• dev – the target device that tensors should be uploaded to.
• loss – the loss function used to evaluate model fidelity.
• optimizer – the optimizer used for back propagation.
• loader – the data loader used to get transformed training samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

thelper.train.detect module

Object detection trainer/evaluator implementation module.

class thelper.train.detect.ObjDetectTrainer(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Bases: thelper.train.base.Trainer

Trainer interface specialized for object detection.

This class implements the abstract functions of thelper.train.base.Trainer required to train/evaluate a model for object detection (i.e. 2D bounding box regression). It also provides a utility function for fetching i/o packets (input images, bounding boxes) from a sample, and that converts those into tensors for forwarding and loss estimation.

See also

thelper.train.base.Trainer

__init__(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Receives session parameters, parses tensor/target keys from task object, and sets up metrics.

eval_epoch(model, epoch, dev, loader, metrics, output_path)

Evaluates the model using the provided objects.

Parameters:

• model – the model to evaluate that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• dev – the target device that tensors should be uploaded to.
• loader – the data loader used to get transformed valid/test samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

supports_detection = True

train_epoch(model, epoch, dev, loss, optimizer, loader, metrics, output_path)

Trains the model for a single epoch using the provided objects.

Parameters:

• model – the model to train that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• dev – the target device that tensors should be uploaded to.
• loss – the loss function used to evaluate model fidelity.
• optimizer – the optimizer used for back propagation.
• loader – the data loader used to get transformed training samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

thelper.train.regr module

Regression trainer/evaluator implementation module.

class thelper.train.regr.RegressionTrainer(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Bases: thelper.train.base.Trainer

Trainer interface specialized for generic (n-dim) regression.

This class implements the abstract functions of thelper.train.base.Trainer required to train/evaluate a model for generic regression (i.e. n-dim target value prediction). It also provides a utility function for fetching i/o packets (input tensors, target values) from a sample, and that converts those into tensors for forwarding and loss estimation.

See also

thelper.train.base.Trainer

__init__(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Receives session parameters, parses tensor/target keys from task object, and sets up metrics.

eval_epoch(model, epoch, dev, loader, metrics, output_path)

Evaluates the model using the provided objects.

Parameters:

• model – the model to evaluate that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• dev – the target device that tensors should be uploaded to.
• loader – the data loader used to get transformed valid/test samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

supports_regression = True

train_epoch(model, epoch, dev, loss, optimizer, loader, metrics, output_path)

Trains the model for a single epoch using the provided objects.

Parameters:

• model – the model to train that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• dev – the target device that tensors should be uploaded to.
• loss – the loss function used to evaluate model fidelity.
• optimizer – the optimizer used for back propagation.
• loader – the data loader used to get transformed training samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

thelper.train.segm module

Segmentation trainer/evaluator implementation module.

class thelper.train.segm.ImageSegmTrainer(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Bases: thelper.train.base.Trainer

Trainer interface specialized for image segmentation.

This class implements the abstract functions of thelper.train.base.Trainer required to train/evaluate a model for image segmentation (i.e. pixel-level classification/labeling). It also provides a utility function for fetching i/o packets (images, class labels) from a sample, and that converts those into tensors for forwarding and loss estimation.

See also

thelper.train.base.Trainer

__init__(session_name, session_dir, model, task, loaders, config, ckptdata=None)

Receives session parameters, parses image/label keys from task object, and sets up metrics.

eval_epoch(model, epoch, dev, loader, metrics, output_path)

Evaluates the model using the provided objects.

Parameters:

• model – the model to evaluate that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• dev – the target device that tensors should be uploaded to.
• loader – the data loader used to get transformed valid/test samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

supports_segmentation = True

train_epoch(model, epoch, dev, loss, optimizer, loader, metrics, output_path)

Trains the model for a single epoch using the provided objects.

Parameters:

• model – the model to train that is already uploaded to the target device(s).
• epoch – the epoch index we are training for (0-based).
• dev – the target device that tensors should be uploaded to.
• loss – the loss function used to evaluate model fidelity.
• optimizer – the optimizer used for back propagation.
• loader – the data loader used to get transformed training samples.
• metrics – the dictionary of metrics/consumers to update every iteration.
• output_path – directory where output files should be written, if necessary.

thelper.train.utils module

Training/evaluation utilities module.

This module contains utilities and tools used to instantiate training sessions. It also contains the prediction consumer interface used by metrics and loggers to receive iteration data during training. See thelper.optim.metrics for more information on metrics.

class thelper.train.utils.ClassifLogger(top_k=1, conf_threshold=None, class_names=None, target_name=None, viz_count=0, report_count=None, log_keys=None, force_softmax=True, format=None)

Bases: thelper.ifaces.PredictionConsumer, thelper.ifaces.ClassNamesHandler, thelper.ifaces.FormatHandler

Classification output logger.

This class provides a simple logging interface for accumulating and saving the predictions of a classifier. By default, all predictions will be logged. However, a confidence threshold can be set to focus on “hard” samples if necessary. It also optionally offers tensorboardX-compatible output images that can be saved locally or posted to tensorboard for browser-based visualization.

Usage examples inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the example consumer; it is used for lookup/printing only
    "logger": {
        # this type is used to instantiate the confusion matrix report object
        "type": "thelper.train.utils.ClassifLogger",
        "params": {
            # log the three 'best' predictions for each sample
            "top_k": 3,
            # keep updating a set of 10 samples for visualization via tensorboardX
            "viz_count": 10
        }
    },
    # ...
}
# ...

Variables:

• top_k – number of ‘best’ predictions to keep for each sample (along with the gt label).
• conf_threshold – threshold used to eliminate uncertain predictions.
• class_names – holds the list of class label names provided by the dataset parser. If it is not provided when the constructor is called, it will be set by the trainer at runtime.
• target_name – name of the targeted label (may be ‘None’ if all classes are used).
• target_idx – index of the targeted label (may be ‘None’ if all classes are used).
• viz_count – number of tensorboardX images to generate and update at each epoch.
• report_count – number of samples to print in reports (use ‘None’ if all samples must be printed).
• log_keys – list of metadata field keys to copy from samples into the log for each prediction.
• force_softmax – specifies whether a softmax operation should be applied to the prediction scores obtained from the trainer.
• score – array used to store prediction scores for logging.
• true – array used to store groundtruth labels for logging.
• meta – array used to store metadata pulled from samples for logging.
• format – output format of the produced log (supports: text, CSV)

__init__(top_k=1, conf_threshold=None, class_names=None, target_name=None, viz_count=0, report_count=None, log_keys=None, force_softmax=True, format=None)

Receives the logging parameters & the optional class label names used to decorate the log.

class_names

Returns the list of class names considered “of interest” by the derived class.

render()

Returns an image of predicted outputs as a numpy-compatible RGBA image drawn by pyplot.

report_csv()

Returns the logged metadata of predicted samples.

The returned object is a print-friendly CSV string that can be consumed directly by tensorboardX. Note that this string might be very long if the dataset is large (i.e. it will contain one line per sample).

report_json()

report_text()

Must be implemented by inheriting classes. Default report text representation.

reset()

Toggles a reset of the internal state, emptying storage arrays.

supports_classification = True

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)

Receives the latest predictions and target values from the training session.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.train.utils.ClassifReport(class_names=None, sample_weight=None, digits=4, format=None)

Bases: thelper.ifaces.PredictionConsumer, thelper.ifaces.ClassNamesHandler, thelper.ifaces.FormatHandler

Classification report interface.

This class provides a simple interface to sklearn.metrics.classification_report so that all count-based metrics can be reported at once under a string-based representation.

Usage example inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the example consumer; it is used for lookup/printing only
    "report": {
        # this type is used to instantiate the classification report object
        "type": "thelper.train.utils.ClassifReport",
        # we do not need to provide any parameters to the constructor, defaults are fine
        "params": {
            # optional parameter that will indicate output as JSON is desired, plain 'text' otherwise
            "format": "json"
        }
    },
    # ...
}
# ...

Variables:

• class_names – holds the list of class label names provided by the dataset parser. If it is not provided when the constructor is called, it will be set by the trainer at runtime.
• pred – queue used to store the top-1 (best) predicted class indices at each iteration.
• format – output format of the produced log (supports: text, JSON)

__init__(class_names=None, sample_weight=None, digits=4, format=None)

Receives the optional class names and arguments passed to the report generator function.

Parameters:

• class_names – holds the list of class label names provided by the dataset parser. If it is not provided when the constructor is called, it will be set by the trainer at runtime.
• sample_weight – sample weights, forwarded to sklearn.metrics.classification_report.
• digits – metrics output digit count, forwarded to sklearn.metrics.classification_report.
• format – output format of the produced log.

gen_report(as_dict=False)

report_json()

Returns the classification report as a JSON formatted string.

report_text()

Returns the classification report as a multi-line print-friendly string.

reset()

Toggles a reset of the metric’s internal state, emptying queues.

supports_classification = True

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)

Receives the latest predictions and target values from the training session.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.train.utils.ConfusionMatrix(class_names=None, draw_normalized=True)

Bases: thelper.ifaces.PredictionConsumer, thelper.ifaces.ClassNamesHandler

Confusion matrix report interface.

This class provides a simple interface to sklearn.metrics.confusion_matrix so that a full confusion matrix can be easily reported under a string-based representation. It also offers a tensorboardX-compatible output image that can be saved locally or posted to tensorboard for browser-based visualization.

Usage example inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the example consumer; it is used for lookup/printing only
    "confmat": {
        # this type is used to instantiate the confusion matrix report object
        "type": "thelper.train.utils.ConfusionMatrix",
        # we do not need to provide any parameters to the constructor, defaults are fine
        "params": {
            # optional parameter that will indicate output as JSON is desired, plain 'text' otherwise
            "format": "json"
        }
    },
    # ...
}
# ...

Variables:

• matrix – report generator function, called at evaluation time to generate the output string.
• class_names – holds the list of class label names provided by the dataset parser. If it is not provided when the constructor is called, it will be set by the trainer at runtime.
• draw_normalized – defines whether rendered confusion matrices should be normalized or not.
• pred – queue used to store the top-1 (best) predicted class indices at each iteration.

__init__(class_names=None, draw_normalized=True)

Receives the optional class label names used to decorate the output string.

Parameters:

• class_names – holds the list of class label names provided by the dataset parser. If it is not provided when the constructor is called, it will be set by the trainer at runtime.
• draw_normalized – defines whether rendered confusion matrices should be normalized or not.

render()

Returns the confusion matrix as a numpy-compatible RGBA image drawn by pyplot.

report()

Returns the confusion matrix as a multi-line print-friendly string.

reset()

Toggles a reset of the metric’s internal state, emptying queues.

supports_classification = True

supports_segmentation = True

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)

Receives the latest predictions and target values from the training session.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.train.utils.DetectLogger(top_k=None, conf_threshold=None, iou_threshold=None, class_names=None, target_name=None, viz_count=0, report_count=None, log_keys=None, format=None)

Bases: thelper.ifaces.PredictionConsumer, thelper.ifaces.ClassNamesHandler, thelper.ifaces.FormatHandler

Detection output logger.

This class provides a simple logging interface for accumulating and saving the bounding boxes of an object detector. By default, all detections will be logged. However, a confidence threshold can be set to focus on strong predictions if necessary.

Usage examples inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the example consumer; it is used for lookup/printing only
    "logger": {
        # this type is used to instantiate the confusion matrix report object
        "type": "thelper.train.utils.DetectLogger",
        "params": {
            # (optional) log the three 'best' detections for each target
            "top_k": 3
        }
    },
    # ...
}
# ...

Variables:

• top_k – number of ‘best’ detections to keep for each target bbox (along with the target label). If omitted, lists all bounding box predictions by the model after applying IoU and confidence thresholds.
• conf_threshold – threshold used to eliminate uncertain predictions (if they support confidence). If confidence is not supported by the model bbox predictions, this parameter is ignored.
• iou_threshold – threshold used to eliminate predictions too far from target (regardless of confidence). If omitted, will ignore only completely non-overlapping predicted bounding boxes (\(IoU=0\)). If no target bounding boxes are provided (prediction-only), this parameter is ignored.
• class_names – holds the list of class label names provided by the dataset parser. If it is not provided when the constructor is called, it will be set by the trainer at runtime.
• target_name – name of the targeted label (may be ‘None’ if all classes are used).
• target_idx – index of the targeted label (may be ‘None’ if all classes are used).
• viz_count – number of tensorboardX images to generate and update at each epoch.
• report_count – number of samples to print in reports (use ‘None’ if all samples must be printed).
• log_keys – list of metadata field keys to copy from samples into the log for each prediction.
• bbox – array used to store prediction bounding boxes for logging.
• true – array used to store groundtruth labels for logging.
• meta – array used to store metadata pulled from samples for logging.
• format – output format of the produced log (supports: text, CSV, JSON)

__init__(top_k=None, conf_threshold=None, iou_threshold=None, class_names=None, target_name=None, viz_count=0, report_count=None, log_keys=None, format=None)

Receives the logging parameters & the optional class label names used to decorate the log.

class_names

Returns the list of class names considered “of interest” by the derived class.

gen_report()

Returns the logged metadata of predicted bounding boxes per sample target in a JSON-like structure.

For every target bounding box, the corresponding best-sorted detections are returned. Sample metadata is appended to every corresponding sub-target if any where requested.

If report_count was specified, the returned report will be limited to that requested amount of targets.

See also

DetectLogger.group_bbox() for formatting, sorting and filtering details.

group_bbox(target_bboxes, detect_bboxes)

Groups a sample’s detected bounding boxes with target bounding boxes according to configuration parameters.

Returns a list of detections grouped by target(s) with following format:

[
    {
        "target": <associated-target-bbox>,
        "detect": [
            {
                "bbox": <detection-bbox>,
                "iou": <IoU(detect-bbox, target-bbox)>
            },
            ...
        ]
    },
    ...
]

The associated target bounding box and \(IoU\) can be None if no target was provided (ie: during inference). In this case, the returned list will have only a single element with all detections associated to it. A single element list can also be returned if only one target was specified for this sample.

When multiple ground truth targets were provided, the returned list will have the same length and ordering as these targets. The associated detected bounding boxes will depend on IoU between target/detection combinations.

All filtering thresholds specified as configuration parameter will be applied for the returned list. Detected bounding boxes will also be sorted by highest confidence (if available) or by highest IoU as fallback.

render()

Returns an image of predicted outputs as a numpy-compatible RGBA image drawn by pyplot.

report_csv()

Returns the logged metadata of predicted bounding boxes.

The returned object is a print-friendly CSV string.

Note that this string might be very long if the dataset is large or if the model tends to generate a lot of detections. The string will contain at least \(N_sample \cdot N_target\) lines and each line will have up to \(N_bbox\) detections, unless limited by configuration parameters.

report_json()

Returns the logged metadata of predicted bounding boxes as a JSON formatted string.

report_text()

Must be implemented by inheriting classes. Default report text representation.

reset()

Toggles a reset of the internal state, emptying storage arrays.

supports_detection = True

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)

Receives the latest predictions and target values from the training session.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.train.utils.PredictionCallback(callback_func, callback_kwargs=None)

Bases: thelper.ifaces.PredictionConsumer

Callback function wrapper compatible with the consumer interface.

This interface is used to hide user-defined callbacks into the list of prediction consumers given to trainer implementations. The callbacks must always be compatible with the list of arguments defined by thelper.typedefs.IterCallbackParams, but may also receive extra arguments defined in advance and passed to the constructor of this class.

Variables:

• callback_func – user-defined function to call on every update from the trainer.
• callback_kwargs – user-defined extra arguments to provide to the callback function.

__init__(callback_func, callback_kwargs=None)

Initialize self. See help(type(self)) for accurate signature.

update(*args, **kwargs)

Forwards the latest prediction data from the training session to the user callback.

class thelper.train.utils.SegmOutputGenerator(prefix='segm', extension='.tif', format='json', class_names=None, color_map=None, dontcare=None)

Bases: thelper.ifaces.PredictionConsumer, thelper.ifaces.ClassNamesHandler, thelper.ifaces.FormatHandler, thelper.ifaces.ColorMapHandler

segmentation output generator.

This class produces segmented output samples from model inference and provided inputs. Outputs are saved into specified directory or in configured save directory by the session runner.

Usage examples inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the example consumer; it is used for lookup/printing only
    "segmenter": {
        # this type is used to instantiate the confusion matrix report object
        "type": "thelper.train.utils.SegmOutputGenerator",
        "params": {
            # (optional) prefix value to saved output files
            "prefix": "segm",
            # (optional) extension of the resulting image file (default: .tif)
            "extension": ".tif",
            # format of the generated metadata file with indices, labels and class name mapping
            "format": "json",
            # (optional) list of class names or map of output index to label name
            # if not provided, will attempt to retrieve applicable ones from model task
            "class_names": ["class-1", "class-2"],
            # (optional) color map of labels to colors to apply for generating output predictions
            # if not provided, will use default sets for distinctive colors
            "color_map": {}
            # (optional) name of the class label to consider as 'ignore' class
            # defaults to black if not specified in color mapping and if applicable for employed model
            "dontcare": "background"
        }
    },
    # ...
}
# ...

Variables:

• prefix – prefix to output sample names.
• extension – extension of the generated output samples.
• format – metadata output format.
• class_names – labels associated to output pixel indices for reporting.
• color_map – mapping of indices to colors applied to segmented output samples.
• dontcare – index of class to ignored, default to black color if not provided in color map.

__init__(prefix='segm', extension='.tif', format='json', class_names=None, color_map=None, dontcare=None)

Receives the parameters and optional class label names used to output mapping.

report_csv()

Returns the class label/name/color mapping metadata.

The returned object is a print-friendly CSV string.

report_json()

report_text()

Must be implemented by inheriting classes. Default report text representation.

reset()

Toggles a reset of the internal state.

supports_segmentation = True

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)

Receives the latest predictions and target values from the training session.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

thelper.train.utils.create_consumers(config)

Instantiates and returns the prediction consumers defined in the configuration dictionary.

All arguments are expected to be handed in through the configuration via a dictionary named ‘params’.

thelper.train.utils.create_trainer(session_name, save_dir, config, model, task, loaders, ckptdata=None)

Instantiates the trainer object based on the type contained in the config dictionary.

The trainer type is expected to be in the configuration dictionary’s trainer field, under the type key. For more information on the configuration, refer to thelper.train.base.Trainer. The instantiated type must be compatible with the constructor signature of thelper.train.base.Trainer. The object’s constructor will be given the full config dictionary and the checkpoint data for resuming the session (if available).

If the trainer type is missing, it will be automatically deduced based on the task object.

Parameters:

• session_name – name of the training session used for printing and to create internal tensorboardX directories.
• save_dir – path to the session directory where logs and checkpoints will be saved.
• config – full configuration dictionary that will be parsed for trainer parameters and saved in checkpoints.
• model – model to train/evaluate; should be compatible with thelper.nn.utils.Module.
• task – global task interface defining the type of model and training goal for the session.
• loaders – a tuple containing the training/validation/test data loaders (a loader can be None if empty).
• ckptdata – raw checkpoint to parse data from when resuming a session (if None, will start from scratch).

Returns:

The fully-constructed trainer object, ready to begin model training/evaluation.

See also

thelper.train.base.Trainer