thelper.optim package¶

Optimization/metrics package.

This package contains metrics implementations used to monitor training sessions and evaluate models, and optimization methods used to control the learning behavior of these models.

Submodules¶

thelper.optim.eval module¶

Evaluation classes/funcs module.

This module contains procedures used to evaluate models and prediction results on specific tasks or datasets. These procedures may be used as part of metric classes (defined in thelper.optim.metrics) or high-level debug/drawing utilities.

thelper.optim.eval.compute_average_precision(precision, recall, method='all-points')[source]¶

Computes the average precision given an array of precision and recall values.

This function is inspired from the ‘Object Detection Metrics’ repository of Rafael Padilla. See https://github.com/rafaelpadilla/Object-Detection-Metrics for more information. The original code is distributed under the MIT License, Copyright (c) 2018 Rafael Padilla.

Parameters:

precision – list of precision values for the evaluated predictions of a class.
recall – list of recall values for the evaluated predictions of a class.
method – the evaluation method to use; can be the the latest & official PASCAL VOC toolkit approach (“all-points”), or the 11-point approach (“11-points”) described in the original paper (“The PASCAL Visual Object Classes(VOC) Challenge”).

Returns:

A 4-element tuple containing the average precision, rectified precision/recall arrays, and the indices used for the integral.

thelper.optim.eval.compute_bbox_iou(bbox1, bbox2)[source]¶: Computes and returns the Intersection over Union (IoU) of two bounding boxes.

thelper.optim.eval.compute_mask_iou(mask1, mask2, class_indices=None, dontcare=None)[source]¶: Computes and returns a map of Intersection over Union (IoU) scores for two segmentation masks.

thelper.optim.eval.compute_pascalvoc_metrics(pred_bboxes, gt_bboxes, task, iou_threshold=0.5, method='all-points')[source]¶

Computes the metrics used by the VOC Pascal 2012 challenge.

This function is inspired from the ‘Object Detection Metrics’ repository of Rafael Padilla. See https://github.com/rafaelpadilla/Object-Detection-Metrics for more information. The original code is distributed under the MIT License, Copyright (c) 2018 Rafael Padilla.

Parameters:

pred_bboxes – list of bbox predictions generated by the model under evaluation.
gt_bboxes – list of groundtruth bounding boxes defined by the dataset.
task – task definition object that holds a vector of all class names.
iou_threshold – Intersection Over Union (IOU) threshold for true/false positive classification.
method – the evaluation method to use; can be the the latest & official PASCAL VOC toolkit approach (“all-points”), or the 11-point approach (“11-points”) described in the original paper (“The PASCAL Visual Object Classes(VOC) Challenge”).

Returns:

A dictionary containing evaluation information and metrics for each class. Each entry contains –

precision: array with the precision values;
recall: array with the recall values;
AP: average precision;
interpolated precision: interpolated precision values;
interpolated recall: interpolated recall values;
total positives: total number of ground truth positives;
total TP: total number of True Positive detections;
total FP: total number of False Negative detections.

thelper.optim.losses module¶

class thelper.optim.losses.FocalLoss(gamma=2, alpha=0.5, weight=None, ignore_index=255)[source]¶

Bases: sphinx.ext.autodoc.importer._MockObject

Note

Contributed by Mario Beaulieu <mario.beaulieu@crim.ca>.

thelper.optim.metrics module¶

Metrics module.

This module contains classes that implement metrics used to monitor training sessions and evaluate models. These metrics should all inherit from thelper.optim.metrics.Metric to allow them to be dynamically instantiated by the framework from a configuration file, and evaluated automatically inside a training session. For more information on this, refer to thelper.train.base.Trainer.

class thelper.optim.metrics.Accuracy(top_k=1, max_win_size=None)[source]¶

Bases: thelper.optim.metrics.Metric

Classification accuracy metric interface.

This is a scalar metric used to monitor the label prediction accuracy of a model. By default, it works in top-k mode, meaning that the evaluation result is given by:

\[\text{accuracy} = \frac{\text{nb. correct predictions}}{\text{nb. total predictions}} \cdot 100\]

When \(k>1\), a ‘correct’ prediction is obtained if any of the model’s top \(k\) predictions (i.e. the \(k\) predictions with the highest score) match the groundtruth label. Otherwise, if \(k=1\), then only the top prediction is compared to the groundtruth label. Note that for binary classification problems, \(k\) should always be set to 1.

This metric’s goal is to maximize its value \(\in [0,100]\) (a percentage is returned).

Usage example inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the example metric; it is used for lookup/printing only
    "top_5_accuracy": {
        # this type is used to instantiate the accuracy metric
        "type": "thelper.optim.metrics.Accuracy",
        # these parameters are passed to the wrapper's constructor
        "params": {
            # the top prediction count to check for a match with the groundtruth
            "top_k": 5
        }
    },
    # ...
}
# ...

Todo: add support for ‘dont care’ target value?

Variables:

top_k – number of top predictions to consider when matching with the groundtruth (default=1).
max_win_size – maximum moving average window size to use (default=None, which equals dataset size).
correct – total number of correct predictions stored using an array for window-based averaging.
total – total number of predictions stored using an array for window-based averaging.
warned_eval_bad – toggles whether the division-by-zero warning has been flagged or not.

__init__(top_k=1, max_win_size=None)[source]¶

Receives the number of predictions to consider for matches (top_k) and the moving average window size (window_size).

Note that by default, if max_win_size is not provided here, the value given to max_iters on the first update call will be used instead to fix the sliding window length. In any case, the smallest of max_iters and max_win_size will be used to determine the actual window size.

eval()[source]¶

Returns the current accuracy (in percentage) based on the accumulated prediction counts.

Will issue a warning if no predictions have been accumulated yet.

goal¶: Returns the scalar optimization goal of this metric (maximization).

reset()[source]¶: Toggles a reset of the metric’s internal state, deallocating count arrays.

supports_classification = True¶

supports_segmentation = True¶

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)[source]¶

Receives the latest class prediction and groundtruth labels from the training session.

This function computes and accumulate the number of correct and total predictions in the internal arrays, cycling over the iteration index if the maximum window length is reached.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.optim.metrics.AveragePrecision(target_class=None, iou_threshold=0.5, method='all-points', max_win_size=None)[source]¶

Bases: thelper.optim.metrics.Metric

Object detection average precision score from PascalVOC.

This metric is computed based on the evaluator function implemented in thelper.optim.eval. It can target a single class at a time, or produce the mean average precision for all classes.

Usage example inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the example metric; it is used for lookup/printing only
    "mAP": {
        # this type is used to instantiate the AP metric
        "type": "thelper.optim.metrics.AveragePrecision",
        # these parameters are passed to the wrapper's constructor
        "params": {
            # no parameters means we will compute the mAP
        }
    },
    # ...
}
# ...

Variables:

target_class – name of the class to target; if ‘None’, will compute mAP instead of AP.
iou_threshold – Intersection Over Union (IOU) threshold for true/false positive classification.
method – the evaluation method to use; can be the the latest & official PASCAL VOC toolkit approach (“all-points”), or the 11-point approach (“11-points”) described in the original paper (“The PASCAL Visual Object Classes(VOC) Challenge”).
max_win_size – maximum moving average window size to use (default=None, which equals dataset size).
preds – array holding the predicted bounding boxes for all input samples.
targets – array holding the target bounding boxes for all input samples.

__init__(target_class=None, iou_threshold=0.5, method='all-points', max_win_size=None)[source]¶

Initializes metric attributes.

Note that by default, if max_win_size is not provided here, the value given to max_iters on the first update call will be used instead to fix the sliding window length. In any case, the smallest of max_iters and max_win_size will be used to determine the actual window size.

eval()[source]¶

Returns the current accuracy (in percentage) based on the accumulated prediction counts.

Will issue a warning if no predictions have been accumulated yet.

goal¶: Returns the scalar optimization goal of this metric (maximization).

live_eval¶: Returns whether this metric can/should be evaluated at every backprop iteration or not.

reset()[source]¶: Toggles a reset of the metric’s internal state, deallocating bbox arrays.

supports_detection = True¶

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)[source]¶

Receives the latest bbox predictions and targets from the training session.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.optim.metrics.ExternalMetric(metric_name, metric_type, metric_goal, metric_params=None, target_name=None, class_names=None, max_win_size=None, force_softmax=True, live_eval=True)[source]¶

Bases: thelper.optim.metrics.Metric, thelper.ifaces.ClassNamesHandler

External metric wrapping interface.

This interface is used to wrap external metrics and use them in the training framework. The metrics of sklearn.metrics are good candidates that have been used extensively with this interface in the past, but those of other libraries might also be compatible.

Along with the name of the class to import and its constructor’s parameters, the user must provide a handling mode that specifies how prediction and groundtruth data should be handled in this wrapper. Also, extra arguments such as target label names, goal information, and window sizes can be provided for specific use cases related to the selected handling mode.

For now, two metric handling modes (both related to classification) are supported:

classif_best: the wrapper will accumulate the predicted and groundtruth classification labels forwarded by the trainer and provide them to the external metric for evaluation. If a target label name is specified, then only classifications related to that label will be accumulated. This is the handling mode required for count-based classification metrics such as accuracy, F-Measure, precision, recall, etc.

classif_score: the wrapper will accumulate the prediction score of the targeted label along with a boolean that indicates whether this label was the groundtruth label or not. This is the handling mode required for score-based classification metrics such as when computing the area under the ROC curve (AUC).

Usage examples inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the first example metric; it is used for lookup/printing only
    "f1_score_reject": {
        # this type is used to instantiate the wrapper
        "type": "thelper.optim.metrics.ExternalMetric",
        # these parameters are passed to the wrapper's constructor
        "params": {
            # the external class to import
            "metric_name": "sklearn.metrics.f1_score",
            # the parameters passed to the external class's constructor
            "metric_params": {},
            # the wrapper metric handling mode
            "metric_type": "classif_best",
            # the target class name (note: dataset-specific)
            "target_name": "reject",
            # the goal type of the external metric
            "metric_goal": "max"
        }
    },
    # this is the name of the second example metric; it is used for lookup/printing only
    "roc_auc_accept": {
        # this type is used to instantiate the wrapper
        "type": "thelper.optim.metrics.ExternalMetric",
        # these parameters are passed to the wrapper's constructor
        "params": {
            # the external class to import
            "metric_name": "sklearn.metrics.roc_auc_score",
            # the parameters passed to the external class's constructor
            "metric_params": {},
            # the wrapper metric handling mode
            "metric_type": "classif_score",
            # the target class name (note: dataset-specific)
            "target_name": "accept",
            # the goal type of the external metric
            "metric_goal": "max"
        }
    },
    # ...
}
# ...

Variables:

metric_goal – goal of the external metric, used for monitoring. Can be min or max.
metric_type – handling mode of the external metric. Can only be one of the predetermined values.
metric – type of the external metric that will be instantiated when eval is called.
metric_params – dictionary of parameters passed to the external metric on instantiation.
target_name – name of the targeted label. Used only in handling modes related to classification.
target_idx – index of the targeted label. Used only in handling modes related to classification.
class_names – holds the list of class label names provided by the dataset parser. If it is not provided when the constructor is called, it will be set by the trainer at runtime.
force_softmax – specifies whether a softmax operation should be applied to the prediction scores obtained from the trainer. Only used with the “classif_score” handling mode.
max_win_size – maximum moving average window size to use (default=None, which equals dataset size).
pred – queue used to store predictions-related values for window-based averaging.
target – queue used to store groundtruth-related values for window-based averaging.

__init__(metric_name, metric_type, metric_goal, metric_params=None, target_name=None, class_names=None, max_win_size=None, force_softmax=True, live_eval=True)[source]¶

Receives all necessary arguments for wrapper initialization and external metric instantiation.

See thelper.optim.metrics.ExternalMetric for information on arguments.

class_names¶: Returns the list of class names considered “of interest” by the derived class.

eval()[source]¶: Returns the external metric’s evaluation result.

goal¶: Returns the scalar optimization goal of this metric (user-defined).

live_eval¶

Returns whether this metric can/should be evaluated at every backprop iteration or not.

By default, this returns True, but implementations that are quite slow may return False.

reset()[source]¶: Toggles a reset of the metric’s internal state, emptying pred/target queues.

supports_classification = True¶

supports_segmentation = True¶

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)[source]¶

Receives the latest predictions and target values from the training session.

The handling of the data received here will depend on the current metric’s handling mode.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.optim.metrics.IntersectionOverUnion(target_names=None, global_score=True, max_win_size=None)[source]¶

Bases: thelper.optim.metrics.Metric

Computes the intersection over union over image classes.

It can target a single class at a time, or produce the mean IoU (mIoU) for a number of classes. It can also average IoU scores from each images, or sum up all intersection and union areas and compute a global score.

Usage example inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the example metric; it is used for lookup/printing only
    "mIoU": {
        # this type is used to instantiate the IoU metric
        "type": "thelper.optim.metrics.IntersectionOverUnion",
        # these parameters are passed to the wrapper's constructor
        "params": {
            # no parameters means we will compute the mIoU with global scoring
        }
    },
    # ...
}
# ...

Variables:	target_names – name(s) of the class(es) to target; if ‘None’ or list, will compute mIoU instead of IoU. max_win_size – maximum moving average window size to use (default=None, which equals dataset size). inters – array holding the intersection areas or IoU scores for all input samples. unions – array holding the union areas for all input samples.

__init__(target_names=None, global_score=True, max_win_size=None)[source]¶

Initializes metric attributes.

Note that by default, if max_win_size is not provided here, the value given to max_iters on the first update call will be used instead to fix the sliding window length. In any case, the smallest of max_iters and max_win_size will be used to determine the actual window size.

eval()[source]¶

Returns the current IoU ratio based on the accumulated counts.

Will issue a warning if no predictions have been accumulated yet.

goal¶: Returns the scalar optimization goal of this metric (maximization).

reset()[source]¶: Toggles a reset of the metric’s internal state, deallocating bbox arrays.

supports_segmentation = True¶

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)[source]¶

Receives the latest bbox predictions and targets from the training session.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.optim.metrics.MeanAbsoluteError(reduction='mean', max_win_size=None)[source]¶

Bases: thelper.optim.metrics.Metric

Mean absolute error metric interface.

This is a scalar metric used to monitor the mean absolute deviation (or error) for a model’s predictions. This regression metric can be described as:

\[e(x, y) = E = \{e_1,\dots,e_N\}^\top, \quad e_n = \left| x_n - y_n \right|,\]

where \(N\) is the batch size. If reduction is not 'none', then:

\[\begin{split}\text{MAE}(x, y) = \begin{cases} \operatorname{mean}(E), & \text{if reduction } = \text{mean.}\\ \operatorname{sum}(E), & \text{if reduction } = \text{sum.} \end{cases}\end{split}\]

x and y are tensors of arbitrary shapes with a total of n elements each.

Usage example inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the example metric; it is used for lookup/printing only
    "mae": {
        # this type is used to instantiate the error metric
        "type": "thelper.optim.metrics.MeanAbsoluteError",
        "params": {
            "reduction": "mean"
        }
    },
    # ...
}
# ...

Todo: add support for ‘dont care’ target value?

Variables:	max_win_size – maximum moving average window size to use (default=None, which equals dataset size). reduction – string representing the tensor reduction strategy to use. errors – array of error values stored for window-based averaging. warned_eval_bad – toggles whether the division-by-zero warning has been flagged or not.

__init__(reduction='mean', max_win_size=None)[source]¶

Receives the reduction strategy and the moving average window size (window_size).

Note that by default, if max_win_size is not provided here, the value given to max_iters on the first update call will be used instead to fix the sliding window length. In any case, the smallest of max_iters and max_win_size will be used to determine the actual window size.

eval()[source]¶

Returns the current (average) mean absolute error based on the accumulated values.

Will issue a warning if no predictions have been accumulated yet.

goal¶: Returns the scalar optimization goal of this metric (minimization).

reset()[source]¶: Toggles a reset of the metric’s internal state, deallocating the errors array.

supports_regression = True¶

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)[source]¶

Receives the latest predictions and target values from the training session.

This function computes and accumulates the L1 distance between predictions and targets in the internal array, cycling over the iteration index if the maximum window length is reached.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.optim.metrics.MeanSquaredError(reduction='mean', max_win_size=None)[source]¶

Bases: thelper.optim.metrics.Metric

Mean squared error metric interface.

This is a scalar metric used to monitor the mean squared deviation (or error) for a model’s predictions. This regression metric can be described as:

\[e(x, y) = E = \{e_1,\dots,e_N\}^\top, \quad e_n = \left( x_n - y_n \right)^2,\]

where \(N\) is the batch size. If reduction is not 'none', then:

\[\begin{split}\text{MSE}(x, y) = \begin{cases} \operatorname{mean}(E), & \text{if reduction } = \text{mean.}\\ \operatorname{sum}(E), & \text{if reduction } = \text{sum.} \end{cases}\end{split}\]

x and y are tensors of arbitrary shapes with a total of n elements each.

Usage example inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the example metric; it is used for lookup/printing only
    "mse": {
        # this type is used to instantiate the error metric
        "type": "thelper.optim.metrics.MeanSquaredError",
        "params": {
            "reduction": "mean"
        }
    },
    # ...
}
# ...

Todo: add support for ‘dont care’ target value?

Variables:	max_win_size – maximum moving average window size to use (default=None, which equals dataset size). reduction – string representing the tensor reduction strategy to use. errors – array of error values stored for window-based averaging. warned_eval_bad – toggles whether the division-by-zero warning has been flagged or not.

__init__(reduction='mean', max_win_size=None)[source]¶

Receives the reduction strategy and the moving average window size (window_size).

Note that by default, if max_win_size is not provided here, the value given to max_iters on the first update call will be used instead to fix the sliding window length. In any case, the smallest of max_iters and max_win_size will be used to determine the actual window size.

eval()[source]¶

Returns the current (average) mean squared error based on the accumulated values.

Will issue a warning if no predictions have been accumulated yet.

goal¶: Returns the scalar optimization goal of this metric (minimization).

reset()[source]¶: Toggles a reset of the metric’s internal state, deallocating the errors array.

supports_regression = True¶

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)[source]¶

Receives the latest predictions and target values from the training session.

This function computes and accumulates the mean squared error between predictions and targets in the internal array, cycling over the iteration index if the maximum window length is reached.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.optim.metrics.Metric[source]¶

Bases: thelper.ifaces.PredictionConsumer

Abstract metric interface.

This interface defines basic functions required so that thelper.train.base.Trainer can figure out how to instantiate, update, and optimize a given metric while training/evaluating a model.

All metrics, by definition, must be ‘optimizable’. This means that they should return a scalar value when ‘evaluated’ and define an optimal goal (-inf or +inf). If this is not possible, then the class should probably be derived using the more generic thelper.ifaces.PredictionConsumer instead.

eval()[source]¶

Returns the metric’s evaluation result.

The returned value should be a scalar. As a model improves, this scalar should get closer to the optimization goal (defined through the ‘goal’ attribute). This value will be queried at the end of each training epoch by the trainer.

goal¶

Returns the scalar optimization goal of the metric.

The returned goal can be the minimize or maximize members of thelper.optim.metrics.Metric if the class’s evaluation returns a scalar value, and None otherwise. The trainer will check this value to see if monitoring the metric’s evaluation result progression is possible.

live_eval¶

Returns whether this metric can/should be evaluated at every backprop iteration or not.

By default, this returns True, but implementations that are quite slow may return False.

maximize = inf¶: Possible value of the goal attribute of this metric.

minimize = -inf¶: Possible value of the goal attribute of this metric.

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)[source]¶

Receives the latest prediction and groundtruth tensors from the training session.

The data given here will be “consumed” internally, but it should NOT be modified. For example, a classification accuracy metric might accumulate the correct number of predictions in comparison to groundtruth labels, but never alter those predictions. The iteration/epoch indices may be used to ‘reset’ the internal state of this object when needed (for example, at the start of each new epoch).

Remember that input, prediction, and target tensors received here will all have a batch dimension!

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.optim.metrics.PSNR(data_range=1.0, max_win_size=None)[source]¶

Bases: thelper.optim.metrics.Metric

Peak Signal-to-Noise Ratio (PSNR) metric interface.

This is a scalar metric used to monitor the change in quality of a signal (or image) following a transformation. For more information, see its definition on [Wikipedia].

The PSNR (in decibels, dB) between a modified signal \(x\) and its original version \(y\) is defined as:

\[\text{PSNR}(x, y) = 10 * \log_{10} \Bigg( \frac{R^2}{\text{MSE}(x, y)} \Bigg)\]

where \(\text{MSE}(x, y)\) returns the mean squared error (see thelper.optim.metrics.MeanSquaredError for more information), and \(R\) is the maximum possible value for a single element in the input signal (i.e. its maximum “range”).

Usage example inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the example metric; it is used for lookup/printing only
    "psnr": {
        # this type is used to instantiate the metric
        "type": "thelper.optim.metrics.PSNR",
        "params": {
            "data_range": "255"
        }
    },
    # ...
}
# ...

Variables:	max_win_size – maximum moving average window size to use (default=None, which equals dataset size). data_range – maximum value of an element in the target signal. psnrs – array of psnr values stored for window-based averaging. warned_eval_bad – toggles whether the division-by-zero warning has been flagged or not.

__init__(data_range=1.0, max_win_size=None)[source]¶

Receives all necessary initialization arguments to compute signal PSNRs,

See thelper.optim.metrics.PSNR for information on arguments.

eval()[source]¶

Returns the current (average) PSNR based on the accumulated values.

Will issue a warning if no predictions have been accumulated yet.

goal¶: Returns the scalar optimization goal of this metric (maximization).

reset()[source]¶: Toggles a reset of the metric’s internal state, deallocating the psnrs array.

supports_regression = True¶

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)[source]¶

Receives the latest predictions and target values from the training session.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

class thelper.optim.metrics.ROCCurve(target_name, target_tpr=None, target_fpr=None, class_names=None, force_softmax=True, sample_weight=None, drop_intermediate=True)[source]¶

Bases: thelper.optim.metrics.Metric, thelper.ifaces.ClassNamesHandler

Receiver operating characteristic (ROC) computation interface.

This class provides an interface to sklearn.metrics.roc_curve and sklearn.metrics.roc_auc_score that can produce various types of ROC-related information including the area under the curve (AUC), the false positive and negative rates for various operating points, and the ROC curve itself as an image (also compatible with tensorboardX).

By default, evaluating this metric returns the Area Under the Curve (AUC). If a target operating point is set, it will instead return the false positive/negative prediction rate of the model at that point.

Usage examples inside a session configuration file:

# ...
# lists all metrics to instantiate as a dictionary
"metrics": {
    # ...
    # this is the name of the first example; it will output the AUC of the "reject" class
    "roc_reject_auc": {
        # this type is used to instantiate the ROC metric
        "type": "thelper.optim.metrics.ROCCurve",
        # these parameters are passed to the constructor
        "params": {
            # the name of the class to evaluate
            "target_name": "reject"
        }
    },
    # this is the name of the second example; it will output the FPR at TPR=0.99
    "roc_reject_0.99tpr": {
        # this type is used to instantiate the ROC metric
        "type": "thelper.optim.metrics.ROCCurve",
        # these parameters are passed to the constructor
        "params": {
            # the name of the class to evaluate
            "target_name": "reject",
            # the target true positive rate (TPR) operating point
            "target_tpr": 0.99
        }
    },
    # ...
}
# ...

Variables:

target_inv – used to target all classes except the named one(s); experimental!
target_name – name of targeted class to generate the roc curve/auc information for.
target_tpr – target operating point in terms of true positive rate (provided in constructor).
target_fpr – target operating point in terms of false positive rate (provided in constructor).
target_idx – index of the targeted class, mapped from target_name using the class_names list.
class_names – holds the list of class label names provided by the dataset parser. If it is not provided when the constructor is called, it will be set by the trainer at runtime.
force_softmax – specifies whether a softmax operation should be applied to the prediction scores obtained from the trainer.
curve – roc curve generator function, called at evaluation time to generate the output string.
auc – auc score generator function, called at evaluation time to generate the output string.
score – queue used to store prediction score values for window-based averaging.
true – queue used to store groundtruth label values for window-based averaging.

__init__(target_name, target_tpr=None, target_fpr=None, class_names=None, force_softmax=True, sample_weight=None, drop_intermediate=True)[source]¶

Receives the target class/operating point info, log parameters, and roc computation arguments.

Parameters:

target_name – name of targeted class to generate the roc curve/auc information for.
target_tpr – target operating point in terms of true positive rate (provided in constructor).
target_fpr – target operating point in terms of false positive rate (provided in constructor).
class_names – holds the list of class label names provided by the dataset parser. If it is not provided when the constructor is called, it will be set by the trainer at runtime.
force_softmax – specifies whether a softmax operation should be applied to the prediction scores obtained from the trainer.
sample_weight – passed to sklearn.metrics.roc_curve and sklearn.metrics.roc_auc_score.
drop_intermediate – passed to sklearn.metrics.roc_curve.

class_names¶: Returns the list of class names considered “of interest” by the derived class.

eval()[source]¶

Returns the evaluation result (AUC/TPR/FPR).

If no target operating point is set, the returned value is the AUC for the target class. If a target TPR is set, the returned value is the FPR for that operating point. If a target FPR is set, the returned value is the TPR for that operating point.

goal¶: Returns the scalar optimization goal of this metric (variable based on target op point).

live_eval¶: Returns whether this metric can/should be evaluated at every backprop iteration or not.

render()[source]¶: Returns the ROC curve as a numpy-compatible RGBA image drawn by pyplot.

reset()[source]¶: Toggles a reset of the metric’s internal state, emptying queues.

supports_classification = True¶

supports_segmentation = True¶

update(task, input, pred, target, sample, loss, iter_idx, max_iters, epoch_idx, max_epochs, output_path, **kwargs)[source]¶

Receives the latest predictions and target values from the training session.

The exact signature of this function should match the one of the callbacks defined in thelper.train.base.Trainer and specified by thelper.typedefs.IterCallbackParams.

thelper.optim.schedulers module¶

Schedulers module.

This module contains classes used for scheduling learning rate changes while training a model. All classes defined here should derive from torch.optim.lr_scheduler._LRScheduler to remain torch- compatible.

class thelper.optim.schedulers.CustomStepLR(optimizer, milestones, last_epoch=-1)[source]¶

Bases: sphinx.ext.autodoc.importer._MockObject

Sets the learning rate of each parameter group using a dictionary of preset scaling factors for epoch-based milestones.

This class can be useful for tuning the learning rate scheduling behavior of a training session beyond what is already possible using PyTorch’s existing LR scheduler classes. Note that all epoch indices are assumed to be 0-based.

Usage example in Python:

# Assuming the optimizer uses lr = 0.05, we hard-code a slow startup...
# lr = 0.00625   if epoch < 2        (1/8 scale before epoch 2)
# lr = 0.0125    if 2 <= epoch < 3   (1/4 scale before epoch 3)
# lr = 0.025     if 3 <= epoch < 4   (1/2 scale before epoch 4)
# lr = 0.05      if 4 <= epoch < 30  (default scale between epoch 4 and 30)
# lr = 0.005     if 30 <= epoch < 80 (1/10 scale past epoch 30)
# lr = 0.0005    if epoch >= 80      (1/100 scale past epoch 80)
scheduler = CustomStepLR(optimizer, milestones={
    0: 1/8,
    2: 1/4,
    3: 1/2,
    4: 1,
    30: 0.1,
    80: 0.01
})
for epoch in range(100):
    # note: we can call the scheduler's step function before optimizing, as
    # we provide it with the epoch index directly --- this is not the case
    # typically with default pytorch schedulers (and it changed beyond v1.1)
    scheduler.step(epoch)
    train(...)
    validate(...)

Usage example inside a session configuration file:

# ...
# lists the model optimization parameters for the training session
"optimization": {
    # lists the optimizer arguments (type, parameters, LR, ...)
    "optimizer": {
        # ...
    },
    # lists the scheduler arguments (field can be omitted if no scheduler is needed)
    "scheduler": {
        # the type used to instantiate the scheduler
        "type": "thelper.optim.schedulers.CustomStepLR",
        # the parameters passed to the scheduler's constructor
        "params": {
            # by default, the optimizer is passed automatically;
            # we only need to specify the extra parameters here
            "milestones": {
                "1": 1,  # after epoch 1, scale the LR by 1
                "10": 0.1, # after epoch 10, scale the LR by 0.1
                "20": 0.01,  # ... and so on
                "30": 0.001,
                "40": 0.0001
            }
        }
    }
},
# ...

Variables:	stages – list of epochs where a new scaling factor is to be applied. scales – list of scaling factors to apply at each stage. milestones – original milestones map provided in the constructor.

thelper.optim.utils module¶

Optimization/metrics utility module.

This module contains utility functions and tools used by the other modules of this package.

thelper.optim.utils.create_loss_fn(config, model, loader=None, uploader=None)[source]¶

Instantiates and returns the loss function to use for training.

The default way to specify the loss function to use is to provide a callable type to instantiate as well as its initialization parameters. For example:

# ...
"loss": {
    # if we want to use PyTorch's cross entropy loss:
    #  >>> loss_fn = torch.nn.CrossEntropyLoss(**params)
    #  >>> ...
    #  >>> loss = loss_fn(pred, target)
    "type": "torch.nn.CrossEntropyLoss"
    "params": {
        "weight": [ ... ],
        "reduction": "mean",
        # ...
    }
},
# ...

The loss function can also be queried from a member function of the model class, as such:

# ...
"loss": {
    # to query the model for the loss function:
    #  >>> loss_fn = model.get_loss_fn(**params)
    #  >>> ...
    #  >>> loss = loss_fn(pred, target)
    "model_getter": "get_loss_fn"
    "params": {
        # ...
    }
},
# ...

If the model is supposed to compute its own loss, we suggest creating a specialized trainer class. In that case only, the ‘loss’ field can be omitted from the session configuration file.

If the task is related to image classification or semantic segmentation, the classes can be weighted based on extra parameters in the loss configuration. The strategy used to compute the weights is related to the one in thelper.data.samplers.WeightedSubsetRandomSampler. The exact parameters that are expected for class reweighting are the following:

weight_distribution (mandatory, toggle): the dictionary of weights assigned to each class, or the rebalancing strategy to use. If omitted entirely, no class weighting will be performed.
weight_param_name (optional, default=”weight”): name of the constructor parameter that expects the weight list.
weight_max (optional, default=inf): the maximum weight that can be assigned to a class.
weight_min (optional, default=0): the minimum weight that can be assigned to a class.
weight_norm (optional, default=True): specifies whether the weights should be normalized or not.

This function also supports an extra special parameter if the task is related to semantic segmentation: ignore_index. If this parameter is found and not None (integer), then the loss function will ignore the given value when computing the loss of a sample. The exact parameters that are expected in this case are the following:

ignore_index_param_name (optional, default=”ignore_index”): name of the constructor parameter that expects the ignore value.
ignore_index_label_name (optional, default=”dontcare”): name of the label to pass the ignore value from.

thelper.optim.utils.create_metrics(config)[source]¶

Instantiates and returns the metrics defined in the configuration dictionary.

All arguments are expected to be handed in through the configuration via a dictionary named ‘params’.

thelper.optim.utils.create_optimizer(config, model)[source]¶

Instantiates and returns the optimizer to use for training.

By default, the optimizer will be instantiated with the model parameters given as the first argument of its constructor. All supplementary arguments are expected to be handed in through the configuration via a dictionary named ‘params’.

thelper.optim.utils.create_scheduler(config, optimizer)[source]¶

Instantiates and returns the learning rate scheduler to use for training.

All arguments are expected to be handed in through the configuration via a dictionary named ‘params’.

thelper.optim.utils.get_lr(optimizer)[source]¶: Returns the optimizer’s learning rate, or 0 if not found.

thelper.optim package¶

Submodules¶

thelper.optim.eval module¶

thelper.optim.losses module¶

thelper.optim.metrics module¶

thelper.optim.schedulers module¶

thelper.optim.utils module¶

Table Of Contents

This Page

Navigation

thelper.optim package¶

Submodules¶

thelper.optim.eval module¶

thelper.optim.losses module¶

thelper.optim.metrics module¶

thelper.optim.schedulers module¶

thelper.optim.utils module¶

Navigation