eegproc.deep_learning.cross_validation.callbacks module

Keras callbacks that report validation and held-out-user metrics during training.

class eegproc.deep_learning.cross_validation.callbacks.EvaluationLevelValidationMetrics(X_val, y_val, subject_ids_val, trial_ids_val, evaluation_level, batch_size=None, ece_bins=15)[source]

Bases: Callback

Add complete window- or trial-level validation metrics to epoch logs.

This is used by nested subject-calibration LOSO folds so source-model epoch selection is based only on source-validation subjects. The outer target subject and all of its calibration/evaluation trials remain untouched.

Parameters:

  • X_val (ndarray)

  • y_val (ndarray)

  • subject_ids_val (ndarray)

  • trial_ids_val (ndarray)

  • evaluation_level (Literal['window', 'trial'])

  • batch_size (int | None, default: None)

  • ece_bins (int, default: 15)

on_epoch_end(epoch, logs=None)[source]

Called at the end of an epoch.

Subclasses should override for any actions to run. This function should only be called during TRAIN mode.

Parameters:

  • epoch (int) – Integer, index of epoch.

  • logs (dict | None, default: None) – Dict, metric results for this training epoch, and for the validation epoch if validation is performed. Validation result keys are prefixed with val_. For training epoch, the values of the Model’s metrics are returned. Example: {‘loss’: 0.2, ‘accuracy’: 0.7}.

Return type:

None

class eegproc.deep_learning.cross_validation.callbacks.HeldOutUserOracleMetrics(X_target, y_target, subject_ids_target, trial_ids_target, target_subject, evaluation_level, batch_size=None, decision_threshold=0.5, diagnostic_thresholds=None, ece_bins=15, calibration_shots=None, calibration_fold=None)[source]

Bases: Callback

Print reporting-only metrics on the outer held-out user each epoch.

This callback deliberately keeps its values out of the Keras logs dictionary. Consequently, the oracle metrics cannot be consumed by EarlyStopping, checkpoint selection, or any other training callback. The target data are used only for inference after an epoch has finished; they never enter a gradient calculation.

Parameters:

  • X_target (ndarray)

  • y_target (ndarray)

  • subject_ids_target (ndarray)

  • trial_ids_target (ndarray)

  • evaluation_level (Literal['window', 'trial'])

  • batch_size (int | None, default: None)

  • decision_threshold (float, default: 0.5)

  • diagnostic_thresholds (list[float] | tuple[float, ...] | None, default: None)

  • ece_bins (int, default: 15)

  • calibration_shots (int | None, default: None)

  • calibration_fold (int | None, default: None)

on_epoch_end(epoch, logs=None)[source]

Called at the end of an epoch.

Subclasses should override for any actions to run. This function should only be called during TRAIN mode.

Parameters:

  • epoch (int) – Integer, index of epoch.

  • logs (dict | None, default: None) – Dict, metric results for this training epoch, and for the validation epoch if validation is performed. Validation result keys are prefixed with val_. For training epoch, the values of the Model’s metrics are returned. Example: {‘loss’: 0.2, ‘accuracy’: 0.7}.

Return type:

None

on_train_begin(logs=None)[source]

Called at the beginning of training.

Subclasses should override for any actions to run.

Parameters:

logs (dict | None, default: None) – Dict. Currently no data is passed to this argument for this method but that may change in the future.

Return type:

None

class eegproc.deep_learning.cross_validation.callbacks.TrialValidationMetrics(X_val, y_val, subject_ids_val, trial_ids_val, batch_size=None)[source]

Bases: Callback

Compute deterministic trial-level validation metrics each epoch.

Hierarchical models are scored directly from one output per trial. Legacy window models are still aggregated within each (subject_id, trial_id) pair. The resulting values are added to the Keras epoch logs as val_trial_f1, val_trial_balanced_accuracy, and val_trial_loss so callbacks such as EarlyStopping can monitor them.

Parameters:

  • X_val (ndarray)

  • y_val (ndarray)

  • subject_ids_val (ndarray)

  • trial_ids_val (ndarray)

  • batch_size (int | None, default: None)

on_epoch_end(epoch, logs=None)[source]

Called at the end of an epoch.

Subclasses should override for any actions to run. This function should only be called during TRAIN mode.

Parameters:

  • epoch (int) – Integer, index of epoch.

  • logs (dict | None, default: None) – Dict, metric results for this training epoch, and for the validation epoch if validation is performed. Validation result keys are prefixed with val_. For training epoch, the values of the Model’s metrics are returned. Example: {‘loss’: 0.2, ‘accuracy’: 0.7}.

Return type:

None