eegproc.deep_learning.cross_validation.calibration module

Subject-calibration cross-validation entry point.

eegproc.deep_learning.cross_validation.calibration.subject_calibration_cv(model_builder_function, feature_array, label_array, subject_id_array, trial_id_array, fixed_config=None, source_epochs=100, source_batch_size=64, calibration_epochs=20, calibration_batch_size=6, *, pretrained_model=None, calibration_trials=6, calibration_folds=3, calibration_levels=None, calibration_selection_shots=None, calibration_learning_rate=0.0001, calibration_optimizer='adamw', calibration_weight_decay=0.0, calibration_seed=42, stratify_calibration=True, allow_overlapping_calibration_folds=False, validation_subjects_per_fold=0, validation_seed=42, early_stopping_patience=None, early_stopping_min_delta=0.0, early_stopping_monitor='val_loss', early_stopping_mode='auto', restore_best_weights=True, evaluation_level='trial', metrics=('accuracy', 'f1', 'precision', 'recall', 'macro_f1', 'macro_precision', 'macro_recall', 'balanced_accuracy', 'roc_auc', 'brier_score', 'ece'), ece_bins=15, decision_threshold=0.5, prediction_diagnostics_thresholds=None, prediction_diagnostics=False, prediction_diagnostics_metric='accuracy', prediction_diagnostics_every_n_epochs=1, prediction_diagnostics_max_samples=256, prediction_diagnostics_threshold_tolerance=0.01, prediction_diagnostics_seed=42, n_prediction_latent_samples=0, latent_sampling_seed=None, log_predictions=True, log_variational_intervals=False, n_uncertainty_samples=30, ci_level=0.95, source_use_class_weight=False, calibration_use_class_weight=False, source_fit_kwargs=None, calibration_fit_kwargs=None, calibration_verbose=False, calibration_print_every_n_epochs=1, verbose=0, n_jobs=1, gpu_ids=None, cpus_per_worker=None, max_subjects=None, target_subjects=None, source_model_output_dir=None)

Strict LOSO pretraining followed by one or more calibration levels.

Pass exactly one of model_builder_function and pretrained_model. The pretrained path accepts one already-loaded, target-specific LOSO model, requires one explicit target_subjects entry, and skips source fitting.

For each target subject, a fresh subject-independent source model is first trained using the other subjects. When validation_subjects_per_fold is positive, a seeded subset of those source subjects is excluded from gradient updates and used only for source checkpoint selection. The target subject is never used to construct, optimize, or select that source model. This is important for models that learn data-dependent adjacency: the four training_* arrays contain only gradient-training source subjects, so its fixed mutual-information adjacency remains training-only.

The source model is evaluated zero-shot on all target trials once. Pass calibration_levels=((3, 6), (6, 3), ...) to request (shots, folds) pairs. If omitted, the legacy calibration_trials/calibration_folds pair is used. The same source checkpoint is restored for every level and fold; it is never retrained between calibration levels. Exact partitions are used when shots * folds equals the retained trial count, otherwise seeded repeated holdouts evaluate on every non-calibration trial. For each fold:

1. restore the exact source-pretrained weights;

2. evaluate zero-shot on the trials not used for calibration;

3. call model.prepare_for_subject_calibration(...);

4. fine-tune only the model-defined calibration parameters, printing reporting-only oracle predictions on every non-calibration trial after each epoch;

5. evaluate the calibrated model on the same held-out target trials.

When source_model_output_dir is provided, the exact source-trained 0-shot model for every LOSO target is saved there in native .keras format before calibration begins. Calibrated fold continuations are not saved. Load these source models with compile=False before calling the model’s calibration preparation hook, which installs a fresh optimizer. Model-specific calibration settings may optionally keep reconstruction decoders trainable and add their weighted loss to the calibration objective.

Thus the default DREAMER protocol remains 6 calibration trials + 12 evaluation trials repeated three times, while arbitrary valid shot/fold pairs can be evaluated in the same LOSO run.

model_builder_function contract

In addition to fixed_config model kwargs, the builder must accept: training_features, training_labels, training_subject_ids, and training_trial_ids. The first of these is where v6 should estimate its training-only MI graph.

prepare_for_subject_calibration contract

The returned model must implement:

model.prepare_for_subject_calibration(
    learning_rate=...,
    optimizer_name=...,
    weight_decay=...,
)

The method owns the architecture-specific freezing policy and must compile a fresh optimizer. A typical policy freezes the shared representation layers — graph, spectral and temporal encoders, plus any subject-invariance machinery — while the classification head stays trainable; reconstruction decoders may also remain trainable when the model’s calibration settings request it.

Aggregation

Metrics are first averaged across folds within each subject and shot level, then across subjects. Calibration folds are never treated as independent subjects. JSON output retains every requested metric at every shot level; terminal output reports final mean accuracy, balanced accuracy, and Brier score for 0-shot and each calibrated level.

type model_builder_function:

Optional[Callable[..., Model]]

param model_builder_function:

type feature_array:

ndarray

param feature_array:

type label_array:

ndarray

param label_array:

type subject_id_array:

ndarray

param subject_id_array:

type trial_id_array:

ndarray

param trial_id_array:

type fixed_config:

dict | None, default: None

param fixed_config:

type source_epochs:

int, default: 100

param source_epochs:

type source_batch_size:

int, default: 64

param source_batch_size:

type calibration_epochs:

int, default: 20

param calibration_epochs:

type calibration_batch_size:

int, default: 6

param calibration_batch_size:

type pretrained_model:

Model | None, default: None

param pretrained_model:

type calibration_trials:

int, default: 6

param calibration_trials:

type calibration_folds:

int, default: 3

param calibration_folds:

type calibration_levels:

list[tuple[int, int]] | tuple[tuple[int, int], ...] | None, default: None

param calibration_levels:

type calibration_selection_shots:

int | None, default: None

param calibration_selection_shots:

type calibration_learning_rate:

float, default: 0.0001

param calibration_learning_rate:

type calibration_optimizer:

str, default: 'adamw'

param calibration_optimizer:

type calibration_weight_decay:

float, default: 0.0

param calibration_weight_decay:

type calibration_seed:

int | None, default: 42

param calibration_seed:

type stratify_calibration:

bool, default: True

param stratify_calibration:

type allow_overlapping_calibration_folds:

bool, default: False

param allow_overlapping_calibration_folds:

type validation_subjects_per_fold:

int, default: 0

param validation_subjects_per_fold:

type validation_seed:

int | None, default: 42

param validation_seed:

type early_stopping_patience:

int | None, default: None

param early_stopping_patience:

type early_stopping_min_delta:

float, default: 0.0

param early_stopping_min_delta:

type early_stopping_monitor:

str, default: 'val_loss'

param early_stopping_monitor:

type early_stopping_mode:

Literal['auto', 'min', 'max'], default: 'auto'

param early_stopping_mode:

type restore_best_weights:

bool, default: True

param restore_best_weights:

type evaluation_level:

Literal['window', 'trial'], default: 'trial'

param evaluation_level:

type metrics:

list[str] | tuple[str, ...], default: ('accuracy', 'f1', 'precision', 'recall', 'macro_f1', 'macro_precision', 'macro_recall', 'balanced_accuracy', 'roc_auc', 'brier_score', 'ece')

param metrics:

type ece_bins:

int, default: 15

param ece_bins:

type decision_threshold:

float, default: 0.5

param decision_threshold:

type prediction_diagnostics_thresholds:

list[float] | tuple[float, ...] | None, default: None

param prediction_diagnostics_thresholds:

type prediction_diagnostics:

bool, default: False

param prediction_diagnostics:

type prediction_diagnostics_metric:

str, default: 'accuracy'

param prediction_diagnostics_metric:

type prediction_diagnostics_every_n_epochs:

int, default: 1

param prediction_diagnostics_every_n_epochs:

type prediction_diagnostics_max_samples:

int, default: 256

param prediction_diagnostics_max_samples:

type prediction_diagnostics_threshold_tolerance:

float, default: 0.01

param prediction_diagnostics_threshold_tolerance:

type prediction_diagnostics_seed:

int | None, default: 42

param prediction_diagnostics_seed:

type n_prediction_latent_samples:

int, default: 0

param n_prediction_latent_samples:

type latent_sampling_seed:

int | None, default: None

param latent_sampling_seed:

type log_predictions:

bool, default: True

param log_predictions:

type log_variational_intervals:

bool, default: False

param log_variational_intervals:

type n_uncertainty_samples:

int, default: 30

param n_uncertainty_samples:

type ci_level:

float, default: 0.95

param ci_level:

type source_use_class_weight:

bool, default: False

param source_use_class_weight:

type calibration_use_class_weight:

bool, default: False

param calibration_use_class_weight:

type source_fit_kwargs:

dict | None, default: None

param source_fit_kwargs:

type calibration_fit_kwargs:

dict | None, default: None

param calibration_fit_kwargs:

type calibration_verbose:

bool, default: False

param calibration_verbose:

type calibration_print_every_n_epochs:

int, default: 1

param calibration_print_every_n_epochs:

type verbose:

int, default: 0

param verbose:

type n_jobs:

int, default: 1

param n_jobs:

type gpu_ids:

list[int] | tuple[int, ...] | None, default: None

param gpu_ids:

type cpus_per_worker:

int | None, default: None

param cpus_per_worker:

type max_subjects:

int | None, default: None

param max_subjects:

type target_subjects:

list[int] | tuple[int, ...] | None, default: None

param target_subjects:

type source_model_output_dir:

str | PathLike[str] | None, default: None

param source_model_output_dir:

rtype:

dict