eegproc.deep_learning.cross_validation.loso module

Leave-one-subject-out cross-validation entry points.

eegproc.deep_learning.cross_validation.loso.fixed_loso_cv(model_builder_function, feature_array, label_array, subject_id_array, trial_id_array, fixed_config, n_epochs, batch_size, *, preprocessing_strategy=None, evaluation_level='trial', selection_metric='balanced_accuracy', selection_level='trial', maximize_metric=None, metrics=('accuracy', 'f1', 'precision', 'recall', 'macro_f1', 'macro_precision', 'macro_recall', 'balanced_accuracy'), log_predictions=True, log_variational_intervals=False, n_prediction_latent_samples=0, latent_sampling_seed=None, n_uncertainty_samples=30, ci_level=0.95, decision_threshold=0.5, prediction_diagnostics=False, prediction_diagnostics_every_n_epochs=1, prediction_diagnostics_max_samples=256, prediction_diagnostics_threshold_tolerance=0.01, prediction_diagnostics_seed=42, verbose=0, extra_fit_kwargs=None, n_jobs=1, gpu_ids=None, cpus_per_worker=None, max_folds=None, alternate_subject_sets=False, alternating_subject_seed=42, use_mldg=False, mldg_meta_train_subjects=6, mldg_meta_test_subjects=2, mldg_samples_per_subject=4, mldg_seed=42)

Evaluate one fixed configuration with strict LOSOCV and no validation.

Every fold trains for exactly n_epochs on all non-test subjects. No validation subjects are removed, no validation data are passed to Keras, no early-stopping callback is installed, and the supplied decision threshold is applied unchanged to every held-out subject.

This is intended as a post-selection diagnostic after another CV run has already chosen the hyperparameters, epoch count, and threshold. It does not perform another hyperparameter or threshold search.

Parameters:

• model_builder_function (Callable[..., Model])

• feature_array (ndarray)

• label_array (ndarray)

• subject_id_array (ndarray)

• trial_id_array (ndarray)

• fixed_config (dict)

• n_epochs (int)

• batch_size (int)

• preprocessing_strategy (Optional[Callable], default: None)

• evaluation_level (Literal['window', 'trial'], default: 'trial')

• selection_metric (str, default: 'balanced_accuracy')

• selection_level (Literal['window', 'trial'], default: 'trial')

• maximize_metric (bool | None, default: None)

• metrics (list[str] | tuple[str, ...], default: ('accuracy', 'f1', 'precision', 'recall', 'macro_f1', 'macro_precision', 'macro_recall', 'balanced_accuracy'))

• log_predictions (bool, default: True)

• log_variational_intervals (bool, default: False)

• n_prediction_latent_samples (int, default: 0)

• latent_sampling_seed (int | None, default: None)

• n_uncertainty_samples (int, default: 30)

• ci_level (float, default: 0.95)

• decision_threshold (float, default: 0.5)

• prediction_diagnostics (bool, default: False)

• prediction_diagnostics_every_n_epochs (int, default: 1)

• prediction_diagnostics_max_samples (int, default: 256)

• prediction_diagnostics_threshold_tolerance (float, default: 0.01)

• prediction_diagnostics_seed (int | None, default: 42)

• verbose (int, default: 0)

• extra_fit_kwargs (dict | None, default: None)

• n_jobs (int, default: 1)

• gpu_ids (list[int] | tuple[int, ...] | None, default: None)

• cpus_per_worker (int | None, default: None)

• max_folds (int | None, default: None)

• alternate_subject_sets (bool, default: False)

• alternating_subject_seed (int | None, default: 42)

• use_mldg (bool, default: False)

• mldg_meta_train_subjects (int, default: 6)

• mldg_meta_test_subjects (int, default: 2)

• mldg_samples_per_subject (int, default: 4)

• mldg_seed (int | None, default: 42)

Return type:

dict

eegproc.deep_learning.cross_validation.loso.loso_cv(model_builder_function, feature_array, label_array, subject_id_array, trial_id_array=None, n_epochs=50, batch_size=2, hyperparameters=None, preprocessing_strategy=None, evaluation_level='trial', selection_metric='f1', selection_level='trial', maximize_metric=None, metrics=('accuracy', 'f1', 'precision', 'recall', 'macro_f1', 'macro_precision', 'macro_recall', 'balanced_accuracy', 'roc_auc'), log_predictions=True, log_variational_intervals=False, n_prediction_latent_samples=0, latent_sampling_seed=None, n_uncertainty_samples=30, ci_level=0.95, validation_subjects_per_fold=0, validation_seed=42, early_stopping_patience=5, early_stopping_min_delta=0.0, early_stopping_monitor='val_loss', early_stopping_mode='min', restore_best_weights=True, prediction_diagnostics=False, prediction_diagnostics_every_n_epochs=1, prediction_diagnostics_max_samples=256, prediction_diagnostics_threshold_tolerance=0.01, prediction_diagnostics_seed=42, decision_thresholds=(0.5,), threshold_selection_metric='f1', threshold_selection_level='trial', verbose=0, extra_fit_kwargs=None, n_jobs=1, gpu_ids=None, cpus_per_worker=None, max_folds=None, alternate_subject_sets=False, alternating_subject_seed=42, use_mldg=False, mldg_meta_train_subjects=6, mldg_meta_test_subjects=2, mldg_samples_per_subject=4, mldg_seed=42)

Run a flat hyperparameter search using complete LOSO evaluations.

For every Cartesian-product hyperparameter configuration, each unique subject is held out exactly once. The configuration is therefore evaluated on the same complete set of subject-wise folds. After all configurations finish, one global configuration is selected from its mean LOSO metric.

This is not nested cross-validation: the held-out LOSO results are used both to compare configurations and to report the selected configuration’s cross-validation performance. This behavior is intentional for a practical flat LOSO hyperparameter search.

Return type:

dict

Hyperparameter grid

Scalar values may be supplied directly or as candidate lists/tuples. The Cartesian product is evaluated with a complete LOSO run per configuration.

Sequence-valued encoder settings preserve one complete architecture before the Cartesian product is expanded. sequence_hyperparameter_depths specifies the nesting depth of one value, resolving CNN1D/CNN2D ambiguity for keys such as kernel_sizes. GCN gcn_units and temporal/spatial pooling schedules are preserved in the same way. One additional outer list level enumerates multiple architecture candidates.

n_epochs and batch_size provide defaults and are overridden when hyperparameters contains epochs or batch_size.

Seeded validation and early stopping

When validation_subjects_per_fold is positive, that many subjects are sampled deterministically from each outer-training pool. They are excluded from gradient updates and passed to model.fit as validation_data. The same fold-local validation subjects are reused for every hyperparameter configuration, while the LOSO test subject remains untouched. This adds no extra fits; it only changes each fit from train/test to train/validation/test.

Selection

selection_level determines whether configurations are ranked using window- or trial-level scores. Hierarchical rank-4 inputs require trial-level selection. For binary tasks, selection_metric='f1' uses the binary convention: class 1 is positive. precision and recall follow the same convention. Explicit macro_* metrics and balanced_accuracy remain available for class-balanced diagnostics, while roc_auc uses the class-1 probability. Classification metrics are maximized; probability loss and joint loss are minimized unless maximize_metric is explicitly supplied. Ties use lower between-subject standard deviation, lower mean log loss, then the earlier grid index.

Returned results

config_results contains per-fold and aggregate metrics for every configuration. Top-level prediction logs, user metrics, and fold metadata correspond only to the globally selected configuration. Selected fold metrics remain available through config_results[best_config_index].

Concurrency

LOSO folds for one configuration run concurrently. The next configuration starts after the current configuration’s folds complete. With one worker per GPU, this prevents multiple models from competing for the same GPU while bounding parent-process memory to approximately one configuration’s logs.

Smoke testing

max_folds deterministically limits every configuration to the first N sorted subjects. Leave it as None for complete LOSO evaluation.

type model_builder_function:

Callable[..., Model]

param model_builder_function:

type feature_array:

ndarray

param feature_array:

type label_array:

ndarray

param label_array:

type subject_id_array:

ndarray

param subject_id_array:

type trial_id_array:

ndarray | None, default: None

param trial_id_array:

type n_epochs:

int, default: 50

param n_epochs:

type batch_size:

int, default: 2

param batch_size:

type hyperparameters:

dict | None, default: None

param hyperparameters:

type preprocessing_strategy:

Optional[Callable], default: None

param preprocessing_strategy:

type evaluation_level:

Literal['window', 'trial'], default: 'trial'

param evaluation_level:

type selection_metric:

str, default: 'f1'

param selection_metric:

type selection_level:

Literal['window', 'trial'], default: 'trial'

param selection_level:

type maximize_metric:

bool | None, default: None

param maximize_metric:

type metrics:

list[str] | tuple[str, ...], default: ('accuracy', 'f1', 'precision', 'recall', 'macro_f1', 'macro_precision', 'macro_recall', 'balanced_accuracy', 'roc_auc')

param metrics:

type log_predictions:

bool, default: True

param log_predictions:

type log_variational_intervals:

bool, default: False

param log_variational_intervals:

type n_prediction_latent_samples:

int, default: 0

param n_prediction_latent_samples:

type latent_sampling_seed:

int | None, default: None

param latent_sampling_seed:

type n_uncertainty_samples:

int, default: 30

param n_uncertainty_samples:

type ci_level:

float, default: 0.95

param ci_level:

type validation_subjects_per_fold:

int, default: 0

param validation_subjects_per_fold:

type validation_seed:

int | None, default: 42

param validation_seed:

type early_stopping_patience:

int | None, default: 5

param early_stopping_patience:

type early_stopping_min_delta:

float, default: 0.0

param early_stopping_min_delta:

type early_stopping_monitor:

str, default: 'val_loss'

param early_stopping_monitor:

type early_stopping_mode:

Literal['auto', 'min', 'max'], default: 'min'

param early_stopping_mode:

type restore_best_weights:

bool, default: True

param restore_best_weights:

type prediction_diagnostics:

bool, default: False

param prediction_diagnostics:

type prediction_diagnostics_every_n_epochs:

int, default: 1

param prediction_diagnostics_every_n_epochs:

type prediction_diagnostics_max_samples:

int, default: 256

param prediction_diagnostics_max_samples:

type prediction_diagnostics_threshold_tolerance:

float, default: 0.01

param prediction_diagnostics_threshold_tolerance:

type prediction_diagnostics_seed:

int | None, default: 42

param prediction_diagnostics_seed:

type decision_thresholds:

list[float] | tuple[float, ...], default: (0.5,)

param decision_thresholds:

type threshold_selection_metric:

Literal['accuracy', 'f1', 'balanced_accuracy', 'binary_f1'], default: 'f1'

param threshold_selection_metric:

type threshold_selection_level:

Literal['window', 'trial'], default: 'trial'

param threshold_selection_level:

type verbose:

int, default: 0

param verbose:

type extra_fit_kwargs:

dict | None, default: None

param extra_fit_kwargs:

type n_jobs:

int, default: 1

param n_jobs:

type gpu_ids:

list[int] | tuple[int, ...] | None, default: None

param gpu_ids:

type cpus_per_worker:

int | None, default: None

param cpus_per_worker:

type max_folds:

int | None, default: None

param max_folds:

type alternate_subject_sets:

bool, default: False

param alternate_subject_sets:

type alternating_subject_seed:

int | None, default: 42

param alternating_subject_seed:

type use_mldg:

bool, default: False

param use_mldg:

type mldg_meta_train_subjects:

int, default: 6

param mldg_meta_train_subjects:

type mldg_meta_test_subjects:

int, default: 2

param mldg_meta_test_subjects:

type mldg_samples_per_subject:

int, default: 4

param mldg_samples_per_subject:

type mldg_seed:

int | None, default: 42

param mldg_seed: