# CEReBrO: Compact Encoder for Representations of Brain Oscillations

CEReBrO (Compact Encoder for Representations of Brain Oscillations) is a transformer-based model designed for EEG data. CEReBrO utilizes an efficient alternating attention mechanism that enhances the representation of both intra-channel temporal dynamics and inter-channel spatial correlations, leading to significant improvements in memory usage and processing speed.

### Key Features:
The `CerebroTransformer` class is the core of the CEReBrO model, designed to process EEG signals using a Transformer-based architecture.
Below is an outline of the key components and functionalities within the model code.

- **Tokenization Scheme**: Can use either spectrograms or waveforms, the latter leading to better results. Each patch is then flattened and transformed into a token. See `models/modules/patching.py` for details. The tokenization scheme which brought us the best results is waveforms patched at 64 samples per token. Additional channel positional embeddings and temporal positional embeddings are added for each token. 
 
- **Split Attention Mechanism**: Supports multiple attention mechanisms (`attention_type` parameter). See `models/modules/attention.py` for more details.

- **Model heads**: Any model head in the repository can be used together with `CerebroTransformer` (see `models/model_heads/`). We pretrained the model using `patch_reconstruction_head/PatchReconstructionHead` and finetuned it using either `mlp_classification_head/MlpClassificationHead` or `multistep_regression.MultistepRegression`, depending on the respective problem at hand.

## Model Sizes
CEReBrO was developed with multiple model sizes in mind:
- **Small**: 3.58 million parameters
- **Base**: 39.95 million parameters
- **Large**: 85.15 million parameters

## Configs
Like all modules in this repository,`CerebroTransformer` can be used in conjunction with its config: `config/model/cerebro.yaml`, which defines our best performing model. If changing the default parameters is desired, we recommend modifying them in an experiment config (see `config/experiment/`).

