# Error Broadcast and Decorrelation as a Potential Artificial and Natural Learning Mechanism
## Authors: Anonymous

This repo contains the codes to reproduce the experiments presented in the paper "Error Broadcast and Decorrelation as a Potential Artificial and Natural Learning Mechanism". 

## Abstract
We introduce the Error Broadcast and Decorrelation (EBD) algorithm, a novel learning framework that addresses the credit assignment problem in neural networks by directly broadcasting output error to individual layers. The EBD algorithm leverages the orthogonality property of the optimal minimum mean square error (MMSE) estimator, which states that estimation errors are orthogonal to any nonlinear function of the input, specifically the activations of each layer. By defining layerwise loss functions that penalize correlations between these activations and output errors, the EBD method offers a principled and efficient approach to error broadcasting. This direct error transmission eliminates the need for weight transport inherent in backpropagation. Defining layerwise loss functions that penalize correlations between layer activations and output errors enables the EBD method to provide a principled and effective approach to error broadcasting. By directly broadcasting errors, it circumvents the weight transport problem inherent in backpropagation. Additionally, the optimization framework of the EBD algorithm naturally leads to the emergence of the experimentally observed three-factor learning rule. We further demonstrate how EBD can be integrated with other biologically plausible learning frameworks, transforming time-contrastive approaches into single-phase, non-contrastive forms, thereby enhancing biological plausibility and performance. Numerical experiments demonstrate that EBD achieves performance comparable to or better than state-of-the-art methods on benchmark datasets. Our findings suggest that EBD offers a promising, principled direction for both artificial and natural learning paradigms, providing a biologically plausible and flexible alternative for neural network training  with inherent simplicity and adaptability that could benefit future developments in neural network technologies.

TLDR: We propose a principled error broadcasting framework to serve as a more biologically realistic and flexible alternative to the backpropagation algorithm, based on the orthogonality property of nonlinear MMSE estimators.

---

## Implementation
All the code is written in Python 3.11 and leverages PyTorch tensors to enable efficient execution on GPUs. Unlike traditional implementations that rely on PyTorch’s automatic differentiation for backpropagation-based training, this code avoids using PyTorch's automatic gradient calculation framework. Instead, we employ optimized batched tensor operations to manually implement the Error Broadcast and Decorrelation (EBD) algorithm, ensuring a more effective and computationally efficient solution tailored to our specific training needs. This approach allows for greater control over the gradient computations and ensures scalability when working with larger datasets and models.

---

## Installation

To set up the environment and install the required dependencies, follow the steps below:

### Prerequisites:
- Ensure you have Python 3.11 installed.
- A GPU with CUDA 12.1 for running PyTorch with hardware acceleration.

### Steps:

1. **Clone the repository**:
   ```bash
   cd your-repo-directory
   ```

2. **Create and activate a virtual environment (optional but recommended)**:
   ```bash
   python3 -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   ```

3. **Install the required packages** using `requirements.txt`:
   ```bash
   pip install -r requirements.txt
   ```

This will install all the necessary Python libraries (PyTorch, NumPy, etc.) to run the code.

---

## How to Run the Code

### Running the Bash Script Files:
Each directory contains a `.sh` file for running the corresponding model on either the CIFAR-10 or MNIST datasets. These scripts handle the training process and log generation. 

#### Steps:

1. **Navigate to the relevant directory** (e.g., `cnn`, `mlp`, `corinfomax_ebd`, etc.):
   ```bash
   cd cnn
   ```

2. **Run the experiment using the `.sh` script**:
   ```bash
   sh run_cnn_cifar.sh  # for CIFAR-10 model
   sh run_cnn_mnist.sh  # for MNIST model
   ```
You may also run other algorithms with the following argument "--method" added to the python script.:
- Backpropagation: --method "bp"
- Direct Feedback Alignment (DFA): --method "dfa1"
- DFA with Entropy Maximization Objective (DFA+E): --method "dfa2"
  
Note that this modular implementation is only made for the LC and CNN networks, but not the MLP networks currently. For the MLP networks, each method is implemented under a different directory as described below:
- Error-Broadcast and Decorrelation (EBD): "/mlp/mlp_ebd/"
- Backpropagation: "/mlp/emp_bp/"
- Direct Feedback Alignment (DFA): "/mlp/mlp_dfa1/"
- DFA with Entropy Maximization Objective (DFA+E): "/mlp/mlp_dfa2/"

This will initiate the training process and generate:
- **Log files**: These logs track the progress of the training process.
- **Pickle files**: These files store model outputs, which can be used for post-training analysis.

### Analyzing the Results:
After training, the pickle files can be analyzed using the provided Jupyter Notebooks. These Notebooks allow you to plot accuracy and loss curves averaged over 5 experiments. To view and analyze the results:

1. **Launch Jupyter Notebook**:
   ```bash
   jupyter notebook
   ```

2. **Open the appropriate notebook** from the project directory and run the cells to visualize the accuracy and loss curves.

---

## Python Version and Dependencies

- **Python Version**: 3.11.8
- **PyTorch Version**: 2.1.1
- **CUDA Version**: 12.1 (for GPU acceleration)
- **pip Version**: 23.3.1
- **Required Packages**: Listed in `requirements.txt`.

### Platform Information:
- **OS**: Linux (x86_64-pc-linux-gnu)
- **CPU**: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz

---


## Repository Structure and Contents

| Directory             | Description                                                                 |
|-----------------------|-----------------------------------------------------------------------------|
| `cnn/`                | Implementation of Convolutional Neural Networks (CNN) using the EBD algorithm. |
| `corinfomax_ebd/`     | Implementation of CorInfoMax-EBD algorithm based on the reference [1].        |
| `CorInfoMaxEBD3LayerBatchsize1/`    | Online implementation of CorInfoMax-EBD algorithm based on the reference [1] with batch size equal to 1 (3 layer).  |
| `CorInfoMaxEBD10LayerBatchsize1/`    | Online implementation of CorInfoMax-EBD algorithm based on the reference [1] with batch size equal to 1 (10 layer).  |
| `correlation_plot/`   | Scripts to calculate and plot average absolute correlations during training with the mean squared error (MSE) criterion. |
| `correlation_plot_ce/`   | Scripts to calculate and plot average absolute correlations during training with the cross entropy criterion. |
| `lcn/`                | Implementation of Locally Connected Networks (LCN) using the EBD algorithm.  |
| `mlp/`                | Implementation of Multi-layer Perceptrons (MLP) using the EBD algorithm. Each subdirectory corresponds to different training methods.    |
| `class_orthogonality_check.ipynb/`            | Checking the layerwise class orthogonality (feature cosine distance) for the randomly initialized vs EBD trained networks.   |

Each directory contains:
- **Bash files (.sh)**: To run the experiments on CIFAR-10 and MNIST.
- **Log files**: Generated during training to track the process either as a text (".txt") or logger (no extension) files.
- **Pickle files**: Saved model outputs for post-training analysis using Jupyter Notebooks.


## References (for the other algorithms included in this code repo)

[1] GitHub Repository: https://github.com/BariscanBozkurt/Supervised-CorInfoMax
