# Long-Short Transformer for Long-Range Arena

This folder contains the source code for the WikiText-103 language modeling task in [Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals]

It is built on this [repository](https://github.com/IDSIA/lmtool-fwp)

## Dataset Setup

Please find the instruction for data download and preprocessing on this [repository](https://github.com/IDSIA/lmtool-fwp).

## Scripts

Run the following script to train the models
  ```angular2html
  bash script.sh
  ```