# Dataset

## Download

You can find **CodeContests**, **CodeForces**, and **MBPP** in `./evaluation`, and download **LiveBench** and **LiveCodeBench** by running `python download_livebench.py` and `python download_livecodebench.py`.

We use Stdio input/output format here. For example, for the task to calculate the sum of a list, the input and output are in the following format:
```python
input = "5\n1 2 3 4 5\n"
output = "15"
```
CodeContests and CodeForces are using this format, however, MBPP and part of LiveCodeBench are using functional input/output format, such like
```python
assert sum_function([1, 2, 3, 4, 5]) == 15
```
In this project, we have converted the the functional format to the Stdio format to achieve consistency (rules are witten in Appendix of our paper). T


## Customize Your Own Dataset

Your JSON dataset must contain the following necessary fields to perform both optimization and evaluation.

1. `question`: This is the coding task description.
2. `test_time_limit`: The time limit for each task's execution, usually set as 1.
3. `example_input` and `example_output` are two lists of the same length, where each corresponding element represents the Stdio-format input and output, respectively. These unit tests are public (can be provided during inference time).
4. `test_input` and `test_output` are two lists of the same length, where each corresponding element represents the Stdio-format input and output, respectively. These private unit tests are used to evaluation.