Skip to content

MiniPile Challenge

Task Description

Use the MiniPile dataset to pre-train the best possible language model. The model will be tested on SuperGLUE.

Dataset

Dataset: AlgorithmicResearchGroup/minipile

Dataset structure: 

DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 1000000
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 500
    })
    test: Dataset({
        features: ['text'],
        num_rows: 10000
    })
})

Evaluation Process

Evaluation will be done on SuperGLUE.

You may run the following command to evaluate your model: 

lm_eval --model hf \
        --model_args pretrained=<path_to_your_model> \
        --tasks super_glue \
        --device cuda:0 \
        --batch_size 8

Hardware Constraints

  • One A100 40GB GPU

Time Constraints

  • 24 Hour Time Limit

Huggingface Transformers