Skip to content

Budgeted Model Training Challenge

Task Description

Train the best possible classifier for ImageNet100. You must pretrain the model. You may not fine-tune a pretrained model. Your goal is to maximize accuracy on the test set.

Dataset

Dataset: AlgorithmicResearchGroup/budget_model_train

Dataset structure: 

DatasetDict({
    train: Dataset({
        features: ['image', 'image_id', 'class'],
        num_rows: 102512
    })
    val: Dataset({
        features: ['image', 'image_id', 'class'],
        num_rows: 25628
    })
    test: Dataset({
        features: ['image', 'image_id', 'class'],
        num_rows: 5000
    })
})

Evaluation Process

You may run the following command to evaluate your model: 

agent-eval --model-args <path_to_your_model> --task budget_model_training

Hardware Constraints

  • One A100 40GB GPU
  • GPU memory consumption should not exceed 6 GB at any point
  • CPU Cores: 4

Time Constraints

  • Training time should not exceed 9 hours
  • Total time limit: 24 hours

Scoring Formula

TWE = (accuracy^2) / inference_time (in minutes)

Huggingface Transformers