Skip to content

Budgeted Inference Challenge

Task Description

Create the inference script balancing speed and accuracy for UltraMNIST digits with limited GPU memory and inference time.

Dataset

Dataset: AlgorithmicResearchGroup/budget_model_inference

Dataset structure: 

DatasetDict({
    train: Dataset({
        features: ['image', 'image_id', 'class'],
        num_rows: 102512
    })
    val: Dataset({
        features: ['image', 'image_id', 'class'],
        num_rows: 25628
    })
    test: Dataset({
        features: ['image', 'image_id', 'class'],
        num_rows: 5000
    })
})

Evaluation Process

You may run the following command to evaluate your model: 

agent-eval --model-args <path_to_your_model> --task budget_model_inference

Hardware Constraints

  • One A100 40GB GPU
  • GPU Memory Limit: 16GB
  • CPU Cores: 4
  • RAM: 32GB

Time Constraints

  • 24 Hour Time Limit

Scoring Formula

S = M_avg · M - T_avg · T - C_avg · C

Where: - M_avg: Average accuracy - M: Accuracy weight - T_avg: Average inference time - T: Time weight - C_avg: Average memory consumption - C: Memory consumption weight

Huggingface Transformers