in the loop

Gemini 1.5 Pro (May 2024)
#176 of 274 scored.

GoogleMay 14, 2024Closed

Capabilities Index

Range 121.1–129.2

126.9

Context

Tokens

—

Input

Per 1M tokens

—

Output

Per 1M tokens

—

Benchmarks

How it scores.

  • GPQA Diamond27.8%

    150th of 186

  • Mock AIME6.7%

    144th of 176

Everything else Epoch has for it

  • BBH85.6%
  • MMLU81.2%
  • MATH level 540.8%
  • DTBench27.5%

On the index

Around it.

  1. #173GPT-4 Turbo (Apr 2024)OpenAI127.3
  2. #174Claude 3.5 HaikuAnthropic127.2
  3. #175Mistral Small 3Mistral · $0.05 in127.1
  4. #176Gemini 1.5 Pro (May 2024)Google126.9
  5. #177Claude 3 OpusAnthropic126.9
  6. #178GPT-4o miniOpenAI · $0.15 in126.6
  7. #179GPT-4 Turbo (Nov 2023)OpenAI126.5

Sources