2024 Findings - NAACL 149

Sep 2, 2026

Document analyzes model performance on standardized exams and their sub-tasks across languages (Chinese and English). It reports aggregate statistics such as number of participants, token averages, and accuracy under multiple prompting regimes including zero-shot, zero-shot CoT, few-shot, few-shot CoT, plus separate prompting templates. Coverage spans geography, biology, history, chemistry, physics, English, math, logic, law analytics/logic/reading, LSAT sections, SAT math/English, and Gaokao and SAT benchmarks, presenting comparative results by task and setting.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Evaluation Setup

Exam and Task Coverage

Prompting Strategies

Zero-shot Prompting

Zero-shot CoT

Few-shot CoT

Results and Analysis

Task-level Comparisons

| Exams | \#Participants | Language | Tasks | Subject | \# Instance | \#Avg. Token | | --- | --- | --- | --- | --- | --- | --- | | Gaokao | 12M | Chinese | GK-geography
GK-biology
GK-history
GK-chemistry
GK-physics
GK-En
GK-Ch
GK-Math-QA
GK-Math-Cloze | Geography Biology History Chemistry Physics English Chinese Math
Math | 199
210
243
207
200
306
246
351
118 | 144
141
116
113
124
356
935
68
60 | | SAT | 1.7M | English | SAT-En.
SAT-Math | English Math | 206
220 | 656
54 | | Lawyer Qualification Test | 820K | Chinese | JEC-QA-KD
JEC-QA-CA | Law
Law | 1000
1000 | 146
213 | | Law School
Admission Test (LSAT) | 170K | English | LSAT-AR
LSAT-LR
LSAT-RC | Law-Analytics
Law-Logic
Law-Reading | 230
510
260 | 154
178
581 | | Civil Service Examination | 2M
2M | English
Chinese | LogiQA-en
LogiQA-ch | Logic
Logic | 651
651 | 144
242 | | GREGMAT | 340K
150K | English
English | AQuA-RAT | Math | 254 | 77 | | AMC
AIME | 300K
3000 | English
English | MATH | Math | 1000 | 40 | | Task/Model | Human | Zero-Shot | | | Zero-Shot CoT | | | Few-Shot | | | Few-Shot CoT | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | Avg. Top | TD | CG | G4 | TD | CG | G4 | TD | CG | G4 | TD | CG | G4 | | AQuA-RAT | 85 100 | 29.9 | 31.9 | 40.6 | 42.1 | 55.9 | 73.2 | 30.3 | 31.1 | 50.8 | 47.2 | 60.6 | 74.0 | | MATH | 40 90 | 11.9 | 26.4 | 35.7 | 19.1 | 31.9 | 47.7 | 10.3 | 14.8 | 15.1 | 15.1 | 30.1 | 25.3 | | LogiQA (English) | 86 95 | 22.7 | 35.0 | 49.3 | 36.9 | 39.9 | 57.8 | 43.5 | 43.5 | 63.9 | 37.5 | 38.9 | 62.7 | | LogiQA (Chinese) | 88 96 | 40.3 | 41.0 | 58.8 | 36.7 | 38.9 | 57.5 | 43.2 | 46.2 | 65.0 | 40.0 | 38.6 | 61.9 | | JEC-QA-KD | 71 78 | 21.9 | 21.1 | 33.4 | 18.4 | 21.2 | 31.9 | 22.4 | 27.6 | 41.3 | 23.6 | 23.4 | 40.4 | | JEC-QA-CA | 58 85 | 21.0 | 22.0 | 31.1 | 16.7 | 19.6 | 29.8 | 22.2 | 25.1 | 37.4 | 16.1 | 20.0 | 34.7 | | LSAT-AR | 56 91 | 21.7 | 24.4 | 35.2 | 23.9 | 22.6 | 34.4 | 22.6 | 25.7 | 33.9 | 22.6 | 25.2 | 31.7 | | LSAT-LR | 56 91 | 47.5 | 52.6 | 80.6 | 50.0 | 52.6 | 80.6 | 60.4 | 59.2 | 85.9 | 51.2 | 52.2 | 84.5 | | LSAT-RC | 56 91 | 64.7 | 65.4 | 85.9 | 57.6 | 62.1 | 85.1 | 70.6 | 67.7 | 87.7 | 64.3 | 57.6 | 87.7 | | SAT-Math | 66 94 | 35.5 | 42.7 | 64.6 | 54.6 | 70.9 | 95.0 | 44.6 | 40.9 | 71.8 | 55.5 | 65.0 | 89.6 | | SAT-English | 66 94 | 74.8 | 81.1 | 88.8 | 75.7 | 77.7 | 85.9 | 84.0 | 81.1 | 88.8 | 76.7 | 78.2 | 85.9 | | GK-Cn | 65 85 | 43.9 | 39.0 | 53.3 | 35.4 | 33.7 | 44.7 | 25.6 | 41.5 | 61.4 | 29.3 | 37.8 | 51.6 | | GK-En | 69 91 | 81.4 | 84.9 | 91.9 | 83.0 | 84.3 | 92.5 | 86.9 | 86.3 | 93.8 | 80.7 | 84.6 | 93.1 | | GK-geography | 65 85 | 53.3 | 59.8 | 76.9 | 48.7 | 55.8 | 72.4 | 59.8 | 63.8 | 75.9 | 52.3 | 61.8 | 76.4 | | GK-history | 64 85 | 47.3 | 59.7 | 77.4 | 37.0 | 50.2 | 76.5 | 49.0 | 57.6 | 77.8 | 51.9 | 58.4 | 78.2 | | GK-biology | 68 89 | 40.5 | 52.9 | 75.7 | 30.0 | 42.4 | 71.9 | 44.3 | 52.4 | 80.0 | 32.9 | 50.0 | 72.9 | | GK-chemistry | 66 86 | 27.1 | 38.7 | 51.7 | 24.6 | 33.8 | 52.2 | 32.4 | 44.0 | 54.6 | 35.8 | 33.8 | 54.1 | | GK-physics | 71 94 | 22.0 | 33.0 | 39.0 | 18.5 | 29.5 | 45.5 | 31.0 | 33.5 | 43.5 | 27.5 | 36.5 | 54.5 | | GK-Math-QA | 73 96 | 28.2 | 36.5 | 47.0 | 28.8 | 33.3 | 50.7 | 27.6 | 31.3 | 39.9 | 33.1 | 31.6 | 49.0 | | GK-Math-Cloze | 73 96 | 17.0 | 7.6 | 16.1 | 4.2 | 5.1 | 15.3 | 5.9 | 5.9 | 11.0 | 5.93 | 8.5 | 16.1 | | Average | 67 91 | 38.1 | 42.9 | 56.4 | 37.4 | 43.2 | 58.4 | 41.2 | 44.4 | 59.2 | 40.4 | 45 | 61.3 | | | | | | --- | --- | --- | | | Zero-shot Prompting | | | Input: [Question]
Among A to D, the answer is:
Output: | | | | | | --- | | - Prompting
Input: Here are the answers for the questions in exams.
􀀡1 : [Question 1] The answer is [Answer 1]
􀀡2 : [Question 2] The answer is [Answer 2]
…
􀀡􀀤 : [Question 􀀤] The answer is [Answer 􀀤][Qu

FAQs

这份文档主要评估什么？

围绕多类标准化考试及其子任务评估模型表现，包括数学、逻辑、法学、以及中英文科目相关题型。

文档中使用了哪些提示策略？

包含零样本提示（Zero-shot）、零样本思维链（Zero-shot CoT）、少样本提示（Few-shot）与少样本思维链（Few-shot CoT），并展示对应的提示输入模板。

结果通常以哪些指标呈现？

按任务汇总给出正确率/Top表现等统计，并在不同提示方式与模型设置下进行对比，同时也记录参与者规模与平均token等信息。

Sep 2, 2026

Uploaded by Genevieve