Findings ACL 1602 - Syllogism and Conditional Reasoning with Diagram Prompts

Sep 3, 2026

Upload by Xiajie

Research results analyze multimodal reasoning performance across syllogism reasoning and conditional reasoning tasks using different prompt and diagram strategies. Experiments compare multiple vision-language models and vary inputs such as text-only, Venn diagrams, Euler diagrams, and linear diagrams. Additional ablations evaluate the impact of including instruction on model accuracy, reporting separate valid, invalid, and total outcomes. Findings highlight substantial gains or degradations depending on model choice and diagram type, especially under instruction-based settings.

LanguageEnglish

CategoryGeneral

FormatPDF

Pages18

Reading Time6 min

AccessFree to download

Experimental Setup

Models and Prompt Variants

Diagram Types (Venn/Euler/Linear)

Valid/Invalid/Total Results

Ablation Study

Instruction vs No Instruction

| | Syllogism Reasoning | | | Conditional Reasoning | | | | --- | --- | --- | --- | --- | --- | --- | | | Valid | Invalid | Total | Valid | Invalid | Total | | Llava-1.5-7b | 72.40 | 25.65 | 49.03 | 55.00 | 80.00 | 73.75 | | +Venn Diagram | 49.35 | 58.44 | 53.90 | 47.50 | 85.83 | 76.25 | | +Euler Diagram | 68.18 | 40.58 | 54.38 | 52.50 | 82.50 | 75.00 | | +Linear Diagram | 75.97 | 29.87 | 52.92 | 47.50 | 79.17 | 71.25 | | Qwen2-VL-7B-Instruct | 91.23 | 31.49 | 61.36 | 47.50 | 86.67 | 76.88 | | +Venn Diagram | 93.83 | 28.25 | 61.04 | 80.00 | 70.00 | 72.50 | | +Euler Diagram | 93.18 | 41.88 | 67.53 | 90.00 | 74.17 | 78.12 | | +Linear Diagram | 95.78 | 32.79 | 64.29 | 77.50 | 72.50 | 73.75 | | LLama3.2-vision-11B | 59.74 | 54.87 | 57.31 | 77.50 | 70.83 | 72.50 | | +Venn Diagram | 80.52 | 25.32 | 52.92 | 82.50 | 72.50 | 75.00 | | +Euler Diagram | 75.00 | 27.27 | 51.14 | 67.50 | 70.00 | 69.38 | | +Linear Diagram | 77.92 | 39.31 | 58.12 | 77.50 | 74.17 | 75.00 | | | Valid | Invalid | Total | | --- | --- | --- | --- | | Only text 91.23 31.49 61.36 | | | | | Venn Diagram | | | | | Only Diagram | 94.81 | 22.40 | 58.60 | | Text + Diagram | 93.83 | 28.25 | 61.04 | | Euler Diagram | | | | | Only Diagram | 90.58 | 33.12 | 61.85 | | Text + Diagram | 93.18 | 41.88 | 67.53 | | Linear Diagram | | | | | Only Diagram | 93.18 | 33.12 | 63.15 | | Text + Diagram | 95.78 | 32.79 | 64.29 | | | Valid | Invalid | Total | | --- | --- | --- | --- | | Llava-1.5-7b | | | | | w/o instruction | 78.57 | 32.14 | 55.36 | | w/ instruction | 68.18 | 40.58 | 54.38 | | Qwen2-VL-7B-Instruct | | | | | w/o instruction | 97.40 | 15.91 | 56.66 | | w/ instruction | 93.18 | 41.88 | 67.53 | | LLama3.2-vision-11B | | | | | w/o instruction | 66.56 | 36.04 | 51.30 | | w/ instruction | 75.00 | 27.27 | 51.14 | | | Venn | Euler | Linear | | --- | --- | --- | --- | | Llava-1.5-7b | 7/14 | 4/4 | 10/20 | | Qwen2-VL-7B-Instruct | 26/48 | 32/48 | 36/50 | | LLama3.2-vision-11B | 23/35 | 8/16 | 26/44 |

FAQs

Which tasks are evaluated in the findings?

The findings evaluate syllogism reasoning and conditional reasoning, with results reported as valid, invalid, and total outcomes.

How do Venn, Euler, and linear diagrams affect performance?

Performance changes across models depending on the diagram type, with text-only, Venn diagrams, Euler diagrams, and linear diagrams producing different valid/invalid/total patterns.

What is the role of instruction in the ablation results?

The ablation compares settings with instruction versus without instruction, showing that instruction can improve or reduce accuracy depending on the model and task configuration.

Sep 3, 2026

Upload by Xiajie