PRAE Coffee Lecture - Critical Evaluation of AI-generated Information

Aug 25, 2026

Uploaded by Connor

A lecture on critically evaluating information generated by GenAI systems, using a live comparison between ChatGPT and Claude. It revisits fundamentals of how generative AI works, emphasizing prompt input, statistical/probabilistic mechanisms, and output generation. The session analyzes why identical questions can yield different animal lists, focusing on translation differences, source usage, and methodological variation. It concludes with an outlook on applying information-evaluation principles to AI-generated content.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

AccessFree to download

Agenda: Our Journey Today

Warm Up

GenAI Basics

How Does GenAI Work?

Evaluation of Information

Evaluation of GenAI-generated content

Outlook

Critical Evaluation of AI-generated Information The ROBOT Test and Other Tools Lukas Tschopp 24.09.2024 Critical Evaluation of AI-generated Information, Lukas Tschopp Seite 1 Agenda: Our Journey Today Warm Up 1. GenAI Basics 2. Evaluation of Information 3. Evaluation of GenAI-generated content 4. Outlook 24.09.2024 Critical Evaluation of AI-generated Information, Lukas Tschopp Seite 2 Warm Up Let’s have a look at one example ChatGPT: What are the ten most endangered animals in Africa? 24.09.2024 Critical Evaluation of AI-generated Information, Lukas Tschopp Seite 3 Warm Up Let’s have a look at one example Claude: What are the ten most endangered animals in Africa? 24.09.2024 Critical Evaluation of AI-generated Information, Lukas Tschopp Seite 4 Warm Up Let’s compare the answers | ChatGPT | | Claude | | --- | --- | --- | | Addax | | Addax | | African Forest Elephant | | African Penguin | | African Wild Dog | | African Wild Dog | | Black Rhinoceros | | Black Rhino | | Ethiopian Wolf | | Cheetah | | Grey Crowned Crane | | Ethiopian Wolf | | Hawksbill Turtle | | Lemur | | Mountain Gorilla | | Mountain Gorilla | | Northern White Rhinoceros | | Pangolin | | Pangolin | | Rothschild's Giraffe | | Match = 6 | | --- | | Mismatch = 4 | 24.09.2024 Critical Evaluation of AI-generated Information, Lukas Tschopp Seite 5 Warm Up Let’s share thoughts | Match = 6 | | --- | | Mismatch = 4 | Why is there a mismatch between ChatGPT’s answers and Claude’s? Please share your thoughts in the Zoom chat. 24.09.2024 Critical Evaluation of AI-generated Information, Lukas Tschopp Seite 6 Warm Up Possible Answers When we use the same input in ChatGPT and Claude, the results are not the same. Why is that? Some possible answers: • Our input has been translated differently • Our input was not precise enough • ChatGPT and Claude use different sources to answer our question • ChatGPT and Claude use the same source but a different approach • ChatGPT and Claude are not designed for such tasks • 24.09.2024 Critical Evaluation of AI-generated Information, Lukas Tschopp Seite 7 1. GenAI Basics How Does GenAI Work? | What are the ten most
endangered animals in Africa? | | --- | | Prompt / Input | | GenAI such as ChatGPTor Claude | | --- | | Statistics / Probability | | | | List of ten animals. | | --- | | Result / Output / text, image, video | 24.09.2024 Critical Evaluation of AI-generated Information, Lukas Tschopp Seite 8 1. GenAI Basics Let’s share thoughts Let’s have a second look at our previous example: What are the implications of knowing that there is a set of data that will be used along with statistics/probabilities to answer my question? Please share your thoughts in the Zoom chat. 24.09.2024 Critical Evaluation of AI-generated Information, Lukas Tschopp Seite 9 1. GenAI Basics Possible Answers What are the implications of knowing that there is a data set that will be used along with statistics/probabilities to answer my question? • I don't know what data set ChatGPT and Claude are using • I don't know if the data contains reliable information • I don't know what information ChatGPT and Claude base their answers on • I don't know if there are other AI tools better suited to my topic • … 24.09.2024 Critical Evaluation of AI-generated Information, Lukas Tschopp Seite 10

FAQs

Why can ChatGPT and Claude produce different answers to the same question?

Even with the same input, outputs may differ due to translation variations, imprecise prompts, different underlying sources, or different approaches to the same data.

What GenAI components are highlighted when explaining how GenAI works?

The lecture contrasts prompt/input, the statistical/probability foundations used by GenAI, and the resulting output text, images, or videos.

How should someone think about the reliability of GenAI-generated information?

The session stresses uncertainty about which data sets and sources the tools use and whether the information in those data sets is reliable, encouraging careful evaluation before accepting results.

Aug 25, 2026

Uploaded by Connor