Significance of ExperimentalResults - Using a Permutation Test (P-Test)

Q: When is a difference between the control group and treatment group likely caused by the treatment?

A difference is likely caused by the treatment when the observed gap between group means is statistically significant under an appropriate null hypothesis for randomization.

Q: Why can permutation tests be used without assuming the data are normally distributed?

Permutation tests rely on resampling by randomly reassigning observations to groups, building the reference distribution from simulations rather than from a normal model.

Q: How do you decide significance using a permutation test p-value?

Use the p-value thresholds: p>0.10 means not significant (fail to reject the null), 0.05<p≤0.10 is marginally significant, 0.01<p≤0.05 is significant, and p≤0.01 is highly significant (reject the null).

Sep 14, 2026

Uploaded by William

Notes on experimental results focus on when differences between a control group and a treatment group are likely caused by the treatment, and how statistical testing evaluates that claim. The material reviews common core expectations for randomized experiments, compares approaches using normal assumptions (z-test) versus permutation tests without distribution requirements, and applies P-tests through null/alternative hypotheses, simulation randomization, and interpretation of p-values to decide significance.

LanguageEnglish

CategoryExam

FormatPDF

Pages4

Reading Time10 min

AccessFree to download

Significance of Experimental Results

Using a Permutation Test (P-Test)

Hypotheses and Simulation Randomization

Test Statistic, p-Value, and Significance

# Significance of ExperimentalResults Going Deeper Essential question:In an experinent,ahenis the difvence betueen the controlgroup and trearnenr group likely to be caursed by he treatnenr? Essential question:In an experiment,when is a difference between the control groupand treatment group likely to be caused by the treatment? COMMON Standards for CORE Mathematical Content CC.9-12.5.JC.5 Use data from a randomized experiment to compare twotreatments;usesimulationsto decide if differences between parazneters are signiflcant. ## Using a Permutation Test(P-Test) | | SAT Scores | | | | | | --- | --- | --- | --- | --- | --- | | Treatment Group | 1440 | 1610 | 1430 | 1700 | 1690 | | | 1570 | 1480 | 1620 | 1780 | 2010 | | Control Group | 1150 | 1500 | 1050 | 1600 | 1460 | | | 1860 | 1350 | 1750 | 1680 | 1330 | | | SAT Scores | | | | | | --- | --- | --- | --- | --- | --- | | Treatment Group | 1440 | 1610 | 1430 | 1700 | 1690 | | | 1570 | 1480 | 1620 | 1780 | 2010 | | Control Group | 1150 | 1500 | 1050 | 1600 | 1460 | | | 1860 | 1350 | 1750 | 1680 | 1330 | Do students that take an SAT prep courseactually improve their SAT scores? Using a Permutation Test(P-Test) | | SAT Scores | | | | | | --- | --- | --- | --- | --- | --- | | Treatment Group | 1440 | 1610 | 1430 | 1700 | 1690 | | | 1570 | 1480 | 1620 | 1780 | 2010 | | Control Group | 1150 | 1500 | 1050 | 1600 | 1460 | | | 1860 | 1350 | 1750 | 1680 | 1330 | b.) Calculate the mean score for each group 3=1633 ₅=1473 买--160 答5 When conducting a z-test,we are actually assuming thatthe data(or the random errors)follow a normaldistribution. Permutation tests(randomization tests)can be used without the normalassumption for the distribution of data. ## Using a Permutation Test(P-Test) a.)State the null and alternative hypothesis interms of the difference of the 2 group means. H₀:The difference of the 2 group means is 0. Ha:The difference between the 2 groupmeans(treatment -control)is positive. If the PREP course has NO effect at all,then if werearrange the data in the two group,we shouldexpect the same results. SIMULATION: We are going to randomize the data before putting theminto 2 groups.We then will find the difference in themeans between the 2 groups. | | SAT Scores | | | | | | --- | --- | --- | --- | --- | --- | | Treatment Group | 1440 | 1610 | 1430 | 1700 | 1690 | | | 1570 | 1480 | 1620 | 1780 | 2010 | | Control Group | 1150 | 1500 | 1050 | 1600 | 1460 | | | 1860 | 1350 | 1750 | 1680 | 1330 | | | SATSCORES | | | | | | --- | --- | --- | --- | --- | --- | | A | 1490 | | | | | | | | | | | | | B | | | | | | | | | | | | | ## Using a Permutation Test(P-Test) thatamstu | Interval | Frequency | | | --- | --- | --- | | Ax<-240 | 2 | | | -240≤x<-160 | 生 | | | -160≤x<-80 | | | | -80≤x<0 | 多4 | | | 0sx<80 | | | | 80sx<160 | 5 | | | 160sx<240 | | | | 240sxyn | | | | Interval | Frequency | | --- | --- | | -320≤x<-240 | 1 | | -240≤x<-160 | 7 | | -160≤x<-80 | 19 | | -80≤x<0 | 26 | | 0≤x<80 | 23 | | 80≤x<160 | 11 | | 160≤x<240 | | | 240≤x<320 | | Using a Permutation Test(P-Test) d.)Here is another way to represent the data.This is a DOTPLOT representing how manytimes our simulated value of 160 or morecould occur in90 simulations Random number generator on graphing calc to select10 data to be on groupA.The rest will be in group B. Make sure you select 10 unique random numbers.Hit enter until you do. A×A-B3Bx Using a Permutation Test(P-Test) e.)Look at the histogram or frequency table tosee how many times our test statistic of 160or more occurred.(this is a sample of what thesimulations would also look like) p va

FAQs

When is a difference between the control group and treatment group likely caused by the treatment?

A difference is likely caused by the treatment when the observed gap between group means is statistically significant under an appropriate null hypothesis for randomization.

Why can permutation tests be used without assuming the data are normally distributed?

Permutation tests rely on resampling by randomly reassigning observations to groups, building the reference distribution from simulations rather than from a normal model.

How do you decide significance using a permutation test p-value?

Use the p-value thresholds: p>0.10 means not significant (fail to reject the null), 0.05<p≤0.10 is marginally significant, 0.01<p≤0.05 is significant, and p≤0.01 is highly significant (reject the null).

Sep 14, 2026

Uploaded by William