Information Theory, Inference, and Learning Algorithms - Version 6.0 - June 26, 2003

Sep 21, 2026

Uploaded by Himbo

Information Theory, Inference, and Learning Algorithms presents a unified view of information theory and machine learning, linking Shannon’s foundations with Bayesian data modelling, Monte Carlo methods, variational approaches, clustering, and neural networks. It is designed for senior undergraduates and graduate students in engineering, science, mathematics, and computing, assuming knowledge of calculus, probability theory, and linear algebra. The book explains how compression and communication concepts connect to inference and learning algorithms through tools shared across modern data and coding tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages640

Reading Time26 h 53 min

AccessFree to download

Preface

Introduction to Information Theory

Probability, Entropy, and Inference

More about Inference

Data Compression

Noisy-Channel Coding

Further Topics in Information Theory

Probabilities and Inference

Neural networks

Sparse Graph Codes

Appendices

Information Theory, Inference, and Learning Algorithms David J. C. MacKay Information Theory, Inference, and Learning Algorithms David J. C. MacKay [mackay@mrao.cam.ac.uk](mackay@mrao.cam.ac.uk) 􀀍c1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003 Version 6.0 (as published) June 26, 2003 Please send feedback on this book via [http://www.inference.phy.cam.ac.uk/mackay/itila/](http://www.inference.phy.cam.ac.uk/mackay/itila/) This book will be [published by C.U.P. in](published by C.U.P. in) September 2003. It will remain viewable on-screen on the above website, in postscript, djvu, and pdf formats. (C. U.P. replace this page with their own page ii.) Contents Preface ................................ v 1 Introduction to Information Theory ................ 3 2 Probability, Entropy, and Inference ................. 22 3 More about Inference ........................ 48 I Data Compression ........................ 65 4 The Source Coding Theorem .................... 67 5 Symbol Codes ............................ 91 6 Stream Codes ............................. 110 7 Codes for Integers .......................... 132 II Noisy-Channel Coding ...................... 137 8 Correlated Random Variables .................... 138 9 Communication over a Noisy Channel ............... 146 10 The Noisy-Channel Coding Theorem ................ 162 11 Error-Correcting Codes and Real Channels ............ 177 III Further Topics in Information Theory ............. 191 12 Hash Codes: Codes for E􀀎cient Information Retrieval ..... 193 13 Binary Codes ............................ 206 14 Very Good Linear Codes Exist ................... 229 15 Further Exercises on Information Theory ............. 233 16 Message Passing ........................... 241 17 Communication over Constrained Noiseless Channels ...... 248 18 Crosswords and Codebreaking ................... 260 19 Why have Sex? Information Acquisition and Evolution ..... 269 IV Probabilities and Inference ................... 281 20 An Example Inference Task: Clustering .............. 284 21 Exact Inference by Complete Enumeration ............ 293 22 Maximum Likelihood and Clustering ................ 300 23 Useful Probability Distributions .................. 311 24 Exact Marginalization ........................ 319 25 Exact Marginalization in Trellises ................. 324 26 Exact Marginalization in Graphs .................. 334 27 Laplace's Method ........................... 341 28 Model Comparison and Occam's Razor .............. 343 29 Monte Carlo Methods ........................ 357 30 E􀀎cient Monte Carlo Methods ................... 387 31 Ising Models ............................. 400 32 Exact Monte Carlo Sampling .................... 413 33 Variational Methods ......................... 422 34 Independent Component Analysis and Latent Variable Modelling ................................. 437 35 Random Inference Topics ...................... 445 36 Decision Theory ........................... 451 37 Bayesian Inference and Sampling Theory ............. 457 V Neural networks ......................... 467 38 Introduction to Neural Networks .................. 468 39 The Single Neuron as a Classi􀀌er .................. 471 40 Capacity of a Single Neuron ..................... 483 41 Learning as Inference ........................ 492 42 Hop􀀌eld Networks .......................... 505 43 Boltzmann Machines ......................... 522 44 Supervised Learning in Multilayer Networks ............ 527 45 Gaussian Processes ......................... 535 46 Deconvolution ............................ 549 VI Sparse Graph Codes ...................... 555 47 Low-Density Parity-Check Codes ................. 557 48 Convolutional Codes and Turbo Codes ............... 574 49 Repeat{Accumulate Codes ..................... 582 50 Digital Fountain Codes ....................... 589 VII Appendic

FAQs

Who is this book intended for?

The book targets senior undergraduates and graduate students in Engineering, Science, Mathematics, and Computing, with prerequisites in calculus, probability theory, and linear algebra.

What central idea motivates the book’s structure?

It unifies information theory and machine learning as two sides of the same coin, emphasizing shared tools across data compression, error-correcting codes, and learning algorithms.

How should readers use the book when deciding what to study first?

Chapter dependencies are indicated in the figure on the next page, and advanced or optional topics within Parts I, II, IV, and V tend to appear toward the end; Part III chapters are optional on first reading.

Sep 21, 2026

Uploaded by Himbo