Information Theory
Home Neural Networks Quantum Computation Presentation Reinforcement Learning Information Theory

 

Presentations
PDF Notes
2005 Coursework Assignments
Coursework Assignments
2004 Notes

Course description

The course introduces the elements of information theory and illustrate their relevance in AI, especially machine learning and pattern recognition. We will also review the mathematical concepts which will be used in the subsequent part of the course. This includes in particular the elements of probability theory, linear algebra, and optimization.

Prerequisites

A good background in university-level mathematics (calculus, basic probability, linear algebra).

Grading

The course has the following assessment components: 1) Written Examination (2.5 hours, 80%) , 2) Coursework Section (4 pieces, 20%). To pass this course, students must obtain at least 40% on the coursework component and an average of at least 50% when the coursework and exam components of a course are weighted together.

Recommended Books and Readings

Information Theory, Inference & Learning Algorithms
by David J. C. MacKay

Feynman Lectures on Computation
by Richard P. Feynman (Editor), Robin W. Allen (Editor), Tony Hey (Editor)

The Elements of Statistical Learning
by T. Hastie, R. Tibshirani, J. H. Friedman

  • Hyvärinen, A. and Oja, E. Independent Component Analysis: A Tutorial
  • T.M. Cover and J.A. Thomas. Elements of Information Theory, Wiley, 1991.
  • Hamming, R. W. Coding and Information Theory. Prentice-Hall, 2nd edition, 1986.
  • A.I. Khinchin. Mathematical foundation of information theory. Dover Pub. Inc., 1957.
  • McEliece, R. J. The Theory of Information and Coding: A Mathematical Framework for Communication. Addison-Wesley, 1977.

    Additional Reading Material (not necessarily on the course's main topics)

    Investigations
    by Stuart A. Kauffman

    (various works by Jim Crutchfield)

    Other papers:

    Adaptive Clustering: Better Representatives with Reinforcement Learning

    Reinforcement Learning: An Introduction
    by Richard S. Sutton, Andrew G. Barto

    Some students may be interested in my publications, a list I keep of vaguely AI-related books I like, or a list I keep on books related to natural inspired computation.

     

  •  

    Home ] Presentations ] PDF Notes ] 2005 Coursework Assignments ] Coursework Assignments ] 2004 Notes ]

    Send mail to robert.elliott.smith@gmail.com with questions or comments about this web site.
    Last modified: November 16, 2006