It looks like you're new here. If you want to get involved, click one of these buttons!

- All Categories 2.4K
- Chat 502
- Study Groups 21
- Petri Nets 9
- Epidemiology 4
- Leaf Modeling 2
- Review Sections 9
- MIT 2020: Programming with Categories 51
- MIT 2020: Lectures 20
- MIT 2020: Exercises 25
- Baez ACT 2019: Online Course 339
- Baez ACT 2019: Lectures 79
- Baez ACT 2019: Exercises 149
- Baez ACT 2019: Chat 50
- UCR ACT Seminar 4
- General 73
- Azimuth Code Project 110
- Statistical methods 4
- Drafts 10
- Math Syntax Demos 15
- Wiki - Latest Changes 3
- Strategy 113
- Azimuth Project 1.1K
- - Spam 1
- News and Information 148
- Azimuth Blog 149
- - Conventions and Policies 21
- - Questions 43
- Azimuth Wiki 718

Options

The blog for the workshop I'm running on "Information and entropy for biological systems" is getting some nice posts, including slides for talks:

- Welcome!
- “Pop” Presentation on a Project on Schrödinger’s Paradox
- Information Theory, Replicator Dynamics, and Distributed Reinforcement Learning
- Optimal Estimation of Entropy
- Link Between Population Genetics and the Boltzmann Distribution
- Introductory Talk (by John Baez)
- Maximum Entropy as a Foundation for Theory Building in Ecology (by John Harte)

I'll post about all this on the Azimuth blog soon, right after David Spivak's series. But if you want you can go look now.

## Comments

It looks like an interesting workshop. I looked at your introduction and Harte's slides. There is one slide I do not like in your talk, the one on joint entropy and mutual information. I don't think you need to bring in X and Y. There is a joint distribution and two marginal distributions. The rest follows from H() which you've already defined. I put a diagram at the bottom of http://www.azimuthproject.org/azimuth/show/Sandbox which you might like to use.

The Wikipedia page on mutual information calls X and Y random variables, then treats them as sets too, which is confusing.

As is traditional with entropy, there is a sign error. You have I(X;Y) = H(X,Y)-H(X)-H(Y). Wikipedia, under Relation to other quantities, has H(X)+H(Y)-H(X,Y).

`It looks like an interesting workshop. I looked at your introduction and Harte's slides. There is one slide I do not like in your talk, the one on joint entropy and mutual information. I don't think you need to bring in X and Y. There is a joint distribution and two marginal distributions. The rest follows from H() which you've already defined. I put a diagram at the bottom of http://www.azimuthproject.org/azimuth/show/Sandbox which you might like to use. The Wikipedia page on mutual information calls X and Y random variables, then treats them as sets too, which is confusing. As is traditional with entropy, there is a sign error. You have I(X;Y) = H(X,Y)-H(X)-H(Y). Wikipedia, under Relation to other quantities, has H(X)+H(Y)-H(X,Y).`

Yes, I know. I mainly brought in these random variables because 1) everybody talks that way, so I should explain the way people talk, 2) Wikipedia provided this picture for free.

I wanted to avoid the word "marginal" since that's yet another piece of jargon to explain, and I never need it later. That's why I said:

... called marginals, but I didn't want to say that...

I'll see if your picture helps me out.

Thanks a million! They're right: H(X)+H(Y)-H(X,Y) is nonnegative and it measures the "amount of information the random variables X and Y have in common". For example, if X = Y it's the same as H(X).

`> I don't think you need to bring in X and Y. There is a joint distribution and two marginal distributions. The rest follows from H() which you've already defined. Yes, I know. I mainly brought in these random variables because 1) everybody talks that way, so I should explain the way people talk, 2) Wikipedia provided this picture for free. I wanted to avoid the word "marginal" since that's yet another piece of jargon to explain, and I never need it later. That's why I said: > A 'pair of random variables' $X$ and $Y$ is a probability distribution on a set $S \times T$. This gives probability distributions on $S$ and on $T$. ... called marginals, but I didn't want to say that... > We may thus define three entropies: the **joint entropy** $H(X,Y)$ and the **individual entropies** $H(X)$ and $H(Y)$. I'll see if your picture helps me out. > As is traditional with entropy, there is a sign error. You have I(X;Y) = H(X,Y)-H(X)-H(Y). Wikipedia, under Relation to other quantities, has H(X)+H(Y)-H(X,Y). Thanks a million! They're right: H(X)+H(Y)-H(X,Y) is nonnegative and it measures the "amount of information the random variables X and Y have in common". For example, if X = Y it's the same as H(X).`

Graham: your picture is good for explaining how a probability distribution on $S \times T$ projects down to give probability distributions on $S$ and on $T$, but I think the slide I've got, together with the words I'll say, will do a decent job of rapidly explaining the buzzword "mutual information" - which is what I really needed to explain.

`Graham: your picture is good for explaining how a probability distribution on $S \times T$ projects down to give probability distributions on $S$ and on $T$, but I think the slide I've got, together with the words I'll say, will do a decent job of rapidly explaining the buzzword "mutual information" - which is what I really needed to explain.`

Hi Graham, I thought your picture would help me understand mutual information better but the perspective meant I couldn't see or work out what the represented numbers might be.

`Hi Graham, I thought your picture would help me understand mutual information better but the perspective meant I couldn't see or work out what the represented numbers might be.`

Jim,

My diagram illustrates a joint probability distribution and marginal distributions. See http://en.wikipedia.org/wiki/Joint_probability_distribution which has a picture for the continuous case.

Once you have got the hang of joint and marginal distributions, mutual information is a small extra step.

`Jim, My diagram illustrates a joint probability distribution and marginal distributions. See [http://en.wikipedia.org/wiki/Joint_probability_distribution](http://en.wikipedia.org/wiki/Joint_probability_distribution) which has a picture for the continuous case. Once you have got the hang of joint and marginal distributions, mutual information is a small extra step.`

It was a great workshop, and I'll blog about it as soon as I recover! All the talk slides are available now, from here:

Azimuth Blog, 6 April 2015.It took a while to extract them from some of the speakers.

Videos of talks will be made available by the end of next week, supposedly.

Now I'm going to Penn State to speak in John Roe's class "Mathematics for Sustainability".

`It was a great workshop, and I'll blog about it as soon as I recover! All the talk slides are available now, from here: * [Information and entropy in biological systems (part 3)](https://johncarlosbaez.wordpress.com/2015/04/06/information-and-entropy-in-biological-systems-part-3/), _Azimuth Blog_, 6 April 2015. It took a while to extract them from some of the speakers. Videos of talks will be made available by the end of next week, supposedly. Now I'm going to Penn State to speak in John Roe's class "Mathematics for Sustainability".`