Information In The Non-Stationary Case

@misc{citeulike:2940943, abstract = {Information estimates such as the ``direct method'' of Strong et al. (1998) sidestep the difficult problem of estimating the joint distribution of response and stimulus by instead estimating the difference between the marginal and conditional entropies of the response. While this is an effective estimation strategy, it tempts the practitioner to ignore the role of the stimulus and the meaning of mutual information. We show here that, as the number of trials increases indefinitely, the direct (or ``plug-in'') estimate of marginal entropy converges (with probability 1) to the entropy of the time-averaged conditional distribution of the response, and the direct estimate of the conditional entropy converges to the time-averaged entropy of the conditional distribution of the response. Under joint stationarity and ergodicity of the response and stimulus, the difference of these quantities converges to the mutual information. When the stimulus is deterministic or non-stationary the direct estimate of information no longer estimates mutual information, which is no longer meaningful, but it remains a measure of variability of the response distribution across time.}, author = {Vu, Vincent Q. and Yu, Bin and Kass, Robert E. }, citeulike-article-id = {2940943}, eprint = {0806.3978}, keywords = {information, mutual-information, statistics}, month = {Jun}, posted-at = {2008-06-29 05:58:45}, priority = {2}, title = {Information In The Non-Stationary Case}, url = {http://arxiv.org/abs/0806.3978}, year = {2008} }

TY - ELEC ID - citeulike:2940943 L3 - citeulike-article-id:2940943 TI - Information In The Non-Stationary Case N2 - Information estimates such as the ``direct method'' of Strong et al. (1998) sidestep the difficult problem of estimating the joint distribution of response and stimulus by instead estimating the difference between the marginal and conditional entropies of the response. While this is an effective estimation strategy, it tempts the practitioner to ignore the role of the stimulus and the meaning of mutual information. We show here that, as the number of trials increases indefinitely, the direct (or ``plug-in'') estimate of marginal entropy converges (with probability 1) to the entropy of the time-averaged conditional distribution of the response, and the direct estimate of the conditional entropy converges to the time-averaged entropy of the conditional distribution of the response. Under joint stationarity and ergodicity of the response and stimulus, the difference of these quantities converges to the mutual information. When the stimulus is deterministic or non-stationary the direct estimate of information no longer estimates mutual information, which is no longer meaningful, but it remains a measure of variability of the response distribution across time. KW - information KW - mutual-information KW - statistics AU - Vu, Vincent AU - Yu, Bin AU - Kass, Robert PY - 2008/Jun/24/ UR - http://arxiv.org/abs/0806.3978 ER -