Directed Information, Causal Estimation, and Communication in Continuous Time

A notion of directed information between two continuous-time processes is proposed. A key component in the definition is taking an infimum over all possible partitions of the time interval, which plays a role no less significant than the supremum over “space” partitions inherent in the definition of mutual information. Properties and operational interpretations in estimation and communication are then established for the proposed notion of directed information. For the continuous-time additive white Gaussian noise channel, it is shown that Duncan's classical relationship between causal estimation error and mutual information continues to hold in the presence of feedback upon replacing mutual information by directed information. A parallel result is established for the Poisson channel. The utility of this relationship is demonstrated in computing the directed information rate between the input and output processes of a continuous-time Poisson channel with feedback, where the channel input process is constrained to be constant between events at the channel output. Finally, the capacity of a wide class of continuous-time channels with feedback is established via directed information, characterizing the fundamental limit on reliable communication.

• Publication year

  2011

• Venue

  IEEE Transactions on Information Theory

• Publication date

  2011-09-02

• Fields of study

  Mathematics, Computer Science, Engineering

• Identifiers
  DOI 10.1109/TIT.2012.2227677arXiv 1109.0351
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• I and J

  2012cited by this paper
• Capacity Theorems for Discrete, Finite-State Broadcast Channels With Feedback and Unidirectional Receiver Cooperation

  2010cited by this paper
• Mutual information, relative entropy, and estimation in the Poisson channel

  2010influential reference
• The Relationship Between Causal and Noncausal Mismatched Estimation in Continuous-Time AWGN Channels

  2010cited by this paper
• Lecture Notes on Network Information Theory

  2010cited by this paper
• The Poisson channel with side information

  2009cited by this paper
• Interpretations of Directed Information in Portfolio Theory, Data Compression, and Hypothesis Testing

  2009cited by this paper
• Brownian Motion and Stochastic Calculus

  2008influential reference
• On directed information and gambling

  2008cited by this paper
• A Coding Theorem for a Class of Stationary Channels with Feedback

  2007cited by this paper
• Feedback Capacity of the Compound Channel

  2007cited by this paper
• On the Role of Feedforward in Gaussian Sources: Point-to-Point Source Coding and Multiple Description Source Coding

  2007cited by this paper
• Scanning and Sequential Decision Making for Multidimensional Data—Part II: The Noisy Case

  2007cited by this paper
• Capacity Region of the Finite-State Multiple-Access Channel With and Without Feedback

  2007cited by this paper
• Source Coding With Feed-Forward: Rate-Distortion Theorems and Error Exponents for a General Source

  2007cited by this paper
• Scanning and Sequential Decision Making for Multidimensional Data

  2007cited by this paper
• Scanning and Sequential Decision Making for Multidimensional Data - Part II: The Noisy Case

  2007cited by this paper
• The Capacity of Channels With Feedback

  2006cited by this paper
• Feedback Capacity of Stationary Gaussian Channels

  2006cited by this paper
• Capacity of the Trapdoor Channel With Feedback

  2006cited by this paper
• Finite State Channels With Time-Invariant Deterministic Feedback

  2006cited by this paper
• The capacity of finite-State Markov Channels With feedback

  2005cited by this paper
• Conservation of mutual and directed information

  2005cited by this paper
• Mutual Information and Conditional Mean Estimation in Poisson Channels

  2004cited by this paper
• Mutual information and minimum mean-square error in Gaussian channels

  2004cited by this paper
• Combinatorial optimization. Polyhedra and efficiency.

  2003cited by this paper
• Network Information Theory

  2001cited by this paper
• Statistics of Random Processes Ii: Applications

  2000cited by this paper
• Capacity results for the discrete memoryless network

  1999cited by this paper
• Coding for computing

  1995cited by this paper
• Coding theorems for a continuous-time Gaussian channel with feedback

  1994cited by this paper
• On the reliability function of the ideal Poisson channel with noiseless feedback

  1993cited by this paper
• Information theory - for continuous systems

  1993cited by this paper
• CAUSALITY, FEEDBACK AND DIRECTED INFORMATION

  1990influential reference
• On channel capacity per unit cost

  1990cited by this paper
• Capacity and error-exponent for the direct detection photon channel-Part II

  1988cited by this paper
• Capacity and error exponent for the direct detection photon channel-Part I

  1988cited by this paper
• Point processes and queues, martingale dynamics

  1983cited by this paper
• Capacity and cutoff rate for Poisson-type channels

  1980cited by this paper
• A Definition of Conditional Mutual Information for Arbitrary Ensembles

  1978cited by this paper
• The Capacity of a Channel of the Poisson Type

  1978cited by this paper
• On optical data communication via direct detection of light pulses

  1976cited by this paper
• Mutual information of the white Gaussian channel with and without feedback

  1971influential reference
• Capacity of a continuous memoryless channel with feedback

  1971influential reference
• ON THE CALCULATION OF MUTUAL INFORMATION

  1970cited by this paper
• Information Theory and Reliable Communication

  1969cited by this paper
• Rate Distortion Theory for Sources with Abstract Alphabets and Memory

  1968influential reference
• Information and information stability of random variables and processes

  1964cited by this paper
• Coding Theorems of Information Theory

  1962cited by this paper
• On the Shannon theory of information transmission in the case of continuous signals

  1956cited by this paper

• Massively Multimodal Foundation Models: A Framework for Capturing Interactions with Specialized Mixture-of-Experts

  2025cites this paper
• CTBN-PH: A continuous-time Bayesian network for individualised diagnostic risk prediction

  2025cites this paper
• Rate-cost tradeoffs in continuous-time control with a biomolecular application

  2025influential citation
• Capacity Bounds for Poset-Causal Channels Via Group Symmetry

  2025cites this paper
• Causality Testing, Directed Information and Spike Trains

  2024cites this paper
• Continuous-Time Distributed Filtering With Sensing and Communication Constraints

  2023cites this paper
• An Information-Theoretic Analysis of Discrete-Time Control and Filtering Limitations by the I-MMSE Relationships

  2023cites this paper
• Temporally Causal Discovery Tests for Discrete Time Series and Neural Spike Trains

  2023cites this paper
• On the complexity of linear systems: an approach via rate distortion theory and emulating systems

  2023cites this paper
• Continuous-Time Channel Gain Control for Minimum-Information Kalman-Bucy Filtering

  2022cites this paper
• An Information-Theoretic Analysis of Continuous-Time Control and Filtering Limitations by the I-MMSE Relationships

  2022cites this paper
• Generic Variance Bounds on Estimation and Prediction Errors in Time Series Analysis: An Entropy Perspective

  2019cites this paper
• Anomaly Detection and Exploratory Causal Analysis for SAP HANA

  2019cites this paper
• Information Theoretic Causal Effect Quantification

  2019cites this paper
• Information thermodynamics for interacting stochastic systems without bipartite structure

  2019cites this paper
• A Survey on MIMO Transmission With Finite Input Signals: Technical Challenges, Advances, and Future Trends

  2018cites this paper
• Fundamental limits and achievable performance in biomolecular control

  2018cites this paper
• Remote State Estimation over Packet Dropping Links in the Presence of an Eavesdropper

  2017cites this paper
• Robust Sparse Approximations for Stochastic Dynamical Systems

  2017cites this paper
• On Continuous-Time Gaussian Channels

  2017cites this paper
• Dynamic reconstruction of influence graphs with adaptive directed information

  2017cites this paper
• A View of Information-Estimation Relations in Gaussian Networks

  2017cites this paper
• Optimal sensor design and zero-delay source coding for continuous-time vector Gauss-Markov processes

  2017cites this paper
• Entropy Production and Information Flow for Markov Diffusions with Filtering

  2017cites this paper
• Testing for Causality in Continuous Time Bayesian Network Models of High-Frequency Data

  2016cites this paper
• Learning Network of Multivariate Hawkes Processes: A Time Series Approach

  2016cites this paper
• Transfer Entropy and Directed Information in Gaussian diffusion processes

  2016cites this paper
• Essays in high-dimensional nonlinear time series analysis

  2016cites this paper
• Dynamic Directed Influence Networks: A Study of Campaigns on Twitter

  2016cites this paper
• Information theory in systems biology. Part I: Gene regulatory and metabolic networks.

  2016cites this paper
• Inferring causal connectivity in epileptogenic zone using directed information

  2015cites this paper
• Coding Theorem and Converse for Abstract Channels With Time Structure and Memory

  2015cites this paper
• Identifying Seizure Onset Zone From the Causal Connectivity Inferred Using Directed Information

  2015cites this paper
• Estimating the Directed Information and Testing for Causality

  2015cites this paper
• Ac ce pt ed M an us cr ip t Information Theory in Systems Biology : Gene Regulatory and Metabolic Networks

  2015cites this paper
• LQG Control with Minimal Information: Three-Stage Separation Principle and SDP-based Solution Synthesis

  2015cites this paper
• LQG Control With Minimum Directed Information: Semidefinite Programming Approach

  2015cites this paper
• FBLG: a simple and effective approach for temporal dependence discovery from time series data

  2014cites this paper
• Relations Between Information and Estimation in the Presence of Feedback

  2014cites this paper
• Extensions of the I-MMSE Relationship to Gaussian Channels With Feedback and Memory

  2014cites this paper
• To Feed or Not to Feedback

  2014cites this paper
• Extensions of the I-MMSE relation

  2014influential citation
• Semidefinite Programming Approach to Gaussian Sequential Rate-Distortion Trade-Offs

  2014cites this paper
• Directed Information Between Connected Leaky Integrate-and-Fire Neurons

  2014cites this paper
• Minimax Filtering via Relations between Information and Estimation

  2013influential citation
• An Examination of Practical Granger Causality Inference

  2013cites this paper
• Minimax Filtering Regret via Relations Between Information and Estimation

  2013cites this paper
• Inferring causality in networks of WSS processes by pairwise estimation methods

  2013cites this paper
• Efficient Methods to Compute Optimal Tree Approximations of Directed Information Graphs

  2013cites this paper
• Optimal Coding and Control for Linear Gaussian Systems Over Gaussian Channels Under Quadratic Cost

  2013cites this paper
• The Interplay Between Information and Estimation Measures

  2013cites this paper
• Minimax filtering regret via relations between information and estimation

  2013cites this paper
• On the mutual information of time reversal for non-stationary channels

  2013cites this paper
• Adaptive waveform scheduling in radar: an information theoretic approach

  2012cites this paper
• Waveform scheduling via directed information in cognitive radar

  2012cites this paper
• Pointwise Relations Between Information and Estimation in Gaussian Noise

  2012cites this paper
• Pointwise relations between information and estimation in Gaussian noise

  2011cites this paper
• To feed or not to feed back

  2010cites this paper
• Interpretations of Directed Information in Portfolio Theory, Data Compression, and Hypothesis Testing

  2009cites this paper
• M ASSIVELY M ULTIMODAL F OUNDATION M ODELS : A F RAMEWORK FOR C APTURING D EPENDENCIES WITH S PECIALIZED M IXTURE - OF -E XPERTS

  year unknowncites this paper