How Can AI Predict Missing Real‑Time Data?

When data arrive late, decisions may rely on incomplete information. Nowcasting uses what is already available to estimate the full picture. A recent study embeds nowcasting in an Expectation‑Maximisation (EM) loop, alternating an estimation (E) step with an optimisation (M) step.

The breakthrough replaces traditional linear models with machine‑learning algorithms—neural networks and XGBoost—to model both event occurrence and reporting delays. These methods can handle hundreds of variables (age, region, weather, etc.) without oversimplification.

In the E step, the model predicts how many unreported events have already happened, based on learned patterns. In the M step, the neural network or XGBoost parameters are updated to improve predictions. The cycle repeats until estimates stabilise.

Simulated tests show that, when relationships are non‑linear, this approach outperforms classic EM methods. The real‑world case study involves Argentinian COVID‑19 reports: XGBoost again delivered the most accurate nowcasts, cutting error by about 15 % compared to linear models.

The work proves that merging EM with machine learning can make forecasts faster and more precise, valuable for health, finance, or emergency management.

The traditional EM algorithm maximises the complete‑data likelihood ε(θ|X,Y) by computing the expected log‑likelihood with respect to the posterior distribution of the missing data. Introducing non‑linear models makes the likelihood non‑convex, complicating global maximisation with Newton‑Raphson‑type methods. Neural networks or gradient boosting approximate the likelihood via differentiable loss functions (e.g., binary cross‑entropy). In the M step, optimisation proceeds via back‑propagation (networks) or gradient boosting (XGBoost), using gradient descent with L2 regularisation to prevent overfitting. Boosted trees capture higher‑order interactions without explicit feature engineering. In the E step, the expectation E[Z|X,θ^{(t)}] is computed, where Z denotes unreported events; this is obtained from the predictive distribution of the M model, typically a Bernoulli with probability p_i(θ^{(t)}). The EM cycle repeats until convergence, monitoring changes in log‑likelihood or loss. High‑dimensional settings demand early stopping and cross‑validation for stability and generalisation. Epidemiological applications also require handling temporal autocorrelation, often modelled with ARIMA components or recurrent neural networks (RNN/LSTM) integrated in the M step, enhancing predictive power on time series with reporting delays.