A calmer, karma, CARMA algorithmic chameleon to improve noisy or incomplete data

August 25, 2025 by David Bradley, Inderscience

Collected at: https://techxplore.com/news/2025-08-calmer-karma-carma-algorithmic-chameleon.html

A novel algorithmic system that works subtly in the background to mutual benefit, and adapts quickly to local conditions, could be useful in data processing where noise terms can be replaced with useful estimates of their values.

In the unpredictable world of data-driven modeling, some algorithms charge through problems like rhinos, others blend in and adapt like chameleons. A new approach to a long-standing challenge in system identification, how to work with missing and noisy data, falls firmly into the latter camp and is discussed in the International Journal of Modelling, Identification and Control.

The method is designed for Controlled Autoregressive Moving Average (CARMA) models, mathematical structures widely used to capture and forecast the behavior of dynamic systems in fields as diverse as control engineering, economics, and climate science.

These models work best when both input and output data are complete and reliable. In reality, such an ideal is rarely achieved, if ever. Network interruptions, faulty sensors, and environmental disruptions frequently leave gaps in the record, while background noise, often with patterns of its own, can distort what remains.

Conventional algorithms might falter under such conditions, producing biased results or unstable models that bear little resemblance to the real system. The new research takes a niftier approach, quietly adjusting to the data landscape and turning potential setbacks into advantages.

Its ingenuity lies in the combination of three distinct techniques. An auxiliary model estimates the unmeasured components of the system, extracting useful signals from what would otherwise be statistical clutter, an almost karmic reversal of bad data into good. An interpolation method then fills in the missing inputs by inferring plausible values from surrounding measurements. Finally, the process is quickened using Nesterov Accelerated Gradient optimization, a mathematically elegant way of anticipating the best next step rather than taking each one blindly.

Together, these steps form the Interpolation-based Nesterov Accelerated Gradient (INAG) algorithm, a system that not only produces more accurate parameter estimates but does so faster than comparable methods, even in the presence of “colored noise,” random fluctuations with structure and memory.

For engineers, better system identification means better control, whether in regulating industrial processes, stabilizing power grids, or fine-tuning autonomous vehicles. For economists and climate scientists, it offers a way to make more reliable forecasts from incomplete or noisy data, potentially improving policy and planning.

More information: Huitong Lu et al, The Nesterov accelerated gradient algorithm for CARMA models with lost input data based on interpolation method, International Journal of Modelling, Identification and Control (2025). DOI: 10.1504/IJMIC.2025.147954