Additive smooth models for big data

Motivated by applications in air-pollution monitoring and electricity grid management, this talk will discuss the development of methods for estimating generalized additive models having of order 10^4 coefficients and 10s of cariance parameters, from of order 10^8 observations. The strategy combines 4 elements: (i) the use of rank reduced smoothers, (ii) fine scale discretization of covariates, (iii) an efficient approach to marginal likelihood optimization, that avoids computation of numerically awkward log determinant terms and (iv) marginal likelihood optimization algorithms that make good use of numerical linear algebra methods with reasonable scalability on modern multi-core processors. 10^4 fold speed ups can be achieved relative to the previous state of the art methods. This enables us to estimate spatio-temporal models for UK `black smoke' air pollution data over the last 4 decades at a daily resolution, where previously an annual resolution was challenging.