Inference for Unknown Functions

Problems arise in many areas of statistics where we require inference about an unknown function, and do not wish to make assumptions about the form of that function. As a result, the problem of inference about unknown functions falls into the general area of nonparametric statistics.

In regression analysis, we wish to estimate the regression function. Simple regression theory assumes that this function is linear, in which case it is described by just two parameters, the slope and intercept. If we wish to allow for a more complex form of regression, we might consider a quadratic regression function, which requires three parameters to describe it. However, in nonparametric regression we do not make any assumptions about the form of the function. Since it can be any function at all, in principle it would require an infinite number of parameters in order to specify it.

The research area of inference about unknown functions deals with such problems, where we wish to make inference about a function but do not wish to make restrictive assumptions about its form. We consider the problem from a Bayesian perspective, in which it is necessary to express prior information about the function. In the case of simple linear regression we had two unknown parameters, and so Bayesian analysis would require the formulation of a prior distribution for those two parameters. If the form of the function is unknown, we need a prior distribution for the whole function, which is effectively infinite-dimensional.

The theory of stochastic processes offers a solution to what appears to be an enormously complex task, of specifying a prior distribution for a function. Research into Bayesian inference for unknown functions at Sheffield has principally modeled the function in question as a Gaussian process, which can be thought of as an infinite-dimensional multivariate normal distribution.

The applications in which unknown functions arise are very diverse, and this is reflected in recent and ongoing research at Sheffield. Unknown functions arise in particular in several of the other research areas of the Bayesian statistics cluster.

  • The field of uncertainty in computer models is based on modeling a computer program as a function that takes the program's inputs as arguments and produces the program's output as the function value.
  • Research is going on into radiocarbon dating in Bayesian archaeology, where the unknown function is the radiocarbon calibration curve.
  • A novel approach to elicitation is under development in which the subject's personal probability distribution is represented by an unknown density function.

Another application is in interpolating pollution monitoring stations, where the unknown function is a mapping that represents non-isotropy in the region over which the monitoring stations are placed. There are clear links to environmental statistics.

Notice that one of the research topics in the area of stochastic modelling in ecology involves Bayesian inference about stochastic processes. The difference between that work and the ideas of inference about unknown functions is that the stochastic processes in the ecological applications are real processes in time or space (not just representing prior information about some idealised function), and we make inference about parameters that describe those processes (as well as about the realisation of the process itself)

Recent publications

All of the recent publications in the area of uncertainty in computer models can be considered as examples of inference about unknown functions.

A precursor of that work treated the problem of numerical integration. In this case the function can literally be any function that we wish to integrate. So the inference that is required is an estimate of the integral. A particular application is where the function is a posterior distribution from some Bayesian analysis. The most recent paper in this area of Bayesian numerical analysis addressed efficient ways to apply Bayesian numerical integration to the integration of a posterior density function.

  • Kennedy, M. and O'Hagan, A. (1996). Iterative rescaling for Bayesian quadrature. In Bayesian Statistics 5, J. M. Bernardo et al (eds.). Oxford University Press, 639-645.

Radiocarbon dating can be said to have revolutionised archaeology, allowing objects to be dated with an accuracy that previously would have been unimaginable. However, the method relies on calibrating radiocarbon ages against ages determined by other means (usually tree-ring dating). It is therefore necessary to use a calibration curve to convert radiocarbon ages to calendar ages. Délil Gomez Portugal Aguilar, a research student of Tony O'Hagan has applied techniques of Bayesian inference for unknown functions to develop improved estimation of the calibration curve. A piece-wise linear curve that reflects the way that calibration is usually treated at present is constructed in the following paper. The work connects with other research of the cluster in Bayesian archaeology.

  • Gomez Portugal Aguilar, D., Litton, C. D. and O'Hagan, A. (2002). A new piece-wise linear radiocarbon calibration curve with more realistic variance. Radiocarbon 44, 195-212

Ongoing research

Work on estimating the radiocarbon calibration curve has continued in collaboration with Caitlin Buck. A paper presenting an improved smooth curve has been written, and further applications are under development.

Another of Tony O'Hagan's research students, Alexandra Schmidt, has been addressing the interpolation of pollution monitoring stations. Such stations form networks that provide much useful data on pollution across the regions that they cover. However, the problem arises of interpolating the measurements at these stations to estimate what the pollution levels are at other points in the region that lie between stations. Such problems of spatial interpolation are generally tackled by the method of geostatistics known as "kriging". Unfortunately, this method relies on an assumption of homogeneity, that the correlation structure between points in the region depends only on how far apart the points are. This is readily demonstrated not to hold in the case of pollution monitoring. Alex Schmidt's work builds on an idea of earlier authors to deform the geographical locations of the stations so that homogeneity applies. Alex then represents the deformation as an unknown function and develops appropriate Bayesian inference. The result is a more complete and sophisticated analysis of this problem than was available before. One paper is submitted to the Journal of the Royal Statistical Society and another is planned. This work also falls within the environmental statistics cluster.

Tony O'Hagan, Jeremy Oakley and Délil Gomez Portugal Aguilar have also recently devised a nonparametric Bayesian method for elicitation of an expert's prior knowledge in the form of a probability distribution. The idea is to represent the density function of this distribution as an unknown function.

Other ongoing research in this area is described in the uncertainty in computer models research pages.

PhD Topics

Research in this area would be supervised by Tony O'Hagan.

Alternative processes for prior distributions

Research at Sheffield in this area has concentrated on modeling the unknown function as a Gaussian process. This is a very flexible family of processes to serve as representing prior information about the function. Nevertheless, other processes are of interest, and would be needed in particular situations. A common problem, for instance, is that the Gaussian process is too smooth and does not allow sudden changes in the function. This research topic will explore heavier-tailed processes, or hybrid mixtures that allow the function to have different behaviour in some regions than in others.

Local computation

An important limitation of Gaussian process methods is that where there are many observations on the function the computation of the posterior inferences becomes extremely demanding. This is because posterior beliefs about the value of the function at any point are influenced by every observation, no matter how distant from the point of interest. This research topic will consider approximations involving a smaller number of neighbouring points.

Other applications

There are also numerous possibilities for applications. Several current areas of application are described in the research page concerning inference about unknown functions, and there are opportunities to apply these methods to other kinds of unknown function arising in a variety of fields.