Stats on and off
I had a chance to talk with some folks at the Santa Cruz workshop & the CCA that are thinking deeply about some problems that I've long struggled with, briefly described below. While I didn't necessarily come away with concrete solutions (other than 'we're working on it', or 'here's some additional tests we can try'), it is an encouraging experience to know that really smart people are working on it, and a sobering one to know that a lot of these things aren't really available in papers even if the astrostats community generally considers them common knowledge. I want to put together a whitepaper on Bayesian inference circa 2025 so that students and fellow researchers have a slightly better experience setting up their inference problems and interpreting their results.
How do we deal with being prior dominated in Bayesian analysis when folks overwhelmingly just use the median estimates from a fit in their analysis? The easiest way to reconcile this is to just have people use the uncertainties in their downstream analysis, so that when something is prior dominated it just doesn't add any information to the fit.
If we do want to try and get more information out of the observations, then we could try a heirarchical model, or do a population level analysis to try and recover information about meta-or hyper-parameters that individual observations don't really constrain. A good test that would be nice to see in more papers is if we use a flat, a Jeffreys and a well-motivated normal prior for the same problem and show whether the posterior is/isn't strongly affected by it.How do we run implicit likelihood inference-type problems when we don't have a perfect forward model? What can we learn of systematic and measurement uncertainties so that we can make a noise model that's 'good enough'? There's a ton of work on this, and might need to be its own, longer, post. The tl;dr is that there are ways of determining population-level noise properties that help with this.
How do we work with iterative denoising (when using higher SNR observations to denoise lower SNR observations) when the distribution of objects changes across the SNR landscape? (e.g. some objects/populations only showing up at really low SNR) No good solution to this one. Just more data so we have a better estimate of the tails, or splitting things into clustered populations in data-space while doing the denoising procedure?
How can we best reconcile large representative datasets with low average SNR (e.g. from blind surveys) with hand-picked observations of unique/interesting objects with very high data quality (e.g. IFU observations of mergers)? Ideally, the first thing I want here it to be able to learn the selection function (i.e. what is the likelihood of seeing object X in a population of galaxies at redshift z?). No good solution to this one either. Some finance(?) literature on learning population means etc. from outlier samples of distributions with heavy tails, but requires a bunch of analytic assumptions. There must be something to be done here with normalizing flows though.
How do we best deal with missing data when working with fitting large datasets? Lots of solutions. Simformer. Imputation. Masking. Always check with data-space posterior predictive methods.
To be continued!