Let's look at some data from the Fermi Large Area Telescope (LAT), a gamma-ray telescope orbiting the earth since 2008. A gamma-ray telescope is similar in spirit and purpose to more familiar telescopes, such as the Hubble Space Telescope. Gamma-rays, despite the name, are photons, quanta of electromagnetic radiation, just like the visible light used to create many Hubble images. The difference is the energy. We'll use units of electron volts, or eV, to measure the energy of light. A photon of red light has an energy of around 1.8 eV; of blue light, about 3.1 eV. The image shown below was created from Fermi LAT data for photons between 1 and 2 billion eV, which we abbreviate GeV (for giga-electron volts). The physics creating gamma-rays of this energy are associated with very high energy phenomena, such as supernova explosions and supermassive black holes. Gamma-ray astronomy provides a window to allow us to better understand the high-energy physics occurring in the universe.
This false-color image represents the number of photons from each $0.1^\circ \times 0.1^\circ$ square over the entire sky, for 13+ years of observation. The image is shown in Galactic coordinates, so that the plane of the Milky Way galaxy is oriented horizontally through the middle, and the center of the Galaxy in at $(0^\circ, 0^\circ)$. When we consider the entire sky, it is effectively a sphere, and to plot a sphere on a flat surface like a computer screen requires a projection. We chose the Plate Carrée projection, because it makes it easy to locate things in Galactic coordinates by eye, at the price increased distortion as you move farther from the Galactic plane (this is why objects appear horizontally stretched near the top and bottom of the image). There were around 20 million photons detected over those 13 years (more accurately, 20 million "made the cut" in the course of processing the data). That may sound like a lot, but if you were to look at the bright blue sky, the number of photons entering your eye each second would be something like $10^{14}$, or 100 trillion. By contrast, map above equates to about $0.05$ photons per second, or roughly one photon between 1 and 2 GeV every 20 seconds.
Okay, so there aren't so many gamma-ray photons. But so what? We still have a nice picture of the sky, right? Well...maybe. The answer really depends on what scientific questions you'd like to answer with this data, or more particularly, how you want to reason about the answers. To get some idea why this matter, let's construct a hypothetical map of the gamma-ray sky in the same energy range, consisting of the following components:
The theoretical map we obtain will generally be in terms of photon flux, expressing how many photons per second of some energy we expect to see passing through some area from a given direction in the sky. To estimate what the Fermi LAT would have seen, we need to pass our theoretical flux map through the instrument response function (IRF). The IRF accounts for many factors, including
The resulting theoretical count map is shown below.
This appears largely similar to the data, but one obvious difference is that the original data has a lot more little dots. You may have thought those were individual sources like stars, or perhaps you already guessed the punchline: those dots represent pixels which have detected only a few photons, most of them being a single photon. The data consists of integer counts: 0, 1, 2, 3, ... Our model contains expected counts, which can be any fractional value, such as 1.28953. A good example of the difference is that the model map actually has no areas of exactly 0, with minimum expected counts of about 0.0002. The discreteness of counting photons results in a particular kind of noise in the data, called shot noise.
Noise in general refers to anything observed in the data which impairs our ability to extract accurate information from that data. To get some idea how shot noise in the Fermi LAT data is a headache, let's try comparing our theoretical model of the gamma-ray sky to the actual data. We'll just subtract the model from the data, with the idea of identifying any areas which are different. Those places showing a difference are, of course, interesting targets for further study, since they indicate that we missed some physics in our original model.
There is plenty of interest here, and clearly our theoretical model needs some work. But how do you tell what here is "real" vs just the effects of shot noise? More importantly, if you were to claim that something here was "physics", how do convince other scientists that some feature is worth the time and effort to study further, and not just some artifact of the noise?
One way that people try to handle noisy data is by applying filters. Noise often adds a lot of high-frequency detail to the data, while natural signals or images have a larger proportion of information at lower frequencies or size scales. Let's try applying a low-pass filter to our difference image:
Visually, this seems more pleasing. But scientifically we haven't really advanced the cause. Certain features appear by eye to be more prominent in the filtered image. But can we say they are really there? And if so, with what confidence? And how do we answer more detailed questions which might help to distinguish which areas of future investigation are most interesting?
To really drive this point home, let's do the same exercise for the 500-1000 GeV energy range.
Now we're really in the range scientists often call "count limited". 2340 photons over 13 years amounts to an average of about one photon detected every two days. Even so, we can roughly make out the Galactic plane. There clearly is some information in this data. What does the theoretical model look like for the same energy range?
The small dots in this image really are gamma-ray sources, not photons. They just aren't very bright and were mostly obscured in the 1-2 GeV model by the much brighter theoretical emission away from the Galactic plane. We can try subtracting the model from the data, but this is clearly silly:
What if we try filtering?
Again, this is not helpful.
Some people love mathematics, some hate it. But most take it for granted, without considering why mathematics exists beyond the immediate obvious utility of calculating the tip on your restaurant bill or the correct trajectory to send a probe to another planet. We first take a detour in psychology, to the phenomenon called pareidolia. Take a look at this image:
Do you just see clouds? Or does a face perhaps appear? How about this one:
Some people see a mysterious "face of Jesus" in the foreground, but on closer examination you might discern this is actually a toddler wearing a hat, sitting on Dad's lap, with some vegetation in the background. Pareidolia is the tendency of the human brain to find recognizable patterns, even when they're not really there. More interestingly, visual interpretations vary between individuals, which is why the Rorschach test can provide information about a person's mental state. What do you see in this image? And if you ask others, what do they see?
The brain will go so far as to fill in "missing" information, so that our mental interpretation of visual input matches expectations:
It is very hard to not see the white triangle - until you block the black circles, and it vanishes.
Psychological phenomena such as pareidolia are probably what physicist Richard Feynman had in mind when he said of the practice of science:
"The first principle is that you must not fool yourself and you are the easiest person to fool." - Richard P. Feynman
That said, the ability of our brains to identify patterns is also a gift for scientific investigation, allowing us to perceive potential interesting new hypotheses in data. For example, in the 1-2 GeV difference image between data and model above, you probably noticed two bubble-like structures appearing to emanate from the Galactic center. This certainly sees like something worth investigating, right? But how do you avoid "fooling yourself", and communicate your exciting new discovery to other scientists while avoiding the sort of perceptual ambiguities associated with pareidolia? One obvious path is to just get better data. Take the infamous "face on Mars" image:
This certainly looks like a face, and has fueled a lot of speculation about aliens leaving monuments on Mars, etc. Personally, I see two faces, the usual one near the top center, and a 1940's gangster wearing a fedora on the left, though the idea that aliens would make monuments to Godfather-style mafiosos would be silly, right?
Are we just fooling ourselves? Let's see a picture made from better data, higher resolution without the blurriness above:
The "face" has disappeared, though the improved data shows how we might have fooled ourselves given a particular combination of shadows and poor camera resolution.
Often we don't have access to improved data. For the Fermi LAT data we are analyzing, it's not going to get any better unless we wait a long, long time, either for the instrument to collect more data (it's already been observing for over 13 years), or for the next generation gamma-ray telescope to be constructed and launched. When looking at the speckled images above, particularly the difference between data and model, how can we know that what we see is more than just a trick of perception? And how do we describe our results to others without it becoming simply a battle of opinions?
That brings us back to mathematics. Math provides us a language for reasoning about the world without opinion. That isn't to say there aren't opinions about the assumptions or inputs, or the interpretation of the final answer. But at least we can make clear the path from one to the other. And mathematics provides the language allowing others to unambiguously understand and verify our reasoning. As we explore the various combinations of observed data and scientific hypotheses, we want to employ as much math as practical. Practicality is an important point. Most mathematical problems have no exact solution, and most of those that do are not amenable to solution on human timescales. There's no point in standing on mathematical principles if it will take longer than the lifetime of the universe to arrive at the answer to your scientific question. Instead we usually make assumptions and approximations for the sake of practicality. That is, we make models. A mathematically well-defined model allows you to reason about the relationship of your model to observations of the real world. Are my assumptions correct? Do my approximations allow for accurate predictions of observable behavior? At what point do these break down, perhaps requiring a new model? And most importantly for our purposes: how much can I believe my model given some data?
There will always be opinions, guesses, and interpretations in science. This is how new ideas are generated, new theories to be tested. But we must always push our mathematical models to cover as much ground as possible, otherwise we risk fooling ourselves.
A good cautionary example is the discovery of the "antimatter fountain" at the center of the Milky Way, which generated a lot of excitement and speculation at the time. However, once we started applying some mathematical and statistical analysis it quickly became apparent that the "fountain" could likely be an artifact of the data, in particular the pattern of how observations were conducted. Understandably, "a mistake of arithmetic" was not nearly as exciting as "antimatter shooting out of a black hole", and the scientific community largely continued fooling themselves until a better telescope showed there was no fountain.
© 2023 Dave Dixon