Neanderthals, whose fossils have been found throughout Eurasia, are extinct relatives of modern humans. Neanderthals disappeared from the fossil record approximately 30,000 years ago and a number of hypotheses exist to explain why [1]. An analysis of the 3D structure of the Neanderthal cranium shows a significant size difference between the olfactory bulb of modern humans and Neanderthals, with human olfactory bulbs being larger, suggesting a potential difference in olfactory capabilites between Neanderthals and modern humans [2]. However, a recent study concluded that although there are differences in the size of the olfactory bulb among living humans (smokers versus non-smokers), there was little to no difference detected in the ability to differentiate odors, suggesting that odor detection thresholds may be more important than bulb size [3]. Therefore, although the difference in bulb volume between modern humans and Neanderthals may represent varying olfaction capabilities in the two lineages, morphology alone may not be enough to determine this. Isotope analysis of Neanderthal fossilised teeth suggests a diet high in protein with some plant consumption [4]. Fossils of another hominin, the extinct Denisovan hominin - a sister taxon to Neanderthals, have been found in the Denisova cave in Siberia [5]. Almost no phenotypic information exists for this species, however both Neanderthal and Denisovan genomes have recently been sequenced allowing further exploration of their genomes [1], [6].

Olfactory receptors (ORs) are G protein-coupled receptors located in the olfactory epithelium of the nasal passage, encoded by the largest mammalian gene family [7]. ORs are a highly plastic gene family showing frequent gene birth/loss events [7]. Two OR classes have been described, Class I and II which bind water and air borne odours respectively [8]. ORs are grouped into subfamilies based on sequence similarity [8]. A ‘many – to – many’ binding relationship exists between odorant molecules and ORs, with a single odorant molecule binding to many receptors and a single receptor binding many odorant molecules [21]. The 1000 Genomes project revealed that SNPs can lead to differences in functional and non-functional ORs between modern humans, including instances of ‘rescue’ from pseudogene to functional gene [9]. Loss and gain of OR function correlates with ecological niche adaptation as observed by the high numbers of pseudogenes in cetaceans, assumed driven by adaptation to aquatic environments [8].

Given the physical changes observed between the Neanderthal and modern human olfactory bulb [2] we examined whether OR gene losses vary across modern humans, Neanderthals and Denisovans representing differences in the evolution of olfaction in hominin species.

A total of 10 functional modern human ORs from the HORDE reference database were found as pseudogenes in the high coverage Neanderthal genome (OR2L8, OR4X1, OR5AR1, OR5M11, OR5P3, OR6C74, OR7A5, OR8I2, OR10X1, OR13C4) (Table 1). By comparing these Neanderthal pseudogenes to known modern human OR loss of function SNPs [9], we found 7 of these ORs with identical non-sense mutations present both in Neanderthals and in a subset of modern humans (OR2L8, OR4X1, OR5AR1, OR5M11 OR6C74, OR8I2, OR10X1), while 3 have different Neanderthal specific loss of function mutations (OR5P3, OR7A5, OR13C4) (Figure 2). Three non-functional modern human ORs have functional Neanderthal homologs (OR1P1P, OR2J1P, OR9H1P). These Neanderthal mutations can be found as ‘rescued’ ORs in some modern humans [9].

The Denisovan genome shows evidence of 8 non-functional ORs with functional modern human homologs (OR2L8, OR2S2, OR2T8, OR4C12, OR5AC2, OR5M11, OR8I2, OR51Q1). These ORs cover 4 subfamilies, 1 from Class I and 3 from Class II. Identical non-sense mutations led to loss of function for 4 of these ORs (OR2L8, OR5M11, OR8I2, OR51Q1) in some modern humans [9], while 4 ORs (OR2S2, OR2T8, OR4C12, OR5AC2) have loss of function mutations unique to the Denisovan lineage (Table 1). Additionally, within the Denisovan genome, 5 ORs (OR1P1P, OR2J1P, OR5E1P, OR9H1P, OR13C1P) from Class II lack stop codons despite their presence within modern human OR orthologs. Three of these ORs (OR1P1P, OR2J1P, OR9H1P) have modern human polymorphisms that can ‘rescue’ these ORs.

The mutation leading to non-functionality for OR2L8, OR8I2 and OR5M11 are shared between the Denisovan and Neanderthal, with the same mutation existing in some modern humans. This may indicate that the ancestral state for these ORs in the Neanderthal-Denisovan-modern human most recent common ancestor is non-functional, with modern humans regaining function post-divergence. The mutation that retains function for OR1P1P, OR2J1P and OR9H1P is shared between Neanderthals, Denisovans and some modern humans. This may indicate an ancestral functional state, with a loss of function happening along the modern human lineage. However, without a larger population sampling of extinct genomes for each hominin species, it is difficult to discriminate among alternative evolutionary trajectories.

The deamination of cytosine-to-thymine and guanine-to-adenine can cause a large number of sequencing errors in studies using ancient DNA [18]. Three Neanderthal (OR5AR1, OR6C74, OR10X1) and 3 Denisovan apparent loss of function mutations (OR4C12, OR5AC2, OR51Q1) potentially could have resulted from such deamination (Table 1). While it may be difficult to differentiate between real loss of function and deamination-induced damage, 2 of these ORs (OR6C74, OR10X1) show the same loss in the 1× genome for Neanderthals, suggesting that it is a real mutation event as opposed to deamination. None of the potential deamination events in the 30× Denisovan genome have coverage in the 1× genome version [19], however one of these mutations can be found as a variant of its modern human homolog (OR51Q1).

There are 466 and 433 non-functional ORs in the modern human and chimpanzee genomes respectively. Of these, 99 modern human non-functional ORs have conserved stop codons also present in the chimpanzee, suggesting they were inherited as pseudogenes from the human-chimpanzee common ancestor and are therefore most likely present as inherited pseudogenes in the Neantherthal and Denisovan lineages. These data suggest at least 473 (466 modern human pseudogenes with an additional 10 pseudogenes, minus 3 functional Neanderthal ORs that are pseudogenes in modern humans) and 469 ORs are non-functional within the Neanderthal and Denisovan genomes respectively (Figure 3).

Based on DNA sequence analysis, the divergence time between Denisovans and Neanderthals is 640,000 years [5]. The divergence of Denisovans, Neanderthals and modern humans from their most recent common ancestor is estimated at 804,000 years ago [5]. The relatively recent divergence time between modern humans, Neanderthals and Denisovans suggests a rapid pseudogenization for these ORs. Periods of cold climatic conditions such as those of the middle-late Pleistocene [4] may have caused environmental pressures that could potentially play a role in loss of OR function. Considering that a minimum odorant detection threshold must be met to detect a smell in modern humans [3], in conjuncture with the effect of cold temperatures limiting odor volatility [20], it is possible that the loss of 10 Neanderthal and 8 Denisovan ORs may have had an affect on odorant perception for these hominin species. The loss of 10 ORs may be related to the decreased size of the olfactory bulb in Neanderthals, however inferring how gene loss may affect phenotype in extinct species remains challenging. Further research into how different odors bind to different receptors [21] may shed further light on this issue. This is an ongoing challenge in comparative sensory genomics.

This analysis of OR genes suggests that modern humans, Neanderthals and Denisovans were subject to a number of changes in their respective OR repertoires. As more ancient genomes are sequenced, such analyses may provide a better picture of how sensory perception has evolved, particularly in species for which little phenotypic information is known.