Sex chromosomes differ in their inheritance properties from autosomes and hence may encode complementary information about past demographic events. We compiled and analyzed a range-wide resequencing data set of the red deer (Cervus elaphus), one of the few Eurasian herbivores of the Late Pleistocene megafauna still found throughout much of its historic range. Our analyses of 144 whole genomes reveal striking discrepancies between the population clusters suggested by autosomal and X-chromosomal data. We postulate that the genetic legacy of Late Glacial population structure is better captured and preserved by the X chromosome than by autosomes, for two reasons. First, X chromosomes have a lower Ne and hence lose genetic variation faster during isolation in glacial refugia, causing increased population differentiation. Second, following postglacial recolonization and secondary contact, immigrant males pass on their X chromosomes to female offspring only, which effectively halves the migration rate when gene flow is male mediated. Our study illustrates how a comparison between autosomal and sex chromosomal phylogeographic signals unravels past demographic processes that otherwise would remain hidden.