The male reproductive system is acutely and uniquely sensitive to a variety of toxicities, including those induced by environmental pollutants throughout the lifespan. Early life hormonal and morphological development results in several especially sensitive critical windows of toxicity risk associated with lifelong decreased reproductive health and fitness. Male factor infertility can account for over 40% of infertility in couples seeking treatment, and 44% of infertile men are diagnosed with idiopathic male infertility. Human environmental exposures are poorly understood due to limited available data. The latency between maternal and in utero exposure and a diagnosis in adulthood complicates the correlation between environmental exposures and infertility. The results from this review include recommendations for more and region specific monitoring of polycyclic aromatic hydrocarbon (PAH) exposure, longitudinal and clinical cohort considerations of exposure normalization, gene-environment interactions, in utero exposure studies, and controlled mechanistic animal experiments. Additionally, it is recommended that detailed semen analysis and male fertility data be included as endpoints in environmental exposure cohort studies due to the sensitivity of the male reproductive system to environmental pollutants, including PAHs.