Arginine vasotocin and androgen pathways are associated with mating system variation in North American cichlid fishes

• The fish Herichthys cyanoguttatus forms monogamous pairs to mate.
• In a close relative, Herichthys minckleyi, dominant males form harems of females.
• The harem-forming species had higher gene expression of V1a2 receptor in the brain.
• The harem-forming species had more 11-ketotestosterone in the blood.
• The monogamous species had more testosterone in the blood.

Neuroendocrine pathways that regulate social behavior are remarkably conserved across divergent taxa. The neuropeptides arginine vasotocin/vasopressin (AVT/AVP) and their receptor V1a mediate aggression, space use, and mating behavior in male vertebrates. The hormone prolactin (PRL) also regulates social behavior across species, most notably paternal behavior. Both hormone systems may be involved in the evolution of monogamous mating systems. We compared AVT, AVT receptor V1a2, PRL, and PRL receptor PRLR1 gene expression in the brains as well as circulating androgen concentrations of free-living reproductively active males of two closely related North American cichlid species, the monogamous Herichthys cyanoguttatus and the polygynous Herichthys minckleyi. We found that H. cyanoguttatus males bond with a single female and together they cooperatively defend a small territory in which they reproduce. In H. minckleyi, a small number of large males defend large territories in which they mate with several females. Levels of V1a2 mRNA were higher in the hypothalamus of H. minckleyi, and PRLR1 expression was higher in the hypothalamus and telencephalon of H. minckleyi. 11-ketotestosterone levels were higher in H. minckleyi, while testosterone levels were higher in H. cyanoguttatus. Our results indicate that a highly active AVT/V1a2 circuit(s) in the brain is associated with space use and social dominance and that pair bonding is mediated either by a different, less active AVT/V1a2 circuit or by another neuroendocrine system.