A dynamic niching genetic algorithm strategy for docking highly flexible ligands

•A multi-solution genetic algorithm for docking highly flexible ligands is proposed.•Multiple minima ligand conformations are found using a phenotype distance measure.•Redocking and cross-docking of five highly flexible HIV-1 ligand molecules are performed.•The method increased the genetic algorithm useful diversity.•Higher probabilities to identify correct binding-modes.

Currently, docking methods are very important tools in structure-based drug design (SBDD). However, despite the great advances in the docking area over the last decades, most methods cannot be used to dock highly flexible ligands successfully. It is even harder when the ligand is cross-docked into different ligand-bound receptor structures. In this work, a new multi-solution genetic algorithm method, named Dynamic Modified Restricted Tournament Selection (DMRTS), was developed for the effective docking of highly flexible ligands. The DMRTS method uses an insertion criterion based on similarity and a dynamic tournament size to preserve good, distinct solutions in the genetic algorithm population. The proposed method was implemented in three different versions of a steady-state genetic algorithm and evaluated for the redocking and cross-docking of five HIV-1 protease ligands, with 12–20 rotatable bonds. The DMRTS method was also tested on a more diverse set of 34 protein–ligand complexes covering 18 different protein families. A performance comparison with three of the currently most used docking programs was also done. The proposed method was evaluated for 25 benchmark functions of the CEC2005 test suite. The results indicated that the DMRTS method can adequately sample the conformational search space, producing a diverse set of high quality solutions. Moreover, it might be a powerful tool for docking studies in SBDD practice, increasing the success rate in finding correct ligand conformations and efficiently exploring distinct and valuable ligand binding modes.