Fractional programming formulation for the vertex coloring problem

•This paper gives a new formulation for the vertex coloring problem.•Decision variables are associated with the pairs of vertices.•The number of colors used is expressed as a linear fractional function.•It shows significantly good performance for dense graphs.•A well-known hard instance DSJC125.9 is solved within 1 minute.

We devise a new formulation for the vertex coloring problem. Different from other formulations, decision variables are associated with pairs of vertices. Consequently, colors will be distinguishable. Although the objective function is fractional, it can be replaced by a piece-wise linear convex function. Numerical experiments show that our formulation has significantly good performance for dense graphs.