Extreme rays of Hankel spectrahedra for ternary forms

The cone of sums of squares is one of the central objects in convex algebraic geometry. Its defining linear inequalities correspond to the extreme rays of the dual convex cone. This dual cone is a spectrahedron, which can be explicitly realized as a section of the cone of positive semidefinite matrices with the linear subspace of Hankel (or middle catalecticant) matrices. In this paper we initiate a systematic study of the extreme rays of Hankel spectrahedra for ternary forms. We show that the Zariski closure of the union of extreme rays is the variety of all Hankel matrices of corank at least 4, an irreducible variety of codimension 10 and we determine its degree. We explicitly construct an extreme ray of maximal rank using the Cayley-Bacharach Theorem for plane curves. We apply our results to the study of the algebraic boundary of the cone of sums of squares. Its irreducible components are dual to varieties of Gorenstein ideals with certain Hilbert functions. We determine these Hilbert functions for some cases of small degree. We also observe surprising gaps in the ranks of Hankel matrices of the extreme rays.