Vortex solitons at the interface separating square and hexagonal lattices

•We demonstrate vortex solitons at the square–hexagonal photonic lattice interface.•A novel type of five-lobe surface vortex solitons is observed.•Different phase structures of surface solutions are studied.•Orbital angular momentum transfer of such solutions is investigated.

Vortex solitons at the interface separating two different photonic lattices – square and hexagonal – are demonstrated numerically. We consider the conditions for the existence of discrete vortex states at such interfaces and develop a concise picture of different scenarios of the vortex solutions behavior. Various vortices with different size and topological charges are considered, as well as various lattice interfaces. A novel type of discrete vortex surface solitons in a form of five-lobe solution is observed. Besides stable three-lobe and six-lobe discrete surface modes propagating for long distances, we observe various oscillatory vortex surface solitons, as well as dynamical instabilities of different kinds of solutions and study their angular momentum. Dynamical instabilities occur for higher values of the propagation constant, or at higher beam powers.