The Brezis-Nirenberg problem for the curl-curl operator

We look for solutions E:Ω→R3 of the problem{∇×(∇×E)+λE=|E|p−2Ein Ων×E=0on ∂Ω on a bounded Lipschitz domain Ω⊂R3, where ∇× denotes the curl operator in R3. The equation describes the propagation of the time-harmonic electric field ℜ{E(x)eiωt} in a nonlinear isotropic material Ω with λ=−μεω2≤0, where μ and ε stand for the permeability and the linear part of the permittivity of the material. The nonlinear term |E|p−2E with p>2 is responsible for the nonlinear polarisation of Ω and the boundary conditions are those for Ω surrounded by a perfect conductor. The problem has a variational structure and we deal with the critical value p, for instance, in convex domains Ω or in domains with C1,1 boundary, p=6=2⁎ is the Sobolev critical exponent and we get the quintic nonlinearity in the equation. We show that there exist a cylindrically symmetric ground state solution and a finite number of cylindrically symmetric bound states depending on λ≤0. We develop a new critical point theory which allows to solve the problem, and which enables us to treat more general anisotropic media as well as other variational problems.