Generalized Serre duality

We introduce a notion of generalized Serre duality on a Hom-finite Krull–Schmidt triangulated category T. This duality induces the generalized Serre functor on T, which is a linear triangle equivalence between two thick triangulated subcategories of T. Moreover, the domain of the generalized Serre functor is the smallest additive subcategory of T containing all the indecomposable objects which appear as the third term of an Auslander–Reiten triangle in T; dually, the range of the generalized Serre functor is the smallest additive subcategory of T containing all the indecomposable objects which appear as the first term of an Auslander–Reiten triangle in T.We compute explicitly the generalized Serre duality on the bounded derived categories of artin algebras and of certain noncommutative projective schemes in the sense of Artin and Zhang. We obtain a characterization of Gorenstein algebras: an artin algebra A is Gorenstein if and only if the bounded homotopy category of finitely generated projective A-modules has Serre duality in the sense of Bondal and Kapranov.