Security of image encryption scheme based on multi-parameter fractional Fourier transform

•We find a serious security problem on the m-MPFRFT.•We present a new generalized fractional Fourier transform.•We proposed an image encryption scheme based on new method.•The encryption scheme has overcome the problem and enlarged the key space.

Recently, multi-parameter fractional Fourier transform (MPFRFT) has been widely applied in the optics cryptosystem, which has attracted more and more researchers’ attention. However, in further study we find a serious security problem on the MPFRFT which is the multi-choice of decryption key corresponding to an encryption key. The existence of multi-decryption-key hinders the application of this algorithm. We present a new generalized fractional Fourier transform, which can overcome the problem and enlarge the key space. The simulation results show that the proposed algorithm has higher security and key sensitivity.