Efficient algorithms for physical layer security in two-way relay systems

In this paper, we study the physical layer security in a two-way relay system (TWRS) consisting of two transceivers, one eavesdropper, and multiple relays. The channel state information (CSI) of the eavesdropper is assumed to be known. We consider the problem of maximizing the secrecy sum rate by optimizing the relay beamforming vector and the transceivers' powers under total power constraint. The optimization problem of the beamforming vector has been formulated in the literature as a product of three Rayleigh quotients which is very difficult to solve, and the best proposed solution was a suboptimal solution to the null space beamforming. Here, we propose two approaches: (1) Optimal solution to the null space beamforming approach, and (2) Ignoring one Rayleigh quotient (IORQ) approach (which has not been proposed before). In the first approach, we convert the nonconvex product of two Rayleigh quotients to a quadratically constrained quadratic program (QCQP) and then to a convex problem with one dimensional search using semidefinite programming (SDP). Then we significantly simplify the problem by providing a new approach that uses the generalized eigenvalues. This new approach can be used to solve all QCQPs with positive definite objective function and two trace constraints. In the second proposed approach, we look for a beamforming vector that does not eliminate the complete information signal at the eavesdropper aiming to increase the whole secrecy sum rate. This approach provides a substantial contribution in improving the secrecy sum especially when the number of relays is low. Numerical results show that the proposed algorithms provide higher secrecy sum rates than the existing algorithms.1