In this paper, a new approach to address bypass control system design for heat exchanger networks (HENs) is proposed based on frequency domain analysis. The classical bypass control system design methodology relies on the calculation of the relative gain array (RGA) which is commonly used as a measure of the interaction. The main drawbacks of this approach are the difficulties of analyzing the unstable process, and the complexity of measuring stability. Instead, the key idea of this work is to take stability margins into consideration when potential bypasses and controlled variables are paired. Firstly, on the basis of dynamic model, small singular values of the return difference matrix are presented for the simultaneous analysis of stability margin and flexibility. Secondly, the non-square relative gain array (ns-RGA) in the frequency domain is applied to study the interactions of potential bypasses and controlled variables, and some pairing rules are proposed. Thirdly parameters of PID controller are designed while phase margin and gain margin are in proper region, then the final pairing scheme is determined. Case studies illustrated the necessity to consider stability margins during the process of the bypass control system.