“Parallel Training Considered Harmful?”: Comparing series-parallel and parallel feedforward network training

Neural network models for dynamic systems can be trained either in parallel or in series-parallel configurations. Influenced by early arguments, several papers justify the choice of series-parallel rather than parallel configuration claiming it has a lower computational cost, better stability properties during training and provides more accurate results. Other published results, on the other hand, defend parallel training as being more robust and capable of yielding more accurate long-term predictions. The main contribution of this paper is to present a study comparing both methods under the same unified framework with special attention to three aspects: (i) robustness of the estimation in the presence of noise; (ii) computational cost; and, (iii) convergence. A unifying mathematical framework and simulation studies show situations where each training method provides superior validation results and suggest that parallel training is generally better in more realistic scenarios. An example using measured data seems to reinforce such a claim. Complexity analysis and numerical examples show that both methods have similar computational cost although series-parallel training is more amenable to parallelization. Some informal discussion about stability and convergence properties is presented and explored in the examples.