This paper explores training and initialization aspects of dynamic neural networks when applied to the nonlinear system identification problem. A well known dynamic neural network structure contains both output states and hidden states. Output states are related to the outputs of the system represented by the network. Hidden states are particularly important in allowing dynamic neural networks to approximate complex nonlinear dynamics. An optimisation based method is proposed in this paper for properly initialising the hidden states of a dynamic neural network, so as to avoid the introduction of bias in the network parameters as a result of incorrect hidden state initialisation. Furthermore, a simple optimisation based method is proposed to initialise the hidden states once the network has been trained. The methods are illustrated with experimental data taken from a laboratory scale pressure plant.