We show that nonlinear latent factors and structures in neural population activity can be modelled in a manner that allows for flexible dynamical inference, causally, non-causally and in the presence of missing neural observations. Further, the developed neural network model improves the prediction of neural activity, behaviour and latent neural structures.