Recurrent neural networks are an important class of models for explaining neural computations. Recently, there has been progress both in training these networks to perform various tasks, and in relating their activity to that recorded in the brain. Despite this progress, there are many fundamental gaps towards a theory of these networks. Neither the conditions for successful learning, nor the dynamics of trained networks are fully understood. I will present the rationale for using such networks for neuroscience research, and a detailed analysis of very simple tasks as an approach to build a theory of general trained recurrent neural networks.