Multi-modal functional magnetic resonance imaging has been widely used for brain research. Conventional data-fusion methods cannot capture complex relationship (eg, nonlinear predictive relationship) between multiple data. This paper aims to develop a neural network framework to extract phenotype related cross-data relationships and use it to study the brain development. We propose a novel method, deep collaborative learning (DCL), to address the limitation of existing methods. DCL first uses a deep network to represent original data and then seeks their correlations, while also linking the data representation with phenotypical information.