For general details on DTI, take a look at this
  https://www.walshmedicalmedia.com/open-access/a-basic-introduction-to-diffusion-tensor-imaging-mathematics-and-imageprocessing-steps-2168-975X-1000229.pdf

The code for FSL is available
  https://github.com/muschellij2/FSL6.0.0/blob/3c3dd651066ee189bc8c290f744ca48cb3d1f156/src/fdt/dtifit.cc


I would ask your FSL specific questions on their mailing list:
  https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=fsl