It looks like your samples are suffering from severe local distortion or folding. If you run an affine only registration and reformat, you should already see reasonably good alignment. Otherwise, the subsequent nonlinear warp may produce distorted results.

Greg's wiki provides a good assessment of registration parameters and their outcomes:

https://flybrain.mrc-lmb.cam.ac.uk/dokuwiki/doku.php?id=warping_manual:start

Registration also has its limits when dealing with distorted samples. MADI3D can help push these limits a bit through interactive manual prealignment, allowing you to translate, rotate, and independently scale your sample along the X, Y, and Z axes before running registration.

Here's the workflow in a nutshell:

First, make sure your sample is loaded into the scene either as a multichannel volume or as individual channels organized into their own group.

Make sure both the template and the sample are visible in the 3D view. Select the group containing your sample channels, or the multichannel volume itself. This ensures that the same transformation is applied consistently.

Open the Transform panel and manually align your sample while inspecting the scene. You can also enable the interactive transform widget. Try to achieve reasonable overall alignment.
Once you're satisfied with the pre-alignment, open the Registration panel. The easiest way to set up the reference is to right-click the template in the scene tree and select"Add as Template to Registration". Then add your sample as well.

Check that the channel roles (especially the Moving registration channel role for the channel you use for registration with the template), registration inputs, and reformat settings are correctly assigned. You can also specify a custom registration output location if desired.

Finally, run the registration.