Hi,

I was wondering if there is a maximum size of image stacks CMTK can work with.

I'm asking because I have been using CMTK successfully over the last couple of months but recently I tried stacks that are bigger than usual (<2GB instead of ca. 500 MB) and I constantly get the following error message:

Caught signal 11, faulty address is 0x191ec5000

When I downsample the same images I have no problem running CMTK.

Thanks,
Michael

Dear Michael,

It would be good to hear from Torsten on this, but it might be worth specifying your cmtk version and platform (or pointing at the specific binaries that you installed). Also what command are you running that produces this error?

If this is during the registration stage, you might be able to play with the image resolution parameters:

Image resolution parameters

--sampling , -s
Image sampling (finest resampled image resolution)
[Default: -1]

--coarsest
Upper limit for image sampling in multiresolution hierarchy
[Default: -1]

--omit-original-data
Do not use original data in full resolution for final registration
stage.


specifically setting --sampling to a sensible value and using --omit-original-data. Very likely you do not need to run the registration on the full resolution data.

Best wishes,

Greg.

Dear Greg,

thanks for the quick reply. I used the most recent version of CMTK (3.2.3) and I tried it on a Mac and on a Windows machine using Cygwin.

Here is an example of my most recent command I used:

"/Applications/Fiji.app/bin/cmtk/munger" -b "/Applications/Fiji.app/bin/cmtk" -a -w -r 01 -X 26 -C 8 -G 80 -R 4 -A '--accuracy 0.4' -W '--accuracy 0.4' -T 6 -s "template/template-1.nrrd" images

I'm pretty sure the error happens during the affine registration so I will try to play around with the parameters you suggested. Here is just an example of what I get:

1-12_12.33.42.command ; exit;
hostname = m-o0-033.neuro.mpg.de; short hostname = m-o0-033.neuro.mpg.de
JOB ID =
Root directory is /Users/mkunst/Documents/CMTK_Alignment
Changing to root directory: /Users/mkunst/Documents/CMTK_Alignment
Setting image root to: images
MYEXEC: CMD = mkdir -p Registration/affine/template-1_test_01_9dof.list exited with value 0
Aff:test_01 INFO: number of threads set to 6 according to environment variable CMTK_NUM_THREADS
Caught signal 11, faulty address is 0x190ec5000
[stack] Execution path:
[stack] 1 libsystem_platform.dylib 0x00007fff86a64eaa _sigtramp + 26
[stack] 2 ??? 0x00007fff5fbfda40 0x0 + 140734799796800
[stack] 3 registration 0x00000001000aaa9d _ZNK4cmtk13TemplateArrayIhE8GetRangeEv + 13
[stack] 4 registration 0x000000010005e222 _ZN4cmtk8VolumeIO4ReadERKSs + 2050
[stack] 5 registration 0x000000010005fd40 _ZN4cmtk8VolumeIO12ReadOrientedERKSsPKc + 32
[stack] 6 registration 0x000000010000512e _ZN4cmtk29AffineRegistrationCommandLineC2EiPPKc + 11294
[stack] 7 registration 0x0000000100002094 _Z6doMainiPPKc + 36
[stack] 8 registration 0x0000000100002210 main + 80
[stack] 9 registration 0x0000000100002064 start + 52
MYEXEC: CMD = /Applications/Fiji.app/bin/cmtk/registration -i -v --dofs 6 --dofs 9 --accuracy 0.4 -o Registration/affine/template-1_test_01_9dof.list template/template-1.nrrd images/test_01.nrrd exited with value 1
Aff:test_01 INFO: number of threads set to 6 according to environment variable CMTK_NUM_THREADS
Caught signal 11, faulty address is 0x18fec5000
[stack] Execution path:
[stack] 1 libsystem_platform.dylib 0x00007fff86a64eaa _sigtramp + 26
[stack] 2 ??? 0x00007fff5fbfda40 0x0 + 140734799796800
[stack] 3 registration 0x00000001000aaa9d _ZNK4cmtk13TemplateArrayIhE8GetRangeEv + 13
[stack] 4 registration 0x000000010005e222 _ZN4cmtk8VolumeIO4ReadERKSs + 2050
[stack] 5 registration 0x000000010005fd40 _ZN4cmtk8VolumeIO12ReadOrientedERKSsPKc + 32
[stack] 6 registration 0x000000010000512e _ZN4cmtk29AffineRegistrationCommandLineC2EiPPKc + 11294
[stack] 7 registration 0x0000000100002094 _Z6doMainiPPKc + 36
[stack] 8 registration 0x0000000100002210 main + 80
[stack] 9 registration 0x0000000100002064 start + 52
MYEXEC: CMD = /Applications/Fiji.app/bin/cmtk/registration -i -v --dofs 6 --dofs 9 --accuracy 0.4 -o Registration/affine/template-1_test_01_9dof.list template/template-1.nrrd images/test_01.nrrd exited with value 1

Ran 0 initial affine registrations of which 0 failed
Ran 2 affine registrations of which 2 failed
Ran 0 warp registrations
Reformatted 0 images
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
[Process completed]

Thanks,
Michael

Hi -

I just took a look at the code to confirm, and sadly CMTK is using 32bit signed integer to index voxels in images, so there is indeed a 2GB limit for byte/char-valued images.

While there is likely a small performance benefit from that, I feel it's probably not worth the inconvenience of being unable to handle large images. So I am going to try and fix this, but it's going to take a little while because as you can imagine the necessary changes will be rather extensive (albeit simple).

If anyone wants to volunteer to help, please raise your hands. Otherwise, I'll keep you posted on the progress here.

BTW - I suspect that regardless of me fixing this, there will remain to be a 4GB-per-image size limit on 32bit platforms (or on 64bit platforms running a 32bit build of CMTK), simply because memory allocation of larger blocks will fail.

Torsten

Hi Torsten,

If it is not too complicated I can help you with updating the code, I just need clear instruction on what to do because my programming experience is fairly limited (mostly writing scripts in R).

Am I correct to assume that the main task is to change the variable types to 64bit signed integers throughout the code?

Best,
Michael

Thanks Michael -

Actually, it turned out to be a little more involved than I thought, but so far it's mostly mechanical. I think I should have something in a couple of days.

I will have to ask you to help me with testing, though, since for the purposes of this project I don't have access to machines with sufficiently large memory.

Torsten

Hey Torsten,

No problem, I can run the data on my machine. Which also brings me to my ne=
xt question. Are 64 GB of RAM enough? The individual stacks are between 2-4 =
GB. If I remember correctly in order for the registration to run I need 3-4 =
the size of the file in available memory but I can=E2=80=99t remember if in the ca=
se of registering multiple brains one has to account for the size of the ent=
ire dataset. I can theoretically increase the RAM to 256 GB.

Best,
Michael

Hi Michael and all:

I just uploaded source code and Linux/Windows/Cygwin builds (all 64bit) for CMTK 3.3.0, which hopefully removes the 2GB volume size limitation. Please give it a spin.

Torsten

About that - if you are doing pairwise registrations, 64GB should be plenty, certainly for affine registration. For nonrigid it also depends on how fine a control point grid you are using, but honestly, if your grid is so fine that you run out of memory, you probably wouldn't live to see the results in your lifetime anyway, simply because the optimization would take forever.

Of course more memory is always better, but often you don't need the entire memory to be physical; having 64GB actual and another 64GB virtual will take slightly longer due to swapping, but will prevent the computation from running out of memory.

But really, the bottom line is, just give it a shot and see what happens ;)

Let me know how it goes with the new release!

Torsten
Originally posted by Michael Kunst:

Thanks for the fast turnaround. I=E2=80=99ll give it a try next week and let you =
know how it works.

Best,
Michael

Hey everybody,

I finally managed to test out the cygwin version and it works on stacks of up to 3.5 GB. Haven't had the chance to try it out on other systems, hopefully I'll be soon able to do that.

Thanks,
Michael