Hello,

I have been running brain registrations using the CMTK GUI generated command files. Brains were registering fine for about a month but now CMTK is generating the affine folder and cmd.sh file and then going idle. Nothing happens and there are no errors. Has anyone else had similar issues?

Thanks,
Rob

Can you give some more details e.g. full set of status messages from CMTK and the layout of your file structure. The munger/CMTK GUI scripts are fussy about filenames, so that could be an issue, but normally that would result in affine and warp output but no reformatted images.

Hello Greg,

CMTK gives the following...

JOB ID =
Root directory is /scratch/projects/zebrafish4autism/r.kozol1/CMTKreg10
Changing to root directory: /scratch/projects/zebrafish4autism/r.kozol1/CMTKreg10
Setting image root to: images
MYEXEC: CMD = mkdir -p Registration/affine/Ref20131120pt14pl2_WTPTZ10_01_9dof.list exited with value 0
Aff:WTPTZ10_01


That is all I get out. Nothing else happens.

Thanks,

Hmm, can you send a directory listing. e.g. what you get out of doing 



Best,

Greg.

[r.kozol1@pegasus CMTKreg10]$ ls -LR
.:
commands images Jobs refbrain reformatted Registration

./commands:
munger_2018-05-18_10.sh
./images:
shank3abCTH2o2_01.nrrd shank3abCTH2o2_02.nrrd WTPTZ10_01.nrrd WTPTZ10_02.nrrd
./Jobs:
16354994.err 16361965.err 16362258.err 16409111.err 16422935.err
16354994.out 16361965.out 16362258.out 16409111.out 16422935.out
16355407.err 16361984.err 16362655.err 16412106.err 16425913.err
16355407.out 16361984.out 16362655.out 16412106.out 16425913.out
16355838.err 16362017.err 16373721.err 16419575.err 16429595.err
16355838.out 16362017.out 16373721.out 16419575.out 16429595.out
16355884.err 16362028.err 16391330.err 16420012.err 16431000.err
16355884.out 16362028.out 16391330.out 16420012.out 16431000.out
16361830.err 16362112.err 16394103.err 16421266.err munger_2018-05-18_10.job
16361830.out 16362112.out 16394103.out 16421266.out
16361891.err 16362163.err 16398550.err 16422926.err
16361891.out 16362163.out 16398550.out 16422926.out
./refbrain:
./reformatted:
Ref20131120pt14pl2_shank3abCTH202_01_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_shank3abCTH202_02_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_shank3abCTPTZ2_01_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_shank3abCTPTZ2_02_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_shank3abCTPTZ5_01_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_shank3abCTPTZ5_02_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_shank3abNTH202_01_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_shank3abNTH202_02_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_shk3abNTLF1_01_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_shk3abNTLF1_02_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_WTH204_01_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_WTH204_02_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_WTPTZ12_01_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_WTPTZ12_02_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_WTPTZ13_01_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_WTPTZ13_02_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_WTPTZ5_01_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_WTPTZ5_02_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_WTPTZ7_01_warp_m0g80c8e1e-1x52r3.nrrd
Ref20131120pt14pl2_WTPTZ7_02_warp_m0g80c8e1e-1x52r3.nrrd
./Registration:
affine warp
./Registration/affine:
Ref20131120pt14pl2_WTh2o1_01_9dof.list Ref20131120pt14pl2_WTPTZ12_01_9dof.list
Ref20131120pt14pl2_WTPTZ10_01_9dof.list Ref20131120pt14pl2_WTPTZ13_01_9dof.list
./Registration/affine/Ref20131120pt14pl2_WTh2o1_01_9dof.list:
cmd.sh
./Registration/affine/Ref20131120pt14pl2_WTPTZ10_01_9dof.list:
cmd.sh
./Registration/affine/Ref20131120pt14pl2_WTPTZ12_01_9dof.list:
cmd.sh registration settings statistics studylist
./Registration/affine/Ref20131120pt14pl2_WTPTZ13_01_9dof.list:
cmd.sh registration settings statistics studylist
./Registration/warp:
Ref20131120pt14pl2_WTPTZ12_01_warp_m0g80c8e1e-1x52r3.list
Ref20131120pt14pl2_WTPTZ13_01_warp_m0g80c8e1e-1x52r3.list
./Registration/warp/Ref20131120pt14pl2_WTPTZ12_01_warp_m0g80c8e1e-1x52r3.list:
cmd.sh level-01.list level-03.list registration.gz statistics
level-00.list level-02.list level-04.list settings studylist
./Registration/warp/Ref20131120pt14pl2_WTPTZ12_01_warp_m0g80c8e1e-1x52r3.list/level-00.list:
registration.gz settings statistics studylist
./Registration/warp/Ref20131120pt14pl2_WTPTZ12_01_warp_m0g80c8e1e-1x52r3.list/level-01.list:
registration.gz settings statistics studylist
./Registration/warp/Ref20131120pt14pl2_WTPTZ12_01_warp_m0g80c8e1e-1x52r3.list/level-02.list:
registration.gz settings statistics studylist
./Registration/warp/Ref20131120pt14pl2_WTPTZ12_01_warp_m0g80c8e1e-1x52r3.list/level-03.list:
registration.gz settings statistics studylist
./Registration/warp/Ref20131120pt14pl2_WTPTZ12_01_warp_m0g80c8e1e-1x52r3.list/level-04.list:
registration.gz settings statistics studylist
./Registration/warp/Ref20131120pt14pl2_WTPTZ13_01_warp_m0g80c8e1e-1x52r3.list:
cmd.sh level-01.list level-03.list registration.gz statistics
level-00.list level-02.list level-04.list settings studylist
./Registration/warp/Ref20131120pt14pl2_WTPTZ13_01_warp_m0g80c8e1e-1x52r3.list/level-00.list:
registration.gz settings statistics studylist
./Registration/warp/Ref20131120pt14pl2_WTPTZ13_01_warp_m0g80c8e1e-1x52r3.list/level-01.list:
registration.gz settings statistics studylist
./Registration/warp/Ref20131120pt14pl2_WTPTZ13_01_warp_m0g80c8e1e-1x52r3.list/level-02.list:
registration.gz settings statistics studylist
./Registration/warp/Ref20131120pt14pl2_WTPTZ13_01_warp_m0g80c8e1e-1x52r3.list/level-03.list:
registration.gz settings statistics studylist
./Registration/warp/Ref20131120pt14pl2_WTPTZ13_01_warp_m0g80c8e1e-1x52r3.list/level-04.list:
registration.gz settings statistics studylist

Hi Rob,

Is your refbrain image still in the right place? The refbrain folder seems to be empty. Best,

Greg.

Hey Greg,

Wow I feel embarrassed. I am running these registrations on a cluster computer and they have monthly maintenance purges. I believe those purges deleted my reference brain. 

I will let you know if that fixes the problem.

Thanks again,
Rob

Hey Greg,

So I fixed the reference brain issue but some brains register and others are still stalling at the first affine step. For instance I have two stacks from the same experiment with the same properties, the first one (shank3abCTH24_01; shank3abCTH24_02) registered fine but the second (shank3abCTH25_01;shank3abCTH25_02) stalled. I cannot figure out the difference.

Thanks,
Rob

Hi Rob,

Is it possible that you have some .lock files in the folder for shank3abCTH25 after interrupting previous runs?

If not then I suggest attaching the results of


again. Incidentally lower case l in -lR gives more information.

Best,

Greg.

Hey Greg,

Sorry for the late reply. Here is the ls -lR from a directory with the same problem.

[r.kozol1@pegasus CMTKreg]$ ls -lR
.:
total 96
drwxrwxrwx 2 r.kozol1 ccsuser 512 Jun 16 23:35 commands
drwxr-xr-x 2 r.kozol1 ccsuser 32768 Jun 27 19:06 images
drwxr-xr-x 2 r.kozol1 ccsuser 32768 Jun 28 00:44 Jobs
drwxr-xr-x 2 r.kozol1 ccsuser 512 Jun 19 21:13 refbrain
drwxr-xr-x 2 r.kozol1 ccsuser 32768 Jun 24 01:26 reformatted
drwxr-xr-x 4 r.kozol1 ccsuser 512 May 15 15:15 Registration

./commands:
total 0
-rwxr-xr-x 1 r.kozol1 ccsuser 320 Jun 13 13:47 munger_2018-05-18_1.sh
./images:
total 424544
-rw-r--r-- 1 r.kozol1 ccsuser 216546431 Jun 27 19:19 WTBG1_01.nrrd
-rw-r--r-- 1 r.kozol1 ccsuser 218154261 Jun 27 19:15 WTBG1_02.nrrd
./Jobs:
total 512
-rw-r--r-- 1 r.kozol1 ccsuser 1331 May 27 14:12 16361889.err
-rw-r--r-- 1 r.kozol1 ccsuser 3722 May 27 14:12 16361889.out
-rw-r--r-- 1 r.kozol1 ccsuser 2150 May 27 17:58 16361964.err
-rw-r--r-- 1 r.kozol1 ccsuser 1595 May 27 17:58 16361964.out
-rw-r--r-- 1 r.kozol1 ccsuser 826 May 27 20:47 16361978.err
-rw-r--r-- 1 r.kozol1 ccsuser 2559 May 27 20:47 16361978.out
-rw-r--r-- 1 r.kozol1 ccsuser 535 May 28 00:53 16362015.err
-rw-r--r-- 1 r.kozol1 ccsuser 2692 May 28 00:53 16362015.out
-rw-r--r-- 1 r.kozol1 ccsuser 1331 May 28 09:42 16362027.err
-rw-r--r-- 1 r.kozol1 ccsuser 3720 May 28 09:42 16362027.out
-rw-r--r-- 1 r.kozol1 ccsuser 330 May 28 16:18 16362111.err
-rw-r--r-- 1 r.kozol1 ccsuser 307 May 28 16:18 16362111.out
-rw-r--r-- 1 r.kozol1 ccsuser 330 May 28 20:41 16362164.err
-rw-r--r-- 1 r.kozol1 ccsuser 204 May 28 20:41 16362164.out
-rw-r--r-- 1 r.kozol1 ccsuser 1331 May 31 14:15 16373719.err
-rw-r--r-- 1 r.kozol1 ccsuser 3874 May 31 14:15 16373719.out
-rw-r--r-- 1 r.kozol1 ccsuser 1331 Jun 2 14:03 16391328.err
-rw-r--r-- 1 r.kozol1 ccsuser 3875 Jun 2 14:03 16391328.out
-rw-r--r-- 1 r.kozol1 ccsuser 1331 Jun 2 19:36 16394102.err
-rw-r--r-- 1 r.kozol1 ccsuser 3874 Jun 2 19:36 16394102.out
-rw-r--r-- 1 r.kozol1 ccsuser 330 Jun 4 19:40 16398549.err
-rw-r--r-- 1 r.kozol1 ccsuser 323 Jun 4 19:40 16398549.out
-rw-r--r-- 1 r.kozol1 ccsuser 330 Jun 6 15:05 16409110.err
-rw-r--r-- 1 r.kozol1 ccsuser 212 Jun 6 15:05 16409110.out
-rw-r--r-- 1 r.kozol1 ccsuser 330 Jun 7 02:30 16410258.err
-rw-r--r-- 1 r.kozol1 ccsuser 212 Jun 7 02:30 16410258.out
-rw-r--r-- 1 r.kozol1 ccsuser 330 Jun 7 15:56 16412094.err
-rw-r--r-- 1 r.kozol1 ccsuser 323 Jun 7 15:56 16412094.out
-rw-r--r-- 1 r.kozol1 ccsuser 39 Jun 15 15:50 16430999.err
-rw-r--r-- 1 r.kozol1 ccsuser 335 Jun 15 15:50 16430999.out
-rw-r--r-- 1 r.kozol1 ccsuser 330 Jun 28 00:44 16632149.err
-rw-r--r-- 1 r.kozol1 ccsuser 305 Jun 28 00:44 16632149.out
-rwxr-xr-x 1 r.kozol1 ccsuser 279 Jun 13 13:47 munger_2018-05-18_1.job
./refbrain:
total 235360
-rw-r--r-- 1 r.kozol1 ccsuser 240983040 Jun 19 21:23 Ref20131120pt14pl2.nrrd
./reformatted:
total 600704
-rw-r--r-- 1 r.kozol1 ccsuser 70134669 Jun 2 14:02 Ref20131120pt14pl2_shank3abCTH201_01_warp_m0g80c8e1e-1x52r3.nrrd
-rw-r--r-- 1 r.kozol1 ccsuser 67580854 Jun 2 14:03 Ref20131120pt14pl2_shank3abCTH201_02_warp_m0g80c8e1e-1x52r3.nrrd
-rw-r--r-- 1 r.kozol1 ccsuser 79469785 Jun 2 19:35 Ref20131120pt14pl2_shank3abCTPTZ1_01_warp_m0g80c8e1e-1x52r3.nrrd
-rw-r--r-- 1 r.kozol1 ccsuser 78265315 Jun 2 19:36 Ref20131120pt14pl2_shank3abCTPTZ1_02_warp_m0g80c8e1e-1x52r3.nrrd
-rw-r--r-- 1 r.kozol1 ccsuser 60518106 May 31 14:14 Ref20131120pt14pl2_shank3abNTH201_01_warp_m0g80c8e1e-1x52r3.nrrd
-rw-r--r-- 1 r.kozol1 ccsuser 65029315 May 31 14:15 Ref20131120pt14pl2_shank3abNTH201_02_warp_m0g80c8e1e-1x52r3.nrrd
-rw-r--r-- 1 r.kozol1 ccsuser 42868805 May 27 14:12 Ref20131120pt14pl2_WTH201_01_warp_m0g80c8e1e-1x52r3.nrrd
-rw-r--r-- 1 r.kozol1 ccsuser 37691353 May 27 14:12 Ref20131120pt14pl2_WTH201_02_warp_m0g80c8e1e-1x52r3.nrrd
-rw-r--r-- 1 r.kozol1 ccsuser 58355215 May 28 09:41 Ref20131120pt14pl2_WTPTZ6_01_warp_m0g80c8e1e-1x52r3.nrrd
-rw-r--r-- 1 r.kozol1 ccsuser 55030973 May 28 09:42 Ref20131120pt14pl2_WTPTZ6_02_warp_m0g80c8e1e-1x52r3.nrrd
./Registration:
total 64
drwxr-xr-x 3 r.kozol1 ccsuser 32768 Jun 27 20:14 affine
drwxr-xr-x 2 r.kozol1 ccsuser 32768 Jun 16 13:24 warp
./Registration/affine:
total 0
drwxr-xr-x 2 r.kozol1 ccsuser 512 Jun 28 00:44 Ref20131120pt14pl2_WTBG1_01_9dof.list
./Registration/affine/Ref20131120pt14pl2_WTBG1_01_9dof.list:
total 0
-rw-r--r-- 1 r.kozol1 ccsuser 229 Jun 27 20:14 cmd.sh
./Registration/warp:
total 0

I don’t see anything obvious. Maybe send the jobs and registration folders and perhaps also check the nrrd header (using unu head or Fiji show image info). Best, Greg.

Here are the Jobs and registration files.

You have log files for 3 jobs there

• 16632149
• 16639860
• 16643252

So you can see that the job is being aborted. If you didn't do this manually then one possibility is that your cluster job manager is aborting the job, perhaps because it is exceeding some system resource (memory, compute time). The two WARNING lines you see there are harmless but do indicate that cmtk registration has loaded the first image (the reference image I think) successfully. I wonder if it is possible that there is something corrupt about your sample image – either that it doesn't open properly or that the spatial calibration is very wrong. I suggest that you look at the nrrd header information using ImageJ's Image ... Show Info command.

Best,

Greg.

Sorry I should have explained those output files. Yes you are right that the cluster aborts the files due to the time limits I put in the job file. However, I have tried giving them more and more time with no results. Usually one brain with two .nrrd files goes through an affine, warp and two registrations in about two hours. Also I have run these on my computer with the same results, CMTK starts, creates an affine directory and then stops. I have left a terminal up for days with no change. 

As for the headers, I have tried and cannot determine anything specific. for example here is one that did not work

NRRD0004
# Created by Nrrd_Writer at Thu Jun 28 15:09:56 EDT 2018
type: uint8
encoding: raw
endian: big
dimension: 3
sizes: 1030 1411 138
space dimension: 3
space directions: (0.891265,0,0) (0,0.891265,0) (0,0,1.976)
space units: "microns" "microns" "microns"
------------------------------------------------------
(Fiji Is Just) ImageJ 2.0.0-rc-67/1.52d; Java 1.8.0_172 [64-bit]; Mac OS X 10.13.5; 360MB of 5333MB (6%)

Title: WTBG5_01.nrrd
Width: 918.0029 microns (1030)
Height: 1228.1632 microns (1378)
Depth: 272.688 microns (138)
Size: 187MB
Resolution: 1.1220 pixels per microns
Voxel size: 0.8913x0.8913x1.976 microns^3
ID: -2
Bits per pixel: 8 (grayscale LUT)
Display range: 0-255
Image: 1/138
No threshold
Magnification: 0.50
ScaleToFit: false
Uncalibrated
Path: /Users/robkozol/Desktop/WT_MS222_06/WT_BG_062518/WTBG5_01.nrrd
Screen location: 12,23 (1440x900)
SetMenuBarCount: 4 (261ms)
Coordinate origin: 0,0,0
No overlay
No selection

Here is one that worked

NRRD0004
# Created by Nrrd_Writer at Wed Jun 27 12:44:09 EDT 2018
type: uint8
encoding: raw
endian: big
dimension: 3
sizes: 1030 1381 138
space dimension: 3
space directions: (0.891265,0,0) (0,0.891265,0) (0,0,1.976)
space units: "microns" "microns" "microns"
------------------------------------------------------
(Fiji Is Just) ImageJ 2.0.0-rc-67/1.52d; Java 1.8.0_172 [64-bit]; Mac OS X 10.13.5; 1853MB of 5333MB (34%)

Title: WTBG4_01.nrrd
Width: 917.1117 microns (1029)
Height: 1214.7942 microns (1363)
Depth: 272.688 microns (138)
Size: 185MB
Resolution: 1.1220 pixels per microns
Voxel size: 0.8913x0.8913x1.976 microns^3
ID: -5
Bits per pixel: 8 (grayscale LUT)
Display range: 0-255
Image: 91/138
No threshold
Magnification: 0.50
ScaleToFit: false
Uncalibrated
Path: /Users/robkozol/Desktop/WT_MS222_06/WT_BG_062518/WTBG4_01.nrrd
Screen location: -40,72 (1440x900)
SetMenuBarCount: 49 (269ms)
Coordinate origin: 0,0,0
No overlay
No selection

The only difference I see are the height sizes because I am stitching two images together, but that doesn't seem to affect those that work (e.g. files are shorter and longer than the reference and work fine). 

Otherwise I agree that there is something odd about the files, but I cannot seem to figure it out.

Thanks,
Rob

Hi Rob,

Is it possible that some of your images are truncated? It seems that you are saving them uncompressed (ie encoding raw) but I notice that for one of your broken files the channel 1 and 2 images are different sizes:

-rw-r--r-- 1 r.kozol1 ccsuser 216546431 Jun 27 19:19 WTBG1_01.nrrd
-rw-r--r-- 1 r.kozol1 ccsuser 218154261 Jun 27 19:15 WTBG1_02.nrrd

The difference of 1,607,830 bytes does not seem compatible with a small difference in the header. Fiji will actually read truncated images without complaining, but cmtk/the nrrd libraries will need the whole image. I suggest you share some of your images offline.

Best,

Greg.