Zhi,

in the TeraConverter command-line tool, the output format is not tiled by default. Which means that one single (very big) image will be produced. For TIFF, which can write up to 4 GB files, it is simply too big in your case. Vaa3D raw should handle it, but somehow it generates an error (the error comes from the Vaa3D raw code...).

To solve these issues, you just have to enable tiling. You can set tile dimensions along x, y, and z with the --width, --height, and --depth parameters respectively. I suggest to set them to 256 pixels each. I understand this might not be very clear since there is no documentation for the TeraConverter command-line. That's why there is the TeraConverter GUI :-).