Hi Alfonso,

Thank you for looking into this so quickly and for getting back with such a detailed response!

Your results seem to align with what we found for the resting-state analyses on our data. Using the Schaefer 400 parcellation, we were also not finding great differences in optimal range between the two different calculations. 

Unfortunately, a lot of the work we did was on a 110-ROI network, looking at task-based functional connectivity using the gPPI analysis (which might be wrong for different reasons, happy to hear your thoughts on that too). We were hoping to look at changes in network topology based on two task conditions. There, the optimal range with the correct formula would probably be 0.09 to 0.13, but the range we picked with the incorrect formula was 0.30 to 0.40. I've attached the two graphs.

1. Is there anything wrong with using graph theory on a task-based network analysed with gPPI? I'm aware that the matrix is not symmetrical in gPPI, but I thought that it wouldn't matter at the higher threshold range we were looking at.
   
2.  The results we find are quite different between the two threshold ranges: nothing is significant at the 'optimal' 0.09 to 0.13 range, but we find some interesting things that line up with theory at the 0.30 to 0.40 range. To salvage this, we were hoping to argue that we looked at the 0.20 to 0.40 range (which has very similar results to the 0.30 to 0.40 range) as a secondary more stable plateau (wider, higher AUC). The differences between the two conditions we're looking at are also very subtle (which we expected), so the higher threshold range is more likely to capture the differences. Do you think this is a line of argumentation worth pursuing or is it not convincing at all?
   

Thanks a lot!

Nic

Originally posted by Alfonso Nieto-Castanon:

Hi Nic,

I have looked at this in a bit more detail and you were exactly right. The computation of the local efficiency plus global efficiency reference curve being displayed by the graph theory GUI when the user entered an empty value in the "edge threshold" field (used to find a reasonable range of high local and global efficiency that would indicate a reasonable range of cost threhsold values to use) had the bug that you described and was not displaying the correct values. I have updated a fix to the CONN repository @ github and I will be creating a new updated version @ NITRC as well soon.

To get a closer look at the impact of this error I looked at the difference between the intended curve (showing local efficiency of the graph compared to that of a random graph plus the global efficiency of the graph compared to that of a lattice with the same number of edges) and the one that was being incorrectly displayed by CONN and, fortunately, they appear very similar. I tested this in three different datasets (the NYU test-retest dataset N=25, with three resting state sessions per subject, the FCP1000 Cambridge dataset N=198 with one resting state session per subject, and the HCP test-retest dataset N=43 with eight resting state sessions per subject) and using a couple of different ROI parcellations (the Schaefer parcellation with 400 ROIs, and the Harvard-Oxford atlas with 132 ROIs), and for each dataset I displayed the two LE+GE curves superimposed (in the result figures attached, the curves in red are the incorrect ones, previously displayed by CONN, and the gray curves in the same plots are the new/correct ones, displayed by CONN after the patch). The ranges of recommended/reasonable cost threshold values that I would derive form the two curves are very similar if not identical at least in these scenarios that I have tested so far (e.g. something like 0.03-0.07 for the Schaefer atlas, and something like 0.05-0.15 for the Harvard-Oxford atlas, similarly across all datasets). If possible, could you take a look at the new code on your own data and let me know how the suggested range of cost threhsold values look when comparing the old version with the new one? (in the code I left in line 136 an "if 0" line, if you change it to "if 1" it will compute the old/incorrect values and will display those results as red curves for comparison, as in the figures that I've attached)

Let me know your thoughts, and again thanks a lot for finding and reporting this issue!

Alfonso

 

Originally posted by Alfonso Nieto-Castanon:

Hi Nic,

I think you are totally right, from a first look it seems that those lines should read instead:

        mnet_d_1=mean(x{1},1)-mean(x0{1}{2},1); %title('Difference with lattice')
        mnet_d_2=mean(x{2},1)-mean(x0{2}{1},1); %title('Difference with random');

I will take a closer look and let you know but very good catch and thanks a lot for the help!!

Alfonso

Originally posted by Nicolas Deom:

Hi all,

I believe there is an issue in the code of the conn_network.m function which calculates and plots global efficiency and local efficiency at different threshold ranges and compares with a lattice and random network to help with the decision of picking a cost threshold. I think that the numbers are correct for the first two plots, the problem is in the last one, which shows the optimization for small-world network properties. In the script for the conn_network function, on lines 136 to 138, where the values seem to be calculated for this plot, lines 136 and 137 both use the variable 'mnet_d' which has its value set to local efficiency data at the end of the previous loop (lines 112 to 134). This means that the graph actually plots LE_{data}-GE_{lattice}+LE_{data}-LE_{random} instead of the intended GE_{data}-GE_{lattice}+LE_{data}-LE_{random}. Looking at the numbers from my data seems to confirm this when trying to manually calculate the values in the graph: using the expected formula does not exactly align, while the unintended formula does.

In light of this, I have two questions:

1. Am I reading and understanding the script correctly? Is there indeed an error there?
   
2. What are the implications for work carried out that used a threshold range based on the incorrect formula? Has it been optimized for a basically meaningless value and is therefore uninterpretable, particularly if the threshold selected based on the two versions of the formula widely differ?
   

Thanks for your help!

Nic