Dear Alfonso

I have a question about the meaning of the effect size (the y-axis in the REX Results GUI). Is this effect size in the sense of cohen's d? Or how should the effect size value be interpreted? Because we get nice significant results, but with a very small effect size - as you can see on the picture I added below.

The analysis we did, was thanks to your help and is described here in detail: https://www.nitrc.org/forum/forum.php?th...

Kind regards
Julian

Dear Julian,

Generally effect sizes in REX displays correspond to contrast values, or linear combinations of regressor coefficients, from your secod-level general linear model. For your analyses, looking at patient-control differences in connectivity, those effect sizes (approximately 0.05 in your results)  will be interpretable as average differences in Fisher-transformed correlation values between the patients and control groups. If you prefer to report Cohen's d, in your case that can be easily computed from your analysis T-stats and dofs as d = T / sqrt(dof) (e.g. T=4.11 and dof probably 54 or 52, not totally sure, so Cohen's d in this case is going to be around 0.5 or a "medium-size" effect)

Hope this helps
Alfonso
Originally posted by Julian Roessler:

Dear Alfonso

I have another question about the meaning of the sign of the effect size in the REX displays:
In the seed to voxel analysis "Treatment : Placebo : Score_Total_Treatment : Score_Total_Placebo [1 -1 0 0]"  we found significant positive connectivity to a particular ROI.
Then, our symptomscores can be divided into two subscores. So using the REXtoolbox we did the analysis for each subscore:

(A) Starting with the ROI from our seed2voxel analyis (see above) we opened the REX GUI for this particular ROI and loaded the SPM file of our 2nd level analysis for "Treatment : Placebo : subscore1_Treatment : subscore1_Placebo [0 0 1 -1]", here we got a significant positive effect.

(B) Same as above for: "Treatment : Placebo : subscore2_Treatment : subscore2_Placebo [0 0 1 -1]" here we got a significant negative effect.

How can we interpret the results?
Does the positive effect in (A) mean that the connectivity is even more positive for the subscore1 in interaction with the treatment, and the negative value in (B) that this effect is a less positive in interaction with subscore2,
OR that the connectivity is now negative in interaction with subscore 2 (and thus reflects an anticorrelation for those who score high on subscore2 under the influence of the treatment)?

Kind regards 
Julian

Dear Julian,

The positive effect sizes in the original "treatment/placebo/score_treatment/score_placebo [1 -1 0 0]" analysis indicate that, at the same level of the score covariate in the two groups, connectivity is higher (either stronger positive connectivity or weaker anticorrelations) in treatment patients compared to controls. 

The positive effect sizes in the (A) analyses (treatment/placebo/score1_treatment/score1_placebo [0 0 1 -1]) indicate that the association between score1 and connectivity values is higher (more positive association or less negative association) in treatment patients compared to controls, while the negative effect sizes in the (B) analyses (treatment/placebo/score2_treatment/score2_placebo [0 0 1 -1]) indicate that the association between score2 and connectivity values is higher (more positive association or less negative association) in controls compared to treatment patients.

There is some information missing to be able to fully interpret these results (e.g. you want to know whether you are talking about positive connectivity or anti-correlations to begin with, and also whether the associations between score and connectivity values are positive or negative). In order to get all of that info I would suggest:

1) in the original seed-to-voxel analysis results explorer window select "import values" to import the actual connectivity values for each subject within your suprathreshold cluster. This will be stored as a new second-level covariate (e.g. named "ConnectivityValues")

2) go to Tools.Calculator to explore these values and their associations with the score covariates within each group. For example, selecting in the "predictor variables" list the 'treatment' and 'score1_treatment' covariates (and entering a [0 1] contrast), while selecting in the "outcome variables" list the new 'ConnectivityValues' covariate, will create a scatter plot showing the association between the score1_treatment values and the connectivity values within treatment patients only (so you can infer from this plot whether the connectivity values are actually positive or negative, and whether the association between connectivity and score1 values is positive or negative, which will help you better interpret the differences between groups that you observed)

Hope this helps
Alfonso
Originally posted by Julian Roessler:

Dear Alfonso

thank you for your quick and detailed reply! We did the additional analyses and found the following results for the scatter plots (see attached file). From this can we conclude now that: 
(high scorers = more symptoms, low scorers=less symptoms)

1.a. The low scorers (in both subscore groups) show anticorrelation under placebo
1.b. The high scorers (in both subscore groups) show less anticorrelation under placebo

2.a. The low scorers (in both subscore groups) show less anticorrelation under treatment
2.b. The high scorers in subscore1 show less anticorrelation under treament and the high scorers in subscore2 show more antiorrelation

3. This means the treatment seems disrupt the anticorrelation in low scorers, but seems to be beneficial for high scorers from subscore2 - but not for high scorers from subscore1. But with the interpretation of the REX results effect sizes from the last post, one could derive that the treatment only affects the subscore1. --> We are still unsure how to connect all the information?

Kind Regards 
Julian

Hello Alfonso and Juian, 

wound't then be that the effect size represents beta values?

I came across an old post from K Friston saying "I am using the term 'effect size' in reference to the
'size of the effect' (i.e. a measure of the modeled effect that
reflects the parameter estimate as opposed to the statistic that is
used for inference."
https://www.jiscmail.ac.uk/cgi-bin/wa.ex...


Can that be it?

Thanks, 

Athena

Dear Alfonso 

I have a follow up question, but first I want to thank you again for your previous help. We ran the analyses as you recommended. Now a reviewer pointed out that our analysis as above indicates that our treatment effect are relative changes in connectivity and symptom scores. He/she asked us to give more explanations form a mathematical perspective about fisher transformation correlation values and statistical results.

Where do I find such values?

Kind Regards
Julian

Dear Alfonso,

Could you please clarify for a non-specialist in statistics,

As far as I have understood, you are writing about Cohen's d in comparison of 2 independent samples (patients vs. controls). Why is not Cohen's d computed in the following way: d = 2T / sqrt(dof)? Everywhere in the literature T is multiplied by 2 in independent samples T-test, e.g.:

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4:863. doi:10.3389/fpsyg.2013.00863

Lots of excuses if the question is strange. Thank you very much for your tool, it's very helpful.

Yours sincerely,
Yana


Originally posted by Alfonso Nieto-Castanon:

Dear Yana,

You are totally right, thanks for pointing that out, for a two sample t-test the Cohen's d effect-size can be computed using the general formula in https://www.nitrc.org/forum/message.php?msg_id=33850, which, for equal-size groups, will be approximately twice the value I was describing in the example in my previous message (the |T|/sqrt(dof) value in my previous message corresponds to the Cohen's f effect-size, which are the ones that one would generally use in the context of an arbitrary GLM analysis; and when used to describe a two-sample t-test as implemented in GLM Cohen_d ~= 2 Cohen_f)

Hope this helps, and thanks again
Alfonso
Originally posted by Yana Panikratova:

Dear Alfonso,

Thank you very much for your kind reply, it's very helpful!

I have a follow-up question. When we compare regression between 2 groups (using one-way ANCOVA covariate interaction: 0 0 1 -1 for group1, group2, covariateForGroup1, covariateForGroup2), what measure of effect size is applicable? Can we use Cohen's f = |T|/sqrt(dof) in this case? 

Grateful for your help,

Yana