pairwise comparison in adonis gives the same p-value

[Puumanamana]

Hi,

I'm using permanova tests to evaluate the significance of a given factor in my dataset. Everything runs but I get surprising results, and I'm wondering if it's correct or if there could potentially be an issue with either the package or with the data itself.

Using adonis, I was able to test that my factor ("color") was significant. However, I want to have more details about the significance and compare each pair of levels: For example, I want to test whether the sample in the "Red" color group are statistically different from the samples in the "Purple" color group.

To do that, I extract from my data all samples corresponding to 2 given colors, and I run adonis on the subset:

condition1 <- (metadata$color == 'Blue') | (metadata$color == 'Purple')
adonis(otu_table[condition1, ] ~ metadata$color[condition1], perm=50)

condition2 <- (metadata$color == 'Blue') | (metadata$color == 'Red')
adonis(otu_table[condition2, ] ~ metadata$color[condition2], perm=50)

condition3 <- (metadata$color == 'Red') | (metadata$color == 'Purple')
adonis(otu_table[condition3, ] ~ metadata$color[condition3], perm=50)

My problem is that for all 3 comparisons, I get exactly the same p-value, 0.01961 (everything else is different: R2, F.model, ...). It looks very suspicious to me since I'm comparing different sample groups.
Do you have an idea what could cause this issue?
I also tried the pairwise.adonis() package to do these pairwise comparisons but I got the same problem (and exactly the same p-value).

Thank you for your help

[klv501]

Hi,
I am also having this problem, I have two factors, and when I subset so that I have data with only one level for one of the factors and then test for significant differences between the other factor levels, I always get the same (insignificant) p-value no matter which subset of different data I am using. The further odd thing is that the overall adonis test was highly significant in the first place and all the pairwise comparisons come out as insignificant.

Have you had any luck solving your problem?

[jarioksa]

We haven't found anything reproducible here and therefore nothing has happened (neither any signs of problems).

What is obvious in the original post is that there are only 50 permutations, and none of these was better than the observed in any of these models. The smallest possible P-value is 1/(nperm+1) which for nperm=50 is 1/51 = 0.01960784. It may be that with higher number of permutations you may get different P-values. Try with permutations = 999, for instance.

[klv501]

Hi Jarioksa,

Thanks for your comment, I have been running it with 999 permutations and have just noticed that I get the following message when I run it: 'nperm' >= set of all permutations: complete enumeration.
Set of permutations < 'minperm'. Generating entire set.
(this is repeated three times - probably for each pairwise comparison).
My sample size is very small (9 in total with 3 replicates per level) because I had to subset to avoid other problems. Do you know if this warning message could be causing my strange p-values and is it because of my sample size?
Thanks!

[gavinsimpson]

@klv501 Those aren't warnings, they're messages from the software telling you that i) the number of possible permutations of the data is fewer than the requested 999 permutations, and ii) that the number of possible permutations is smaller than the minperm argument. In both cases the result is that we generate the complete set of permutations of the data and use all of them to assess the results. With so few permutations, it is better to do an exact test rather than generate random permutations as many of the randomly generated permutations would be the same permutation. It is wasteful to do this randomly.

It may well be that a permutation test for a single pair of values has lower power than the omnibus test over all levels, because there are fewer permutations of the smaller subset of data involved in the pair being compared.

In a sense, this is one of the negatives of using permutation-based tests.

[emudrak]

I have a client with this same problem. We have 4 strata, each with 8 plots, and each plot has one of 8 treatments. The overall tests shows significant difference between the 8 treatments:
adonis(weeds ~ Treatment, data = dat, strata=Block)
(this is significant whether or not we include the Block as strata).
But when we use pairwise.adonis with the strata argument,
pairwise.adonis2(weeds ~ Treatment, data = dat, strata="Block")
The blocking reduces the number of possible permutations for a pair of treatments to 16, and none of the pairs are significant.
Running this code without the strata argument releases the restriction and increases the number of possible permutations, and we end up with significant results. But the blocking is part of the design so we feel we should include it.
It was my understanding that including the blocking term would increase the power to test for differences by treatment (by essentially removing block-specific variation from the picture), but the blocking term seems to be a hindrance to the pairwise tests by restriction the number of possible permutations. Is my understanding correct? How do we reconcile this?

[gavinsimpson]

Doing this via strata is no longer the recommended route. You should use permute::how() to define the restrictions on the permutation test and pass the result of that to the permutations argument of adonis2(). I don't recall just now if that works with adonis().

I can't speak to the pairwise stuff as that's a non-vegan function.

Given the number of questions we get asked about this and the issues that other people's software seems to generate for us vegan developers, it might actually be worth my spending a bit of time implementing a pairwise option, but how to do this meaningfully for the general models adonis2() supports (i.e. linear terms, interactions) is a little more complex than what I did for betadisper().

[elisingularis]

Hello, since I'm having the same trouble you addressed in this discussion one year ago, I wonder if there is any updates to solve it. I'm having significant differences among bacterial communities diversity of different samples when I'm applying the adonis function. But then when I run the pairwise.adonis function I lose any significant differences. Here below the scripts I'm using

meso_tr <- microbiome::transform(Meso, "clr")
meso_dist_matrix <- phyloseq::distance(Meso, method ="bray")
vegan::adonis2(meso_dist_matrix ~ phyloseq::sample_data(meso_tr)$Site)
pairwise.adonis(meso_dist_matrix, phyloseq::sample_data(meso_tr)$Site)

Thanks for any help you can provide!

[jarioksa]

@elisingularis From the vegan point of view there is no problem and certainly nothing to be fixed. You have the problem that you do not have sufficient data for your test. adonis2 uses permutation tests, and the smallest possible P-value is defined by the number of permutations, and if you have only a couple of distinct permutations, the smallest possible P-value is necessarily high, and many P-values may be identical. If you have a single class variable, the number of distinct permutations is determined by the number of distinct samples you can pick from that variable. The number of distinct samples still goes down with pairwise.adonis where you pick just two levels for each pairwise comparison. However, pairwise.adonis is not vegan function, and we are unable to help in its use (because we do not know that function very well).

[klv501]

@gavinsimpson Thanks for your reply, I think that my small sample size from sub-setting is definitely the issue. Sorry, i'm very new to this type of multivariate analysis - would you be able to recommend an exact test for this as you mention? I don't believe that a MANOVA would be appropriate as I have many OTUs (dependent variables), many with 0 relative abundance values.
Thanks!

[gavinsimpson]

Your code is already doing an exact permutation test for you; those messages are telling you that we're generating the full set of permutations and using them all for the test. As such there is no randomness to the permutation test depending on which permutations of the data are formed; all possible permutations were formed. In that sense it is exact.

[Puumanamana]

@jarioksa Thank you. I actually tried many values for nperm (if think up to 1000 or more), and the p-values were always the same. The reason is (I think) that my groups are really homogeneous and any permutations yielded a worse score than the original order.

[jarioksa]

It is not rare to get the lowest possible P-value in permutation tests. When people use 999 permutations and get P = 0.001, they easily accept this. If the number of permutations +1 is less even, they are alerted for the strange values. So 1000 permutations has a lower limit of 0.00099901 which seems to beg for an explanation. With small data sets – or reduced subsets of data – you are bound the get more often weird probabilities at the lower limit.

[jarioksa]

Another case with abundant ties in P-values is with factor predictors. We permute observations, but if predictor variable is a factor, many permutations give identical allocations to factor classes. In that case the number of distinct allocations can be much lower than the number of permutations. This concerns particularly models with one factor predictor – with several factors you already have more distinct combinations of factor levels.