# Multiple-Comparison Issue

• When you test multiple times, you should not believe nominal p-values of individual tests. 検定を複数行った場合には、個々の検定のp値をそのまま使って解釈できない
```p <- runif(10^5)
hist(p)
plot(sort(p),pch=20,cex=0.1)
alpha <- 0.05
abline(v=length(p)*alpha,col="red")
abline(h=alpha,col="blue")
```
• Multiple tests (100 tests) 100個の検定
```n <- 100
m <- 1000
ps <- matrix(runif(n*m),n,m)
min.ps <- apply(ps,2,min)
hist(min.ps)
plot(sort(min.ps),pch=20,cex=0.1)
alpha <- 0.05
abline(v=length(min.ps)*alpha,col="red")
abline(h=alpha,col="blue")
```
• False Disovery Rate
• q-value not p-value
• Some tests should be "significant"; many genes should be significantly diffrently expressed between cancer and normal cells.
• The smallest p-value should be small enough, but the 2nd, 3rd,... smallest could be not so small and they should be judged by different threshold.
```n.null <- 80
n.alt <- 20
p <- runif(n.null)
p <- c(p,pchisq(rchisq(n.alt,df=1,ncp=10),df=1,lower.tail=FALSE))
p <- sort(p)
plot(p)
```plot(q,pch=20)