GEO data mining / R and Bioconductor

GEO Data Mining Tutorial

When you have a GEO accession number for the first time, it is often unclear which file to download, whether the values need log2, and how to match groups to samples. This page uses the pancreatic cancer microarray dataset GSE16515 as an example, from download to differentially expressed genes, a volcano plot and a heatmap. Each step gives its purpose and a way to check the result; errors are collected in the troubleshooting section at the end.

Short answer

To run a differential analysis on GEO data, complete four steps: download the Series Matrix expression matrix, check whether the values are already on a log2 scale, convert probes to gene symbols, and extract the groups from the sample information and check that the sample order matches. Use limma to compare two groups for microarray data; raw RNA-seq counts need a count-based method. A group mismatch raises no error and only gives wrong results, so check each step.

About this page

Tutorial
What this page solves
You have found a pancreatic cancer microarray dataset on GEO and want to compare gene expression between tumor and normal tissue. You do not know which file to download, whether the values need log2, or how to assign each sample to its group.
Who it is for
Medical graduate students and researchers who are doing a differential analysis on GEO data for the first time. They know some R but have not analysed microarray data before.
What you will have
You will end up with an expression matrix annotated with gene symbols, a full limma result table for all genes, a volcano plot and a heatmap, and checks for whether the groups and annotation are correct.
Covers
  • Download the Series Matrix of a microarray dataset from GEO and decide whether the values need log2.
  • Convert probes to gene symbols, run a two-group limma analysis, and draw a volcano plot and a heatmap.
  • Common problems such as failed downloads, group mismatches and annotation conflicts (see the troubleshooting section).
  • Approaches for RNA-seq data, merging several datasets and using TCGA as a control (see the choices section).
Does not cover
  • Other analyses such as single-cell analysis and WGCNA; see the related guides.
  • Enrichment analysis (GO, KEGG, GSEA); see the differential expression and enrichment tutorial.
  • Wet-lab validation such as qPCR and immunohistochemistry.
  • Survival analysis and prognostic models built on GEO data.

Main steps

Step 1 · Choose the file

Why this stepOne GEO study (accession starting with GSE) usually comes with several files. Choosing the wrong one means redoing every later step, so first decide which file this example uses.

This example is the pancreatic cancer microarray dataset GSE16515, with 52 samples on platform GPL570. The platform is the chip model; it determines which genes each probe corresponds to.

For microarray data, download the Series Matrix file, whose name ends in _series_matrix.txt.gz. It is the expression matrix processed by the submitter, together with the sample information. For other file types, see the choice “What to do when the data is not a Series Matrix”.

Step 2 · Download

Why this stepEvery later step reads this file. Download it to a fixed folder so that you do not have to download it again after closing R.

The first download of the GSE16515 Series Matrix (6.4 MB) took 8.3 seconds. getGPL = FALSE skips the 85.6 MB GPL570 annotation file, because step 4 uses an annotation package instead.

If the download fails or returns something other than data, look up the exact error in the download entries of the troubleshooting section. On a poor connection, use the manual download method in the choices section.

Download the Series Matrix and cache it in a fixed folderr
library(GEOquery)
options(timeout = 600)                        # default is 60 s; large files get cut off
Sys.setenv(VROOM_CONNECTION_SIZE = 500000L)   # avoids the connection buffer error
dir.create("geo_cache", showWarnings = FALSE)

# Fixed cache directory: the default is tempdir(), which is wiped when R exits
gse <- getGEO("GSE16515", destdir = "geo_cache",
              GSEMatrix = TRUE, getGPL = FALSE)  # skip the 85.6 MB GPL570 annotation
length(gse)        # multi-platform Series return several ExpressionSets
eset <- gse[[1]]
dim(eset)          # 54613 probes x 52 samples
annotation(eset)   # "GPL570"

Step 3 · Preprocessing

Why this steplimma needs values on a log scale. A Series Matrix may already be log2-transformed or may contain linear values, so judge from the numbers themselves rather than from the description.

The GSE16515 description says GC-RMA, a method whose output is usually log2. However, the largest value in the matrix is 65045, so the values are actually linear.

The code below applies the GEO2R rule. When the result is TRUE, take log2 first, then check the distribution of each sample with a boxplot. The reason for this rule is in the explanation section.

GEO2R's log2 rule and a distribution checkr
library(limma)
ex <- exprs(eset)
# The rule GEO2R uses
qx <- as.numeric(quantile(ex, c(0, .25, .5, .75, .99, 1), na.rm = TRUE))
LogC <- (qx[5] > 100) || (qx[6] - qx[1] > 50 && qx[2] > 0)
qx; LogC
# GSE16515: 2.28 7.11 18.17 86.87 3283.25 65045 -> TRUE
if (LogC) { ex[ex <= 0] <- NaN; ex <- log2(ex) }

# Check distributions: sample medians in the boxplot should roughly line up
boxplot(ex, las = 2, outline = FALSE)
# Only if they clearly do not: quantile normalization (what GEO2R's Force normalization does)
# ex <- normalizeBetweenArrays(ex, method = "quantile")

Step 4 · Annotation

Why this stepMatrix rows are probe IDs, but plots and comparison with other datasets need gene symbols. One probe can match several genes and one gene can have several probes, so you need one consistent rule.

This example uses the annotation package hgu133plus2.db for GPL570. After removing probes without annotation and probes that match several genes, keep the probe with the highest mean expression for each gene.

For choosing an annotation package or GPL table, see the choice “Choose the annotation package by platform”. For why the handling of multiple probes changes the result, see the explanation section.

Probe to gene: drop multi-gene probes and keep one probe per gener
library(hgu133plus2.db)   # annotation package for GPL570; see the table for other platforms
pk <- AnnotationDbi::select(hgu133plus2.db, keys = rownames(ex),
                            columns = "SYMBOL", keytype = "PROBEID")
multi <- names(which(table(pk$PROBEID) > 1))          # probes mapping to several genes: 1528
pk <- pk[!is.na(pk$SYMBOL) & !pk$PROBEID %in% multi, ] # drop unannotated and multi-gene probes
ex <- ex[pk$PROBEID, ]; sym <- pk$SYMBOL               # 43112 probes, 20826 genes left

# Several probes per gene: keep the probe with the highest mean (11041 genes have >1 probe)
o <- order(sym, -rowMeans(ex))
keep <- o[!duplicated(sym[o])]
ex_gene <- ex[keep, ]; rownames(ex_gene) <- sym[keep]
# Alternative, averaging: ex_gene <- avereps(ex, ID = sym)

# Platforms without a package: use the GPL table (getGEO("GPLxxx") or .annot.gz on FTP)
# gpl  <- getGEO("GPL570", destdir = "geo_cache")
# anno <- Table(gpl)[, c("ID", "Gene Symbol")]
# anno <- anno[anno$`Gene Symbol` != "" & !grepl("///", anno$`Gene Symbol`), ]

Step 5 · Groups

Why this steplimma matches groups to columns by position. If the group vector is off by one, every result is wrong and no error is shown, so check each item after extraction.

Group information is usually stored in a column named like xxx:ch1, so use grep to find the column name first. In this example, the tissue:ch1 column has 16 normal and 36 tumor tissues.

The first level of the factor is the reference group, and the comparison is written as tumor relative to normal. If the sample count does not match the paper or the source of the groups is unclear, see the group entries in the troubleshooting section.

Extract groups and run three checksr
pd <- pData(eset)
grep(":ch1$", colnames(pd), value = TRUE)   # group labels usually live in xxx:ch1 columns
table(pd[["tissue:ch1"]])
# Normal Tissue in Pancreatic Cancer Sample 16 / Tumor Tissue ... 36

group <- factor(ifelse(grepl("Tumor", pd[["tissue:ch1"]]), "Tumor", "Normal"),
                levels = c("Normal", "Tumor"))   # first level is the reference
# Three checks: sample names match, no mismatches in the cross-table, group sizes match the paper
stopifnot(identical(rownames(pd), colnames(ex_gene)))
table(group, pd[["tissue:ch1"]])
patient <- sub("^Pancreatic Sample ?([0-9]*)-.*$", "\\1", pd$title)  # pairing, used later

Step 6 · Differential analysis

Why this steplimma fits a linear model for each gene across the two groups, estimating the fold change and giving an adjusted P value. The design and the contrast set the direction of the comparison, so write them down first.

design is the design matrix of the groups, and makeContrasts sets the direction of the comparison. This example is tumor minus normal, so logFC > 0 means higher expression in tumor.

The filter is adj.P < 0.05 and |logFC| > 1. For the reason for these thresholds, other choices, and an equivalent way to write a two-group comparison, see the choices section.

limma analysis: two-group comparison (measured on this page)r
design <- model.matrix(~ 0 + group)
colnames(design) <- levels(group)                       # Normal, Tumor
contr <- makeContrasts(Tumor - Normal, levels = design) # logFC > 0 means up in tumor
fit  <- lmFit(ex_gene, design)
fit2 <- eBayes(contrasts.fit(fit, contr), trend = TRUE)
tt   <- topTable(fit2, number = Inf)                    # BH adjustment by default
deg  <- subset(tt, adj.P.Val < 0.05 & abs(logFC) > 1)
table(sign(deg$logFC))                                  # measured here: -1 399, 1 1136

Step 7 · Plots

Why this stepUse the plots to check whether the result makes sense. The volcano plot shows fold change against significance, and the heatmap shows whether the differential genes separate the two groups.

The volcano plot uses the same P value on its y axis as the filter (adj.P throughout this page), and labels the 10 genes with the smallest adj.P. The heatmap takes 25 up-regulated and 25 down-regulated genes, with samples ordered by group.

If the heatmap shows genes in only one direction, see the heatmap entry in the troubleshooting section. For plotting details, see the reference section.

Volcano plot and heatmap (GSE16515, plotting took 1.9 s)r
library(ggplot2); library(ggrepel); library(pheatmap)
v <- tt; v$gene <- rownames(v)
v$change <- with(v, ifelse(adj.P.Val < 0.05 & logFC > 1, "Up",
                    ifelse(adj.P.Val < 0.05 & logFC < -1, "Down", "NS")))
lab <- head(v[v$change != "NS", ][order(v$adj.P.Val[v$change != "NS"]), ], 10)
p <- ggplot(v, aes(logFC, -log10(adj.P.Val), colour = change)) +
  geom_point(size = 0.8, alpha = 0.6) +
  scale_colour_manual(values = c(Down = "#2b6cb0", NS = "grey75", Up = "#c53030")) +
  geom_vline(xintercept = c(-1, 1), linetype = 2) +
  geom_hline(yintercept = -log10(0.05), linetype = 2) +   # same P on the axis as in the filter
  geom_text_repel(data = lab, aes(label = gene), size = 3, colour = "black") +
  labs(x = "log2 fold change (Tumor vs Normal)", y = "-log10 adjusted P") + theme_bw()
ggsave("volcano.png", p, width = 6, height = 5, dpi = 300)

# Heatmap: the 25 up- and 25 down-regulated genes with the smallest adj.P
up <- head(rownames(v)[v$change == "Up"][order(v$adj.P.Val[v$change == "Up"])], 25)
dn <- head(rownames(v)[v$change == "Down"][order(v$adj.P.Val[v$change == "Down"])], 25)
o  <- order(group)
ann <- data.frame(group = group[o], row.names = colnames(ex_gene)[o])
pheatmap(ex_gene[c(up, dn), o], scale = "row", cluster_cols = FALSE,
         annotation_col = ann, show_colnames = FALSE, fontsize_row = 6,
         breaks = seq(-2, 2, length.out = 101),   # cap z-scores so outliers do not flatten the scale
         filename = "heatmap.png", width = 7, height = 8)

Choose by your situation

Hand it to an agent

The download, annotation, differential analysis and plotting steps are fixed, so they suit a scientific agent. Choosing the dataset, defining the groups and interpreting the results remain your responsibility.

Example instruction: “Run a differential analysis of tumor versus normal pancreatic tissue using GSE16515 and GSE15471. For each dataset, decide whether log2 is needed, annotate with hgu133plus2.db and keep the probe with the highest mean expression, run limma separately, then merge the matrices and put the batch in the design matrix. Use adj.P < 0.05 and |logFC| > 1. Output the intersection of the differential genes from both datasets, PCA before and after batch correction, a volcano plot, and a heatmap with 25 up and 25 down genes.”

  1. 01

    Download and check

    Download the Series Matrix with GEOquery and cache it; if the interface is blocked, switch to FTP. Record the sample count, the platform and the data_processing description.

  2. 02

    Preprocessing

    Decide on log2 with the GEO2R rule, draw a boxplot, complete the probe annotation, and report probe and gene counts before and after annotation.

  3. 03

    Differential analysis and plots

    Extract the groups and check the sample order, run limma, and output the full gene result table, the volcano plot and the heatmap.

  4. 04

    Merge and validate

    Merge the datasets with the batch in the design matrix; output the PCA and an overlap table of differential genes from the two datasets.

  5. 05

    Adversarial review

    Check that the groups match the paper, that the logFC direction is correct, that log2 was not applied twice, and that the heatmap includes both up- and down-regulated genes.

  • The project keeps the cached raw files, R scripts, sessionInfo, result tables and all figures.
  • You still need to check that the dataset fits your research question, that the groups and sample exclusions match the paper, and that the thresholds suit your paper.
  • Experimental validation such as qPCR and immunohistochemistry is up to you.

Choices

GEO also provides other file types. The table below shows when each one fits; first check which kind of data you have.

The NCBI-generated counts are produced as follows: HISAT2 aligns reads to GRCh38 (GCA_000001405.15), featureCounts counts them, and gene annotation is Annotation Release 109.20190905. Runs with alignment rates below 50% and single-cell data are not counted, and counts are summed when one GSM has several runs. GEO2R uses this same set of counts with DESeq2 for RNA-seq.

Species coverage: the official page (revised 2026-07-08) still says mouse data is “expected in 2025”. On 2026-10-11 this page searched E-utilities for "rnaseq counts"[Filter]: there were 0 Series with only mouse samples, and about 27,700 records with this label, all human.

GDS status: counting by year in E-utilities, 279 were added in 2014, 64 in 2015, 2 in 2016, and none since 2018. Old tutorials that use getGEO("GDSxxxx") only return old data.

FileContentsWhen to useWatch out
Series Matrix (GSE…_series_matrix.txt.gz)Submitter-processed expression matrix and sample metadataDefault for microarrays; GEO2R uses it tooThe scale is up to the submitter, so check for log2; multi-platform GSEs have several files
Raw CEL files (GSE…_RAW.tar)Raw Affymetrix scan signalsWhen Series Matrix values are odd or missing, or you want uniform RMA across datasetsRun RMA with oligo or affy; files are large and slow to download
NCBI-generated RNA-seq raw countsGene × sample integer matrix from NCBI's uniform alignment and countingHuman bulk RNA-seq where the submitter gave no counts or only FPKMHuman only for now; may not match the paper; the bundled FPKM and TPM are not for differential testing
Submitter supplementary files (suppl)Counts, FPKM, TPM or other formatsWhen you need to match the paper exactlyFormats vary; check column names and whether values are integers before reading
GDSCurated dataset with predefined groupsReproducing pre-2015 datasetsEssentially no new GDS since 2016; recent data have none

Choices

If the getGEO web interface is blocked or the download times out, download the file manually from FTP and read the local file.

A mirror commonly used in China is geoChina() from the AnnoProbe package by Biotrainee, which downloads the submitter-processed ExpressionSet (.Rdata) from a Tencent Cloud server. This page tested the server on 2026-10-11 and it was still reachable. Its GSE list is stored in the package, and the repository was last updated in November 2022, so datasets published later are not in the mirror.

GEOquery version difference: this page was measured with GEOquery 2.74.0 (Bioconductor 3.20). The development version on GitHub (2.81.x) plans to change the return type of getGEO from ExpressionSet to SummarizedExperiment; then exprs() and pData() must be replaced with assay() and colData().

Manual download with resume, then read locallybash
# FTP path rule: replace the last three digits of the GSE number with nnn
# (GSE16515 -> GSE16nnn; accessions below 1000 use GSEnnn)
# -c resumes partial downloads; rerun the same command after an interruption
wget -c https://ftp.ncbi.nlm.nih.gov/geo/series/GSE16nnn/GSE16515/matrix/GSE16515_series_matrix.txt.gz
# Platform annotation (GPL570 is below 1000, so the folder is GPLnnn)
wget -c https://ftp.ncbi.nlm.nih.gov/geo/platforms/GPLnnn/GPL570/annot/GPL570.annot.gz
# Raw CEL files
wget -c https://ftp.ncbi.nlm.nih.gov/geo/series/GSE16nnn/GSE16515/suppl/GSE16515_RAW.tar

# Read the local file in R without going online
Rscript -e 'library(GEOquery); e <- getGEO(filename = "GSE16515_series_matrix.txt.gz", getGPL = FALSE); print(dim(e))'

Choices

The annotation package is determined by the chip platform. If no matching package exists, use the GPL table.

Annotation packages are newer than GPL tables. On GPL570, the two disagree on 3874 of the 41039 single-gene probes, mostly because genes have been renamed, for example FAM122C is now PABIR3 and CCDC11 is now CFAP53. State the annotation source and version in the methods.

PlatformAnnotation packageNotes
GPL570 HG-U133 Plus 2.0hgu133plus2.db54,675 probes; the most common
GPL96 HG-U133Ahgu133a.db22,283 probes
GPL571 HG-U133A_2hgu133a2.db
GPL6244 HuGene 1.0 SThugene10sttranscriptcluster.dbAnnotated by transcript cluster
GPL6947 / GPL10558 Illumina HumanHT-12illuminaHumanv3.db / illuminaHumanv4.dbMatrix row names are ILMN_ probe IDs
Agilent and other platforms without a packageGPL table or .annot.gz on FTPThe Gene Symbol column may contain " /// ", meaning one probe maps to several genes

Choices

The threshold decides how many differential genes you get. Paired samples and fold-change thresholds each have their own code, so choose according to your design.

Reason for the thresholds: adj.P uses the BH method to control the false discovery rate, and 0.05 is the default in GEO2R. |logFC| > 1 is a convention with no statistical basis. The limma topTable help states that when P value and fold change are weakly correlated, filtering by P first and then by fold change lets the false discovery rate exceed the nominal level, and treat is recommended when a fold-change threshold is needed. If you use the conventional thresholds in a paper, state them and give the full gene table in the supplement.

The number of differential genes depends strongly on sample size: for the same tissue type, GSE15471 (78 samples) gives 2306 up-regulated genes and GSE16515 (52 samples) gives 1136. When up- and down-regulated counts are unbalanced, handle up and down genes separately in the heatmap and enrichment analysis.

A two-group comparison can also be written as ~group and then take coef = 2. This page checked on GSE15471 that the largest difference in logFC between the two forms is 8.8 × 10⁻¹⁵, so use whichever you can read easily. For more than two groups, use ~0 + group and state each contrast clearly.

treat and partly paired design (measured on this page)r
# Build the fold-change threshold into the test (the approach limma recommends)
tr <- topTreat(treat(contrasts.fit(fit, contr), lfc = 1, trend = TRUE), number = Inf)
sum(tr$adj.P.Val < 0.05)                                # measured here: 166

# Paired samples: if every sample is paired, put patient in the design
# design <- model.matrix(~ patient + group)
# Partly paired (GSE16515 has 16 pairs plus 20 unpaired tumors):
corfit <- duplicateCorrelation(ex_gene, design, block = patient)
fitb <- lmFit(ex_gene, design, block = patient, correlation = corfit$consensus)
ttb  <- topTable(eBayes(contrasts.fit(fitb, contr), trend = TRUE), number = Inf)
Cutoff or setting (GSE16515)UpDownNotes
adj.P < 0.05 and |logFC| > 11136399Most common in the literature
adj.P < 0.05 and |logFC| > 0.585 (1.5-fold)25651001Common with few samples or small effects
adj.P < 0.05 only43395992With large samples nearly every gene is significant
treat(lfc = 1), adj.P < 0.05166 totalTests whether the fold change is significantly above 2
eBayes(trend = FALSE), cutoff as row 11134397trend has little effect on arrays
Partly paired, duplicateCorrelation, cutoff as row 11134403Within-patient correlation 0.136, 16 pairs

Choices

Differential analysis of RNA-seq needs raw integer counts. If GEO only has FPKM or TPM, first check whether NCBI generated raw counts; if not, you must download the fastq files from SRA and quantify them again.

The example measured on 2026-10-11 is GSE164073, with 18 samples (3 ocular surface tissues × infection/control × 3 replicates) and the design matrix ~ tissue + infection. After filterByExpr, 16933 genes remain; voom gives 54 up-regulated and 118 down-regulated genes (adj.P < 0.05, |logFC| > 1), taking 2.4 seconds.

Read NCBI raw counts and run limma-voom (GSE164073, run for this page)r
library(GEOquery); library(edgeR); library(limma); library(org.Hs.eg.db)
acc <- "GSE164073"
hasRNASeqQuantifications(acc)   # TRUE means NCBI-generated counts exist
# Download raw counts directly (this URL form is exempt from reCAPTCHA since 2026-09-02)
url <- paste0("https://www.ncbi.nlm.nih.gov/geo/download/?type=rnaseq_counts&acc=", acc,
              "&format=file&file=", acc, "_raw_counts_GRCh38.p13_NCBI.tsv.gz")
f <- file.path("geo_cache", paste0(acc, "_raw_counts.tsv.gz"))
if (!file.exists(f)) download.file(url, f, mode = "wb")
cnt <- as.matrix(read.delim(f, row.names = 1, check.names = FALSE))  # row names are Entrez Gene IDs
sym <- mapIds(org.Hs.eg.db, rownames(cnt), "SYMBOL", "ENTREZID")     # replaces the blocked annotation file

pd  <- pData(getGEO(acc, destdir = "geo_cache", getGPL = FALSE)[[1]])[colnames(cnt), ]
inf <- factor(ifelse(grepl("SARS", pd[["infection:ch1"]]), "CoV2", "mock"), levels = c("mock", "CoV2"))
tis <- factor(pd[["tissue:ch1"]])
design <- model.matrix(~ tis + inf)

y <- DGEList(cnt)
keep <- filterByExpr(y, design = design)          # 39376 -> 16933 genes
y <- normLibSizes(y[keep, , keep.lib.sizes = FALSE])
v <- voom(y, design)
tt <- topTable(eBayes(lmFit(v, design)), coef = "infCoV2", number = Inf)
tt$SYMBOL <- sym[rownames(tt)]
DataMethodNotes
Microarray (log2 intensities)limmalmFit + eBayes
RNA-seq counts, 3–10 samples per groupDESeq2, edgeR or limma-voomResults overlap heavily; GEO2R uses DESeq2
RNA-seq counts with very different depths or a need for sample weightslimma-voom (voomWithQualityWeights)Fast; design matrices written as for arrays
RNA-seq counts, dozens or more population samples per groupWilcoxon rank-sum test (after edgeR normalization)Li et al. 2022 reported excess false positives from DESeq2 and edgeR on such data
Only FPKM or TPMNo count-model differential testinglimma-trend on log2(TPM + 1) is exploratory at best; NCBI also notes FPKM and TPM are unsuitable for quantitative cross-sample comparison

Choices

Before merging, confirm that the platforms are the same and that each dataset is already on a log2 scale. Batch effects are often larger than biological differences: after merging GSE16515 and GSE15471 (both GPL570), the first principal component explains 48.9% of the variance and mainly separates the two datasets.

The correct approach is to put the batch in the design matrix of the differential analysis and keep analysis and plotting separate. A corrected matrix is used only for PCA and heatmaps, not for limma. If you must use ComBat, include mod so that the group effect is protected.

Correct use of batch: keep testing and plotting separater
library(sva)
# m: merged gene matrix of two same-platform GSEs (each confirmed log2); batch and group are factors
table(batch, group)          # check the design first: does every batch contain both groups?

# Differential expression: put batch in the design
design <- model.matrix(~ group + batch)
fit <- eBayes(lmFit(m, design), trend = TRUE)
tt  <- topTable(fit, coef = "groupTumor", number = Inf)

# Corrected matrix for PCA and heatmaps only (not fed back into limma)
m_plot <- removeBatchEffect(m, batch = batch, design = model.matrix(~ group))
# If you must use ComBat, pass mod to protect the group effect
m_cb <- ComBat(m, batch = batch, mod = model.matrix(~ group))
ApproachBalanced: both datasets have normal and tumorUnbalanced: GSE15471 tumors only
No correction1174 / 6213441 / 296
Batch in the design matrix (recommended)1149 / 2351117 / 404
ComBat(mod = ~group), then limma1144 / 2321108 / 417
removeBatchEffect, then limma1149 / 236Not tested
ComBat without mod, then limma1063 / 220Not tested
Measured for this page; numbers are up / down genes (adj.P < 0.05, |logFC| > 1).

Choices

GEO holds scattered studies submitted by different labs, with different platforms and processing. TCGA is a cancer cohort processed with one pipeline, with full clinical and survival data, but it has few normal controls.

TCGA has few normal samples, so a common approach is to merge GTEx normal tissues. The two use different sequencing and processing pipelines, so after merging, batch and group are completely confounded, as explained in the batch section. Conclusions from such a comparison need validation with a GEO dataset or an experiment.

Current TCGAbiolinks usage (not run for this page; GDC API query checked)r
library(TCGAbiolinks)
q <- GDCquery(project = "TCGA-PAAD",
              data.category = "Transcriptome Profiling",
              data.type = "Gene Expression Quantification",   # without this line you get twice the files
              workflow.type = "STAR - Counts")                # HTSeq - Counts no longer exists
GDCdownload(q, files.per.chunk = 20)   # chunked; rerunning skips files already downloaded
se  <- GDCprepare(q)
cnt <- SummarizedExperiment::assay(se, "unstranded")         # use this assay for DESeq2/edgeR
table(se$sample_type)                  # TCGA-PAAD: Primary Tumor 178, Solid Tissue Normal 4
ItemGEOTCGA (GDC)
SourceThousands of independent studies, arrays and sequencing33 cancer types, uniform sequencing and processing
Normal controlsMany datasets include matched normal tissueFew: TCGA-PAAD has 178 primary tumors and 4 normal tissues
Clinical dataUp to the submitter, often only the groupSurvival, stage, treatment and more
Expression dataSeries Matrix, CEL, countsSTAR - Counts (unstranded, TPM, FPKM and other columns)
Typical useFinding DEGs, external validationPrognostic models, survival analysis, cross-validation with GEO

Concepts and judgment

Explanation

The basis for deciding on log2 is the distribution of the values. A description can be wrong; the values are not.

GEO2R decides automatically using only the grouped samples, and applies or skips log2 to the whole matrix. The code in step 3 uses the same rule, which relies on quantiles rather than on the description.

The number of differential genes changes a lot with the approach used on the same matrix. For measured results, see “Measured comparison of three log2 approaches” in the reference section.

Explanation

When a gene has several probes, the choice of probe changes the number of differential genes directly.

The way multiple probes are handled changes the number of differential genes

On GSE16515 with the same threshold, keeping the probe with the highest mean expression gives 1136 up and 399 down; averaging with avereps gives 834 up and 294 down; keeping the probe with the largest interquartile range gives 1237 up and 431 down. 450 genes are significant only under the maximum-mean method and 43 only under the averaging method; the difference mainly comes from low-expression probes pulling down the mean for the same gene. State which method you used in the methods.

Why keeping the probe with the highest mean expression is recommended

Keep the probe with the highest mean expression (AnnoProbe’s filterEM uses the highest median, with similar results). Averaging mixes in probes that are not expressed or not specific, which shrinks the differences. Running limma at the probe level and then merging to genes gives 1771 genes, more than analysing genes directly, because a gene is counted if any of its probes is significant.

Explanation

A heatmap of differential genes almost always separates the two groups, so it cannot be used as evidence that the groups are reliable.

With cluster_cols = TRUE, a heatmap of differential genes almost always separates the two groups, because the differential genes were selected for exactly that difference. To show the difference between groups, use a PCA based on all genes.

Reference and appendix

Reference

The table gives measured results for GSE16515 and GSE15471, using the same thresholds as the main text (|logFC| > 1, adj.P < 0.05).

ApproachMeasured on GSE16515 / GSE15471 (|logFC| > 1, adj.P < 0.05)
Log2-transform linear values, then limma (correct)GSE16515: 1136 up, 399 down
Run limma on linear valuesGSE16515: 2804 up, 3868 down, median |logFC| 4.51
Apply log2(x+1) to data already in log2GSE15471: 0 (correct handling gives 2306 up, 348 down)

Reference

Truncating z values

After scale = "row", some z values reach 4 or more, which compresses the colour scale so most cells look alike. Setting breaks to -2 to 2 shows values beyond that in the darkest colour.

Labelling genes on the volcano plot

Only the 10 genes with the smallest adj.P are labelled, using ggrepel to avoid overlap. The top genes here are LAMB3, SLC2A1, TMPRSS4, MET and S100P, which match genes known to be up-regulated in pancreatic cancer, a first check that the result is plausible.

Reference

GEO data-mining papers are often returned for a few recurring reasons, listed below. Fix as many as you can before submission.

ObjectionResponse
Small sample sizeReport samples per group; use limma's empirical Bayes method (more stable than t-tests with few samples); merge same-platform datasets with batch in the design matrix; frame conclusions as candidate genes
No validation setAnalyse an independent GSE separately and report DEG overlap and direction agreement. Here GSE16515 and GSE15471 gave 1507 and 1534 DEGs, with 787 shared, all in the same direction
Platforms differ and cannot be mergedRun differential expression per dataset and intersect, or integrate rankings with RobustRankAggreg; merge matrices only within a platform
Unclear batch-effect handlingShow PCA colored by batch before and after correction; state that batch is in the design matrix and removeBatchEffect is used only for plots
Arbitrary cutoffsState the adj.P method and the |logFC| cutoff; add treat results or a sensitivity analysis
Bioinformatics onlyValidate key genes by qPCR, immunohistochemistry or TCGA and HPA data; validate prognostic models in an external cohort
Not reproducibleProvide GSE accessions, platform, R and package versions, the full script and results for all genes

Reference

The following points come from community posts on CSDN, Jianshu and Biotrainee. Each is supported by the original error text, source code or reproduction on this page.

The connection buffer error

A CSDN post shows the error “The size of the connection buffer (131072) was not large enough” when getGEO reads GSE94994. After setting Sys.setenv("VROOM_CONNECTION_SIZE" = 99999999) the file reads correctly. The path “Using locally cached version” in the error text points to the Rtmp temp folder, which means the author did not set destdir.

The probe handling rule in AnnoProbe

For a probe that matches several genes, filterEM keeps only one annotation, and the source code comments admit this is a problem. For a gene with several probes, it keeps the probe with the highest median. When using idmap + filterEM, the methods can say “keeping the probe with the highest median expression”.

Reading a GPL table

A Biotrainee post uses GPL570 as an example: skip the description lines beginning with # before reading the table; when Gene Symbol contains " /// " the post takes the first gene. Some platform pages only allow online viewing without a download button, so download the file from the annot folder on FTP.

log2(dat + 1) in templates

A widely shared template writes dat <- log2(dat + 1) right after the boxplot. The original author used linear data; copying this onto log2 data gives 0 differential genes in this page’s test. When copying, replace this line with the rule above.

Troubleshooting

Find your error message or symptom below. Entries follow the order of the steps and roughly how often each problem comes up. Each entry gives the cause and the fix, with code attached where it can be copied directly.

Error: The downloaded content looks like an HTML page, not GEO data

Cause: Since late August 2026 NCBI returns a reCAPTCHA verification page (HTTP 200) for scripted requests under /geo/, and GEOquery parses it as data.

Fix: Since 2026-09-02 NCBI has opened SOFT text and file download links that include acc. If the error persists, use getGPL = FALSE, or download from FTP and read with filename. FTP is not affected.

Error: The size of the connection buffer (131072) was not large enough

Cause: vroom uses a default buffer that is too small for very long lines.

Fix: Set a larger buffer before calling getGEO, and increase it again if needed. The CSDN post on this error is in the reference section.

Sys.setenv(VROOM_CONNECTION_SIZE = 500000L)
Error: Timeout of 60 seconds was reached, or the downloaded file is incomplete

Cause: R’s default download timeout is 60 seconds, so large files are cut off partway.

Fix: Set options(timeout = 600). If it still fails, resume with wget -c; see the manual download method in the choices section.

Every restart of R downloads the file again

Cause: The default cache folder is tempdir(), which is deleted when R closes.

Fix: Add destdir to getGEO to point to a fixed folder. When you see Using locally cached version, the cached copy is being read.

Cannot connect to the server

Cause: The default download method does not work with the current network.

Fix: Set options('download.file.method.GEOquery' = 'libcurl') and try again. If it still fails, download the file manually.

A gene is missing from the annotation package, or its name differs from the GPL table

Cause: Annotation packages are newer than GPL tables, and genes have been renamed; for example, FAM122C is now PABIR3. Mixing two sources makes the gene names not match.

Fix: Choose one annotation source (hgu133plus2.db in this example), state the package name and version in the methods, and do not mix sources.

The probe or gene count after annotation is far off from what the platform should give

Cause: The wrong platform was chosen, or the matrix row names are not probe IDs (for example ILMN_ IDs on Illumina platforms).

Fix: Check the platform ID with annotation(eset), and use head(rownames(ex)) to confirm the row names match the probe IDs in the annotation package.

The Gene Symbol column contains " /// ", meaning one probe maps to several genes

Cause: The GPL table has 2796 such probes and hgu133plus2.db has 1528. Community posts often keep the first gene name, and AnnoProbe’s source code admits this handling is a problem.

Fix: Remove these probes before the differential analysis, as in the step 4 code.

Thousands of differential genes, with a very large median |logFC|

Cause: Linear values were analysed by limma without log2. For GSE16515 this gives 2804 up and 3868 down genes, with median |logFC| of 4.51.

Fix: Apply the step 3 rule; when the result is TRUE, take log2 before running limma.

ex[ex <= 0] <- NaN
ex <- log2(ex)
Data already on log2 scale gives almost no differential genes

Cause: A template runs log2(dat + 1) unconditionally, which compresses fold changes to almost zero on data that is already log2. GSE15471 (already log2) gave 0 differential genes when processed this way; correct processing gives 2306 up and 348 down.

Fix: Delete the unconditional log2 line and use the step 3 rule instead. The source of the template is in the reference section.

A NaN warning appears when taking log2

Cause: The data contain negative values or many zeros, which is common for MAS5 background-subtracted data or for Illumina data that were not log-transformed. Taking log2 directly produces NaN.

Fix: First count the negative values and zeros. If the data cannot be log2-transformed, use the raw CEL files and reprocess with RMA; see the choices section.

Thousands of differential genes drop to 0 after grouping

Cause: The group vector was taken from a re-sorted pData, so it no longer matches the matrix columns. In GSE16515, 18 of the 52 labels were misaligned, and the differential genes fell from 1535 to 0.

Fix: Do not sort pData, and do not build groups after merging with an external table. Check the sample order with stopifnot(identical(rownames(pd), colnames(ex_gene))).

All samples are assigned to one group, for example all as tumor

Cause: Groups were judged from source_name or title. In GSE15471 every source_name is pancreas, so all 78 samples would be called tumor; the groups are actually in the sample:ch1 column.

Fix: Find the group column with grep(":ch1$", colnames(pd)), then check the sample count of each group with table().

The sign of logFC is opposite to what you expected

Cause: Factor levels are sorted alphabetically by default. With control and case, case comes first and the logFC direction is reversed. Also, GEO2R has treated the group defined first as the test group since November 2020, the opposite of before, so logFC signs in old tutorial screenshots may be reversed.

Fix: Set the reference level explicitly. When comparing with old tutorial screenshots, follow the current GEO2R rule.

group <- factor(group, levels = c("Normal", "Tumor"))
The sample count does not match the paper, for example 78 instead of 36

Cause: The GSE may contain several platforms, or cell lines and technical replicates. The 78 samples in GSE15471 come from 36 patients, and the 6 samples with _rep in the title are repeat arrays from 3 patients.

Fix: First check whether length(gse) is greater than 1. Do not treat repeat arrays as independent samples; merge them by patient and tissue with avereps, or use duplicateCorrelation with patient as the block. See the choices section for the code.

All 50 genes in the heatmap are up-regulated

Cause: Taking the top 50 by adj.P picks up-regulated genes, which are more numerous and more significant in this dataset, so all 50 fall on the up side.

Fix: Take 25 up-regulated and 25 down-regulated genes so both directions are shown. The code is in step 7.

After merging, there are unusually many differential genes and P values are much smaller than expected

Cause: Correcting the matrix with ComBat or removeBatchEffect before limma overstates the residual degrees of freedom, so P values are too small. The removeBatchEffect help page says it is for plotting and exploration only.

Fix: Put the batch in the limma design matrix and run the analysis on the original matrix; use the corrected matrix only for PCA and heatmaps.

design <- model.matrix(~ group + batch)
After ComBat, the differences between groups shrink noticeably

Cause: Without mod, ComBat also removes group differences that overlap with the batch. In this page’s balanced design, 86 fewer up-regulated genes were found. When batch and group are unbalanced, ComBat with mod can still make downstream tests overconfident (Nygaard et al., 2016); in this page’s unbalanced case, 5 permutations of group labels gave 0 to 1 false positive genes after ComBat.

Fix: If you must use ComBat, include mod = model.matrix(~ group). For differential analysis, prefer putting the batch in the design matrix.

m_cb <- ComBat(m, batch = batch, mod = model.matrix(~ group))
table(batch, group) shows that one dataset contains only one group

Cause: Batch and group are completely confounded, so no method can separate them. All group differences then come from differences between datasets, so this kind of merged analysis is not valid. Reviewers can spot this from table(batch, group).

Fix: Do not merge these datasets for comparison; use datasets that each contain both groups.

getRNASeqData fails with invalid 'row.names' length

Cause: The download link for the human annotation file Human.GRCh38.p13.annot.tsv.gz has no acc parameter and is still blocked by reCAPTCHA, so GEOquery reads a verification HTML page. Measured on 2026-10-11 with GEOquery 2.74.0.

Fix: The raw counts matrix itself can be downloaded. Annotate it with org.Hs.eg.db using Entrez IDs; in this example 37692 of 39376 GeneIDs map to gene symbols.

sym <- mapIds(org.Hs.eg.db, rownames(cnt), "SYMBOL", "ENTREZID")
Only FPKM or TPM files are available, and you want to run a differential analysis directly

Cause: DESeq2, edgeR and limma-voom all need integer counts. FPKM and TPM are not suitable for comparing across samples.

Fix: First check with hasRNASeqQuantifications() whether NCBI generated raw counts; if not, download the fastq files from SRA and quantify again. With only TPM, log2(TPM + 1) and limma-trend can be used only for exploration.

GDCdownload fetches twice the expected number of files

Cause: GDCquery was run without data.type, so the results include two data types.

Fix: Add data.type = "Gene Expression Quantification".

GDCquery finds no HTSeq data, or the legacy argument fails

Cause: Since Data Release 32 (2022-03-29), GDC has removed HTSeq counts and kept only STAR - Counts. The Legacy Archive was retired on 2023-05-04, and old hg19 code no longer works.

Fix: Set workflow.type to "STAR - Counts" and do not use legacy = TRUE.

GDCprepare fails with: Column `disease_response` doesn't exist

Cause: A known problem in TCGAbiolinks 2.37.x and 2.38.0 for some projects. Users on GitHub issues report that version 2.34 works.

Fix: Go back to version 2.34, or download the organised GDC TCGA expression and clinical matrices from UCSC Xena.

GDCquery_clinic no longer works

Cause: A change to the GDC interface in early 2025 broke this function, and the related issue is still open.

Fix: The maintainers suggest installing the devel branch of TCGAbiolinks from GitHub.

GDCdownload fails with Error in if (ret == 1) break

Cause: A known TCGAbiolinks problem; issue #656 is still unresolved.

Fix: Use method = "client" to download through the GDC Data Transfer Tool. Download in chunks with files.per.chunk = 20; after a failure, rerunning skips files that are already downloaded.

geoChina prints: Your GSE may not be expression by array

Cause: The GSE list in the AnnoProbe mirror has not been updated since November 2022, so newer GSEs are not in it.

Fix: Download with getGEO directly; see step 2.

References

FAQ

How do I tell whether GEO data need a log2 transform?

Use the GEO2R rule: compute quantiles of the expression matrix and log2-transform if the 99th percentile exceeds 100, or if max minus min exceeds 50 and the 25th percentile is positive. Log2 array data usually lie between 2 and 16. Do not rely on the sample description; GSE16515 says GC-RMA but stores linear values.

For genes with several probes, should I take the maximum or the mean?

Keep the probe with the highest mean expression. On this page averaging gave about 400 fewer DEGs than the highest-mean method, because low-expressed probes pull the mean down. Whichever you choose, state it in Methods.

What if getGEO fails or is very slow?

Set options(timeout = 600) and destdir, and use getGPL = FALSE to skip the large platform annotation. If it still fails, download series_matrix.txt.gz from ftp.ncbi.nlm.nih.gov with wget -c and read it with getGEO(filename = ...). Since late August 2026 NCBI's web endpoints challenge scripts with reCAPTCHA; FTP is unaffected.

How should ComBat be used when merging GEO datasets?

For differential expression, put batch in the limma design matrix (~ group + batch); ComBat beforehand is not needed. Use ComBat or removeBatchEffect output for PCA and heatmaps; if you must use ComBat, pass mod = model.matrix(~ group). If a dataset contains only one group, batch and group are confounded and the datasets cannot be compared after merging.

Can limma be used for RNA-seq data?

Yes: normalize raw counts with edgeR and run limma-voom. FPKM and TPM are not suitable for count-model differential testing. For human data, hasRNASeqQuantifications() tells you whether GEO has NCBI-generated raw counts.

What should I do when TCGAbiolinks downloads fail?

Make sure workflow.type is "STAR - Counts" and add data.type = "Gene Expression Quantification"; HTSeq and the legacy argument no longer work. If GDCprepare reports that the disease_response column does not exist, go back to version 2.34 or download curated matrices from UCSC Xena.

Hand GEO data mining off to Scientify

Give the GSE accessions, groups and cutoffs; the science agent runs download, annotation, limma, batch handling and plotting in an isolated cloud computer and keeps every script, results table and log. New users get $5 of free credit.