---
title: "What I Learned Benchmarking Single-Cell Foundation Models in Liatir"
description: "I compared scGPT and Geneformer with PCA, Harmony, and scVI on two single-cell RNA-seq datasets. The results were useful, but not the sweeping foundation-model win one might expect."
date: 2026-10-21
topic: ai-for-science
tags: ["liatir","single-cell-rna-seq","foundation-models","bioinformatics","scgpt","geneformer","reproducibility"]
production: human
source: https://notes.suffro.com/posts/liatir-single-cell-foundation-models-benchmark
---

# What I Learned Benchmarking Single-Cell Foundation Models in Liatir

I recently completed a [scientific showcase in Liatir](https://liatir.com/showcases/single-cell-foundation-benchmark) comparing pretrained single-cell foundation models with methods that have been used in transcriptomics for years. I wanted the study to answer a practical question: if I already have a single-cell RNA-seq dataset, do large pretrained models give me more useful cell representations than PCA, Harmony, or scVI?

The answer was not a simple yes.

scGPT performed well at predicting cell types in the PBMC dataset. But on the Pancreas dataset, the established methods were stronger on several important measurements. Geneformer did not stand out on either dataset under this protocol. And the results changed depending on whether I looked at cell-type prediction, clustering, or batch mixing.

That is precisely why I think the study is worth sharing. A useful biological representation has more than one job, and a model's pretraining does not remove the need to measure those jobs separately.

## Why compare these methods?

Single-cell RNA sequencing produces a high-dimensional gene-expression profile for each cell. To analyze thousands of those profiles, we usually transform them into smaller numerical representations, or embeddings. Ideally, cells of the same biological type remain recognizable, and technical differences between experiments do not dominate the structure.

There are several ways to get such representations.

PCA is a relatively simple linear reduction technique. Harmony adjusts a PCA representation using batch information. scVI learns a probabilistic latent representation from the dataset being analyzed. Foundation models such as scGPT and Geneformer instead arrive with weights pretrained on large collections of cells, and can produce embeddings without being fine-tuned for this particular analysis.

The question is whether those pretrained embeddings deliver a useful practical advantage. That might show up in better cell-type classification, clearer clustering, improved batch integration, or a favorable tradeoff between quality and compute.

These are different outcomes. Treating them as one score would hide most of the interesting result.

## Two datasets, one fixed protocol

The [study protocol](https://github.com/Liatir/liatir-app/blob/main/showcases/single-cell-foundation-benchmark/protocol.md) fixed scientific parameters before the scores were inspected. I used two public human datasets:

| Dataset | Cells | Genes | Cell types | Batches |
| --- | ---: | ---: | ---: | ---: |
| PBMC | 11,990 | 3,346 | 9 | 2 |
| Pancreas | 16,382 | 19,093 | 14 | 9 technologies |

The comparison included PCA, Harmony, scVI, Geneformer V1 10M, and scGPT whole-human. UCE 4-layer was also requested, but its executions were blocked by runtime constraints, so I did not assign it scientific scores.

All completed methods were evaluated on the same cells. A stratified 80/20 cell-label split with seed 23 supplied training and test labels for the classifiers. For clustering, I used fixed k-means settings rather than tuning them to make any particular representation look good.

One methodological distinction matters: scGPT and Geneformer used pretrained embeddings without fine-tuning, whereas scVI was trained on the evaluation datasets. PCA, Harmony, and scVI also operated on all cells without using their withheld classification labels. This was therefore a **within-dataset, hidden-label evaluation**, not a test of transfer to an entirely unseen laboratory, batch, or dataset.

## scGPT looked good on PBMC, but the metrics disagreed

For cell-type classification I used logistic regression and reported macro-F1, which weights the cell types equally. I also measured adjusted Rand index (ARI), which compares the discovered clusters with the provided biological labels.

These are selected results from the [full measured report](https://github.com/Liatir/liatir-app/blob/main/showcases/single-cell-foundation-benchmark/report/results.md):

| Method | PBMC macro-F1 | PBMC ARI | Pancreas macro-F1 | Pancreas ARI |
| --- | ---: | ---: | ---: | ---: |
| PCA | 0.92880 | 0.75318 | 0.97788 | 0.71291 |
| Harmony | 0.93241 | 0.70619 | 0.94953 | 0.92191 |
| scVI | 0.94916 | 0.54180 | 0.97184 | 0.52441 |
| Geneformer | 0.83946 | 0.28906 | 0.76435 | 0.05019 |
| scGPT | 0.95685 | 0.66926 | 0.89987 | 0.27478 |

On PBMC, scGPT had the highest logistic macro-F1 among the completed methods: 0.95685. scVI followed at 0.94916, while PCA reached 0.92880.

But PCA had the higher PBMC ARI: 0.75318 against scGPT's 0.66926. The embedding that was most useful for this classifier was not the one whose k-means groups aligned most closely with the supplied cell types.

Pancreas made the tradeoffs even clearer. PCA and scVI had macro-F1 scores of 0.97788 and 0.97184, compared with scGPT's 0.89987. Harmony had an ARI of 0.92191, far above scGPT's 0.27478 and Geneformer's 0.05019.

I would not turn those numbers into a universal model ranking. They describe these datasets, model configurations, and evaluation settings. But they do rule out the simple story that the pretrained models uniformly performed better here.

## Batch integration told another story

In single-cell analysis, experimental batches can produce patterns unrelated to the biology we care about. The difficulty is to reduce those technical differences without also erasing genuine differences between cell types.

The study measured local batch mixing using scaled iLISI, along with graph connectivity and other batch-related metrics.

Harmony had the strongest scaled iLISI of the completed methods on both datasets: 0.82647 on PBMC and 0.21006 on Pancreas. For comparison, scGPT scored 0.74429 and 0.00637 respectively. On PBMC, scVI had particularly strong cell-type graph connectivity at 0.93785.

Higher batch mixing alone is not automatically better. It needs to be interpreted alongside preservation of biological cell types. That is why I kept these measurements separate instead of averaging them into an attractive but difficult-to-interpret leaderboard.

The [showcase page](https://liatir.com/showcases/single-cell-foundation-benchmark) includes the original UMAP figures, but the scores were calculated from the full embeddings, not from the two-dimensional plots. A visually tidy UMAP is useful for inspection, not a substitute for evaluation.

## The computational cost matters too

The study also recorded representation time and resource use. Those measurements include preprocessing, imports, model loading, inference or training, and saving the embeddings. They exclude downloads, environment installation, common evaluation, and UMAP generation.

On the same Apple M1 CPU setup, generating PBMC representations took about 7.5 seconds with PCA, 15.2 seconds with Harmony, 249 seconds with scVI, 804 seconds with Geneformer, and 4,994 seconds with scGPT. These are measurements of the complete representation pipelines under that particular resource profile, not isolated neural-network inference benchmarks.

For Pancreas, the unfinished scGPT computation was completed on a different computer with an RTX 4060 Ti, using Liatir under Linux in WSL2. Its fresh GPU representation run took about 516 seconds. I have deliberately not treated that as a direct speed comparison with the earlier Mac CPU runs: the machines, settings, and resource ceilings were different.

The failed and interrupted work was not erased to produce a cleaner performance story. In particular, UCE was blocked by an original Mac memory limit and by the absence of a signed Linux runtime target on the later PC. Those are execution constraints, **not evidence that UCE produces poor embeddings**.

## What this tested in Liatir itself

There was a second purpose to the showcase. I built Liatir as a local-first bioinformatics environment, and I wanted a scientific workflow that exercised more than a polished demonstration screen.

The study used the desktop application's single-cell workflow for dataset preparation, model installation, execution, job monitoring, results, plots, and exports. Runs retain their own identities, and completed work can be reused only when the recorded data, code, split, and embedding checksums match.

The [validation record](https://github.com/Liatir/liatir-app/blob/main/showcases/single-cell-foundation-benchmark/report/reproduction-validation.md) documents an independent reproduction of the completed **evaluation metrics**, matching the saved biological and batch scores within the stated numerical tolerances. It also records exact source-count checks and interruption/resume tests. This does not mean every model was independently retrained or that the entire study was independently replicated by another laboratory.

The source repository contains the protocol, data provenance, small result tables, figures, and test evidence. I also published the [complete reproducibility archive on Zenodo](https://doi.org/10.5281/zenodo.23187931), including larger artifacts kept outside Git.

## What I take away from the experiment

There are important boundaries to this study. It covers two datasets and one random seed, without confidence intervals. The splits were within each dataset, not leave-one-batch-out or leave-one-study-out evaluations. Foundation-model pretraining overlap was not ruled out. The PBMC data contain a selected gene subset, and the Pancreas source includes fractional quantification values that complicate integer-count assumptions.

Those limitations matter. I cannot use these results to claim that a particular model family is broadly superior or inferior across single-cell biology.

What I can say is narrower and more useful: **in this benchmark, pretraining did not confer a consistent advantage over established methods**. scGPT was competitive for PBMC cell-type prediction; PCA, Harmony, and scVI remained strong elsewhere; and the answer depended substantially on which biological or technical property I measured.

For future comparisons, I would want multiple seeds, more datasets, genuinely unseen studies or batches, and compute measurements on comparable hardware. Until then, I would choose an embedding method based on the downstream scientific task and measured constraints, not the size or novelty of its pretraining story.

For anyone who wants to inspect the actual work rather than just the summary, the [Liatir scientific showcase](https://liatir.com/showcases/single-cell-foundation-benchmark) links the figures, methodology, source, and complete archive.

## Sources

- [Liatir scientific showcase: Single-cell foundation models vs established baselines](https://liatir.com/showcases/single-cell-foundation-benchmark)
- [Study source and evidence package](https://github.com/Liatir/liatir-app/tree/main/showcases/single-cell-foundation-benchmark)
- [Frozen protocol and scientific evaluation settings](https://github.com/Liatir/liatir-app/blob/main/showcases/single-cell-foundation-benchmark/protocol.md)
- [Measured scientific results and computational costs](https://github.com/Liatir/liatir-app/blob/main/showcases/single-cell-foundation-benchmark/report/results.md)
- [Limitations and interpretation boundaries](https://github.com/Liatir/liatir-app/blob/main/showcases/single-cell-foundation-benchmark/report/limitations.md)
- [Tested reproduction and execution evidence](https://github.com/Liatir/liatir-app/blob/main/showcases/single-cell-foundation-benchmark/report/reproduction-validation.md)
- [Complete study reproducibility archive (Zenodo)](https://doi.org/10.5281/zenodo.23187931)