arXiv Recommender
Hybrid arXiv paper recommender on a 28,000-paper Computer Science snapshot from OpenAlex. Compares popularity, TF-IDF, a MiniLM sentence-transformer, implicit-feedback ALS on the citation graph, and a hand-tuned linear blend, with bootstrap-CI held-out evaluation, exact FAISS inner-product serving, and a side-by-side interactive dashboard.
Problem
Research is moving faster than any one person can read. The default arXiv "related papers" surface is content similarity in one shape, but academic search has two strong signals: text (titles and abstracts) and citation relationships. A credible recommender tests both, reports their performance honestly, and stays fast enough to serve from a single VPS.
The project needed to demonstrate that end to end: a real scholarly dataset, multiple algorithmic families implemented properly, a held-out evaluation harness with confidence intervals, low-latency serving, and an explainable interactive demo.
Users and decisions
Researchers and graduate students are the intended users. They can compare content- and citation-based recommendations to decide which papers to read next and inspect why the models surfaced them.
The interactive demo is built for portfolio demonstration on a public OpenAlex snapshot of recent Computer Science arXiv papers. It is not a production literature-discovery service.
Five models behind one protocol, scored on one split
Five recommender models are implemented behind a common protocol: a popularity baseline by cited-by count, a classical TF-IDF model over title, abstract, authors, and topic, a MiniLM (sentence-transformers/all-MiniLM-L6-v2) content tower over abstracts, implicit-feedback ALS over a bipartite citation graph (citing papers as users, cited papers as items), and a hybrid that blends min-max normalised scores from all four.
Every model is evaluated on the same held-out test set with precision, recall, MAP, NDCG (all with 1,000-sample bootstrap 95% confidence intervals), coverage, diversity, and intra-list similarity. A FastAPI service on a Linux VPS exposes scored similar-items queries from a FAISS IndexFlatIP exact inner-product index. A Next.js frontend on Cloudflare Pages renders the four algorithms side by side so the differences are immediately visible.
Architecture
The blend leads at 0.199 MAP@10; the neural tower lost to TF-IDF
Evaluated as a similar-items task. For each test seed we hold out 10% of its outgoing citation edges before training, then ask whether the held-out citations appear in the top 10 when the seed is used as the query. Metrics carry 1,000-sample bootstrap 95% confidence intervals.
Evidence for
0.199MAP@10 for the hybrid blend (95% CI 0.186 to 0.213), against 0.149 for TF-IDF (0.137 to 0.161). The intervals do not overlap, so the blend is separated from the strongest single model rather than merely ahead of it.Evidence for
1.34×Hybrid over TF-IDF on MAP@10. This is the comparison that carries information. The often-quoted 20x figure is against a popularity ranker scoring 0.0098, which returns the same globally most-cited papers for every query and is uninformative by construction.Evidence against
0.121 vs 0.149The MiniLM neural tower loses to plain TF-IDF on MAP@10, with non-overlapping intervals. A 22M-parameter general-purpose encoder does not beat term matching on a corpus this domain-specific, which is the most useful negative result the benchmark produced.Tools used
Key features
- Five recommender algorithms behind a single typed protocol so the evaluation harness treats them identically.
- Bootstrap 95% confidence intervals on every ranking metric; coverage, diversity, and intra-list similarity reported alongside accuracy.
- Citation-graph collaborative filtering: citing papers as users, cited papers as items, implicit ALS over a bipartite matrix.
- Sentence-transformer content tower (MiniLM, 384-d, L2-normalised) on titles plus abstracts so the same FAISS index pattern works for both content and collaborative similarity.
- Cold-seed handling: the hybrid blend down-weights ALS when the seed has very few train edges, content carries the recommendation.
- FastAPI endpoints with Pydantic-validated I/O, request logging into the
ops.request_logtable, and a healthcheck that lists loaded algorithms. - Next.js dashboard on Cloudflare Pages with KPI tiles, a search box, a seed-paper card, and four side-by-side algorithm result columns with per-item explanations and live latency badges.
- VPS deploy artefacts: a systemd unit for the API, a weekly refresh timer that re-pulls OpenAlex and retrains, an nginx site with TLS via Let's Encrypt, and a bash bootstrap script for the Postgres role and schema.
- Property-based tests on every ranking metric (precision, recall, MAP, NDCG bounded in [0, 1] across hundreds of generated cases) and pinned tests for ALS, TF-IDF, and top-k correctness.
- Reproducibility via a single
RANDOM_SEEDthat controls the train/test split, ALS init, and bootstrap sampling.
Tradeoffs and constraints
OpenAlex is the source of truth for both metadata and the citation graph. Most papers cite older work outside our 2019 plus subset, so the in-set citation graph is sparser than a full graph would be (roughly 1.6 edges per paper). Collaborative ALS therefore underperforms content here. The hybrid is set up to absorb a denser graph without other code changes; a real production deployment would widen the date window or pull a second hop of citations as shadow nodes.
The MiniLM encoder is intentionally small (22M parameters, 384-d output) so the full leaderboard regenerates in under ten minutes on CPU. A real deployment would swap to a stronger SPECTER-style scholarly encoder and run on GPU, and would extend the hybrid head from a fixed linear blend to a tiny trainable model fit on a held-out validation slice.
Methodology
Appropriate use: portfolio demonstration of recommender systems engineering on a public scholarly dataset.
Inappropriate use: as a production literature-search service or as ground truth for hiring, funding, or editorial decisions; the snapshot is point-in-time and the citation graph is intentionally restricted to a closed subset.
Limitations
The recommender operates on an OpenAlex snapshot of Computer Science arXiv papers published since 2019 with at least three citations. The citation graph is restricted to in-subset edges, so collaborative coverage is lower than a full-graph deployment would be. Cover and PDF links resolve through the original source URLs and may rot over time.
The hybrid blend weights are hand-set (45% neural / 35% ALS / 15% TF-IDF / 5% popularity), not learned. Replacing the fixed blend with a small linear head trained on a held-out validation slice is the obvious next step.
The popularity baseline is close to null, so lift over it overstates the result. On a similar-items task, a popularity ranker returns roughly the same globally most-cited papers for every seed, so it is not a weak baseline but an uninformative one. Any multiple measured against it is inflated by construction. The comparison that carries information is the neural and hybrid models against TF-IDF, which is a real content baseline, and that is the number to read off the leaderboard.
The evaluation target partly favors the content model. Holding out 10 percent of outgoing citation edges and predicting them from title and abstract embeddings rewards the same textual similarity the encoder was trained on. Sparse in-set citations explain part of why ALS underperforms here; this near-tautology explains the rest.
Decisions and rejected alternatives
MiniLM over a SPECTER-style scholarly encoder, and the benchmark says I was wrong. I chose a 22M-parameter, 384-dimension general-purpose encoder so the leaderboard rebuilds in under ten minutes on CPU, expecting to trade some quality for iteration speed. The measured trade is worse than that: the neural tower scores MAP@10 0.121 against TF-IDF's 0.149, with non-overlapping confidence intervals. A general-purpose sentence encoder loses to term matching on a corpus this domain-specific. It still earns a place in the blend because it contributes signal TF-IDF does not, which is why the hybrid clears both, but the case for a scholarly encoder like SPECTER is now empirical rather than theoretical.
Exact inner-product search over an approximate index. FAISS IndexFlatIP is brute force. At 28,436 vectors and 384 dimensions the index is 43.7 MB and an exact scan already answers in double-digit milliseconds, so IVF or HNSW would have introduced recall error to solve a latency problem that does not exist yet. Approximate search earns its place two or three orders of magnitude further up; claiming it here would have been resume-driven engineering.
A fixed hand-set blend over a learned head. The 45/35/15/5 weighting is judgment, not fitting. With one held-out split and a sparse citation graph, a learned head had a real chance of fitting the split rather than the problem, and I could not have told the difference from a single number. The honest cost is that I cannot claim the blend is optimal, only that it is stable and explainable.
Still open. A learned hybrid head with a proper validation slice. A second hop of citation edges for cold seeds. An author-recommendation surface over the same ALS factors. A nightly OpenAlex refresh, and the promotion canary that refuses a new model unless MAP@10 sits within the bootstrap CI of the prior best.