D-academic-agent — Academic Agent
- TypeScript
- Electron
- MCP
- ONNX / HuggingFace Transformers
- DeBERTa-NLI
- SQLite / better-sqlite3
- React
- AI SDK
- Vitest
- scite / Consensus APIs
- OAuth 2.1 PKCE
- Zod
A local-first academic evidence workspace that audits claims and citations against source evidence with a deterministic NLI judge. It ships as an Electron desktop app plus MCP-capable core, with external scite/Consensus evidence search kept explicitly opt-in.
At a glance
Role Sole builder — architecture, core pipeline, Electron app, MCP server, evaluation framework
Problem
Citation auditing and literature review are error-prone at scale: LLMs hallucinate support where none exists, and 'supports' from a lexical judge is meaningless if it can't detect contradiction. Every claim needs a grounded locator, not a model guess.
Solution
I built a local-first pipeline that ingests papers, retrieves evidence, judges claims with deterministic NLI, and exposes verdicts with snippets and locators. External evidence search stays opt-in, so private drafts and libraries are not sent out by default.
Impact
317 offline tests · 42-claim hand-labeled gold set · latest packaged release for macOS arm64 / x64 / universal
My contribution
- Draft Citation Audit — per-sentence: each citation is resolved to a source, judged (supports / weakly_supports / unsupported / contradicts / unclear) with a verbatim snippet, a structured char-span locator, a confidence, and a suggested rewrite; corpus counter-evidence surfaces when other sources push back
- Thesis Review — decompose thesis → retrieve evidence → judge each piece → consensus verdict (supported / contested / refuted / insufficient) with ratio and decisiveness over a supporting-vs-contradicting evidence map
- Hybrid retrieval — BM25 + dense vectors (local ONNX: all-MiniLM / bge-small / nomic, role-aware) fused with reciprocal-rank fusion; locator survival from page to char span
- Offline-first judge — deterministic DeBERTa-NLI (zero external calls) as default; OpenAI-compatible and Ollama one-click preset for cloud/local LLM judges
- Persistent library — PDF drag-drop ingest (GROBID section-aware + unpdf fallback); local SQLite (Source / Chunk / vectors); re-index on embedder switch
- Iterative agent-loop checker — when top evidence is inconclusive, examines next-ranked candidates with traced steps; failure-mode is measured and exposed as a diagnostic
- Three-axis evaluation: accuracy (macro-F1 + per-class + confusion matrix), faithfulness (answer_groundedness), and governance (policy_compliance: grounded-locator rate + snippet-only rate + outbound chars)
- Provider ablation runner — compares hash+mock vs all-MiniLM+NLI vs agentic on all three axes; local-vs-cloud and single-shot-vs-agentic trade-offs, quantified
- Writing Desk — paste a paragraph: it splits and classifies claims (causal / comparison / method …) and flags needs_citation / overclaimed / contradicted against your library, with safer-wording rewrites; per claim, find external evidence via an editable, disclosed query
- External research & reference health (opt-in) — search scite / Consensus in-app for candidate external evidence; surface scite's citation analysis as a risk badge (no major concerns / some pushback / notable concerns / retracted or serious notice) with support / contradict / mention counts and retraction notices, in three places: on search results, on your imported library papers (catch a retracted work before you cite it), and on a paper's GROBID-parsed bibliography
- Reusable MCP OAuth — an OAuth 2.1 PKCE + Dynamic Client Registration browser sign-in (built on the MCP SDK) for OAuth-gated MCP servers: a 127.0.0.1 loopback flow, tokens stored only in the keychain, the worker handed nothing but a bearer access token
- Desktop release pipeline — packaged the latest public release for macOS arm64 / x64 / universal so reviewers can inspect the workspace as a real app instead of only a source checkout
Technical evidence
- Shared core, two faces: pure-TS engine in src/ (ingest · retrieve · check · plan · eval · trace); MCP server and Electron app are thin adapters — src/ never imports Electron
- Determinism from offline mocks (HashEmbedder / MockJudge / MockPlanner); model-loading tests are environment-gated so CI stays fully offline
- Gold set: 42 hand-labeled claims with integrity-gated sources.lock.json; gold must be human-labeled, never self-graded by the model under test
- Opt-in egress with a hard boundary — external calls fire only on an explicit action and send only the query or public DOIs, never the corpus or draft (the one exception, Writing-Desk evidence search, sends an editable, disclosed query); OAuth tokens stay in the keychain, reach the worker only as a bearer scalar, and are redacted at every boundary
- Secrets in OS keychain (Electron safeStorage) — never in config, git, traces, or logs; 317 tests across retrieval, checking, eval, trace, and the external-research layer, all offline (live-network smokes are environment-gated)
- Electron packaging covers macOS arm64 / x64 / universal artifacts in the latest packaged release, with source-run fallback kept available for non-macOS targets
The pipeline — plan → retrieve → judge → report
The core loop is plan → retrieve → judge → report, observable end to end through versioned trace events. A thesis is decomposed into sub-queries; hybrid retrieval gathers evidence across the corpus; the judge rules on one claim at a time from the retrieved snippet only; the report is a grounded evidence map with a verdict.
Three entry points share the engine: a per-sentence Draft Citation Audit that resolves each citation to a source and judges the cited snippet; a Thesis Review that decomposes a thesis and returns a consensus verdict over a supporting-vs-contradicting evidence map; and a Writing Desk that splits a paragraph into claims, flags the uncited and overclaimed against your library, and can pull external evidence for any one of them.
Why a deterministic NLI judge
The verdict is only as strong as the judge. A lexical judge can't detect contradiction — everything looks 'supported'. The deterministic DeBERTa-NLI judge surfaces the real split between support and refutation, fully offline. This is measured against a gold set, not assumed.
Shared core, two faces
A pure-TypeScript engine (ingest · retrieve · check · plan · eval · trace) is wrapped by two thin adapters — an MCP server and an Electron desktop app. The iron rule is that src/ never imports Electron, so the core stays headless and testable. Determinism comes from offline mocks (HashEmbedder / MockJudge / MockPlanner), and model-loading tests are environment-gated so CI runs fully offline.
External research, explicitly opt-in
Bind scite or Consensus in Settings and the agent can reach beyond your library, but only on demand. Every external call is gated behind an explicit action and sends only the submitted query or public DOIs; the corpus never leaves the machine, and in the Writing Desk the claim-derived query is editable and disclosed before submission. scite's citation analysis comes back as reference health: a risk badge plus support / contradict / mention counts and retraction notices, surfaced on search results, on the papers you've imported (so you can catch a retracted work before you cite it), and on a paper's GROBID-parsed bibliography. Consensus contributes to candidate evidence search, not reference health.
Connecting an OAuth-gated MCP server reuses the same discipline: a standard OAuth 2.1 PKCE + Dynamic Client Registration browser flow (built on the MCP SDK) with a 127.0.0.1 loopback capture, tokens kept only in the OS keychain, and the worker receives only a bearer access token. External results are always framed as candidates — never the app's own verdict.
Evaluation, and an honest failure mode
A frozen gold set of 42 hand-labeled claims, integrity-gated against a sources lock file, drives a three-axis report: accuracy (macro-F1 plus a confusion matrix), faithfulness (answer groundedness), and governance (grounded-locator rate, snippet-only rate, total outbound characters). A provider-ablation runner quantifies the local-vs-cloud and single-shot-vs-agentic trade-offs.
The agent-loop checker has a measured failure mode: with a noisy judge it over-commits. Rather than hide that, it's exposed as a diagnostic — the honest framing throughout is that this is a seed set, not an authoritative benchmark.
Screenshots
