← Back to Horizons

ResearchArticleAI workflows

Considerations for an AI Research Workflow.

What properties should an AI research workflow have? Three answers — Grounded, Personalized, Sustainable — and what we learned building and benchmarking our own.

Director, Horizons OfficeAssociate Professor, School of Computing, NUSAI Scientist, Horizons OfficeDate28 August 2026Read~8 minutes

At the Horizons Office in the NUS School of Computing, we have been folding AI assistants into our daily work: literature review, coding, analysis, and writing. The tools are genuinely useful. But wiring them together into an actual workflow — one that a research group depends on every day — raises a design question that we think deserves care: what properties should an AI research workflow have?

Here, we discuss three such properties. The workflow should be Grounded in the literature, Personalized to our goals, and Sustainable over the lifetime of the research it supports. The initials spell GPS, which is fitting: the point of all three is to keep a research group from getting lost — in fabricated citations, in generic advice, or in dependencies we cannot re-run three years later. Below we take each in turn, including what we learned building and benchmarking our own setup.

01

Grounded in the literature

Language models generate fluent text whether or not the underlying claim is true. The failure mode most damaging to research is the plausible fabrication (aka hallucination): the citation that does not exist, the theorem attributed to the wrong paper, a wrong benchmark number.

We require that claims of consequence be tied to retrievable sources — links we can follow, quotations we can check against the PDF — sourced from the literature itself instead of the model's training-data recall. Our research agent reflects this: it searches our own PostgreSQL discovery database of papers and researchers, reads the stored PDFs directly, and falls back to web search. Answering from model memory alone is the last resort.

We have also learned that grounding must be enforced mechanically, not assumed. Every model behind our agent passes through the same schema validation (Pydantic), bounded retries, and citation checks. These checks earn their keep: they routinely catch responses — particularly from smaller local models — that read fluently while silently breaking a tool call, a JSON contract, or a citation. A useful test of grounding is whether the assistant can point to the specific table or theorem, and whether that pointer survives inspection. Retrieval narrows the hallucination problem but it does not eliminate it. These checks enforce structure, retrievability, and citation presence; they do not establish that the cited source actually supports the claim. The judgment about what a source establishes remains with us. This is the same standard we would apply to a new student or intern.

02

Personalized to our goals

A generic assistant answers the question asked. A useful research assistant answers the question asked in the context of what we are trying to achieve: the research agenda, the standards of the venues we write for, the conventions of our codebase, the mathematical framing we prefer, and what we have already tried and discarded.

In practice, this means treating context as infrastructure — maintained and carried across sessions, not re-typed into each new one. Persistent memory, project files, and standing instructions let the assistant accumulate an understanding of the group's goals: which problems matter to us, which baselines we consider credible, which claims we would never make without an experiment behind them. The difference is tangible. An assistant that knows we care about calibrated uncertainty in embodied settings reads a paper differently — and drafts differently — than one that does not.

Two cautions are in order. First, an assistant tuned to our preferences can drift toward telling us what we want to hear. Agreement is not evidence, and a workflow should preserve pushback, not suppress it. Second, personalization data is often sensitive: unpublished ideas, student work, reviews in progress. This constrains where that context can live, a point we return to next.

03

Sustainable over time

The third property concerns time. Research artifacts have long lifetimes: a paper should be re-runnable years later, a PhD spans 3–4 years, and internal services accumulate dependents. A workflow built entirely on proprietary hosted models sits uneasily with these timescales, for three reasons:

  • Version drift. A hosted model behind a version string is not a fixed object; weights, scaffolding, and decoding defaults can change. "We used provider-model-x" under-specifies the experiment.
  • Deprecation. Model retirement follows a commercial cadence, not a thesis cadence. When a baseline endpoint is withdrawn, published comparisons become unrepeatable.
  • Accumulated coupling. Prompts, tool schemas, and output formats specialise to a particular model over time, quietly raising the cost of ever switching.

Hosted frontier models remain ahead on the hardest reasoning tasks and carry no infrastructure burden; for exploratory work they are frequently the right choice, and they remain part of our stack. Our claim is that a sustainable workflow considers dependency explicitly, and maintains an exit.

This is where open-weight models enter. The 2026 landscape — DeepSeek V4, Qwen 3.6/3.7/3.8, GLM-5.2, Kimi, Gemma 4, Llama 4 — is close enough to the closed frontier that capability alone no longer settles the question for many workloads. A downloaded checkpoint is a frozen, archivable artifact: precisely the object a reproducible pipeline needs, and one that can also hold the sensitive personalization context on our own hardware. (One point of nuance: these are open-weight, not open-source. Training data and pipelines are generally not released, and licences vary per checkpoint.)

The State-of-Computing (SoC) Research Agent: What we are building

Our research agent (Python 3.12, FastAPI, LangGraph) is written against a single backend contract covering messages, tools, structured output, and errors. Thin adapters connect that contract to OpenAI, Gemini, and a locally hosted Ollama-compatible endpoint, so the same agent loop runs across providers (or can be modified per-provider for increased performance). This adapter layer is the sustainability property made concrete: any model, hosted or local, is a swappable component, not a foundation.

We've been experimenting with Qwen 3.8 (dense 27B, a 4-bit MLX checkpoint with multi-token prediction, thinking disabled), serving both the research agent and our ingestion pipeline on an M3 Ultra Mac Studio (96 GB unified memory, 819 GB/s). An earlier pilot used the Qwen 3.6 35B mixture-of-experts model, which activates only ~3B parameters per token and should in principle be cheaper per generated token. Note that its archived runs used different prompts, workloads, and serving configurations, so we cannot report a reliable speed comparison between the two.

M3 Ultra Mac Studio and attached storage used for local AI model serving
Our local inference setup: an M3 Ultra Mac Studio with attached storage.

What we can report is a comparison of the Mac Studio against a single NVIDIA A100 80 GB on the same Qwen 3.8 model. It taught us that the inference engine matters about as much as the hardware:

ConfigurationMac Studio decodeA100 decode
Same GGUF Q4_K_M via Ollama on both15.9 tok/s46.4 tok/s
MLX-optimized on Mac vs Ollama on A100 (~1K-token input)54.8 tok/s44.2 tok/s

Under similar settings, the A100 decodes roughly 3x faster. With an Apple-native MLX runtime, the same Mac out-decodes the A100. The A100 retains a real advantage in prompt processing: on uncached 16K–65K-token inputs it finished requests 2–3x sooner overall (about 81s vs 279s at 65K) even while decoding slightly slower. A growing conversation with a valid KV prefix cache only processes newly added tokens, which should narrow that gap — we have not yet measured this directly. We are now benchmarking Ollama, llama.cpp, and oMLX across 1K–128K-token inputs, multi-token prediction, prefix caching, and concurrency. The lesson so far: default settings are likely under-optimized, and matching engine to hardware and workload requires actual measurement.

We are also evaluating DeepSeek V4 Flash under Salvatore Sanfilippo's ds4/DwarfStar engine, whose recommended ds4f-q2 configuration targets 96–128 GB machines like ours. However, a 2-bit-class configuration sits below the comfort line the quantization literature supports, so further experiments and evaluations are needed.

What it costs

One major cost of local inference is quantization. Liu et al. (COLM 2025) find, across the DeepSeek-R1-distilled families, that 8-bit and 4-bit weight-only quantization are near-lossless (≤1–2%), while 3-bit is risky, with degradation up to 4× larger on hard tasks than easy ones. Red Hat's separate evaluation is consistent. Below 4 bits the picture worsens sharply and becomes method-dependent: one study reports 3-bit AWQ dropping a 1.5B model's MATH-500 accuracy from 85.6% to 47.0% while inflating chain-of-thought 4.5×, where GPTQ at the same bit-width retains 71.6%. For full-size DeepSeek models, Q4_K_M appears safe on most reasoning benchmarks. Our working rule: 8-bit is safe, 4-bit is generally acceptable, and sub-4-bit is a research variable, not a deployment default.

Beyond quantization: long-context prefill is one weakness of the Mac platform (our own numbers show it), concurrency is limited, clustering frameworks remain immature, and someone has to own/maintain the hardware. Local inference converts a variable per-token cost into a fixed capital and staffing cost. These should be considered before building it out.

Practical reading

—

In Summary

The three properties reinforce one another. Grounding determines whether the assistant's outputs can be trusted enough to enter a paper; personalization determines whether it works on our problems; sustainability determines whether any of it survives contact with time. Our current design makes each concrete: retrieval over our own paper database with mechanically enforced schema and citation checks; persistent, locally-held context for goals and conventions; a single backend contract with per-provider adapters so that every model is swappable; and a private evaluation suite with unambiguous pass conditions, so that swapping any model — hosted or local — is a measured change.

None of this is finished, and our particular stack is likely not the right one for every group. But we would rather build the workflow deliberately now than discover later that we cannot reproduce our own results. The inference-engine benchmarks are still running; we'll share more in a future article.

AI usage: LLMs (Claude and ChatGPT) were used to find relevant resources and in the editing of this article.

Research← Back to the Horizons Office
DiscussWrite to us →