Research Capture — An Agentic Archival Pipeline

DRAFT — pending Harsh's verification.

This page documents a personal agent pipeline that captures recurring multi-source research into a durable local archive. It is not a product and is not deployed anywhere — it is a set of plain rules a coding agent follows to collect raw material that would otherwise vanish behind login screens and expiring sessions. It is included here because it shows two things this site is built to surface: agent rules that an LLM actually obeys run after run, and the capture-vs-analysis separation — the discipline to archive raw and refuse to interpret until asked, which is the same honesty posture the site's own agent is built around.

1. The problem

Recurring research arrives across several gated, session-protected sources — subscription research platforms, premium newsletters, community channels, and a synced drive folder. None of it is yours: it sits behind logins, it scrolls out of view within days, and the platforms are under no obligation to keep it addressable. An agent that summarizes on the way in loses the original forever — the summary is a lossy, dated, model-version-dependent compression of something it can never re-fetch on the same terms.

The job, then, is not to read the research but to keep it: pull the raw content and the original files into a folder structure that survives, do nothing to them, and leave any interpretation for a separate, explicitly-requested pass later. The archive is the asset; analysis is a different task that reads from the archive but never writes back into it.

2. The capture rules as an agent contract

The whole system is a short list of operating rules written for an agent, not for a human. The rules are stated as imperatives and the agent is expected to hold them across runs without re-negotiating each time:

• First run for a date lands under data/YYYY-MM-DD/. The date is the primary key.
• A same-day re-run never overwrites. If the date folder already exists, the next run becomes data/YYYY-MM-DD_2/, then _3, and so on. The earlier run is immutable unless the user explicitly asks for a correction or cleanup.
• Preserve raw originals and raw content only — by default. Download the original files where possible; where direct download is unavailable or unsuitable, save the exact available content as Markdown.
• Do not summarize, analyze, score, rank, or produce investment conclusions unless explicitly requested as a separate analysis task. This is the load-bearing rule. Capture and analysis are two different skills, two different output folders, and the agent is told which one it is doing before it starts.
• Do not initialize git, do not build an app, do not migrate the existing layout — unless the user approves.

The date-with-_N-suffix scheme is doing two things at once: it is a primitive append-only log (every run is preserved, none silently overwritten), and it is a cheap conflict-resolution policy that needs no locking — a second run for the same date just picks the next free suffix. The agent never has to decide whether a re-run is "important enough" to overwrite; it isn't allowed to overwrite at all.

Reusable pattern: make immutability a folder-naming rule, not a behavior the model has to remember. If the path the agent should write to is mechanically derived from "is there already a folder for this date," the model can't overwrite by accident — the path itself forces a new run.

3. Browser-auth session reuse, with an ask-before-continuing rule

The sources are gated, so the pipeline reuses the user's existing browser-auth sessions by default rather than carrying its own credentials. Three things keep this safe and non-magical:

• Sessions are a default, not a fallback. Capture assumes a live browser session is present for the session-protected sources. It does not spin up its own auth, store tokens, or fall back to public-only scraping.
• If auth is unavailable, expired, or blocked — ask before continuing. The agent stops and asks the user rather than silently degrading to whatever is unauthenticated. It never broadens a premium source to its public-only slice to "make the run succeed"; a blocked source is recorded as a source-status note and the run continues around it.
• Locked or inaccessible content is recorded, not skipped quietly. When a source has no new visible content for the requested date, or a card is locked, the run preserves a source/status note rather than leaving a hole — so the archive is honest about what was seen but empty versus what was never reached.

The ask-before-continuing rule is the same posture as the site's chat agent's honesty rule: when the system can't get the real thing, it says so instead of substituting a plausible-looking inferior. A silent fallback to public-only content would be the archival equivalent of a bluff.

4. The run layout

Every capture run is one self-describing folder. The shape is fixed by the archive contract so a later pass — analysis, validation, or a re-fetch — can treat any run as a typed record without reading its contents first:

data/YYYY-MM-DD[_N]/
  raw/                       browser or connector capture JSON (the raw wire shape)
  raw-files/
    content/                 raw text/Markdown when direct download is unavailable
    originals/               downloaded original files (PDFs, images, charts, screenshots)
    download-manifest.json   every URL download attempt
    download-summary.json    distilled result of those attempts
    download-urls.tsv        the URL list, TSV for machine consumption
  <source>/                  per-source folders for raw captures or status notes
  assets/
  summaries/                 present in legacy runs; NOT created unless explicitly asked
  manifest.json              the run's index, finalized when the run closes

A few details are deliberate:

• raw/ holds the wire shape; raw-files/ holds the human/analyst shape. The raw connector JSON is the ground truth of what came back; the content extracts and originals are derived artifacts that are easier to read. Keeping both means a future re-materialization can start from raw/ without re-fetching.
• Three download records, one per concern. download-manifest.json is the full attempt log; download-summary.json is the distilled result for a human or a model; download-urls.tsv is the flat URL list for any downstream tool. Same data, three shapes, so the right consumer never has to parse the wrong one.
• summaries/ exists in older runs but is not created going forward. The layout carries history: the archive once mixed summary output into capture runs, and the rules were tightened to separate them. The folder is kept on old runs untouched (existing data is never changed without an explicit request) but a new run won't have one unless analysis is explicitly requested.
• Media policy is part of the layout, not a runtime choice. Video and audio are captured as their text surfaces only — transcripts, captions, notes, descriptions — never media files. Static visuals (charts, tables, figures) are captured as originals or screenshots. So the archive holds the things an analyst or a model can read, not binaries that would bloat storage and that no downstream reasoning can ingest directly.

Two utility scripts bridge the two shapes. A materializer walks known raw capture shapes (per-source raw JSON, or a consolidated browser-capture.json) and produces the content extracts plus the download manifests; a downloader then reads those manifests and fetches the linked originals. Validation is a deterministic pre-check — a single script run against a date or run folder audits the shape without invoking a model at all. The point is that the machine-checkable parts of a run (does it have the required folders? do the download records reconcile?) are not the model's job; they are scripts, so a run either passes or fails the same way every time.

5. Config-driven source scope

The set of sources is not restated in the agent's instructions or in the skill bodies. It lives in one canonical config file: a JSON object keyed by source, each entry carrying a label, its per-source capture notes, and — where relevant — its feeds, its tier, and an auth: browser-session marker. Changing scope is editing that file, not rewriting the agent's prompt.

This is the same separation as the build-triage pipeline's signal→action table: the what (which sources, which channels, which access tier) is data; the how (capture raw, don't analyze, ask on auth failure) is the contract. The agent reads scope from the config and follows the contract; the contract does not have to know which sources exist. A source can be added, removed, or re-tiered without touching a single rule.

Per-source capture notes are the place the config gets specific about how to treat a given source — for example, which text surface to use for an audio post, or that a locked item should be a source-status skip rather than a public-only fallback. Those notes live next to the source they describe, so the contract stays generic and the specifics stay addressable.

6. Deliberate non-engineering choices — restraint as a design decision

The most striking thing about the pipeline is what it refuses to be. It is not a git repository. It is not an app. It has no database, no service, no deploy. It is "a loose local archive," and it stays that way by rule: do not initialize git unless requested; keep the current data/ structure unless the user approves a migration; treat the folder as a durable user-owned workspace, not scratch space.

That restraint is a design decision, not neglect. The recurring cost in this domain is losing the raw material, not not having a query engine over it. An app would add maintenance surface — schema migrations, a deploy, a backing store — in exchange for queryability the archive doesn't need yet, because analysis is a separate, on-demand task that reads from the flat files. Git would add history, but the date-with-_N-suffix scheme already gives the one kind of history that matters (run immutability) without the overhead of versioning large binary originals. The pipeline takes the smallest shape that solves the actual problem: a folder tree, a fixed contract, and a few scripts.

Reusable pattern: the cheapest durable artifact is a folder with a contract. Reach for git, a database, or a service only when the flat-file version is demonstrably failing — not because "real systems have those." Adding infrastructure before it's needed is how a capture pipeline turns into a capture platform nobody maintains.

7. What it demonstrates

Taken together, the pipeline is a worked example of two claims this site makes about agent engineering:

• Rules that an LLM actually follows. The capture contract is short, imperative, and mechanically checkable — and it is honored across runs because the rules are written as constraints the model can satisfy without judgment calls (write to a mechanically-derived path; never overwrite; preserve raw; ask on auth failure). The parts that shouldn't depend on the model's judgment don't; the model supplies only the fetch-and-file glue.
• Capture-vs-analysis separation as a trust posture. Refusing to analyze on the way in is the same discipline as the site's chat agent refusing to answer without a citation. The archive stays honest because the capturing agent doesn't get to interpret; a later analysis pass — a different skill, writing to a different folder — is the only thing that produces conclusions, and only when explicitly asked. The orchestration model reinforces this: the workflow skill acts as a coordinator over per-source worker runs that return distilled reports (counts, statuses, paths, failures) — never raw content — and the coordinator reasons over those reports plus the deterministic validator's output. Analysis is a frontier leg; extraction and verification are executor-grade legs. The raw content never has to flow back through the coordinator to be reasoned about.

One honest gap. The pipeline is described here at the level of its rules and layout, which is exactly what the source material captures. What is not in the source: how long a typical run takes, how often runs occur, how much content a run holds, or any before/after measure of the time-saved by automating the capture. A metrics-backed version of this story would need that pulled from real usage, and it is not in the material this page was distilled from. The source-sync pipeline that keeps the synced drive folder fresh is referenced by the design but its implementation was not part of this extraction.

Sources

This page was distilled from the research-capture project's own README.md and AGENTS.md (the operating rules, the run-layout contract, and the orchestration model), cross-checked against config/sources.json for the config-driven scope structure only. All source names have been genericized — no vendor, newsletter, community server or channel, membership tier, or URL from the real system appears here; named sources are described as "subscription research platforms," "premium newsletters," "community channels," and "a synced drive folder." Every architectural claim above traces to that material; nothing was invented beyond it.