Dagster Remembers Every Run. Your AI Steps Still Start Each Run Blank.

Open a Dagster deployment after a few months of scheduled work and the past is all there: every run listed, every event logged, every asset materialization recorded with its lineage. It looks like a system that remembers. Then the next scheduled run starts, an AI step inside a job receives a fresh prompt with none of that history in it, and the gap between a record a person can inspect and a memory a run can use becomes hard to ignore.

Dagster is excellent at the record. What it does not provide is agent memory: the shared, readable past that lets a recurring AI step begin tonight's work already knowing what yesterday's work found, decided, and got corrected on.

What Dagster genuinely keeps

Run storage is real persistence. Dagster keeps a history of runs and per-run event logs, so a failed step at 3 AM can be replayed from the record: which op ran, what it emitted, where it stopped. Compute logs sit alongside that history for the person debugging the failure.

Materialized assets persist too. When an asset is materialized, its output is written through an IO manager to storage, and a later run can load that asset as an input instead of recomputing it. Partitions extend the same idea across days, clients, or regions, so yesterday's partition can feed today's job when the code is written to load it.

Sensor cursors persist as well. A sensor can store a cursor, update it after each evaluation, and pick up next time from exactly where it stopped, which is how a file-watching sensor avoids reprocessing the same files forever.

Each of these is a deliberate, well-built mechanism. None of them is a memory an AI step can read.

Records are addressed to people, not to the next run

Run history and event logs face backward. They exist so an operator can answer "what happened?" after the fact. A new run never opens the previous run's event log as input. It does not scan last night's warnings, inherit last night's conclusions, or notice that the exception it is about to flag is the same exception it flagged, and saw resolved, three nights in a row.

The history grows every night. The runner starts empty every night. Both stay true at the same time, and confusing the first for a fix to the second is how scheduled AI steps end up re-deriving the same findings on a loop.

Assets carry data, not the conversation around it

An asset holds a value: a table, a file, a model, a report. That is the right shape for pipeline data, and loading yesterday's asset is the right move when yesterday's output is genuinely today's input.

An agent's past rarely fits that shape. It includes why a source was distrusted, which definition a reviewer settled on, what was tried and abandoned, and the wording of the decision itself. That context lives in conversations around the work, not inside the asset value. A later run can load the asset perfectly and still know nothing about the reasoning that produced it, so it repeats the argument, re-asks the settled question, or quietly reverses a decision it never knew existed.

Passing context through assets also has to be designed in advance, asset by asset, job by job. The moment the same past is needed by a second job, a different orchestrator, or the assistant used in chat during the day, the asset graph is the wrong place to share it from.

A cursor remembers a position, not a past

The sensor cursor is the closest thing Dagster has to continuity, and its limits show the pattern clearly. A cursor holds one small value, typically the last key or timestamp processed. It answers "where did processing stop?" and nothing else.

That is exactly right for incremental ingestion. It is nowhere near what a recurring AI step needs, which is the accumulated understanding from every previous run: the patterns spotted, the edge cases named, the corrections applied. A position marker overwritten each cycle cannot hold a history, and it was never meant to. Continuity for agents has to live somewhere built to hold the full story, not a bookmark.

Where a shared memory layer fits

The workable pattern is to put memory underneath the orchestration instead of inside it. Vilix AI is a cloud-hosted memory layer built for that position: zero infrastructure to run or maintain, so existing jobs, schedules, and sensors stay exactly as they are.

Vilix AI is reached over MCP, so the memory is not tied to Dagster or to any one tool. The AI step in tonight's job, a step in a different automation tomorrow, and an assistant in a daytime chat all read and write the same memory. One past, available in every place the work happens.

It stores full conversation history, not just distilled facts, so a run can recover what was actually discussed and decided, in context, instead of working from a compressed summary. In practice the rhythm is simple: read memory near the start of the run so the AI step begins informed, and write the run's conclusions back near the end so the next run, in any tool, begins informed too.

There is no commitment attached to trying it. The free plan is free forever, the 7-day Pro trial needs no credit card, and stored memory can be exported or deleted at any time.

Keep the record. Add the memory.

Dagster's run history, asset lineage, and sensor cursors do their jobs well, and a serious pipeline should keep all three. Just do not ask them to be the memory your AI steps think with. Add a shared memory layer the runs read at the start and write at the end, and the schedule stops being a series of first days. The first run does the learning. Every run after it gets to build on it.