Skip to main content
data-engineering is the harness for changes to a data pipeline: a wrong number to fix, a dataset to build, a refactor, a cost or speed change, a migration, one change or a backlog of them. It ships inside the package and loads by name. Its pages call each change asked for a ticket, whether or not a ticketing system exists. dbt on a SQL warehouse is the stack it was worked on: the measurement scripts and the two delivery targets drive dbt and nothing else, and the pages say what to do by hand on another stack. What it holds is the discipline, not the task: the brief, the evaluator, the warehouse and the starting repository are yours.

Where it came from

The harness was written from published practice on data modelling, data quality, pipeline reliability and warehouse cost, then revised after independent reviews. Its rule is that a change is believed when the built data proves it: every assumption becomes a checked declaration before work starts, a fix removes a cause and never patches a symptom, and the proof matches the kind of change. The package’s gate runs check over each bundled harness at the packaged page budget and scans it for a benchmark’s names. The dbt on Snowflake example is a task set up to run on it.

What it ships

Two pages reach every session: instructions/guardrails.md, the standing rules (verify in data, not in code; fix causes; the proof matches the kind of change; production is read and never written by a session; the label is computed, never chosen), and the module’s own page, one each for setup, ideation, implementation, verification, feedback, deployment and learning. The skills, the delegates and the hooks are defined once and listed per module:

The skills

The delegates

root-cause-reviewer reviews one attempt’s diff against the ticket and the proof: a fix removes a demonstrated cause at the transformation responsible for it, and any other kind of change carries the evidence its kind obliges. validation-auditor checks one attempt’s proof against those obligations and the attempt’s own files. Each reports pass, fail or inconclusive with evidence, and neither edits a file.

The hooks

  • session-brief, when a session starts: one paragraph on what setup pinned and where the attempt stands. Before setup has written anything it says nothing.
  • protect-prepared, on the file tools: what setup pinned under prepared/ is read-only, since it is what every attempt is measured against.
  • shared-layer-notice, after a write: a note when the file sits in a shared layer of the project (a staging model, a source, a snapshot, a test), naming what the proof must then carry. It never denies.
  • require-proof, when an implementation session stops: it asks once for the proof file, with the label the measurements earn.

The label

A change’s label is computed from measurements taken before its first edit and after its last, never chosen. safe to merge needs every obligation of the kind of change met by a measurement, taken again by the verification session; anything missing gives unsure with what is missing as the reason; won't do is a premise shown wrong, with the same proof a fix needs. On a dbt project one script takes the measurements (evidence.py, in the validation-report skill). It reads prepared/environment.json, which the setup session writes from the repository’s own documentation: the dbt targets a session builds on and reads production through, and, where the repository has them, a reset command, a check a session may run, and a release command. Every key is optional, and a key left out is a measurement no attempt can take: the label says so. dbt and git are expected on PATH; the scripts use the standard library only.

The delivery targets

The deployment page ships two targets for when the task’s channel is theirs, and kapso deploy names one as Targets describes for a harness target:
pull-request opens the change for review on GitHub through the gh command, with each title and body rendered from the proof. warehouse-release releases what a reviewer merged, through a release command of the repository’s that rehearses a release, publishes it and rolls it back; a repository without such a command has no release path this target can drive, and it says so.

Run a task on it

The brief’s Harness data-engineering line and kapso.load_harness("data-engineering") name it the same way. The campaign then runs as Use a harness describes: the setup session pins the measurement context and, for a backlog, writes the inventory of its tickets; each implementation session closes a ticket by taking a baseline, making the change, measuring it and writing its proof; each attempt is verified by a session that measures every closed ticket again from the attempt’s code and asks the two delegates. Related pages: Use a harness · Agent engineering · Predictive modelling · Write a harness · Harness reference · Deployment targets Kapso is an open-source engine by Leeroo that runs experiments toward an objective, deploys the best solution and learns from every run. Source code: github.com/Leeroo-AI/kapso · Install: pip install leeroo-kapso · Every page as plain text: llms.txt.