How Kapso turns repositories and papers into wiki pages

Two stages turn a raw source into knowledge a campaign can search: ingestors extract structured wiki pages from each source, then the merger folds those pages into the graph. learn_knowledge() runs both.

How do I run the pipeline?

Full pipeline

from kapso import Kapso
from kapso.knowledge_base.learners import KnowledgePipeline, Source

kapso = Kapso()
pipeline = KnowledgePipeline(wiki_dir="data/wikis", index_builder=kapso.index_kg)

# Full pipeline: ingest + merge
result = pipeline.run(Source.Repo("https://github.com/user/repo"))
print(f"Created: {result.created}, Edited: {result.edited}")

index_builder builds the wiki’s search index when it has none yet (the first merge into a fresh wiki_dir); Kapso.index_kg reads the search backends and the collection from the config. kapso.learn_knowledge() wires this itself.

Via Kapso API

from kapso import Kapso, Source

kapso = Kapso()

# Ingest multiple sources
findings = kapso.research("QLoRA best practices")
kapso.learn_knowledge(
    Source.Repo("https://github.com/huggingface/transformers"),
    findings,
    wiki_dir="data/wikis",
)

Extract only (no merge)

# Get WikiPages without modifying KG
result = pipeline.run(
    Source.Repo("https://github.com/user/repo"),
    skip_merge=True,
)
pages = result.extracted_pages

What can I ingest?

The Source namespace provides typed wrappers for knowledge inputs:

Source Type	Description	Status
`Source.Repo(url, branch=None)`	Git repository, its default branch unless `branch` names one	Implemented
`Source.Solution(solution)`	Completed experiment	Basic
`Source.Idea(query, source, content)`	Research idea	Implemented
`Source.Implementation(query, source, content)`	Code implementation	Implemented
`Source.ResearchReport(query, content)`	Research report	Implemented

from kapso.knowledge_base.types import Source

# Repository source
repo = Source.Repo("https://github.com/user/repo")  # or branch="develop"

# Solution source (from evolve())
solution = kapso.evolve(goal="...")
sol_source = Source.Solution(solution)

# Research sources (from research())
findings = kapso.research("topic", mode=["idea", "implementation"])
# findings.ideas is List[Source.Idea]
# findings.implementations is List[Source.Implementation]

Stage 1: Ingestors

Ingestors extract WikiPages from sources. Each source type has a dedicated ingestor.

IngestorFactory

from kapso.knowledge_base.learners.ingestors import IngestorFactory

# Create by type
ingestor = IngestorFactory.create("repo")
pages = ingestor.ingest(Source.Repo("..."))

# Auto-detect from source
ingestor = IngestorFactory.for_source(source)

# List available ingestors
IngestorFactory.list_ingestors()  # ["repo", "solution", "idea", "implementation", "researchreport"]

RepoIngestor

The most sophisticated ingestor, using a two-branch multi-phase pipeline: Phase 0: Repository Understanding (pre-phase)

Parse repo structure, generate _RepoMap.md with AST info
Agent fills in natural language understanding for each file
Subsequent phases read this file instead of re-exploring

Branch 1: Workflow-Based Extraction

Phase 1a: Anchoring - Find workflows from README/examples, write Workflow pages + rough WorkflowIndex
Phase 1b: Anchoring Context - Enrich WorkflowIndex with detailed implementation context
Phase 2: Excavation+Synthesis - Trace imports, write Implementation-Principle pairs together
Phase 3: Enrichment - Mine constraints/tips, write Environment/Heuristic pages
Phase 4: Audit - Validate graph integrity, fix broken links
Phase 4b: Repo Builder - Publish each Workflow as a runnable GitHub repository. Runs after the deterministic validation, and only when learner.ingestor.publish_workflows is on (or github_org / is_private is passed); gh authenticates with GH_TOKEN

Branch 2: Orphan Mining (runs after Branch 1)

Phase 5a: Triage (code) - Deterministic filtering into AUTO_KEEP/AUTO_DISCARD/MANUAL_REVIEW
Phase 5b: Review (agent) - Agent evaluates MANUAL_REVIEW files
Phase 5c: Create (agent) - Agent creates wiki pages for approved files
Phase 6: Orphan Audit - Validate orphan nodes
Phase 6b: Verify (code) - Every file that must get a page is marked done and every review has a decision; undone work gets one targeted audit pass with the findings, and the run fails if any remains

Sessions. Every phase is a Claude Code session holding five file tools (Read, Write, Edit, Glob, Grep) and, for the merge, the knowledge tools, and nothing else: the built-in set is pinned with the CLI’s --tools flag and only Kapso’s own MCP servers are mounted, so a repository’s code or README cannot make the session run anything, reach the network, delegate to a sub-agent or touch the account’s connectors. The one exception is the Repo Builder, which also gets Bash for git and gh. Staging and resume. A run stages everything under <wiki_dir>/_staging/<repo>/<branch>: the clone, the repository map, the phase reports and the pages, which are copied into wiki_dir at the end. Each finished phase leaves a marker in _phases/. A run that stops, on a failed phase or a killed process, keeps the directory, and learning the same source again continues from the first unfinished phase; a completed run’s directory is replaced by the next run of that source. cleanup removes the clone when a run completes, cleanup_staging the whole directory.

Research ingestors

Research ingestors convert web research results into WikiPages using a three-phase agentic pipeline: Phase 1: Planning

Analyzes content and decides what pages to create
Writes _plan.md with page decisions

Phase 2: Writing

Creates wiki pages following section definitions
Writes pages to staging directory

Phase 3: Auditing

Validates pages and fixes issues
Ensures graph integrity

There are three specialized research ingestors:

Ingestor	Source Type	Description
`IdeaIngestor`	`idea`	Extracts principles from research ideas
`ImplementationIngestor`	`implementation`	Extracts implementations from code examples
`ResearchReportIngestor`	`researchreport`	Extracts comprehensive knowledge from reports

ideas = kapso.research("QLoRA best practices", mode="idea")  # List[Source.Idea]

# Ingest into WikiPages (auto-detects ingestor type)
ingestor = IngestorFactory.for_source(ideas[0])
pages = ingestor.ingest(ideas[0])

# Or create by type
ingestor = IngestorFactory.create("idea")
pages = ingestor.ingest(Source.Idea(query="...", source="...", content="..."))

Stage 2: Knowledge merger

The merger uses a hierarchical sub-graph-aware algorithm with a single Claude Code agent call. It processes connected pages as units, respecting the knowledge graph DAG structure.

Wiki hierarchy

The Knowledge Graph follows a top-down DAG structure:

Principle (Core Node - The Theory)
├── implemented_by → Implementation (MANDATORY 1+)
└── uses_heuristic → Heuristic (optional)

Implementation (The Code)
├── requires_env → Environment (optional)
└── uses_heuristic → Heuristic (optional)

Environment (Leaf - Target Only)
Heuristic (Leaf - Target Only)

Merge algorithm

The merger executes a 5-phase process: Phase 1: Sub-Graph Detection

Parses outgoing_links to build an adjacency list
Finds root nodes (no incoming edges from proposed pages)
Groups connected components into sub-graphs

Phase 2: Planning (Top-Down)

For each sub-graph, makes merge decisions starting from root:
- Root decision: Search for similar pages of same type → MERGE or CREATE_NEW
- Children decisions (recursive):
  - If parent = CREATE_NEW → child inherits CREATE_NEW (no search needed)
  - If parent = MERGE → search only among target’s children → MERGE or CREATE_NEW
- Special case: Heuristics with multiple parents use lowest parent for scoped search
Computes execution order (bottom-up): Environment → Heuristic → Implementation → Principle
Records deferred edges (which parent adds edge after processing)

Phase 3: Execution (Bottom-Up)

Processes nodes in computed order
For CREATE_NEW: Get page structure, prepare content, call kg_index
For MERGE: Get page structure, fetch target, merge content intelligently, call kg_edit
Updates outgoing_links to point to processed children’s result_page_id

Phase 4: Audit

Verifies nodes exist (CREATE_NEW/MERGE)
Verifies edges (parent has edge to child’s result_page_id)
Handles failures with retries (max 3)

Phase 5: Finalize

Collects all result_page_id values
Categorizes as created (CREATE_NEW) or edited (MERGE)
Writes final summary to _merge_plan.md

Edge types

From	Edge Type	To
Principle	`implemented_by`	Implementation
Principle	`uses_heuristic`	Heuristic
Implementation	`requires_env`	Environment
Implementation	`uses_heuristic`	Heuristic

Merge rules

Beyond the phase mechanics above: search is same-type only (Principles among Principles, never a Principle merged with an Implementation), and a merge keeps existing edges, only ever adding new ones.

MCP tools used

The merger uses these MCP tools via the kg-graph-search server:

Tool	Purpose
`search_knowledge`	Find similar pages in the graph
`get_wiki_page`	Read existing page content by title
`get_page_structure`	Get sections definition for a page type
`kg_index`	Create new page in the graph
`kg_edit`	Update existing page in the graph

Using the merger

from kapso.knowledge_base.learners import KnowledgeMerger

merger = KnowledgeMerger(index_builder=kapso.index_kg)
result = merger.merge(
    proposed_pages=pages,
    wiki_dir="data/wikis",
)

print(f"Created: {len(result.created)}")
print(f"Edited: {len(result.edited)}")
print(f"Failed: {len(result.failed)}")

Merge modes

The merger operates in two modes based on whether a KG index exists:

No Index Mode: Creates all pages as new, writes to wiki directory, then builds the index through index_builder (a merger without one fails here)
Merge Mode: Runs agentic hierarchical merge using MCP tools

In merge mode the result is settled from the store after the session, not from the plan file the session writes: created are the proposed pages the store now holds, edited the pages the session’s kg_edit calls named, and the remaining proposed pages were folded into those, so their files (published into wiki_dir before the merge) are removed and the index file’s page count follows the store. A page the session indexed that the store does not hold fails the merge, as does a proposed page that vanished while nothing was edited. Index detection order:

Explicit kg_index_path from config
Auto-detect .index file in wiki directory

Merge result

@dataclass
class MergeResult:
    total_proposed: int       # Pages proposed for merge
    subgraphs_processed: int  # Sub-graphs detected and processed
    created: List[str]        # New page IDs created
    edited: List[str]         # Existing pages updated
    failed: List[str]         # Pages that failed to process
    errors: List[str]         # Error messages
    plan_path: Optional[Path] # Path to merge plan file
    
    @property
    def success(self) -> bool  # True if no errors and no failures
    
    @property
    def total_processed(self) -> int  # created + edited count

WikiPage structure

@dataclass
class WikiPage:
    id: str                 # Unique page identifier
    page_type: str          # Page category
    overview: str           # Brief summary (the "card" content)
    content: str            # Full page content
    description: str        # Detailed text used for embedding
    domains: List[str]      # Topic domains
    sources: List[Dict]     # Source references
    last_updated: str       # Last update timestamp
    outgoing_links: List[Dict]  # Graph connections

How do I run it from the CLI?

The command runs Kapso.learn_knowledge(), so the preflight, the models and the search backends come from the config.

# Learn from a GitHub repository (its default branch)
python -m kapso.knowledge_base.learners https://github.com/user/repo

# Specify a branch
python -m kapso.knowledge_base.learners https://github.com/user/repo --branch develop

# Extract only (don't merge)
python -m kapso.knowledge_base.learners https://github.com/user/repo --extract-only

# Custom wiki directory
python -m kapso.knowledge_base.learners https://github.com/user/repo --wiki-dir ./my_wikis

# Your own config
python -m kapso.knowledge_base.learners https://github.com/user/repo --config ./config.yaml

CLI options

Option	Short	Description
`--branch`	`-b`	Git branch (default: the repository’s default branch)
`--extract-only`	`-e`	Only extract, don’t merge
`--wiki-dir`	`-w`	Wiki directory path
`--config`	`-c`	Config file (default: the packaged config)
`--verbose`	`-v`	Enable verbose logging

What does it return?

@dataclass
class PipelineResult:
    sources_processed: int
    total_pages_extracted: int
    merge_result: Optional[MergeResult]
    extracted_pages: List[WikiPage]
    errors: List[str]

    @property
    def created(self) -> int
    @property
    def edited(self) -> int
    @property
    def success(self) -> bool

Knowledge graph

What the graph holds

Search backends

How it is queried

Related pages: Knowledge graph · Search backends · Knowledge graph Kapso is an open-source framework by Leeroo that builds software toward measurable goals through experiment campaigns. Source code: github.com/Leeroo-AI/kapso · Install: pip install leeroo-kapso · Every page as plain text: llms.txt.

Getting started

Evolve system

Knowledge graph

Trajectory learning

Research

Deployment

Benchmarks

How Kapso turns repositories and papers into wiki pages

How do I run the pipeline?

Full pipeline

Via Kapso API

Extract only (no merge)

What can I ingest?

Stage 1: Ingestors

IngestorFactory

RepoIngestor

Research ingestors

Stage 2: Knowledge merger

Wiki hierarchy

Merge algorithm

Edge types

Merge rules

MCP tools used

Using the merger

Merge modes

Merge result

WikiPage structure

How do I run it from the CLI?

CLI options

What does it return?

Knowledge graph

Search backends

​How do I run the pipeline?

​Full pipeline

​Via Kapso API

​Extract only (no merge)

​What can I ingest?

​Stage 1: Ingestors

​IngestorFactory

​RepoIngestor

​Research ingestors

​Stage 2: Knowledge merger

​Wiki hierarchy

​Merge algorithm

​Edge types

​Merge rules

​MCP tools used

​Using the merger

​Merge modes

​Merge result

​WikiPage structure

​How do I run it from the CLI?

​CLI options

​What does it return?

​Related

Knowledge graph

Search backends

How do I run the pipeline?

Full pipeline

Via Kapso API

Extract only (no merge)

What can I ingest?

Stage 1: Ingestors

IngestorFactory

RepoIngestor

Research ingestors

Stage 2: Knowledge merger

Wiki hierarchy

Merge algorithm

Edge types

Merge rules

MCP tools used

Using the merger

Merge modes

Merge result

WikiPage structure

How do I run it from the CLI?

CLI options

What does it return?

Related