|
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778 |
- # Semantic Splitting Strategy
-
- When the source content is large (exceeds ~15,000 tokens) or a token_budget requires it, split the distillate into semantically coherent sections rather than arbitrary size breaks.
-
- ## Why Semantic Over Size-Based
-
- Arbitrary splits (every N tokens) break coherence. A downstream workflow loading "part 2 of 4" gets context fragments. Semantic splits produce self-contained topic clusters that a workflow can load selectively — "give me just the technical decisions section" — which is more useful and more token-efficient for the consumer.
-
- ## Splitting Process
-
- ### 1. Identify Natural Boundaries
-
- After the initial extraction and deduplication (Steps 1-2 of the compression process), look for natural semantic boundaries:
- - Distinct problem domains or functional areas
- - Different stakeholder perspectives (users, technical, business)
- - Temporal boundaries (current state vs future vision)
- - Scope boundaries (in-scope vs out-of-scope vs deferred)
- - Phase boundaries (analysis, design, implementation)
-
- Choose boundaries that produce sections a downstream workflow might load independently.
-
- ### 2. Assign Items to Sections
-
- For each extracted item, assign it to the most relevant section. Items that span multiple sections go in the root distillate.
-
- Cross-cutting items (items relevant to multiple sections):
- - Constraints that affect all areas → root distillate
- - Decisions with broad impact → root distillate
- - Section-specific decisions → section distillate
-
- ### 3. Produce Root Distillate
-
- The root distillate contains:
- - **Orientation** (3-5 bullets): what was distilled, from what sources, for what consumer, how many sections
- - **Cross-references**: list of section distillates with 1-line descriptions
- - **Cross-cutting items**: facts, decisions, and constraints that span multiple sections
- - **Scope summary**: high-level in/out/deferred if applicable
-
- ### 4. Produce Section Distillates
-
- Each section distillate must be self-sufficient — a reader loading only one section should understand it without the others.
-
- Each section includes:
- - **Context header** (1 line): "This section covers [topic]. Part N of M from [source document names]."
- - **Section content**: thematically-grouped bullets following the same compression rules as a single distillate
- - **Cross-references** (if needed): pointers to other sections for related content
-
- ### 5. Output Structure
-
- Create a folder `{base-name}-distillate/` containing:
-
- ```
- {base-name}-distillate/
- ├── _index.md # Root distillate: orientation, cross-cutting items, section manifest
- ├── 01-{topic-slug}.md # Self-contained section
- ├── 02-{topic-slug}.md
- └── 03-{topic-slug}.md
- ```
-
- Example:
- ```
- product-brief-distillate/
- ├── _index.md
- ├── 01-problem-solution.md
- ├── 02-technical-decisions.md
- └── 03-users-market.md
- ```
-
- ## Size Targets
-
- When a token_budget is specified:
- - Root distillate: ~20% of budget (orientation + cross-cutting items)
- - Remaining budget split proportionally across sections based on content density
- - If a section exceeds its proportional share, compress more aggressively or sub-split
-
- When no token_budget but splitting is needed:
- - Aim for sections of 3,000-5,000 tokens each
- - Root distillate as small as possible while remaining useful standalone
|