Skill Store 33

Indexion Segment

Split text into contextual chunks for RAG/embedding pipelines.

by trkbt10 · Apache-2.0

The skill

Split text into contextual chunks for RAG/embedding pipelines. Document segmentation and section extraction using window, tfidf, punctuation, or hybrid strategies chosen by intent.

indexion segment

Split text into contextual segments using divergence-based, TF-IDF, or punctuation strategies.

When to Use

  • User needs to chunk text for RAG or embedding pipelines
  • User wants to split a document into meaningful sections
  • User asks to segment text for processing
  • Preparing text for similarity analysis at sub-document level

Usage

# Default window divergence strategy
indexion segment <input-file> <output-dir>

# TF-IDF based segmentation
indexion segment --strategy=tfidf <input-file> <output-dir>

# Punctuation-based segmentation
indexion segment --strategy=punctuation <input-file> <output-dir>

# Custom segment sizes
indexion segment --min-size=200 --max-size=3000 --target-size=800 document.txt output/

# Custom divergence threshold
indexion segment --threshold=0.5 document.txt output/

# Adaptive threshold mode (default)
indexion segment --adaptive document.txt output/

# Hybrid NCD+TF-IDF mode
indexion segment --hybrid --ncd-weight=0.6 --tfidf-weight=0.4 document.txt output/

# Custom window size
indexion segment --window-size=5 document.txt output/

# Custom output prefix
indexion segment --prefix=chunk document.txt output/

Options

Option Default Description
--strategy=NAME window Strategy: window, tfidf, punctuation
--min-size=INT 100 Minimum segment characters
--max-size=INT 2000 Maximum segment characters
--target-size=INT 500 Target segment characters
--threshold=FLOAT 0.42 Divergence threshold
--window-size=INT 3 Window size
--adaptive true Adaptive threshold mode
--hybrid false NCD+TF-IDF hybrid mode
--ncd-weight=FLOAT 0.5 NCD weight in hybrid mode
--tfidf-weight=FLOAT 0.5 TF-IDF weight in hybrid mode
--prefix=NAME segment Output file prefix

Strategies

Strategy Description
window (default) Sliding window divergence detection
tfidf TF-IDF based topic change detection
punctuation Punctuation/sentence boundary based

Workflow

  1. Run indexion segment <input-file> <output-dir> to split text with defaults
  2. Adjust --threshold and --target-size to tune segmentation granularity
  3. Use --hybrid mode for better accuracy on mixed-content documents

Use this skill

Connect your agent to the Skill Store MCP server and ask for it by ref: trkbt10/indexion-segment-58988529041.

Download SKILL.md · Markdown version

Details

Publisher
trkbt10
Category
Data & Analytics › Data pipelines & ETL
License
Apache-2.0
Updated
2026-09-22

References

More from trkbt10

  • Indexion Agent Orient Generate and use a pre-edit structure brief so coding agents learn likely owners, consumer surfac…
  • Indexion Documentation Documentation analysis — assess coverage, detect code-to-doc drift with plan reconcile, visualize…
  • Indexion Identity Detect and review name/content drift in code using `indexion identity audit`, then plan verified…
  • Indexion Kgf Debug and inspect KGF specs — view tokenization results, parse trees, and extracted edges from so…

Related skills