Tooluniverse Lab Data Standardization

Lab Instrument Data Standardization

Convert raw instrument output into a standardized, structured format — distinguishing raw measurements from calculated/derived values with explicit traceability — for LIMS/ELN upload, data lakes, or downstream analysis.

Honesty contract (read first)

1. Never fabricate a value. Every number in the output JSON/CSV must come directly from the source file. If a value can't be found or confidently parsed, it is omitted — never guessed.
2. Never claim certified ASM output you didn't produce. Tier 1 (native allotropy parsing) produces genuine Allotrope Simple Model documents. Tier 2 (fallback, this skill's own generic parser) produces a structurally similar but NOT validated document, and is always tagged "tier": "fallback" with an explanatory "note" — never presented as equivalent to a certified conversion.
3. Traceability or it doesn't count. A "calculated" value with no link back to the raw measurement it came from is not emitted as calculated — the fallback parser only classifies a value as calculated when it can also identify at least one raw column in the same row to point at.
4. Say so if a dependency is missing. If allotropy isn't installed, print the pip install allotropy instruction and proceed with the fallback tier — do not pretend Tier 1 ran.

When to use vs. not

Use this skill when the user wants to:

• Convert a plate reader / spectrophotometer / qPCR / cell counter / chromatography export (CSV, TSV, XLSX, or a text-extractable PDF table) into a standardized structured format
• Separate raw instrument readings from calculated/derived values with traceability, ahead of a LIMS or data-lake upload
• Get a flattened 2D CSV version of an ASM-style document for spreadsheet import
• Validate that a produced JSON document is structurally sound (has proper traceability, numeric values, no duplicate IDs)

Do NOT use this skill for (route elsewhere):

• General-purpose tabular cleaning/reshaping unrelated to lab instruments -> tooluniverse-data-wrangling
• FASTQ QC -> tooluniverse-fastq-qc
• VCF / variant files -> tooluniverse-variant-analysis
• Proteomics raw files (mzML, RAW) -> tooluniverse-proteomics-analysis
• Single-cell matrices (h5ad, 10X) -> tooluniverse-single-cell

Two-tier conversion strategy

allotropy is an OPTIONAL dependency — it is not part of ToolUniverse's declared package dependencies. The scripts detect it at runtime via try: import allotropy and never error out if it's missing; they just report the install command and use Tier 2. Never add allotropy to pyproject.toml — it stays opt-in per-user.

See references/supported_instruments.md for exactly what each tier covers, and how to enumerate the installed allotropy Vendor list yourself.

Workflow

1. Identify the input file (CSV/TSV/XLSX/PDF) and, for Excel, which sheet if not the first.
2. Run the converter:

   python scripts/convert_instrument_data.py <input_file> --output-dir <out_dir>
   

   This auto-detects the tier (tries native allotropy first if installed and --vendor is given/inferrable, otherwise uses the fallback parser automatically) and writes <stem>_asm.json and <stem>_flat.csv into <out_dir>.
3. Read the console output. It states which tier ran and how many measurements/calculated values were found. Zero measurements is flagged as a warning, not silently reported as success.
4. Validate the output:

   python scripts/validate_output.py <out_dir>/<stem>_asm.json
   # or, to escalate every warning to a failure:
   python scripts/validate_output.py <out_dir>/<stem>_asm.json --strict
   

5. Re-flatten on demand (if you already have the JSON and just need the CSV regenerated, e.g. after hand-editing the JSON):

   python scripts/flatten_to_csv.py <out_dir>/<stem>_asm.json -o <path>.csv
   

   (Only supported for tier: "fallback" documents — a native allotropy document has its own internal ASM structure and needs allotropy's own flattening utilities, not this script.)
6. Report which tier ran, the measurement/calculated-value counts, any validator warnings/errors, and where the output files were written. Never report a value the tool didn't actually produce.

Options reference (convert_instrument_data.py)

References

• references/schema_overview.md — the JSON document shape and field rules
• references/supported_instruments.md — what Tier 1 vs. Tier 2 actually parse
• references/field_classification_guide.md — raw vs. calculated heuristics, with a table of common calculated fields by instrument type