116 lines
6.2 KiB
Markdown
116 lines
6.2 KiB
Markdown
# Step 9 - manifest, cost tracker, cleanup, final report
|
|
|
|
Moved out of SKILL.md; read it when the pipeline reaches Step 9.
|
|
|
|
### Step 9 - Save manifest, update cost tracker, clean up, and report
|
|
|
|
```bash
|
|
$(cat graphify-out/.graphify_python) -c "
|
|
import json
|
|
from pathlib import Path
|
|
from datetime import datetime, timezone
|
|
from graphify.detect import save_manifest
|
|
|
|
# Save manifest for --update
|
|
detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\"utf-8\"))
|
|
extract = json.loads(Path('graphify-out/.graphify_extract.json').read_text(encoding=\"utf-8\"))
|
|
INPUT_PATH = Path('graphify-out/.graphify_root').read_text(encoding=\"utf-8-sig\").strip()
|
|
# In --update mode, 'all_files' carries the full corpus; 'files' is the changed
|
|
# subset. Full-rebuild mode populates only 'files', so the fallback handles that.
|
|
# root= relativizes the manifest keys to the scan root (same base as the build),
|
|
# so the on-disk manifest is portable across clones/machines and a later --update
|
|
# matches cached files instead of missing every one (#1417).
|
|
#
|
|
# Only stamp semantic files (docs/papers/images) that ACTUALLY produced output:
|
|
# a detected file whose chunk failed or was omitted must stay unstamped so the
|
|
# next --update re-queues it, otherwise it is marked done and its content is lost
|
|
# forever (#2015). This mirrors the library extract path exactly
|
|
# (cli._stamped_manifest_files + clear_semantic + scan_corpus); do not stamp the
|
|
# raw corpus. Code files are always stamped (AST is deterministic); only semantic
|
|
# types are gated on output.
|
|
from graphify.cli import _stamped_manifest_files
|
|
_corpus = detect.get('all_files') or detect['files']
|
|
# A file a subagent only partly read DID produce output, so the stamp filter alone would
|
|
# keep it - but its node set covers just the head of the file, and stamping it means
|
|
# detect_incremental reads it as done and never re-queues it, leaving the truncated graph
|
|
# live forever. Name those files so they stay unstamped and are retried next run (#2225).
|
|
_pf = Path('graphify-out/.graphify_partial.txt')
|
|
_partial = {l.strip() for l in _pf.read_text(encoding=\"utf-8\").splitlines() if l.strip()} if _pf.exists() else set()
|
|
_manifest_files = _stamped_manifest_files(_corpus, extract, Path(INPUT_PATH), partial_source_files=_partial)
|
|
# Files dispatched this run (the changed subset) but NOT stamped above still carry
|
|
# a stale semantic_hash from a prior run; clear it so detect_incremental re-queues
|
|
# them instead of reading them as unchanged (#1948).
|
|
_sem_types = ('document', 'paper', 'image')
|
|
_dispatched = {f for t, fl in detect['files'].items() if t in _sem_types for f in fl}
|
|
_stamped = {f for fl in _manifest_files.values() for f in fl}
|
|
_cleared = _dispatched - _stamped
|
|
# scan_corpus = the RAW full corpus (not the stamp-filtered subset) so in-root
|
|
# files newly excluded since last run are dropped rather than masquerading as
|
|
# deletions; untouched files' prior rows are still preserved (#1908).
|
|
_scan = {f for fl in _corpus.values() for f in fl}
|
|
save_manifest(_manifest_files, root=INPUT_PATH, scan_corpus=_scan, clear_semantic=_cleared or None)
|
|
|
|
# Update cumulative cost tracker
|
|
input_tok = extract.get('input_tokens', 0)
|
|
output_tok = extract.get('output_tokens', 0)
|
|
|
|
cost_path = Path('graphify-out/cost.json')
|
|
if cost_path.exists():
|
|
cost = json.loads(cost_path.read_text(encoding=\"utf-8\"))
|
|
else:
|
|
cost = {'runs': [], 'total_input_tokens': 0, 'total_output_tokens': 0}
|
|
|
|
cost['runs'].append({
|
|
'date': datetime.now(timezone.utc).isoformat(),
|
|
'input_tokens': input_tok,
|
|
'output_tokens': output_tok,
|
|
'files': detect.get('total_files', 0),
|
|
})
|
|
cost['total_input_tokens'] += input_tok
|
|
cost['total_output_tokens'] += output_tok
|
|
cost_path.write_text(json.dumps(cost, indent=2, ensure_ascii=False), encoding=\"utf-8\")
|
|
|
|
print(f'This run: {input_tok:,} input tokens, {output_tok:,} output tokens')
|
|
print(f'All time: {cost[\"total_input_tokens\"]:,} input, {cost[\"total_output_tokens\"]:,} output ({len(cost[\"runs\"])} runs)')
|
|
"
|
|
rm -f graphify-out/.graphify_detect.json graphify-out/.graphify_extract.json graphify-out/.graphify_ast.json graphify-out/.graphify_semantic.json graphify-out/.graphify_analysis.json graphify-out/.graphify_partial.txt
|
|
find graphify-out -maxdepth 1 -name '.graphify_chunk_*.json' -delete 2>/dev/null
|
|
rm -f graphify-out/.needs_update 2>/dev/null || true
|
|
```
|
|
|
|
INPUT_PATH is read from `graphify-out/.graphify_root` (the scan root Steps 4-5 used), so the manifest is relativized to the scan root.
|
|
|
|
Tell the user (omit the obsidian line unless --obsidian was given):
|
|
```
|
|
Graph complete. Outputs in PATH_TO_DIR/graphify-out/
|
|
|
|
graph.html - interactive graph, open in browser
|
|
GRAPH_REPORT.md - audit report
|
|
graph.json - raw graph data
|
|
obsidian/ - Obsidian vault (only if --obsidian was given)
|
|
```
|
|
|
|
If graphify saved you time, consider supporting it: https://github.com/sponsors/safishamsi
|
|
|
|
Replace PATH_TO_DIR with the actual absolute path of the directory that was processed.
|
|
|
|
If PATH_TO_DIR is inside a git repository, add one line to the report: `graphify-out/` was written into the working tree, so unless the repository already ignores it, it will show in `git status`; the user can add `graphify-out/` to `.gitignore`, or commit only the outputs their team should share (see https://docs.graphify.com/guides/team-workflows).
|
|
|
|
Then paste these sections from GRAPH_REPORT.md directly into the chat:
|
|
- God Nodes
|
|
- Surprising Connections
|
|
- Suggested Questions
|
|
|
|
Do NOT paste the full report - just those three sections. Keep it concise.
|
|
|
|
Then immediately offer to explore. Pick the single most interesting suggested question from the report - the one that crosses the most community boundaries or has the most surprising bridge node - and ask:
|
|
|
|
> "The most interesting question this graph can answer: **[question]**. Want me to trace it?"
|
|
|
|
If the user says yes, run `/graphify query "[question]"` on the graph and walk them through the answer using the graph structure - which nodes connect, which community boundaries get crossed, what the path reveals. Keep going as long as they want to explore. Each answer should end with a natural follow-up ("this connects to X - want to go deeper?") so the session feels like navigation, not a one-shot report.
|
|
|
|
The graph is the map. Your job after the pipeline is to be the guide.
|
|
|
|
---
|
|
|