feat(rag): adatta pipeline allo schema chunks AST-based + ottimizza system prompt

config.py:
- EMBED_MODEL: qwen3-embedding:0.6b → bge-m3 (multilingua, migliore su testi accademici)
- SYSTEM_PROMPT: lingua esplicita, anti-allucinazione rafforzata, citazione strutturata
  con percorso sezione, passaggi numerati per spiegazioni, fallback al plurale

ingestion/ingest.py:
- embed su content_for_embedding (prefisso header contestuale)
- store content_original in ChromaDB (testo pulito per retrieval)
- metadata aggiornati: header_path, chunk_index, content_type, flags, start/end_line

rag.py, retrieve.py:
- sostituisce sezione/titolo (schema vecchio) con header_path (schema AST)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-09 09:10:35 +02:00
parent 89d77d35d6
commit 2939d2f8ca
4 changed files with 40 additions and 34 deletions
+16 -6
View File
@@ -139,18 +139,28 @@ def _ingest_stem(stem: str, collection: chromadb.Collection,
for i, chunk in enumerate(chunks, start=1):
t0 = time.monotonic()
vector = embed(chunk["text"], model)
vector = embed(chunk["content_for_embedding"], model)
t1 = time.monotonic()
durations.append(t1 - t0)
hp = chunk.get("header_path", [])
flags = chunk.get("flags", {})
ids.append(f"{stem}__{chunk['chunk_id']}")
embeddings.append(vector)
documents.append(chunk["text"])
documents.append(chunk["content_original"])
metadatas.append({
"source": stem,
"sezione": chunk.get("sezione", ""),
"titolo": chunk.get("titolo", ""),
"sub_index": chunk.get("sub_index", 0),
"source": stem,
"chunk_index": chunk.get("chunk_index", i - 1),
"content_type": chunk.get("content_type", ""),
"header_path": " > ".join(h["text"] for h in hp),
"start_line": chunk.get("start_line", 0),
"end_line": chunk.get("end_line", 0),
"chars": chunk.get("chars", 0),
"has_code": flags.get("has_code", False),
"has_table": flags.get("has_table", False),
"has_math": flags.get("has_math", False),
"is_overflow": flags.get("is_overflow", False),
})
avg = sum(durations) / len(durations)