davide/rag-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(step-6): riconosci _word._ come terminatore valido in verify_chunks

Browse Source 
Rimuove falsi positivi per chunk che terminano con marcatori markdown
di enfasi (_) dopo punteggiatura di fine frase (es. _parola._).
Aggiunge U+2026 (…) alla lista di terminatori accettati.
This commit is contained in:
Davide Grilli
2026-04-15 13:33:39 +02:00
parent 8a36735741
commit 58de94e03b
1 changed files with 7 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
step-6/verify_chunks.py
+7 -3
View File
@@ -26,7 +26,7 @@ from pathlib import Path
MIN_CHARS = 200 MIN_CHARS = 200
MAX_CHARS = 800 MAX_CHARS = 800
PUNCT_END = re.compile("[.!?»)\\]'\u2019\"\u201c\u201d\u2018\u2014\u2013-]$") PUNCT_END = re.compile("[.!?»)\\]'\u2019\"\u201c\u201d\u2018\u2014\u2013\u2026-]$")
# ─── Checks ─────────────────────────────────────────────────────────────────── # ─── Checks ───────────────────────────────────────────────────────────────────
@@ -53,8 +53,12 @@ def ends_incomplete(chunk: dict) -> bool:
text = chunk.get("text", "").rstrip() text = chunk.get("text", "").rstrip()
if not text: if not text:
return False return False
# Controlla l'ultimo carattere non-whitespace # Rimuovi marcatori markdown finali (_ e *) prima di controllare:
return not PUNCT_END.search(text) # pattern come _parola._ o _parola!_ sono frasi complete.
text_check = re.sub(r"[_*]+$", "", text).rstrip()
if not text_check:
return False
return not PUNCT_END.search(text_check)
# ─── Report ─────────────────────────────────────────────────────────────────── # ─── Report ───────────────────────────────────────────────────────────────────