davide/rag-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
102 Commits 1 Branch 0 Tags
f947131f16350f6ac96d34df2d612211a3f636e1
Commit Graph

11 Commits

Author SHA1 Message Date
davide 3b61df73d3 feat(chunks): rilevamento universale has_math/has_table + dollarmath plugin 
- parser.py: integra dollarmath_plugin (graceful fallback) — blocchi $$…$$ → token math_block atomico
- segmenter.py: gestisce math_block → Block(kind=math, atomic=True, plain_text=[formula])
- packer.py: has_table rileva anche <table> in blocchi html; has_math rileva $$ e \begin{ nel contenuto
- +11 test (parser×3, segmenter×3, packer×5) — totale 65 test

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-09 08:40:41 +02:00
davide 1980efb0d6 feat(chunks): aggiunge chunker.py — CLI + orchestrazione pipeline AST-based 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-09 08:21:54 +02:00
davide ed42e1ba7b feat(chunks): aggiunge validator.py — invarianti + metriche 
Implementa validate() che controlla chunk_id duplicati, fence rotti,
size compliance (esclusi overflow) e calcola metriche aggregate.
Fix fixture cfg: min_chars=40 per allinearla ai chunk da 50 chars nel test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-09 08:02:35 +02:00
davide d2b8baa844 feat(chunks): aggiunge packer.py — Block[] → Chunk[] con packing min/target/max 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-08 16:29:24 +02:00
davide 2d2d478a3f feat(chunks): aggiunge segmenter.py — token stream → Block[] 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-08 16:25:49 +02:00
davide 63ca7121b2 feat(chunks): aggiunge parser.py con markdown-it-py 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-08 16:22:58 +02:00
davide bca01dc1c6 feat(chunks): aggiunge ChunkerConfig dataclass 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-08 16:20:59 +02:00
davide be1347378b feat(chunks): aggiunge models.py con Block, Chunk, Diagnostics, ChunkingResult 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-08 16:20:30 +02:00
davide be8ae9f6b8 chore: rimuove chunker legacy, aggiunge dipendenze AST-based pipeline 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-08 16:16:07 +02:00
davide 52e881d09c refactor: rimuove pipeline conversione PDF→MD, delegata a MinerU 
Elimina conversione/ (9 stadi PyMuPDF) e tests/ (tutti unit/integration).
Il repository gestisce ora solo: pulizia MD, chunking, vettorizzazione e RAG.
Aggiorna CLAUDE.md e .gitignore per il nuovo flusso MinerU-first.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-20 14:21:16 +02:00
davide 83e6ba9083 feat: integra pipeline PDF→Markdown a 9 stadi e test suite 
Porta da branch marker la riscrittura completa di conversione/_pipeline/
(9 stadi PyMuPDF) e la suite tests/ senza modificare il resto del progetto
RAG (ollama/, step-5/, step-6/, step-8/, rag.py, retrieve.py, config.py).

requirements.txt: aggiunge PyMuPDF>=1.24.0 e pytest>=8.0, mantiene chromadb,
rimuove opendataloader-pdf e pymupdf4llm.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-11 14:44:16 +02:00