davide
cdda929537
Clone del repo RAG, cella config modificabile che sovrascrive chunks/config.py, esecuzione chunker.py e download automatico di chunks.json. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
304 lines
17 KiB
Plaintext
304 lines
17 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "ozeCGy0fwA4R"
|
||
},
|
||
"source": [
|
||
"# Conversione PDF con MinerU"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "zyMHLIPh3sCk"
|
||
},
|
||
"source": [
|
||
"[](https://github.com/opendatalab/MinerU)\n",
|
||
"[](https://github.com/opendatalab/MinerU)\n",
|
||
"[](https://github.com/opendatalab/MinerU/issues)\n",
|
||
"[](https://github.com/opendatalab/MinerU/issues)\n",
|
||
"[](https://pypi.org/project/mineru/)\n",
|
||
"[](https://pypi.org/project/mineru/)\n",
|
||
"[](https://pepy.tech/project/mineru)\n",
|
||
"[](https://pepy.tech/project/mineru)\n",
|
||
"[](https://mineru.net/OpenSourceTools/Extractor?source=github)\n",
|
||
"[](https://huggingface.co/spaces/opendatalab/MinerU)\n",
|
||
"[](https://www.modelscope.cn/studios/OpenDataLab/MinerU)\n",
|
||
"[](https://colab.research.google.com/gist/myhloli/a3cb16570ab3cfeadf9d8f0ac91b4fca/mineru_demo.ipynb)\n",
|
||
"[](https://arxiv.org/abs/2409.18839)\n",
|
||
"[](https://arxiv.org/abs/2509.22186)\n",
|
||
"[](https://deepwiki.com/opendatalab/MinerU)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "F9dnYLSDwQyR"
|
||
},
|
||
"source": [
|
||
"## 1.Install"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Rw-hwakFwaeY"
|
||
},
|
||
"source": [
|
||
"### Install via pip"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/",
|
||
"height": 1000
|
||
},
|
||
"id": "_3EDplqlxKB_",
|
||
"outputId": "a4c96ba2-6dea-4ef0-aac7-955dbfac0598"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"!pip install --upgrade pip\n",
|
||
"!pip install \"mineru[all]>=3.1.3\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "rSfS77AisXH8"
|
||
},
|
||
"source": [
|
||
"### Check verison"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "MSyCKHmqCp_j",
|
||
"outputId": "eb0aad89-3261-4388-9dfc-2b6ce29a3030"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"mineru, version 3.1.3\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"!mineru -v"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Qh8YKU19CrQY"
|
||
},
|
||
"source": [
|
||
"## 2.Usage"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "3dZlkl3jDNKL"
|
||
},
|
||
"source": [
|
||
"### 2.1 Command line"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "mVRMM_FaDi2d"
|
||
},
|
||
"source": [
|
||
"#### **PDF**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Hyoz1B-tPXcM"
|
||
},
|
||
"source": [
|
||
"#### **Converti un singolo PDF**\n",
|
||
"\n",
|
||
"Su Google Colab il file system è temporaneo: ogni volta che avvii una nuova sessione la cartella `/content/` è vuota.\n",
|
||
"Devi quindi **caricare il tuo PDF prima di eseguire la conversione**.\n",
|
||
"\n",
|
||
"**Come caricare il file:**\n",
|
||
"\n",
|
||
"1. Nel pannello di sinistra clicca sull'icona **File**.\n",
|
||
"2. Trascina il PDF nella cartella `/content/` (radice della sessione), oppure clicca il pulsante *Carica* (icona freccia su).\n",
|
||
"3. Attendi che il caricamento sia completato: il file comparirà nell'elenco.\n",
|
||
"\n",
|
||
"In alternativa puoi caricare via codice:\n",
|
||
"\n",
|
||
"```python\n",
|
||
"from google.colab import files\n",
|
||
"uploaded = files.upload() # apre il selettore file\n",
|
||
"```\n",
|
||
"\n",
|
||
"Una volta caricato, il PDF si trova in `/content/<nome-file>.pdf`.\n",
|
||
"Sostituisci `mio_documento.pdf` nella cella seguente con il nome del tuo file:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"PDF_NAME = \"mio_documento.pdf\" # ← sostituisci con il nome del file che hai caricato\n",
|
||
"import os\n",
|
||
"STEM = os.path.splitext(os.path.basename(PDF_NAME))[0]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"collapsed": true,
|
||
"id": "6RivAWteLeYM",
|
||
"outputId": "dc56bcc0-6e84-432a-8511-e461ff605a51"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"!mineru -p {PDF_NAME} -o ./output -b pipeline"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/",
|
||
"height": 807
|
||
},
|
||
"id": "F6K4E8CfH6_V",
|
||
"outputId": "136175e0-c1b0-4c80-84c4-ce7e7c78e656"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import os, zipfile, shutil\n",
|
||
"\n",
|
||
"# Cerca la cartella di output (backend auto o hybrid_auto)\n",
|
||
"candidates = [\n",
|
||
" f'./output/{STEM}/hybrid_auto',\n",
|
||
" f'./output/{STEM}/auto',\n",
|
||
" f'./output/{STEM}',\n",
|
||
"]\n",
|
||
"output_dir = next((p for p in candidates if os.path.isdir(p)), None)\n",
|
||
"\n",
|
||
"if output_dir is None:\n",
|
||
" print(f'Cartella output non trovata per \"{STEM}\". Percorsi cercati:')\n",
|
||
" for p in candidates:\n",
|
||
" print(' ', p)\n",
|
||
"else:\n",
|
||
" zip_path = f'./{STEM}_output.zip'\n",
|
||
" with zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as zf:\n",
|
||
" for root, dirs, files in os.walk(output_dir):\n",
|
||
" for file in files:\n",
|
||
" full_path = os.path.join(root, file)\n",
|
||
" arcname = os.path.relpath(full_path, start=os.path.dirname(output_dir))\n",
|
||
" zf.write(full_path, arcname)\n",
|
||
" size_mb = os.path.getsize(zip_path) / (1024 * 1024)\n",
|
||
" print(f'ZIP creato: {zip_path} ({size_mb:.1f} MB)')\n",
|
||
"\n",
|
||
" try:\n",
|
||
" from google.colab import files\n",
|
||
" files.download(zip_path)\n",
|
||
" except ImportError:\n",
|
||
" print('(download automatico disponibile solo su Google Colab)')\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "t53WETM90pKr"
|
||
},
|
||
"source": [
|
||
"#### **Converti più PDF nella stessa cartella**\n",
|
||
"\n",
|
||
"Converte tutti i PDF presenti in `/content/` in un'unica esecuzione.\n",
|
||
"\n",
|
||
"Carica i file prima di eseguire la cella (trascina i PDF nel pannello File a sinistra oppure usa `files.upload()`).\n",
|
||
"\n",
|
||
"L'output di ogni PDF viene salvato in `./output/<nome-file>/auto/`.\n",
|
||
"\n",
|
||
"> Usa questa cella al posto di quella precedente se hai più documenti da convertire."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/"
|
||
},
|
||
"id": "4dqBGZiz0orA",
|
||
"outputId": "8233269d-d912-4dd3-9332-d26bd464db7c"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"!mineru -p ./ -o ./output -b pipeline"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": "## 3. Chunking\n\nScarica il repo RAG e suddivide il Markdown in chunk pronti per la vettorizzazione."
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": "!git clone https://santantonio.sytes.net/davide/rag-from-scratch.git /content/rag 2>/dev/null || (cd /content/rag && git pull)"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": "# ── Parametri chunking — modifica qui prima di eseguire ──────────────────────\n\nMAX_CHARS = 1200 # lunghezza massima chunk (caratteri)\nMIN_CHARS = 80 # soglia minima (paragrafi più corti vengono fusi)\nCONTEXT_DEPTH = 3 # livelli heading inclusi nel prefisso (1–3)\nMERGE_SHORT_PARAGRAPHS = True\nSKIP_PRE_HEADING = True # salta contenuto prima del primo heading\nSKIP_HEADINGS = {\n \"indice\",\n \"sommario\",\n \"bibliografia\",\n \"ringraziamenti\",\n \"abbreviazioni\",\n}\n\n# ── Applica al config del repo ────────────────────────────────────────────────\n\nconfig_path = \"/content/rag/chunks/config.py\"\nwith open(config_path) as f:\n src = f.read()\n\nimport re\n\ndef _replace(src, name, value):\n if isinstance(value, bool):\n v = \"True\" if value else \"False\"\n return re.sub(rf\"^{name}\\s*=.*$\", f\"{name}: bool = {v}\", src, flags=re.MULTILINE)\n elif isinstance(value, int):\n return re.sub(rf\"^{name}\\s*=.*$\", f\"{name}: int = {value}\", src, flags=re.MULTILINE)\n elif isinstance(value, set):\n items = \",\\n \".join(f'\"{s}\"' for s in sorted(value))\n block = f\"{name}: set[str] = {{\\n {items},\\n}}\"\n return re.sub(rf\"^{name}.*?^\\}}\", block, src, flags=re.MULTILINE | re.DOTALL)\n return src\n\nfor name, val in [\n (\"MAX_CHARS\", MAX_CHARS),\n (\"MIN_CHARS\", MIN_CHARS),\n (\"CONTEXT_DEPTH\", CONTEXT_DEPTH),\n (\"MERGE_SHORT_PARAGRAPHS\", MERGE_SHORT_PARAGRAPHS),\n (\"SKIP_PRE_HEADING\", SKIP_PRE_HEADING),\n (\"SKIP_HEADINGS\", SKIP_HEADINGS),\n]:\n src = _replace(src, name, val)\n\nwith open(config_path, \"w\") as f:\n f.write(src)\n\nprint(\"Config applicato:\", config_path)\n"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": "import os, sys, shutil\nsys.path.insert(0, \"/content/rag\")\n\n# Copia l'output MinerU dove il chunker si aspetta di trovarlo\nstem_output_src = f\"./output/{STEM}/auto\"\nstem_output_dst = f\"/content/rag/sources/{STEM}_output/auto\"\nif os.path.isdir(stem_output_src):\n os.makedirs(stem_output_dst, exist_ok=True)\n shutil.copytree(stem_output_src, stem_output_dst, dirs_exist_ok=True)\n\n!python /content/rag/chunks/chunker.py --stem {STEM}\n\n# Scarica chunks.json\nchunks_path = f\"/content/rag/chunks/{STEM}/chunks.json\"\nif os.path.exists(chunks_path):\n try:\n from google.colab import files\n files.download(chunks_path)\n except ImportError:\n print(f\"Chunk pronti in: {chunks_path}\")\nelse:\n print(\"chunks.json non trovato — controlla l'output sopra.\")\n"
|
||
}
|
||
],
|
||
"metadata": {
|
||
"accelerator": "GPU",
|
||
"colab": {
|
||
"gpuType": "T4",
|
||
"provenance": []
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"name": "python"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 0
|
||
} |