docs: aggiunge notebook MinerU per Colab e configurazione rapida nel README
- Passo 1 ora presenta opzione Colab (mineru.ipynb) e installazione locale - Notebook adattato per uso reale: variabile PDF_NAME, path dinamico, gestione pagine - Nuova sezione "Configurazione rapida" con parametri chunking e RAG Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -176,6 +176,32 @@ pip install -r requirements.txt
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Configurazione rapida
|
||||||
|
|
||||||
|
Prima di usare la pipeline verifica i parametri nei due file di configurazione:
|
||||||
|
|
||||||
|
**`chunks/config.py`** — parametri di chunking:
|
||||||
|
|
||||||
|
| Parametro | Default | Quando cambiarlo |
|
||||||
|
|-----------|---------|-----------------|
|
||||||
|
| `MAX_CHARS` | 1200 | Aumenta per chunk più lunghi (es. testi narrativi) |
|
||||||
|
| `MIN_CHARS` | 80 | Abbassa se hai molti titoli brevi |
|
||||||
|
| `FRONTMATTER_HEADINGS` | set italiano | Svuota (`set()`) per documenti non italiani |
|
||||||
|
| `OVERLAP_SENTENCES` | 1 | Aumenta a 2 se il retrieval perde contesto tra chunk |
|
||||||
|
|
||||||
|
**`config.py`** — parametri RAG:
|
||||||
|
|
||||||
|
| Parametro | Default | Quando cambiarlo |
|
||||||
|
|-----------|---------|-----------------|
|
||||||
|
| `OLLAMA_MODEL` | `qwen3.5:4b` | Sostituisci con il modello Ollama che vuoi usare |
|
||||||
|
| `EMBED_MODEL` | `nomic-embed-text` | Deve corrispondere al modello usato in ingestion |
|
||||||
|
| `TOP_K` | 6 | Aumenta per recuperare più contesto per domanda |
|
||||||
|
| `OLLAMA_URL` | `localhost:11434` | Cambia se Ollama gira su un altro host/porta |
|
||||||
|
|
||||||
|
> Se cambi `EMBED_MODEL` devi rieseguire l'ingestion con `--force`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Flusso completo
|
## Flusso completo
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -205,15 +231,39 @@ sources/<stem>/auto/
|
|||||||
|
|
||||||
### Passo 1 — Converti il PDF con MinerU
|
### Passo 1 — Converti il PDF con MinerU
|
||||||
|
|
||||||
Usa MinerU per convertire il PDF e posiziona la cartella di output in `sources/`:
|
Per questo step è necessario **MinerU** ([github.com/opendatalab/MinerU](https://github.com/opendatalab/MinerU)).
|
||||||
|
|
||||||
|
**Opzione A — Google Colab (consigliata, nessuna installazione locale)**
|
||||||
|
|
||||||
|
Usa il notebook incluso in questo repository:
|
||||||
|
|
||||||
|
```
|
||||||
|
mineru_demo.ipynb
|
||||||
|
```
|
||||||
|
|
||||||
|
Aprilo su Google Colab, carica il tuo PDF nella sessione e segui le celle in ordine. Al termine scarica la cartella `output/<stem>/` generata.
|
||||||
|
|
||||||
|
**Opzione B — Installazione locale**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install "mineru[all]"
|
||||||
|
mineru -p documento.pdf -o sources/<stem>/
|
||||||
|
```
|
||||||
|
|
||||||
|
Al termine, **copia la cartella completa** prodotta da MinerU dentro `sources/`. La struttura attesa è:
|
||||||
|
|
||||||
```
|
```
|
||||||
sources/
|
sources/
|
||||||
└── <stem>/
|
└── <stem>/
|
||||||
└── auto/
|
└── auto/
|
||||||
├── <stem>_content_list_v2.json
|
├── <stem>.md
|
||||||
├── <stem>_model.json
|
├── <stem>_content_list_v2.json ← richiesto
|
||||||
└── ...
|
├── <stem>_model.json ← raccomandato
|
||||||
|
├── <stem>_middle.json
|
||||||
|
├── <stem>_content_list.json
|
||||||
|
├── <stem>_layout.pdf
|
||||||
|
├── <stem>_span.pdf
|
||||||
|
└── images/
|
||||||
```
|
```
|
||||||
|
|
||||||
### Passo 2 — Chunking (Stage 1 + Stage 2)
|
### Passo 2 — Chunking (Stage 1 + Stage 2)
|
||||||
|
|||||||
+338
@@ -0,0 +1,338 @@
|
|||||||
|
{
|
||||||
|
"cells": [
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "ozeCGy0fwA4R"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"# Conversione PDF con MinerU"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "zyMHLIPh3sCk"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"[](https://github.com/opendatalab/MinerU)\n",
|
||||||
|
"[](https://github.com/opendatalab/MinerU)\n",
|
||||||
|
"[](https://github.com/opendatalab/MinerU/issues)\n",
|
||||||
|
"[](https://github.com/opendatalab/MinerU/issues)\n",
|
||||||
|
"[](https://pypi.org/project/mineru/)\n",
|
||||||
|
"[](https://pypi.org/project/mineru/)\n",
|
||||||
|
"[](https://pepy.tech/project/mineru)\n",
|
||||||
|
"[](https://pepy.tech/project/mineru)\n",
|
||||||
|
"[](https://mineru.net/OpenSourceTools/Extractor?source=github)\n",
|
||||||
|
"[](https://huggingface.co/spaces/opendatalab/MinerU)\n",
|
||||||
|
"[](https://www.modelscope.cn/studios/OpenDataLab/MinerU)\n",
|
||||||
|
"[](https://colab.research.google.com/gist/myhloli/a3cb16570ab3cfeadf9d8f0ac91b4fca/mineru_demo.ipynb)\n",
|
||||||
|
"[](https://arxiv.org/abs/2409.18839)\n",
|
||||||
|
"[](https://arxiv.org/abs/2509.22186)\n",
|
||||||
|
"[](https://deepwiki.com/opendatalab/MinerU)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "F9dnYLSDwQyR"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"## 1.Install"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "Rw-hwakFwaeY"
|
||||||
|
},
|
||||||
|
"source": "### Install via pip"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"colab": {
|
||||||
|
"base_uri": "https://localhost:8080/",
|
||||||
|
"height": 1000
|
||||||
|
},
|
||||||
|
"id": "_3EDplqlxKB_",
|
||||||
|
"outputId": "a4c96ba2-6dea-4ef0-aac7-955dbfac0598"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": "!pip install --upgrade pip\n!pip install \"mineru[all]>=3.1.3\""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "rSfS77AisXH8"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Check verison"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"colab": {
|
||||||
|
"base_uri": "https://localhost:8080/"
|
||||||
|
},
|
||||||
|
"id": "MSyCKHmqCp_j",
|
||||||
|
"outputId": "eb0aad89-3261-4388-9dfc-2b6ce29a3030"
|
||||||
|
},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"mineru, version 3.1.3\n"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"!mineru -v"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "Qh8YKU19CrQY"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"## 2.Usage"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "3dZlkl3jDNKL"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### 2.1 Command line"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "mVRMM_FaDi2d"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"#### **PDF**"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "Hyoz1B-tPXcM"
|
||||||
|
},
|
||||||
|
"source": "#### **Converti un singolo PDF**\n\nSu Google Colab il file system è temporaneo: ogni volta che avvii una nuova sessione la cartella `/content/` è vuota.\nDevi quindi **caricare il tuo PDF prima di eseguire la conversione**.\n\n**Come caricare il file:**\n\n1. Nel pannello di sinistra clicca sull'icona **File**.\n2. Trascina il PDF nella cartella `/content/` (radice della sessione), oppure clicca il pulsante *Carica* (icona freccia su).\n3. Attendi che il caricamento sia completato: il file comparirà nell'elenco.\n\nIn alternativa puoi caricare via codice:\n\n```python\nfrom google.colab import files\nuploaded = files.upload() # apre il selettore file\n```\n\nUna volta caricato, il PDF si trova in `/content/<nome-file>.pdf`.\nSostituisci `mio_documento.pdf` nella cella seguente con il nome del tuo file:"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"PDF_NAME = \"mio_documento.pdf\" # ← sostituisci con il nome del file che hai caricato\n",
|
||||||
|
"import os\n",
|
||||||
|
"STEM = os.path.splitext(os.path.basename(PDF_NAME))[0]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"colab": {
|
||||||
|
"base_uri": "https://localhost:8080/"
|
||||||
|
},
|
||||||
|
"collapsed": true,
|
||||||
|
"id": "6RivAWteLeYM",
|
||||||
|
"outputId": "dc56bcc0-6e84-432a-8511-e461ff605a51"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": "!mineru -p {PDF_NAME} -o ./output -b pipeline"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"colab": {
|
||||||
|
"base_uri": "https://localhost:8080/",
|
||||||
|
"height": 807
|
||||||
|
},
|
||||||
|
"id": "F6K4E8CfH6_V",
|
||||||
|
"outputId": "136175e0-c1b0-4c80-84c4-ce7e7c78e656"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"import os, io\n",
|
||||||
|
"import fitz\n",
|
||||||
|
"import matplotlib.pyplot as plt\n",
|
||||||
|
"import numpy as np\n",
|
||||||
|
"from IPython.display import display\n",
|
||||||
|
"\n",
|
||||||
|
"os.system('pip install -q pymupdf')\n",
|
||||||
|
"\n",
|
||||||
|
"# Cerca il layout PDF nell'output di MinerU (backend auto o hybrid_auto)\n",
|
||||||
|
"candidates = [\n",
|
||||||
|
" f'./output/{STEM}/hybrid_auto/{STEM}_layout.pdf',\n",
|
||||||
|
" f'./output/{STEM}/auto/{STEM}_layout.pdf',\n",
|
||||||
|
"]\n",
|
||||||
|
"pdf_path = next((p for p in candidates if os.path.exists(p)), None)\n",
|
||||||
|
"\n",
|
||||||
|
"if pdf_path is None:\n",
|
||||||
|
" print('Layout PDF non trovato. Percorsi cercati:')\n",
|
||||||
|
" for p in candidates:\n",
|
||||||
|
" print(' ', p)\n",
|
||||||
|
"else:\n",
|
||||||
|
" doc = fitz.open(pdf_path)\n",
|
||||||
|
" n_pages = min(len(doc), 2) # mostra al massimo 2 pagine\n",
|
||||||
|
" fig, axes = plt.subplots(1, n_pages, figsize=(12 if n_pages > 1 else 6, 8))\n",
|
||||||
|
" if n_pages == 1:\n",
|
||||||
|
" axes = [axes]\n",
|
||||||
|
" for idx in range(n_pages):\n",
|
||||||
|
" pix = doc.load_page(idx).get_pixmap()\n",
|
||||||
|
" img = np.frombuffer(pix.samples, dtype=np.uint8).reshape(pix.height, pix.width, pix.n)\n",
|
||||||
|
" axes[idx].imshow(img)\n",
|
||||||
|
" axes[idx].axis('off')\n",
|
||||||
|
" axes[idx].set_title(f'Pagina {idx + 1}')\n",
|
||||||
|
" plt.tight_layout()\n",
|
||||||
|
" plt.show()\n",
|
||||||
|
" doc.close()\n",
|
||||||
|
" print(f'Layout da: {pdf_path}')\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "t53WETM90pKr"
|
||||||
|
},
|
||||||
|
"source": "#### **Converti più PDF nella stessa cartella**\n\nConverte tutti i PDF presenti in `/content/` in un'unica esecuzione.\n\nCarica i file prima di eseguire la cella (trascina i PDF nel pannello File a sinistra oppure usa `files.upload()`).\n\nL'output di ogni PDF viene salvato in `./output/<nome-file>/auto/`.\n\n> Usa questa cella al posto di quella precedente se hai più documenti da convertire."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"colab": {
|
||||||
|
"base_uri": "https://localhost:8080/"
|
||||||
|
},
|
||||||
|
"id": "4dqBGZiz0orA",
|
||||||
|
"outputId": "8233269d-d912-4dd3-9332-d26bd464db7c"
|
||||||
|
},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"\u001b[32m2026-04-23 11:36:57.400\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mmineru.cli.client\u001b[0m:\u001b[36mrun_orchestrated_cli\u001b[0m:\u001b[36m874\u001b[0m - \u001b[1mStarted local mineru-api at http://127.0.0.1:37453\u001b[0m\n",
|
||||||
|
"\u001b[32m2026-04-23 11:37:00.858\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mcreate_app\u001b[0m:\u001b[36m260\u001b[0m - \u001b[1mRequest concurrency limited to 3\u001b[0m\n",
|
||||||
|
"Start MinerU FastAPI Service: http://127.0.0.1:37453\n",
|
||||||
|
"API documentation: http://127.0.0.1:37453/docs\n",
|
||||||
|
"\u001b[32mINFO\u001b[0m: Started server process [\u001b[36m10876\u001b[0m]\n",
|
||||||
|
"\u001b[32mINFO\u001b[0m: Waiting for application startup.\n",
|
||||||
|
"\u001b[32mINFO\u001b[0m: Application startup complete.\n",
|
||||||
|
"\u001b[32mINFO\u001b[0m: Uvicorn running on \u001b[1mhttp://127.0.0.1:37453\u001b[0m (Press CTRL+C to quit)\n",
|
||||||
|
"\u001b[32m2026-04-23 11:37:01.423\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mmineru.cli.client\u001b[0m:\u001b[36mrun_planned_task\u001b[0m:\u001b[36m771\u001b[0m - \u001b[1mSubmitting batch 1/1 | 2 documents, 19 pages in this batch | 19 pages total | task#1 [demo1, demo2]\u001b[0m\n",
|
||||||
|
"2026-04-23 11:37:06.518957: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
|
||||||
|
"WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
|
||||||
|
"E0000 00:00:1776944226.544180 10908 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
|
||||||
|
"E0000 00:00:1776944226.552343 10908 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
|
||||||
|
"W0000 00:00:1776944226.572768 10908 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.\n",
|
||||||
|
"W0000 00:00:1776944226.572813 10908 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.\n",
|
||||||
|
"W0000 00:00:1776944226.572818 10908 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.\n",
|
||||||
|
"W0000 00:00:1776944226.572821 10908 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.\n",
|
||||||
|
"2026-04-23 11:37:06.578247: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
|
||||||
|
"To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
|
||||||
|
"\u001b[32m2026-04-23 11:37:11.673\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mmineru.backend.pipeline.pipeline_analyze\u001b[0m:\u001b[36mdoc_analyze_streaming\u001b[0m:\u001b[36m183\u001b[0m - \u001b[1mPipeline processing-window multi-file run. doc_count=2, total_pages=19, window_size=64, total_batches=1\u001b[0m\n",
|
||||||
|
"\u001b[32m2026-04-23 11:37:17.154\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mmineru.backend.pipeline.pipeline_analyze\u001b[0m:\u001b[36mdoc_analyze_streaming\u001b[0m:\u001b[36m235\u001b[0m - \u001b[1mPipeline processing window batch 1/1: 19/19 pages, batch_pages=19, doc_slices=doc0:1-13,doc1:1-6\u001b[0m\n",
|
||||||
|
"\u001b[32m2026-04-23 11:37:17.158\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mmineru.backend.pipeline.pipeline_analyze\u001b[0m:\u001b[36mbatch_image_analyze\u001b[0m:\u001b[36m328\u001b[0m - \u001b[1mGPU Memory: 15 GB, Batch Ratio: 4. \u001b[0m\n",
|
||||||
|
"\u001b[32m2026-04-23 11:37:17.160\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mmineru.backend.pipeline.model_init\u001b[0m:\u001b[36m__init__\u001b[0m:\u001b[36m207\u001b[0m - \u001b[1mDocAnalysis init, this may take some times......\u001b[0m\n",
|
||||||
|
"Fetching 7 files: 100% 7/7 [00:00<00:00, 72494.14it/s]\n",
|
||||||
|
"Fetching 3 files: 100% 3/3 [00:00<00:00, 20004.63it/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 7244.05it/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 8160.12it/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 7653.84it/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 21183.35it/s]\n",
|
||||||
|
"Fetching 1 files: 0% 0/1 [00:00<?, ?it/s]\n",
|
||||||
|
"models/TabRec/UnetStructure/unet.onnx: 0% 0.00/8.34M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/TabRec/UnetStructure/unet.onnx: 0% 0.00/8.34M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/TabRec/UnetStructure/unet.onnx: 0% 0.00/8.34M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/TabRec/UnetStructure/unet.onnx: 100% 8.34M/8.34M [00:00<00:00, 12.0MB/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 1.06it/s]\n",
|
||||||
|
"Fetching 1 files: 0% 0/1 [00:00<?, ?it/s]\n",
|
||||||
|
"models/TabRec/SlanetPlus/slanet-plus.onn(…): 0% 0.00/7.76M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/TabRec/SlanetPlus/slanet-plus.onn(…): 0% 0.00/7.76M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/TabRec/SlanetPlus/slanet-plus.onn(…): 100% 7.76M/7.76M [00:00<00:00, 18.8MB/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 1.72it/s]\n",
|
||||||
|
"Fetching 1 files: 0% 0/1 [00:00<?, ?it/s]\n",
|
||||||
|
"models/TabCls/paddle_table_cls/PP-LCNet_(…): 0% 0.00/6.78M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/TabCls/paddle_table_cls/PP-LCNet_(…): 0% 0.00/6.78M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/TabCls/paddle_table_cls/PP-LCNet_(…): 100% 6.78M/6.78M [00:00<00:00, 16.3MB/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 1.69it/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 13751.82it/s]\n",
|
||||||
|
"Fetching 1 files: 0% 0/1 [00:00<?, ?it/s]\n",
|
||||||
|
"models/OCR/paddleocr_torch/ch_PP-OCRv5_r(…): 0% 0.00/32.6M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/OCR/paddleocr_torch/ch_PP-OCRv5_r(…): 0% 0.00/32.6M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/OCR/paddleocr_torch/ch_PP-OCRv5_r(…): 0% 0.00/32.6M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/OCR/paddleocr_torch/ch_PP-OCRv5_r(…): 2% 799k/32.6M [00:00<00:07, 3.99MB/s]\u001b[A\n",
|
||||||
|
"models/OCR/paddleocr_torch/ch_PP-OCRv5_r(…): 9% 2.77M/32.6M [00:00<00:04, 7.45MB/s]\u001b[A\n",
|
||||||
|
"models/OCR/paddleocr_torch/ch_PP-OCRv5_r(…): 12% 4.07M/32.6M [00:01<00:04, 6.98MB/s]\u001b[A\n",
|
||||||
|
"models/OCR/paddleocr_torch/ch_PP-OCRv5_r(…): 55% 18.0M/32.6M [00:01<00:00, 31.7MB/s]\u001b[A\n",
|
||||||
|
"models/OCR/paddleocr_torch/ch_PP-OCRv5_r(…): 100% 32.6M/32.6M [00:01<00:00, 23.0MB/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:01<00:00, 1.59s/it]\n",
|
||||||
|
"Fetching 1 files: 0% 0/1 [00:00<?, ?it/s]\n",
|
||||||
|
"models/OriCls/paddle_orientation_classif(…): 0% 0.00/6.79M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/OriCls/paddle_orientation_classif(…): 0% 0.00/6.79M [00:00<?, ?B/s]\u001b[A\n",
|
||||||
|
"models/OriCls/paddle_orientation_classif(…): 100% 6.79M/6.79M [00:00<00:00, 16.5MB/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 1.70it/s]\n",
|
||||||
|
"\u001b[32m2026-04-23 11:37:31.842\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mmineru.backend.pipeline.model_init\u001b[0m:\u001b[36m__init__\u001b[0m:\u001b[36m260\u001b[0m - \u001b[1mDocAnalysis init done!\u001b[0m\n",
|
||||||
|
"\u001b[32m2026-04-23 11:37:31.843\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mmineru.backend.pipeline.pipeline_analyze\u001b[0m:\u001b[36mcustom_model_init\u001b[0m:\u001b[36m83\u001b[0m - \u001b[1mmodel init cost: 14.68275260925293\u001b[0m\n",
|
||||||
|
"Layout Predict: 100% 19/19 [00:02<00:00, 7.77it/s]\n",
|
||||||
|
"MFR Predict: 100% 135/135 [00:06<00:00, 22.08it/s]\n",
|
||||||
|
"Table-ocr det: 100% 7/7 [00:00<00:00, 10.03it/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 15650.39it/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 7825.19it/s]\n",
|
||||||
|
"Table-ocr rec ch: 100% 614/614 [00:03<00:00, 170.47it/s]\n",
|
||||||
|
"Table-wireless Predict: 100% 7/7 [00:02<00:00, 2.92it/s]\n",
|
||||||
|
"Table-wired Predict: 0% 0/6 [00:00<?, ?it/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 10754.63it/s]\n",
|
||||||
|
"Table-wired Predict: 100% 6/6 [00:03<00:00, 1.59it/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 18236.10it/s]\n",
|
||||||
|
"Fetching 1 files: 100% 1/1 [00:00<00:00, 20560.31it/s]\n",
|
||||||
|
"OCR-det ch: 100% 49/49 [00:04<00:00, 11.70it/s]\n",
|
||||||
|
"Seal Predict: 0it [00:00, ?it/s]\n",
|
||||||
|
"Processing pages: 68% 13/19 [00:01<00:00, 10.86it/s]\n",
|
||||||
|
"OCR-rec Predict: 0% 0/6 [00:00<?, ?it/s]\u001b[A\n",
|
||||||
|
"OCR-rec Predict: 100% 6/6 [00:00<00:00, 51.21it/s]\n",
|
||||||
|
"Processing pages: 100% 19/19 [00:02<00:00, 6.65it/s]\n",
|
||||||
|
"\u001b[32m2026-04-23 11:38:03.054\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mmineru.cli.client\u001b[0m:\u001b[36mrun_planned_task\u001b[0m:\u001b[36m807\u001b[0m - \u001b[1mCompleted batch 1/1 | Processed 19/19 pages | 1 of 1 batch finished | task#1 [demo1, demo2]\u001b[0m\n",
|
||||||
|
"\u001b[32mINFO\u001b[0m: Shutting down\n",
|
||||||
|
"\u001b[32mINFO\u001b[0m: Waiting for application shutdown.\n",
|
||||||
|
"\u001b[32mINFO\u001b[0m: Application shutdown complete.\n",
|
||||||
|
"\u001b[32mINFO\u001b[0m: Finished server process [\u001b[36m10876\u001b[0m]\n"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"!mineru -p ./ -o ./output -b pipeline"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"accelerator": "GPU",
|
||||||
|
"colab": {
|
||||||
|
"gpuType": "T4",
|
||||||
|
"provenance": []
|
||||||
|
},
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3",
|
||||||
|
"name": "python3"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"name": "python"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"nbformat": 4,
|
||||||
|
"nbformat_minor": 0
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user