palladum-lightning/doc/COVERAGE.md

Code Coverage Guide

This guide explains how to measure code coverage for Core Lightning's test suite.

Overview

Core Lightning uses Clang's source-based coverage instrumentation to measure which lines of code are executed during tests. This is particularly challenging because:

• CLN is a multi-process application (lightningd + 8 daemon executables)
• Each test spawns multiple nodes, each running multiple daemon processes
• Tests run in parallel (10+ workers)
• Test processes run in temporary directories

Our solution uses LLVM_PROFILE_FILE environment variable with unique naming patterns to prevent profile file collisions across parallel processes.

Local Development Workflow

Prerequisites

• Clang compiler (clang-15 or later)
• LLVM tools: llvm-profdata, llvm-cov

Install on Ubuntu/Debian:

sudo apt-get install clang llvm

Step 1: Build with Coverage Instrumentation

./configure --enable-coverage CC=clang
make clean  # Important: clean previous builds
make

This compiles all binaries with -fprofile-instr-generate -fcoverage-mapping flags.

Step 2: Run Tests with Coverage Collection

Set the coverage directory and run tests:

export CLN_COVERAGE_DIR=/tmp/cln-coverage
mkdir -p "$CLN_COVERAGE_DIR"
uv run pytest tests/ -n 10

You can run a subset of tests for faster iteration:

uv run pytest tests/test_pay.py -n 10

All test processes will write .profraw files to $CLN_COVERAGE_DIR with unique names like 12345-67890abcdef.profraw (PID-signature).

Step 3: Generate Coverage Reports

Merge all profile files and generate HTML report:

make coverage-clang

This runs two scripts:

1. contrib/coverage/collect-coverage.sh - Merges all .profraw files into coverage/merged.profdata
2. contrib/coverage/generate-coverage-report.sh - Generates HTML report from merged profile

Step 4: View the Report

Open the HTML report in your browser:

xdg-open coverage/html/index.html

Or on macOS:

open coverage/html/index.html

The report shows:

• Per-file coverage: Which files have been tested
• Line-by-line coverage: Which lines were executed and how many times
• Summary statistics: Overall coverage percentage

You can also view the text summary:

cat coverage/summary.txt

Step 5: Clean Up

make coverage-clang-clean

This removes the coverage/ directory and $CLN_COVERAGE_DIR.

Complete Example

# Build
./configure --enable-coverage CC=clang
make

# Test
export CLN_COVERAGE_DIR=/tmp/cln-coverage
mkdir -p "$CLN_COVERAGE_DIR"
uv run pytest tests/test_pay.py tests/test_invoice.py -n 10

# Report
make coverage-clang
xdg-open coverage/html/index.html

# Clean
make coverage-clang-clean

Advanced Usage

Running Specific Test Files

For faster development iteration, run only the tests you're working on:

uv run pytest tests/test_plugin.py -n 5

Per-Test Coverage

Coverage data is automatically organized by test name, allowing you to see which code each test exercises:

export CLN_COVERAGE_DIR=/tmp/cln-coverage
mkdir -p "$CLN_COVERAGE_DIR"
uv run pytest tests/test_pay.py tests/test_invoice.py -n 10

This creates a directory structure like:

/tmp/cln-coverage/
  ├── test_pay/
  │   ├── 12345-abc.profraw
  │   └── 67890-def.profraw
  └── test_invoice/
      ├── 11111-ghi.profraw
      └── 22222-jkl.profraw

Generate per-test coverage reports:

# Generate text summaries
./contrib/coverage/per-test-coverage.sh

# Generate HTML reports (optional)
./contrib/coverage/per-test-coverage-html.sh

This creates:

• coverage/per-test/<test>.profdata - Merged profile for each test
• coverage/per-test/<test>.txt - Text summary for each test
• coverage/per-test-html/<test>/index.html - HTML report for each test (if generated)

Merging Multiple Test Runs

You can accumulate coverage across multiple test runs by reusing the same CLN_COVERAGE_DIR:

export CLN_COVERAGE_DIR=/tmp/cln-coverage
mkdir -p "$CLN_COVERAGE_DIR"

# Run different test subsets
uv run pytest tests/test_pay.py -n 10
uv run pytest tests/test_invoice.py -n 10
uv run pytest tests/test_plugin.py -n 10

# Generate combined report (merges all tests)
make coverage-clang

# Or generate per-test reports
./contrib/coverage/per-test-coverage.sh

Manual Collection and Reporting

If you want more control:

# Collect and merge
./contrib/coverage/collect-coverage.sh /tmp/cln-coverage coverage/merged.profdata

# Generate report
./contrib/coverage/generate-coverage-report.sh coverage/merged.profdata coverage/html

Continuous Integration

Coverage is automatically measured nightly on the master branch via the coverage-nightly.yaml GitHub Actions workflow. The workflow:

1. Builds CLN with coverage instrumentation
2. Runs tests with both sqlite and postgres databases
3. Merges coverage from all test runs
4. Uploads results to Codecov.io
5. Saves HTML reports as artifacts (90-day retention)

You can view:

• Codecov dashboard: codecov.io/gh/ElementsProject/lightning
• HTML artifacts: Download from GitHub Actions workflow runs

Troubleshooting

No .profraw files created

Problem: make coverage-clang reports "No .profraw files found"

Solutions:

1. Verify CLN_COVERAGE_DIR is set: echo $CLN_COVERAGE_DIR
2. Verify you built with coverage: ./configure --enable-coverage CC=clang && make
3. Check that tests actually ran successfully

llvm-profdata not found

Problem: llvm-profdata: command not found

Solution: Install LLVM tools:

sudo apt-get install llvm
# Or on macOS:
brew install llvm

Binary not found errors in generate-coverage-report.sh

Problem: Script complains about missing binaries

Solution: Make sure you've run make to build all CLN executables

Coverage shows 0% for some files

Causes:

1. Those files weren't executed by your tests (expected)
2. The binary wasn't instrumented (check build flags)
3. The profile data is incomplete

Corrupt .profraw files

Problem: llvm-profdata merge fails with "invalid instrumentation profile data (file header is corrupt)"

Cause: When test processes crash or timeout, they may leave incomplete/corrupt .profraw files.

Solution: The collect-coverage.sh script automatically validates and filters out bad files:

• Empty files - Processes that crash immediately
• Incomplete files (< 1KB) - Processes killed before writing enough data
• Corrupt files - Files with invalid headers or structure

You'll see output like:

Found 1250 profile files
  Skipping empty file: /tmp/cln-coverage/12345-abc.profraw
  Skipping incomplete file (512 bytes): /tmp/cln-coverage/67890-def.profraw
  Skipping corrupt file: /tmp/cln-coverage/11111-ghi.profraw
Valid files: 1247
Filtered out: 3 files
  - Empty: 1
  - Incomplete (< 1KB): 1
  - Corrupt/invalid: 1
✓ Merged profile: coverage/merged.profdata

To manually review and clean up corrupt files:

./contrib/coverage/cleanup-corrupt-profraw.sh

This will show you which files are corrupt and offer to delete them.

Prevention: Incomplete/corrupt files are unavoidable when tests crash/timeout. The collection script handles this automatically by filtering them out during merge.

Understanding Coverage Metrics

• Lines: Percentage of source code lines executed
• Functions: Percentage of functions that were called
• Regions: Percentage of code regions (blocks) executed
• Hit count: Number of times each line was executed

Aim for:

• >80% line coverage for core functionality
• >60% overall given the complexity of CLN

Remember: 100% coverage doesn't mean bug-free code, but low coverage means untested code paths.

References

• LLVM Source-Based Code Coverage
• llvm-profdata documentation
• llvm-cov documentation