This commit is contained in:
NotRin7
2024-03-15 18:16:03 +01:00
commit 0a6901a7f0
1865 changed files with 515440 additions and 0 deletions

38
ci/README.md Normal file
View File

@@ -0,0 +1,38 @@
## ci scripts
This directory contains scripts for each build step in each build stage.
Currently three stages `lint`, `extended_lint` and `test` are defined. Each stage has its own lifecycle, similar to the
[Travis CI lifecycle](https://docs.travis-ci.com/user/job-lifecycle#the-job-lifecycle). Every script in here is named
and numbered according to which stage and lifecycle step it belongs to.
### Running a stage locally
Be aware that the tests will be built and run in-place, so please run at your own risk.
If the repository is not a fresh git clone, you might have to clean files from previous builds or test runs first.
The ci needs to perform various sysadmin tasks such as installing packages or writing to the user's home directory.
While most of the actions are done inside a docker container, this is not possible for all. Thus, cache directories,
such as the depends cache, previous release binaries, or ccache, are mounted as read-write into the docker container. While it should be fine to run
the ci system locally on you development box, the ci scripts can generally be assumed to have received less review and
testing compared to other parts of the codebase. If you want to keep the work tree clean, you might want to run the ci
system in a virtual machine with a Linux operating system of your choice.
To allow for a wide range of tested environments, but also ensure reproducibility to some extent, the test stage
requires `docker` to be installed. To install all requirements on Ubuntu, run
```
sudo apt install docker.io bash
```
To run the default test stage,
```
./ci/test_run_all.sh
```
To run the test stage with a specific configuration,
```
FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh
```

15
ci/lint/04_install.sh Executable file
View File

@@ -0,0 +1,15 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C
travis_retry pip3 install codespell==1.15.0
travis_retry pip3 install flake8==3.7.8
travis_retry pip3 install yq
SHELLCHECK_VERSION=v0.6.0
curl -sL "https://github.com/koalaman/shellcheck/releases/download/${SHELLCHECK_VERSION}/shellcheck-${SHELLCHECK_VERSION}.linux.x86_64.tar.xz" | tar --xz -xf - --directory /tmp/
export PATH="/tmp/shellcheck-${SHELLCHECK_VERSION}:${PATH}"

9
ci/lint/05_before_script.sh Executable file
View File

@@ -0,0 +1,9 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C
git fetch --unshallow

26
ci/lint/06_script.sh Executable file
View File

@@ -0,0 +1,26 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C
if [ "$TRAVIS_EVENT_TYPE" = "pull_request" ]; then
test/lint/commit-script-check.sh $TRAVIS_COMMIT_RANGE
fi
test/lint/git-subtree-check.sh src/crypto/ctaes
test/lint/git-subtree-check.sh src/secp256k1
test/lint/git-subtree-check.sh src/univalue
test/lint/git-subtree-check.sh src/leveldb
test/lint/git-subtree-check.sh src/crc32c
test/lint/check-doc.py
test/lint/check-rpc-mappings.py .
test/lint/lint-all.sh
if [ "$TRAVIS_REPO_SLUG" = "palladium/palladium" ] && [ "$TRAVIS_EVENT_TYPE" = "cron" ]; then
git log --merges --before="2 days ago" -1 --format='%H' > ./contrib/verify-commits/trusted-sha512-root-commit
travis_retry gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys $(<contrib/verify-commits/trusted-keys) &&
./contrib/verify-commits/verify-commits.py --clean-merge=2;
fi

123
ci/retry/README.md Normal file
View File

@@ -0,0 +1,123 @@
retry - The command line retry tool
------------------------------------------
Retry any shell command with exponential backoff or constant delay.
### Instructions
Install:
retry is a shell script, so drop it somewhere and make sure it's added to your $PATH. Or you can use the following one-liner:
```sh
sudo sh -c "curl https://raw.githubusercontent.com/kadwanev/retry/master/retry -o /usr/local/bin/retry && chmod +x /usr/local/bin/retry"
```
If you're on OS X, retry is also on Homebrew:
```
brew pull 27283
brew install retry
```
Not popular enough for homebrew-core. Please star this project to help.
### Usage
Help:
`retry -?`
Usage: retry [options] -- execute command
-h, -?, --help
-v, --verbose Verbose output
-t, --tries=# Set max retries: Default 10
-s, --sleep=secs Constant sleep amount (seconds)
-m, --min=secs Exponential Backoff: minimum sleep amount (seconds): Default 0.3
-x, --max=secs Exponential Backoff: maximum sleep amount (seconds): Default 60
-f, --fail="script +cmds" Fail Script: run in case of final failure
### Examples
No problem:
`retry echo u work good`
u work good
Test functionality:
`retry 'echo "y u no work"; false'`
y u no work
Before retry #1: sleeping 0.3 seconds
y u no work
Before retry #2: sleeping 0.6 seconds
y u no work
Before retry #3: sleeping 1.2 seconds
y u no work
Before retry #4: sleeping 2.4 seconds
y u no work
Before retry #5: sleeping 4.8 seconds
y u no work
Before retry #6: sleeping 9.6 seconds
y u no work
Before retry #7: sleeping 19.2 seconds
y u no work
Before retry #8: sleeping 38.4 seconds
y u no work
Before retry #9: sleeping 60.0 seconds
y u no work
Before retry #10: sleeping 60.0 seconds
y u no work
etc..
Limit retries:
`retry -t 4 'echo "y u no work"; false'`
y u no work
Before retry #1: sleeping 0.3 seconds
y u no work
Before retry #2: sleeping 0.6 seconds
y u no work
Before retry #3: sleeping 1.2 seconds
y u no work
Before retry #4: sleeping 2.4 seconds
y u no work
Retries exhausted
Bad command:
`retry poop`
bash: poop: command not found
Fail command:
`retry -t 3 -f 'echo "oh poopsickles"' 'echo "y u no work"; false'`
y u no work
Before retry #1: sleeping 0.3 seconds
y u no work
Before retry #2: sleeping 0.6 seconds
y u no work
Before retry #3: sleeping 1.2 seconds
y u no work
Retries exhausted, running fail script
oh poopsickles
Last attempt passed:
`retry -t 3 -- 'if [ $RETRY_ATTEMPT -eq 3 ]; then echo Passed at attempt $RETRY_ATTEMPT; true; else echo Failed at attempt $RETRY_ATTEMPT; false; fi;'`
Failed at attempt 0
Before retry #1: sleeping 0.3 seconds
Failed at attempt 1
Before retry #2: sleeping 0.6 seconds
Failed at attempt 2
Before retry #3: sleeping 1.2 seconds
Passed at attempt 3
### License
Apache 2.0 - go nuts

163
ci/retry/retry Executable file
View File

@@ -0,0 +1,163 @@
#!/usr/bin/env bash
GETOPT_BIN=$IN_GETOPT_BIN
GETOPT_BIN=${GETOPT_BIN:-getopt}
__sleep_amount() {
if [ -n "$constant_sleep" ]; then
sleep_time=$constant_sleep
else
#TODO: check for awk
#TODO: check if user would rather use one of the other possible dependencies: python, ruby, bc, dc
sleep_time=`awk "BEGIN {t = $min_sleep * $(( (1<<($attempts -1)) )); print (t > $max_sleep ? $max_sleep : t)}"`
fi
}
__log_out() {
echo "$1" 1>&2
}
# Parameters: max_tries min_sleep max_sleep constant_sleep fail_script EXECUTION_COMMAND
retry()
{
local max_tries="$1"; shift
local min_sleep="$1"; shift
local max_sleep="$1"; shift
local constant_sleep="$1"; shift
local fail_script="$1"; shift
if [ -n "$VERBOSE" ]; then
__log_out "Retry Parameters: max_tries=$max_tries min_sleep=$min_sleep max_sleep=$max_sleep constant_sleep=$constant_sleep"
if [ -n "$fail_script" ]; then __log_out "Fail script: $fail_script"; fi
__log_out ""
__log_out "Execution Command: $*"
__log_out ""
fi
local attempts=0
local return_code=1
while [[ $return_code -ne 0 && $attempts -le $max_tries ]]; do
if [ $attempts -gt 0 ]; then
__sleep_amount
__log_out "Before retry #$attempts: sleeping $sleep_time seconds"
sleep $sleep_time
fi
P="$1"
for param in "${@:2}"; do P="$P '$param'"; done
#TODO: replace single quotes in each arg with '"'"' ?
export RETRY_ATTEMPT=$attempts
bash -c "$P"
return_code=$?
#__log_out "Process returned $return_code on attempt $attempts"
if [ $return_code -eq 127 ]; then
# command not found
exit $return_code
elif [ $return_code -ne 0 ]; then
attempts=$[$attempts +1]
fi
done
if [ $attempts -gt $max_tries ]; then
if [ -n "$fail_script" ]; then
__log_out "Retries exhausted, running fail script"
eval $fail_script
else
__log_out "Retries exhausted"
fi
fi
exit $return_code
}
# If we're being sourced, don't worry about such things
if [ "$BASH_SOURCE" == "$0" ]; then
# Prints the help text
help()
{
local retry=$(basename $0)
cat <<EOF
Usage: $retry [options] -- execute command
-h, -?, --help
-v, --verbose Verbose output
-t, --tries=# Set max retries: Default 10
-s, --sleep=secs Constant sleep amount (seconds)
-m, --min=secs Exponential Backoff: minimum sleep amount (seconds): Default 0.3
-x, --max=secs Exponential Backoff: maximum sleep amount (seconds): Default 60
-f, --fail="script +cmds" Fail Script: run in case of final failure
EOF
}
# show help for no arguments if stdin is a terminal
if { [ -z "$1" ] && [ -t 0 ] ; } || [ "$1" == '-h' ] || [ "$1" == '-?' ] || [ "$1" == '--help' ]
then
help
exit 0
fi
$GETOPT_BIN --test > /dev/null
if [[ $? -ne 4 ]]; then
echo "Im sorry, 'getopt --test' failed in this environment. Please load GNU getopt."
exit 1
fi
OPTIONS=vt:s:m:x:f:
LONGOPTIONS=verbose,tries:,sleep:,min:,max:,fail:
PARSED=$($GETOPT_BIN --options="$OPTIONS" --longoptions="$LONGOPTIONS" --name "$0" -- "$@")
if [[ $? -ne 0 ]]; then
# e.g. $? == 1
# then getopt has complained about wrong arguments to stdout
exit 2
fi
# read getopts output this way to handle the quoting right:
eval set -- "$PARSED"
max_tries=10
min_sleep=0.3
max_sleep=60.0
constant_sleep=
fail_script=
# now enjoy the options in order and nicely split until we see --
while true; do
case "$1" in
-v|--verbose)
VERBOSE=true
shift
;;
-t|--tries)
max_tries="$2"
shift 2
;;
-s|--sleep)
constant_sleep="$2"
shift 2
;;
-m|--min)
min_sleep="$2"
shift 2
;;
-x|--max)
max_sleep="$2"
shift 2
;;
-f|--fail)
fail_script="$2"
shift 2
;;
--)
shift
break
;;
*)
echo "Programming error"
exit 3
;;
esac
done
retry "$max_tries" "$min_sleep" "$max_sleep" "$constant_sleep" "$fail_script" "$@"
fi

61
ci/test/00_setup_env.sh Executable file
View File

@@ -0,0 +1,61 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
# The root dir.
# The ci system copies this folder.
# This is where the build is done (depends and dist).
BASE_ROOT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )"/../../ >/dev/null 2>&1 && pwd )
export BASE_ROOT_DIR
echo "Setting specific values in env"
if [ -n "${FILE_ENV}" ]; then
set -o errexit;
# shellcheck disable=SC1090
source "${FILE_ENV}"
fi
echo "Fallback to default values in env (if not yet set)"
# The number of parallel jobs to pass down to make and test_runner.py
export MAKEJOBS=${MAKEJOBS:--j4}
# A folder for the ci system to put temporary files (ccache, datadirs for tests, ...)
# This folder only exists on the ci host.
export BASE_SCRATCH_DIR=${BASE_SCRATCH_DIR:-$BASE_ROOT_DIR/ci/scratch/}
# What host to compile for. See also ./depends/README.md
# Tests that need cross-compilation export the appropriate HOST.
# Tests that run natively guess the host
export HOST=${HOST:-$("$BASE_ROOT_DIR/depends/config.guess")}
# Whether to prefer BusyBox over GNU utilities
export USE_BUSY_BOX=${USE_BUSY_BOX:-false}
export RUN_UNIT_TESTS=${RUN_UNIT_TESTS:-true}
export RUN_FUNCTIONAL_TESTS=${RUN_FUNCTIONAL_TESTS:-true}
export TEST_PREVIOUS_RELEASES=${TEST_PREVIOUS_RELEASES:-false}
export RUN_FUZZ_TESTS=${RUN_FUZZ_TESTS:-false}
export CONTAINER_NAME=${CONTAINER_NAME:-ci_unnamed}
export DOCKER_NAME_TAG=${DOCKER_NAME_TAG:-ubuntu:18.04}
# Randomize test order.
# See https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/random.html
export BOOST_TEST_RANDOM=${BOOST_TEST_RANDOM:-1}
export CCACHE_SIZE=${CCACHE_SIZE:-100M}
export CCACHE_TEMPDIR=${CCACHE_TEMPDIR:-/tmp/.ccache-temp}
export CCACHE_COMPRESS=${CCACHE_COMPRESS:-1}
# The cache dir.
# This folder exists on the ci host and ci guest. Changes are propagated back and forth.
export CCACHE_DIR=${CCACHE_DIR:-$BASE_SCRATCH_DIR/.ccache}
# The depends dir.
# This folder exists on the ci host and ci guest. Changes are propagated back and forth.
export DEPENDS_DIR=${DEPENDS_DIR:-$BASE_ROOT_DIR/depends}
# Folder where the build is done (bin and lib).
export BASE_OUTDIR=${BASE_OUTDIR:-$BASE_SCRATCH_DIR/out/$HOST}
export PREVIOUS_RELEASES_DIR=${PREVIOUS_RELEASES_DIR:-$BASE_ROOT_DIR/releases/$HOST}
export SDK_URL=${SDK_URL:-https://palladiumcore.org/depends-sources/sdks}
export WINEDEBUG=${WINEDEBUG:-fixme-all}
export DOCKER_PACKAGES=${DOCKER_PACKAGES:-build-essential libtool autotools-dev automake pkg-config bsdmainutils curl ca-certificates ccache python3 rsync git procps}
export GOAL=${GOAL:-install}
export DIR_QA_ASSETS=${DIR_QA_ASSETS:-${BASE_SCRATCH_DIR}/qa-assets}
export PATH=${BASE_ROOT_DIR}/ci/retry:$PATH
export CI_RETRY_EXE=${CI_RETRY_EXE:retry}

View File

@@ -0,0 +1,28 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export HOST=arm-linux-gnueabihf
# The host arch is unknown, so we run the tests through qemu.
# If the host is arm and wants to run the tests natively, it can set QEMU_USER_CMD to the empty string.
if [ -z ${QEMU_USER_CMD+x} ]; then export QEMU_USER_CMD="${QEMU_USER_CMD:-"qemu-arm -L /usr/arm-linux-gnueabihf/"}"; fi
export DPKG_ADD_ARCH="armhf"
export PACKAGES="python3-zmq g++-arm-linux-gnueabihf busybox libc6:armhf libstdc++6:armhf libfontconfig1:armhf libxcb1:armhf"
if [ -n "$QEMU_USER_CMD" ]; then
# Likely cross-compiling, so install the needed gcc and qemu-user
export PACKAGES="$PACKAGES qemu-user"
fi
export CONTAINER_NAME=ci_arm_linux
# Use debian to avoid 404 apt errors when cross compiling
export DOCKER_NAME_TAG="debian:buster"
export USE_BUSY_BOX=true
export RUN_UNIT_TESTS=true
export RUN_FUNCTIONAL_TESTS=true
export GOAL="install"
# -Wno-psabi is to disable ABI warnings: "note: parameter passing for argument of type ... changed in GCC 7.1"
# This could be removed once the ABI change warning does not show up by default
export PALLADIUM_CONFIG="--enable-glibc-back-compat --enable-reduce-exports CXXFLAGS=-Wno-psabi --enable-werror"

View File

@@ -0,0 +1,15 @@
#!/usr/bin/env bash
#
# Copyright (c) 2020 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export HOST=i686-pc-linux-gnu
export CONTAINER_NAME=ci_i686_centos_7
export DOCKER_NAME_TAG=centos:7
export DOCKER_PACKAGES="gcc-c++ glibc-devel.x86_64 libstdc++-devel.x86_64 glibc-devel.i686 libstdc++-devel.i686 ccache libtool make git python3 python36-zmq which patch lbzip2 dash"
export GOAL="install"
export PALLADIUM_CONFIG="--enable-zmq --with-gui=qt5 --enable-reduce-exports"
export CONFIG_SHELL="/bin/dash"

View File

@@ -0,0 +1,16 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export CONTAINER_NAME=ci_macos_cross
export HOST=x86_64-apple-darwin16
export PACKAGES="cmake imagemagick libcap-dev librsvg2-bin libz-dev libbz2-dev libtiff-tools python3-dev python3-setuptools"
export OSX_SDK=10.14
export RUN_UNIT_TESTS=false
export RUN_FUNCTIONAL_TESTS=false
export GOAL="deploy"
export PALLADIUM_CONFIG="--enable-gui --enable-reduce-exports --enable-werror"

View File

@@ -0,0 +1,17 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export HOST=x86_64-apple-darwin16
export PIP_PACKAGES="zmq"
export RUN_UNIT_TESTS=true
export RUN_FUNCTIONAL_TESTS=false
export GOAL="install"
export PALLADIUM_CONFIG="--enable-gui --enable-reduce-exports --enable-werror"
# Run without depends
export NO_DEPENDS=1
export OSX_SDK=""

View File

@@ -0,0 +1,14 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export CONTAINER_NAME=ci_native_asan
export PACKAGES="clang-8 llvm-8 python3-zmq qtbase5-dev qttools5-dev-tools libevent-dev bsdmainutils libboost-system-dev libboost-filesystem-dev libboost-test-dev libboost-thread-dev libdb5.3++-dev libminiupnpc-dev libzmq3-dev libqrencode-dev"
# Use clang-8 instead of default clang (which is clang-6 on Bionic) to avoid spurious segfaults when running on ppc64le
export NO_DEPENDS=1
export GOAL="install"
export PALLADIUM_CONFIG="--enable-zmq --with-incompatible-bdb --with-gui=qt5 CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER' --with-sanitizers=address,integer,undefined CC=clang-8 CXX=clang++-8"

View File

@@ -0,0 +1,17 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export DOCKER_NAME_TAG="ubuntu:20.04"
export CONTAINER_NAME=ci_native_fuzz
export PACKAGES="clang llvm python3 libevent-dev bsdmainutils libboost-system-dev libboost-filesystem-dev libboost-test-dev libboost-thread-dev"
export NO_DEPENDS=1
export RUN_UNIT_TESTS=false
export RUN_FUNCTIONAL_TESTS=false
export RUN_FUZZ_TESTS=true
export GOAL="install"
export PALLADIUM_CONFIG="--enable-fuzz --with-sanitizers=fuzzer,address,undefined CC=clang CXX=clang++"

View File

@@ -0,0 +1,18 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export DOCKER_NAME_TAG="ubuntu:20.04"
export CONTAINER_NAME=ci_native_fuzz_valgrind
export PACKAGES="clang llvm python3 libevent-dev bsdmainutils libboost-system-dev libboost-filesystem-dev libboost-test-dev libboost-thread-dev valgrind"
export NO_DEPENDS=1
export RUN_UNIT_TESTS=false
export RUN_FUNCTIONAL_TESTS=false
export RUN_FUZZ_TESTS=true
export FUZZ_TESTS_CONFIG="--valgrind"
export GOAL="install"
export PALLADIUM_CONFIG="--enable-fuzz --with-sanitizers=fuzzer CC=clang CXX=clang++"

View File

@@ -0,0 +1,13 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export CONTAINER_NAME=ci_native_nowallet
export PACKAGES="python3-zmq"
export DEP_OPTS="NO_WALLET=1"
export GOAL="install"
export PALLADIUM_CONFIG="--enable-glibc-back-compat --enable-reduce-exports"

View File

@@ -0,0 +1,17 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export CONTAINER_NAME=ci_native_qt5
export PACKAGES="python3-zmq qtbase5-dev qttools5-dev-tools libdbus-1-dev libharfbuzz-dev"
export DEP_OPTS="NO_QT=1 NO_UPNP=1 DEBUG=1 ALLOW_HOST_PACKAGES=1"
export TEST_RUNNER_EXTRA="--coverage --extended --exclude feature_dbcrash" # Run extended tests so that coverage does not fail, but exclude the very slow dbcrash
export RUN_UNIT_TESTS_SEQUENTIAL="true"
export RUN_UNIT_TESTS="false"
export GOAL="install"
export TEST_PREVIOUS_RELEASES=true
export PALLADIUM_CONFIG="--enable-zmq --with-gui=qt5 --enable-glibc-back-compat --enable-reduce-exports --enable-debug CFLAGS=\"-g0 -O2 -funsigned-char\" CXXFLAGS=\"-g0 -O2 -funsigned-char\""

View File

@@ -0,0 +1,18 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export CONTAINER_NAME=ci_native_tsan
export DOCKER_NAME_TAG=ubuntu:16.04
export PACKAGES="clang llvm python3-zmq qtbase5-dev qttools5-dev-tools libevent-dev bsdmainutils libboost-system-dev libboost-filesystem-dev libboost-test-dev libboost-thread-dev libdb5.3++-dev libminiupnpc-dev libzmq3-dev libqrencode-dev"
export NO_DEPENDS=1
export GOAL="install"
export PALLADIUM_CONFIG="--enable-zmq --disable-wallet --with-gui=qt5 CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER' --with-sanitizers=thread --disable-hardening --disable-asm CC=clang CXX=clang++"
# xenial comes with old clang versions that can not parse the sanitizer suppressions files
# Remove unparseable lines as a hacky workaround
sed -i '/^implicit-/d' "${BASE_ROOT_DIR}/test/sanitizer_suppressions/ubsan"

View File

@@ -0,0 +1,15 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export CONTAINER_NAME=ci_native_valgrind
export PACKAGES="valgrind clang llvm python3-zmq libevent-dev bsdmainutils libboost-system-dev libboost-filesystem-dev libboost-test-dev libboost-thread-dev libdb5.3++-dev libminiupnpc-dev libzmq3-dev"
export USE_VALGRIND=1
export NO_DEPENDS=1
export TEST_RUNNER_EXTRA="--exclude rpc_bind" # Excluded for now, see https://github.com/palladium/palladium/issues/17765#issuecomment-602068547
export GOAL="install"
export PALLADIUM_CONFIG="--enable-zmq --with-incompatible-bdb --with-gui=no CC=clang CXX=clang++" # TODO enable GUI

View File

@@ -0,0 +1,25 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export HOST=s390x-linux-gnu
# The host arch is unknown, so we run the tests through qemu.
# If the host is s390x and wants to run the tests natively, it can set QEMU_USER_CMD to the empty string.
if [ -z ${QEMU_USER_CMD+x} ]; then export QEMU_USER_CMD="${QEMU_USER_CMD:-"qemu-s390x"}"; fi
export PACKAGES="python3-zmq"
if [ -n "$QEMU_USER_CMD" ]; then
# Likely cross-compiling, so install the needed gcc and qemu-user
export DPKG_ADD_ARCH="s390x"
export PACKAGES="$PACKAGES g++-s390x-linux-gnu qemu-user libc6:s390x libstdc++6:s390x libfontconfig1:s390x libxcb1:s390x"
fi
# Use debian to avoid 404 apt errors
export CONTAINER_NAME=ci_s390x
export DOCKER_NAME_TAG="debian:buster"
export RUN_UNIT_TESTS=true
export RUN_FUNCTIONAL_TESTS=true
export GOAL="install"
export PALLADIUM_CONFIG="--enable-reduce-exports --with-incompatible-bdb"

View File

@@ -0,0 +1,14 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
export CONTAINER_NAME=ci_win64
export HOST=x86_64-w64-mingw32
export PACKAGES="python3 nsis g++-mingw-w64-x86-64 wine-binfmt wine64"
export RUN_FUNCTIONAL_TESTS=false
export GOAL="deploy"
export PALLADIUM_CONFIG="--enable-reduce-exports --disable-gui-tests"

26
ci/test/03_before_install.sh Executable file
View File

@@ -0,0 +1,26 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
# Add llvm-symbolizer directory to PATH. Needed to get symbolized stack traces from the sanitizers.
PATH=$PATH:/usr/lib/llvm-6.0/bin/
export PATH
BEGIN_FOLD () {
echo ""
CURRENT_FOLD_NAME=$1
echo "travis_fold:start:${CURRENT_FOLD_NAME}"
}
END_FOLD () {
RET=$?
echo "travis_fold:end:${CURRENT_FOLD_NAME}"
if [ $RET != 0 ]; then
echo "${CURRENT_FOLD_NAME} failed with status code ${RET}"
fi
}

109
ci/test/04_install.sh Executable file
View File

@@ -0,0 +1,109 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2020 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
if [[ $DOCKER_NAME_TAG == centos* ]]; then
export LC_ALL=en_US.utf8
fi
if [[ $QEMU_USER_CMD == qemu-s390* ]]; then
export LC_ALL=C
fi
if [ "$TRAVIS_OS_NAME" == "osx" ]; then
export PATH="/usr/local/opt/ccache/libexec:$PATH"
${CI_RETRY_EXE} pip3 install $PIP_PACKAGES
fi
mkdir -p "${BASE_SCRATCH_DIR}"
mkdir -p "${CCACHE_DIR}"
mkdir -p "${PREVIOUS_RELEASES_DIR}"
export ASAN_OPTIONS="detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1"
export LSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/lsan"
export TSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/tsan:log_path=${BASE_SCRATCH_DIR}/sanitizer-output/tsan"
export UBSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/ubsan:print_stacktrace=1:halt_on_error=1:report_error_type=1"
env | grep -E '^(PALLADIUM_CONFIG|BASE_|QEMU_|CCACHE_|WINEDEBUG|LC_ALL|BOOST_TEST_RANDOM|CONFIG_SHELL|(ASAN|LSAN|TSAN|UBSAN)_OPTIONS|TEST_PREVIOUS_RELEASES|PREVIOUS_RELEASES_DIR)' | tee /tmp/env
if [[ $HOST = *-mingw32 ]]; then
DOCKER_ADMIN="--cap-add SYS_ADMIN"
elif [[ $PALLADIUM_CONFIG = *--with-sanitizers=*address* ]]; then # If ran with (ASan + LSan), Docker needs access to ptrace (https://github.com/google/sanitizers/issues/764)
DOCKER_ADMIN="--cap-add SYS_PTRACE"
fi
export P_CI_DIR="$PWD"
if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
echo "Creating $DOCKER_NAME_TAG container to run in"
${CI_RETRY_EXE} docker pull "$DOCKER_NAME_TAG"
DOCKER_ID=$(docker run $DOCKER_ADMIN -idt \
--mount type=bind,src=$BASE_ROOT_DIR,dst=/ro_base,readonly \
--mount type=bind,src=$CCACHE_DIR,dst=$CCACHE_DIR \
--mount type=bind,src=$DEPENDS_DIR,dst=$DEPENDS_DIR \
--mount type=bind,src=$PREVIOUS_RELEASES_DIR,dst=$PREVIOUS_RELEASES_DIR \
-w $BASE_ROOT_DIR \
--env-file /tmp/env \
--name $CONTAINER_NAME \
$DOCKER_NAME_TAG)
DOCKER_EXEC () {
docker exec $DOCKER_ID bash -c "export PATH=$BASE_SCRATCH_DIR/bins/:\$PATH && cd $P_CI_DIR && $*"
}
else
echo "Running on host system without docker wrapper"
DOCKER_EXEC () {
bash -c "export PATH=$BASE_SCRATCH_DIR/bins/:\$PATH && cd $P_CI_DIR && $*"
}
fi
if [ -n "$DPKG_ADD_ARCH" ]; then
DOCKER_EXEC dpkg --add-architecture "$DPKG_ADD_ARCH"
fi
if [[ $DOCKER_NAME_TAG == centos* ]]; then
${CI_RETRY_EXE} DOCKER_EXEC yum -y install epel-release
${CI_RETRY_EXE} DOCKER_EXEC yum -y install $DOCKER_PACKAGES $PACKAGES
elif [ "$CI_USE_APT_INSTALL" != "no" ]; then
${CI_RETRY_EXE} DOCKER_EXEC apt-get update
${CI_RETRY_EXE} DOCKER_EXEC apt-get install --no-install-recommends --no-upgrade -y $PACKAGES $DOCKER_PACKAGES
fi
if [ "$TRAVIS_OS_NAME" == "osx" ]; then
top -l 1 -s 0 | awk ' /PhysMem/ {print}'
echo "Number of CPUs: $(sysctl -n hw.logicalcpu)"
else
DOCKER_EXEC free -m -h
DOCKER_EXEC echo "Number of CPUs \(nproc\):" \$\(nproc\)
DOCKER_EXEC echo $(lscpu | grep Endian)
DOCKER_EXEC echo "Free disk space:"
DOCKER_EXEC df -h
fi
if [ ! -d ${DIR_QA_ASSETS} ]; then
if [ "$RUN_FUZZ_TESTS" = "true" ]; then
DOCKER_EXEC git clone https://github.com/palladium-core/qa-assets ${DIR_QA_ASSETS}
fi
fi
export DIR_FUZZ_IN=${DIR_QA_ASSETS}/fuzz_seed_corpus/
DOCKER_EXEC mkdir -p "${BASE_SCRATCH_DIR}/sanitizer-output/"
if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
echo "Create $BASE_ROOT_DIR"
DOCKER_EXEC rsync -a /ro_base/ $BASE_ROOT_DIR
fi
if [ "$USE_BUSY_BOX" = "true" ]; then
echo "Setup to use BusyBox utils"
DOCKER_EXEC mkdir -p $BASE_SCRATCH_DIR/bins/
# tar excluded for now because it requires passing in the exact archive type in ./depends (fixed in later BusyBox version)
# find excluded for now because it does not recognize the -delete option in ./depends (fixed in later BusyBox version)
# ar excluded for now because it does not recognize the -q option in ./depends (unknown if fixed)
# shellcheck disable=SC1010
DOCKER_EXEC for util in \$\(busybox --list \| grep -v "^ar$" \| grep -v "^tar$" \| grep -v "^find$"\)\; do ln -s \$\(command -v busybox\) $BASE_SCRATCH_DIR/bins/\$util\; done
# Print BusyBox version
DOCKER_EXEC patch --help
fi

42
ci/test/05_before_script.sh Executable file
View File

@@ -0,0 +1,42 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
# Make sure default datadir does not exist and is never read by creating a dummy file
if [ "$TRAVIS_OS_NAME" == "osx" ]; then
echo > $HOME/Library/Application\ Support/Palladium
else
DOCKER_EXEC echo \> \$HOME/.palladium
fi
DOCKER_EXEC mkdir -p ${DEPENDS_DIR}/SDKs ${DEPENDS_DIR}/sdk-sources
if [ -n "$OSX_SDK" ] && [ ! -f ${DEPENDS_DIR}/sdk-sources/MacOSX${OSX_SDK}.sdk.tar.gz ]; then
curl --location --fail $SDK_URL/MacOSX${OSX_SDK}.sdk.tar.gz -o ${DEPENDS_DIR}/sdk-sources/MacOSX${OSX_SDK}.sdk.tar.gz
fi
if [ -n "$OSX_SDK" ] && [ -f ${DEPENDS_DIR}/sdk-sources/MacOSX${OSX_SDK}.sdk.tar.gz ]; then
DOCKER_EXEC tar -C ${DEPENDS_DIR}/SDKs -xf ${DEPENDS_DIR}/sdk-sources/MacOSX${OSX_SDK}.sdk.tar.gz
fi
if [[ $HOST = *-mingw32 ]]; then
DOCKER_EXEC update-alternatives --set $HOST-g++ \$\(which $HOST-g++-posix\)
fi
if [ -z "$NO_DEPENDS" ]; then
if [[ $DOCKER_NAME_TAG == centos* ]]; then
# CentOS has problems building the depends if the config shell is not explicitly set
# (i.e. for libevent a Makefile with an empty SHELL variable is generated, leading to
# an error as the first command is executed)
SHELL_OPTS="CONFIG_SHELL=/bin/bash"
else
SHELL_OPTS="CONFIG_SHELL="
fi
DOCKER_EXEC $SHELL_OPTS make $MAKEJOBS -C depends HOST=$HOST $DEP_OPTS
fi
if [ "$TEST_PREVIOUS_RELEASES" = "true" ]; then
BEGIN_FOLD previous-versions
DOCKER_EXEC contrib/devtools/previous_release.sh -b -t "$PREVIOUS_RELEASES_DIR" v0.17.1 v0.18.1 v0.19.0.1
END_FOLD
fi

46
ci/test/06_script_a.sh Executable file
View File

@@ -0,0 +1,46 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
PALLADIUM_CONFIG_ALL="--disable-dependency-tracking --prefix=$DEPENDS_DIR/$HOST --bindir=$BASE_OUTDIR/bin --libdir=$BASE_OUTDIR/lib"
if [ -z "$NO_DEPENDS" ]; then
DOCKER_EXEC ccache --max-size=$CCACHE_SIZE
fi
BEGIN_FOLD autogen
if [ -n "$CONFIG_SHELL" ]; then
DOCKER_EXEC "$CONFIG_SHELL" -c "./autogen.sh"
else
DOCKER_EXEC ./autogen.sh
fi
END_FOLD
DOCKER_EXEC mkdir -p build
export P_CI_DIR="$P_CI_DIR/build"
BEGIN_FOLD configure
DOCKER_EXEC ../configure --cache-file=config.cache $PALLADIUM_CONFIG_ALL $PALLADIUM_CONFIG || ( (DOCKER_EXEC cat config.log) && false)
END_FOLD
BEGIN_FOLD distdir
# Create folder on host and docker, so that `cd` works
mkdir -p "palladium-$HOST"
DOCKER_EXEC make distdir VERSION=$HOST
END_FOLD
export P_CI_DIR="$P_CI_DIR/palladium-$HOST"
BEGIN_FOLD configure
DOCKER_EXEC ./configure --cache-file=../config.cache $PALLADIUM_CONFIG_ALL $PALLADIUM_CONFIG || ( (DOCKER_EXEC cat config.log) && false)
END_FOLD
set -o errtrace
trap 'DOCKER_EXEC "cat ${BASE_SCRATCH_DIR}/sanitizer-output/* 2> /dev/null"' ERR
BEGIN_FOLD build
DOCKER_EXEC make $MAKEJOBS $GOAL || ( echo "Build failure. Verbose build follows." && DOCKER_EXEC make $GOAL V=1 ; false )
END_FOLD

48
ci/test/06_script_b.sh Executable file
View File

@@ -0,0 +1,48 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
if [ -n "$QEMU_USER_CMD" ]; then
BEGIN_FOLD wrap-qemu
# Generate all binaries, so that they can be wrapped
DOCKER_EXEC make $MAKEJOBS -C src/secp256k1 VERBOSE=1
DOCKER_EXEC make $MAKEJOBS -C src/univalue VERBOSE=1
DOCKER_EXEC "${BASE_ROOT_DIR}/ci/test/wrap-qemu.sh"
END_FOLD
fi
if [ -n "$USE_VALGRIND" ]; then
BEGIN_FOLD wrap-valgrind
DOCKER_EXEC "${BASE_ROOT_DIR}/ci/test/wrap-valgrind.sh"
END_FOLD
fi
bash -c "${CI_WAIT}" & # Print dots in case the tests take a long time to run
if [ "$RUN_UNIT_TESTS" = "true" ]; then
BEGIN_FOLD unit-tests
DOCKER_EXEC LD_LIBRARY_PATH=$DEPENDS_DIR/$HOST/lib make $MAKEJOBS check VERBOSE=1
END_FOLD
fi
if [ "$RUN_UNIT_TESTS_SEQUENTIAL" = "true" ]; then
BEGIN_FOLD unit-tests-seq
DOCKER_EXEC LD_LIBRARY_PATH=$DEPENDS_DIR/$HOST/lib "${BASE_ROOT_DIR}/build/palladium-*/src/test/test_palladium" --catch_system_errors=no -l test_suite
END_FOLD
fi
if [ "$RUN_FUNCTIONAL_TESTS" = "true" ]; then
BEGIN_FOLD functional-tests
DOCKER_EXEC test/functional/test_runner.py --ci $MAKEJOBS --tmpdirprefix "${BASE_SCRATCH_DIR}/test_runner/" --ansi --combinedlogslen=4000 ${TEST_RUNNER_EXTRA} --quiet --failfast
END_FOLD
fi
if [ "$RUN_FUZZ_TESTS" = "true" ]; then
BEGIN_FOLD fuzz-tests
DOCKER_EXEC test/fuzz/test_runner.py ${FUZZ_TESTS_CONFIG} -l DEBUG ${DIR_FUZZ_IN}
END_FOLD
fi

18
ci/test/wrap-qemu.sh Executable file
View File

@@ -0,0 +1,18 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
for b_name in {"${BASE_OUTDIR}/bin"/*,src/secp256k1/*tests,src/univalue/{no_nul,test_json,unitester,object}}; do
# shellcheck disable=SC2044
for b in $(find "${BASE_ROOT_DIR}" -executable -type f -name $(basename $b_name)); do
echo "Wrap $b ..."
mv "$b" "${b}_orig"
echo '#!/usr/bin/env bash' > "$b"
echo "$QEMU_USER_CMD \"${b}_orig\" \"\$@\"" >> "$b"
chmod +x "$b"
done
done

18
ci/test/wrap-valgrind.sh Executable file
View File

@@ -0,0 +1,18 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
for b_name in "${BASE_OUTDIR}/bin"/*; do
# shellcheck disable=SC2044
for b in $(find "${BASE_ROOT_DIR}" -executable -type f -name $(basename $b_name)); do
echo "Wrap $b ..."
mv "$b" "${b}_orig"
echo '#!/usr/bin/env bash' > "$b"
echo "valgrind --gen-suppressions=all --quiet --error-exitcode=1 --suppressions=${BASE_ROOT_DIR}/contrib/valgrind.supp \"${b}_orig\" \"\$@\"" >> "$b"
chmod +x "$b"
done
done

14
ci/test_run_all.sh Executable file
View File

@@ -0,0 +1,14 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The Palladium Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C.UTF-8
set -o errexit; source ./ci/test/00_setup_env.sh
set -o errexit; source ./ci/test/03_before_install.sh
set -o errexit; source ./ci/test/04_install.sh
set -o errexit; source ./ci/test/05_before_script.sh
set -o errexit; source ./ci/test/06_script_a.sh
set -o errexit; source ./ci/test/06_script_b.sh