mirror of
https://projects.blender.org/blender/blender.git
synced 2025-01-22 07:22:12 -05:00
bb8460da9e
This only works with GCC and has only been tested on Linux. The main goal is to automatically generate the code coverage reports on the buildbot and to publish them. With some luck, this motivates people to increase test coverage in their respective areas. Nevertheless, it should be easy to generate the reports locally too (at least on supported software stacks). Usage: 1. Create a **debug** build using **GCC** with **WITH_COMPILER_CODE_COVERAGE** enabled. 2. Run tests. This automatically generates `.gcda` files in the build directory. 3. Run `make/ninja coverage-report` in the build directory. If everything is successful, this will open a browser with the final report which is stored in `build-dir/coverage/report/`. For a bit more control one can also run `coverage.py` script directly. This allows passing in the `--no-browser` option which may be benefitial when running it on the buildbot. Running `make/ninja coverage-reset` deletes all `.gcda` files which resets the line execution counts. The final report has a main entry point (`index.html`) and a separate `.html` file for every source code file that coverage data was available for. This also contains some code that is not in Blender's git repository. We could filter those out, but it also seems interesting (to me anyway), so I just kept it in. Doing the analysis and writing the report takes ~1 min. The slow part is running all tests in a debug build which takes ~12 min for me. Since the coverage data is fairly large and the report also includes the entire source code, file compression is used in two places: * The intermediate analysis results for each file are stored in compressed zip files. This data is still independent from the report html and could be used to build other tools on top of. I could imagine storing the analysis data for each day for example to gather greater insights into how coverage changes over time in different parts of the code. * The analysis data and source code is compressed and base64 encoded embedded into the `.html` files. This makes them much smaller than embedding the data without compression (5-10x). Pull Request: https://projects.blender.org/blender/blender/pulls/126181
85 lines
2.9 KiB
CMake
85 lines
2.9 KiB
CMake
# SPDX-FileCopyrightText: 2014-2023 Blender Authors
|
|
#
|
|
# SPDX-License-Identifier: GPL-2.0-or-later
|
|
|
|
# Always run tests from install path, so all required scripts and libraries
|
|
# are available and we are testing the actual installation layout.
|
|
#
|
|
# Getting the install path of the executable is somewhat involved, as there are
|
|
# no direct CMake generator expressions to get the install paths of executables.
|
|
set(TEST_INSTALL_DIR ${CMAKE_INSTALL_PREFIX_WITH_CONFIG})
|
|
|
|
# Path to Blender and Python executables for all platforms.
|
|
if(MSVC)
|
|
set(TEST_BLENDER_EXE ${TEST_INSTALL_DIR}/blender.exe)
|
|
elseif(APPLE)
|
|
set(TEST_BLENDER_EXE ${TEST_INSTALL_DIR}/Blender.app/Contents/MacOS/Blender)
|
|
else()
|
|
if(WITH_INSTALL_PORTABLE)
|
|
set(TEST_BLENDER_EXE ${TEST_INSTALL_DIR}/blender)
|
|
else()
|
|
set(TEST_BLENDER_EXE ${TEST_INSTALL_DIR}/bin/blender)
|
|
endif()
|
|
endif()
|
|
|
|
# The installation directory's Python is the best one to use. However, it can only be there
|
|
# after the install step, # which means that Python will never be there on a fresh system.
|
|
# To suit different needs, the user can pass `-DTEST_PYTHON_EXE=/path/to/python` to CMake.
|
|
if(NOT TEST_PYTHON_EXE)
|
|
set(TEST_PYTHON_EXE ${PYTHON_EXECUTABLE})
|
|
if(FIRST_RUN)
|
|
message(STATUS "Tests: Using Python executable: ${TEST_PYTHON_EXE}")
|
|
endif()
|
|
elseif(NOT EXISTS ${TEST_PYTHON_EXE})
|
|
message(FATAL_ERROR "Tests: TEST_PYTHON_EXE ${TEST_PYTHON_EXE} does not exist")
|
|
endif()
|
|
|
|
|
|
# Include these arguments before all others, they must not interfere with Python execution.
|
|
set(TEST_PYTHON_EXE_EXTRA_ARGS "")
|
|
|
|
# Check if this a Blender managed Python installation, if so, don't add `*.pyc` files.
|
|
if(DEFINED LIBDIR)
|
|
path_is_prefix(LIBDIR TEST_PYTHON_EXE _is_prefix)
|
|
if(_is_prefix)
|
|
# Keep the Python in Blender's SVN LIBDIR pristine, to avoid conflicts on updating.
|
|
set(TEST_PYTHON_EXE_EXTRA_ARGS "-B")
|
|
endif()
|
|
unset(_is_prefix)
|
|
endif()
|
|
|
|
# For testing with Valgrind
|
|
# set(TEST_BLENDER_EXE valgrind --track-origins=yes --error-limit=no ${TEST_BLENDER_EXE})
|
|
|
|
# Standard Blender arguments for running tests.
|
|
# Specify exit code so that if a Python script error happens, the test fails.
|
|
set(TEST_BLENDER_EXE_PARAMS
|
|
--background --factory-startup --debug-memory --debug-exit-on-error --python-exit-code 1
|
|
)
|
|
|
|
# Python CTests
|
|
if(WITH_BLENDER AND WITH_PYTHON AND NOT WITH_PYTHON_MODULE)
|
|
add_subdirectory(python)
|
|
endif()
|
|
|
|
# Blender as python module tests.
|
|
if(WITH_PYTHON_MODULE)
|
|
add_subdirectory(blender_as_python_module)
|
|
endif()
|
|
|
|
# GTest
|
|
add_subdirectory(gtests)
|
|
|
|
if(WITH_COMPILER_CODE_COVERAGE)
|
|
set(COVERAGE_SCRIPT_PATH ${CMAKE_CURRENT_SOURCE_DIR}/coverage/coverage.py)
|
|
|
|
add_custom_target(coverage-report
|
|
${PYTHON_EXECUTABLE} ${COVERAGE_SCRIPT_PATH} report --build-directory ${CMAKE_BINARY_DIR}
|
|
USES_TERMINAL
|
|
)
|
|
|
|
add_custom_target(coverage-reset
|
|
${PYTHON_EXECUTABLE} ${COVERAGE_SCRIPT_PATH} reset --build-directory ${CMAKE_BINARY_DIR}
|
|
USES_TERMINAL
|
|
)
|
|
endif()
|