Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Files from dependency non-deterministically included in package built on Windows #1375

Open
traversaro opened this issue Jan 23, 2025 · 1 comment
Open

Files from dependency non-deterministically included in package built on Windows #1375

traversaro opened this issue Jan 23, 2025 · 1 comment

Comments

@traversaro
Copy link

In the last weeks I experienced a non-deterministic issue when building Windows package with rattler-build . In particular, what is happening is that sometimes files from the dependency are included in the built packages.

The problem is non-deterministic so it is tricky to reproduce, but I was able to put together a mwe in https://github.com/traversaro/rattler-build-clobber-bomb-mre, that you can run (on Windows) with:

git clone https://github.com/traversaro/rattler-build-clobber-bomb-mre
cd rattler-build-clobber-bomb-mre
pixi run reproduce

What happens (just in some cases) is that the build packages also include some files that are part of the dependencies. For example, see the run in https://github.com/traversaro/rattler-build-clobber-bomb-mre/actions/runs/12931376501/job/36065083136 the log includes:

2025-01-23T14:34:46.1642155Z  │ │   -- Generating done (0.4s)
2025-01-23T14:34:46.1653624Z  │ │   -- Build files have been written to: %SRC_DIR%/build
2025-01-23T14:34:46.2494056Z  │ │   1>
2025-01-23T14:34:46.3053405Z  │ │   -- Install configuration: "Release"
2025-01-23T14:34:46.3054427Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/joint_limiters.xml
2025-01-23T14:34:46.3059494Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits
2025-01-23T14:34:46.3062788Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits
2025-01-23T14:34:46.3067122Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/joint_limiter_interface.hpp
2025-01-23T14:34:46.3073451Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/joint_limits.hpp
2025-01-23T14:34:46.3079327Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/joint_limits_rosparam.hpp
2025-01-23T14:34:46.3086998Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/joint_limits_urdf.hpp
2025-01-23T14:34:46.3092331Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/joint_saturation_limiter.hpp
2025-01-23T14:34:46.3099217Z  │ │   -- Installing: %PREFIX%/Library/lib/joint_limiter_interface.lib
2025-01-23T14:34:46.3106185Z  │ │   -- Installing: %PREFIX%/Library/bin/joint_limiter_interface.dll
2025-01-23T14:34:46.3113286Z  │ │   -- Installing: %PREFIX%/Library/lib/joint_saturation_limiter.lib
2025-01-23T14:34:46.3123299Z  │ │   -- Installing: %PREFIX%/Library/bin/joint_saturation_limiter.dll
2025-01-23T14:34:46.3140403Z  │ │   -- Installing: %PREFIX%/Library/share/ament_index/resource_index/package_run_dependencies/joint_limits
2025-01-23T14:34:46.3154232Z  │ │   -- Installing: %PREFIX%/Library/share/ament_index/resource_index/parent_prefix_path/joint_limits
2025-01-23T14:34:46.3168332Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/environment/ament_prefix_path.bat
2025-01-23T14:34:46.3176745Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/environment/ament_prefix_path.dsv
2025-01-23T14:34:46.3183116Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/environment/path.bat
2025-01-23T14:34:46.3189856Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/environment/path.dsv
2025-01-23T14:34:46.3196663Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/local_setup.bat
2025-01-23T14:34:46.3203574Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/local_setup.dsv
2025-01-23T14:34:46.3210254Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/package.dsv
2025-01-23T14:34:46.3217591Z  │ │   -- Installing: %PREFIX%/Library/share/ament_index/resource_index/packages/joint_limits
2025-01-23T14:34:46.3225316Z  │ │   -- Installing: %PREFIX%/Library/share/ament_index/resource_index/joint_limits__pluginlib__plugin/joint_limits
2025-01-23T14:34:46.3233918Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/export_joint_limitsExport.cmake
2025-01-23T14:34:46.3242039Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/export_joint_limitsExport-release.cmake
2025-01-23T14:34:46.3249146Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/ament_cmake_export_targets-extras.cmake
2025-01-23T14:34:46.3255729Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/ament_cmake_export_dependencies-extras.cmake
2025-01-23T14:34:46.3263694Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/joint_limitsConfig.cmake
2025-01-23T14:34:46.3272786Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/cmake/joint_limitsConfig-version.cmake
2025-01-23T14:34:46.3279068Z  │ │   -- Installing: %PREFIX%/Library/share/joint_limits/package.xml
2025-01-23T14:34:46.3287253Z  │ │   -- Installing: %PREFIX%/Library/include/joint_limits/joint_limits/version.h
2025-01-23T14:34:46.3734995Z  │ │ %SRC_DIR%\build>if errorlevel 1 exit 1 
2025-01-23T14:34:46.3755201Z  │ │
2025-01-23T14:34:46.3755742Z  │ ╰─────────────────── (took 77 seconds)
2025-01-23T14:34:46.7787695Z  │
2025-01-23T14:34:46.7788107Z  │ ╭─ Packaging new files
2025-01-23T14:34:46.7788437Z  │ │ Copying done!
2025-01-23T14:34:46.8068570Z  │ │ Post-processing done!
2025-01-23T14:34:46.8068971Z  │ │ Writing test files
2025-01-23T14:34:46.8075644Z  │ │ Writing metadata for package
2025-01-23T14:34:46.8201346Z  │ │ Copying license files
2025-01-23T14:34:46.8201869Z  │ │ Copying recipe files
2025-01-23T14:34:46.8305865Z  │ │ Creating entry points
2025-01-23T14:34:46.8306264Z  │ │ 
2025-01-23T14:34:46.8306529Z  │ │ Files in package:
2025-01-23T14:34:46.8307190Z  │ │   - Library/bin/joint_limiter_interface.dll
2025-01-23T14:34:46.8307983Z  │ │   - Library/bin/joint_saturation_limiter.dll
2025-01-23T14:34:46.8308735Z  │ │   - Library/cmake/console_bridge-config-version.cmake
2025-01-23T14:34:46.8309498Z  │ │   - Library/cmake/console_bridge-config.cmake
2025-01-23T14:34:46.8310448Z  │ │   - Library/cmake/console_bridge-targets-release.cmake
2025-01-23T14:34:46.8311268Z  │ │   - Library/cmake/console_bridge-targets.cmake
2025-01-23T14:34:46.8312012Z  │ │   - Library/cmake/urdfdom_headers-config-version.cmake
2025-01-23T14:34:46.8312760Z  │ │   - Library/cmake/urdfdom_headers-config.cmake
2025-01-23T14:34:46.8313474Z  │ │   - Library/cmake/urdfdom_headersExport.cmake
2025-01-23T14:34:46.8314363Z  │ │   - Library/include/joint_limits/joint_limits/joint_limiter_interface.hpp
2025-01-23T14:34:46.8315337Z  │ │   - Library/include/joint_limits/joint_limits/joint_limits.hpp
2025-01-23T14:34:46.8316296Z  │ │   - Library/include/joint_limits/joint_limits/joint_limits_rosparam.hpp
2025-01-23T14:34:46.8317206Z  │ │   - Library/include/joint_limits/joint_limits/joint_limits_urdf.hpp
2025-01-23T14:34:46.8317990Z  │ │   - Library/include/joint_limits/joint_limits/joint_saturation_limiter.hpp
2025-01-23T14:34:46.8318753Z  │ │   - Library/include/joint_limits/joint_limits/version.h
2025-01-23T14:34:46.8319211Z  │ │   - Library/lib/joint_limiter_interface.lib
2025-01-23T14:34:46.8319675Z  │ │   - Library/lib/joint_saturation_limiter.lib
2025-01-23T14:34:46.8320287Z  │ │   - Library/share/ament_index/resource_index/joint_limits__pluginlib__plugin/joint_limits
2025-01-23T14:34:46.8321001Z  │ │   - Library/share/ament_index/resource_index/package_run_dependencies/joint_limits
2025-01-23T14:34:46.8321589Z  │ │   - Library/share/ament_index/resource_index/packages/joint_limits
2025-01-23T14:34:46.8322175Z  │ │   - Library/share/ament_index/resource_index/parent_prefix_path/joint_limits
2025-01-23T14:34:46.8323077Z  │ │   - Library/share/joint_limits/cmake/ament_cmake_export_dependencies-extras.cmake
2025-01-23T14:34:46.8323736Z  │ │   - Library/share/joint_limits/cmake/ament_cmake_export_targets-extras.cmake
2025-01-23T14:34:46.8324313Z  │ │   - Library/share/joint_limits/cmake/export_joint_limitsExport-release.cmake
2025-01-23T14:34:46.8324873Z  │ │   - Library/share/joint_limits/cmake/export_joint_limitsExport.cmake
2025-01-23T14:34:46.8325374Z  │ │   - Library/share/joint_limits/cmake/joint_limitsConfig-version.cmake
2025-01-23T14:34:46.8325843Z  │ │   - Library/share/joint_limits/cmake/joint_limitsConfig.cmake
2025-01-23T14:34:46.8326304Z  │ │   - Library/share/joint_limits/environment/ament_prefix_path.bat
2025-01-23T14:34:46.8326765Z  │ │   - Library/share/joint_limits/environment/ament_prefix_path.dsv
2025-01-23T14:34:46.8327176Z  │ │   - Library/share/joint_limits/environment/path.bat
2025-01-23T14:34:46.8327555Z  │ │   - Library/share/joint_limits/environment/path.dsv
2025-01-23T14:34:46.8327921Z  │ │   - Library/share/joint_limits/joint_limiters.xml
2025-01-23T14:34:46.8328279Z  │ │   - Library/share/joint_limits/local_setup.bat
2025-01-23T14:34:46.8329327Z  │ │   - Library/share/joint_limits/local_setup.dsv
2025-01-23T14:34:46.8329683Z  │ │   - Library/share/joint_limits/package.dsv
2025-01-23T14:34:46.8330010Z  │ │   - Library/share/joint_limits/package.xml

The files:

2025-01-23T14:34:46.8308735Z  │ │   - Library/cmake/console_bridge-config-version.cmake
2025-01-23T14:34:46.8309498Z  │ │   - Library/cmake/console_bridge-config.cmake
2025-01-23T14:34:46.8310448Z  │ │   - Library/cmake/console_bridge-targets-release.cmake
2025-01-23T14:34:46.8311268Z  │ │   - Library/cmake/console_bridge-targets.cmake
2025-01-23T14:34:46.8312012Z  │ │   - Library/cmake/urdfdom_headers-config-version.cmake
2025-01-23T14:34:46.8312760Z  │ │   - Library/cmake/urdfdom_headers-config.cmake
2025-01-23T14:34:46.8313474Z  │ │   - Library/cmake/urdfdom_headersExport.cmake

are the problematic ones, as they are not installed by CMake (see the earliers lines that start with -- Installing:), but are instead part of urdfdom-headers, urdfdom and console_bridge packages.

In some local builds, I noticed the same problem with other files:

 │ │   - Library/CMake/yamlConfig.cmake
 │ │   - Library/CMake/yamlConfigVersion.cmake
 │ │   - Library/CMake/yamlTargets-release.cmake
 │ │   - Library/CMake/yamlTargets.cmake

In some builds (for example in https://github.com/traversaro/rattler-build-clobber-bomb-mre/actions/runs/12931317141/job/36064883960) no spurious files are included.
@traversaro
Copy link
Author

A side-effect of this bug is that as the files spuriously included are actually part of the dependencies, as soon as the package is installed, this results in the problematic files being clobbered, and if you are building a deep enough graph of packages, you start seeing an "explosion" of clobbered files, see for example:

2025-01-22T00:08:18.6843107Z  │ │   - Library/CMake/console_bridge-config-version.cmake__clobber-from-ros-jazzy-joint-limits
2025-01-22T00:08:18.6844348Z  │ │   - Library/CMake/console_bridge-config-version.cmake__clobber-from-ros-jazzy-rclcpp-components__clobber-from-ros-jazzy-position-controllers
2025-01-22T00:08:18.6845562Z  │ │   - Library/CMake/console_bridge-config.cmake__clobber-from-ros-jazzy-joint-limits
2025-01-22T00:08:18.6847233Z  │ │   - Library/CMake/console_bridge-config.cmake__clobber-from-ros-jazzy-rclcpp-components__clobber-from-ros-jazzy-position-controllers
2025-01-22T00:08:18.6848450Z  │ │   - Library/CMake/console_bridge-targets-release.cmake__clobber-from-ros-jazzy-joint-limits
2025-01-22T00:08:18.6850263Z  │ │   - Library/CMake/console_bridge-targets-release.cmake__clobber-from-ros-jazzy-rclcpp-components__clobber-from-ros-jazzy-position-controllers
2025-01-22T00:08:18.6851758Z  │ │   - Library/CMake/console_bridge-targets.cmake__clobber-from-ros-jazzy-joint-limits
2025-01-22T00:08:18.6853317Z  │ │   - Library/CMake/console_bridge-targets.cmake__clobber-from-ros-jazzy-rclcpp-components__clobber-from-ros-jazzy-position-controllers
2025-01-22T00:08:18.6854479Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-moveit-resources-prbt-ikfast-manipulator-plugin
2025-01-22T00:08:18.6855924Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-moveit-ros-occupancy-map-monitor
2025-01-22T00:08:18.6857379Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-pilz-industrial-motion-planner-testutils
2025-01-22T00:08:18.6859199Z  │ │   - Library/CMake/urdfdom_headers-config-version.cmake__clobber-from-ros-jazzy-rviz-common
2025-01-22T00:08:18.6860522Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-moveit-resources-prbt-ikfast-manipulator-plugin
2025-01-22T00:08:18.6861649Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-moveit-ros-occupancy-map-monitor
2025-01-22T00:08:18.6863388Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-pilz-industrial-motion-planner-testutils
2025-01-22T00:08:18.6864494Z  │ │   - Library/CMake/urdfdom_headers-config.cmake__clobber-from-ros-jazzy-rviz-common
2025-01-22T00:08:18.6865888Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-moveit-resources-prbt-ikfast-manipulator-plugin
2025-01-22T00:08:18.6867355Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-moveit-ros-occupancy-map-monitor
2025-01-22T00:08:18.6868699Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-pilz-industrial-motion-planner-testutils
2025-01-22T00:08:18.6869943Z  │ │   - Library/CMake/urdfdom_headersExport.cmake__clobber-from-ros-jazzy-rviz-common

from RoboStack/ros-jazzy#16 .

@traversaro traversaro changed the title Files from dependency non-deterministically included in package build on Windows Files from dependency non-deterministically included in package built on Windows Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@traversaro