Context
Balancing the ease of implementation with the correctness of a solution is a complex trade-off. When developing a package to be used in a CI/CD pipeline for multiple repositories, I encountered the challenge of deciding how to handle the package's versioning strategy within the CI process.
Problem
Pinning the package to a specific version in the Jenkins script for each repository ensures that the CI process is stable and predictable. However, this approach necessitates manual intervention for each repository whenever a new package version is released, which can be problematic, especially considering the following pragmatic factors:
- Diverse repositories managed by different teams, where gaining approvals for changes can be time-consuming and laborious.
- The package is still under active development, with new versions released frequently.
On the other hand, always using the latest package version in CI pipelines simplifies updates but risks unexpected disruptions. This approach can eliminate the need for manual updates to many repositories but also introduces the risk of breaking changes, leading to failing CI pipelines across various repositories, which can have adverse consequences:
- Unexpected disruptions for developers in their branches or PRs.
- Resistance from developers, possibly leading to the removal or ignoring of this CI step.
Discussion
Finding a balance between the two approaches is crucial, and importantly, it requires a deeper understanding of the underlying problems and whether we can address them in a more fundamental way. Here are some hidden issues behind this problem:
- Why do cross-team, multi-repo changes intimidate and slow down processes?
- Are there ways to automate the creation of similar changes to multiple repositories?
For the first problem, it might be a management issue where a standard procedure can be devised to guide the process of assigning responsibilities and gaining approvals for cross-repo changes within the organization/team. For the second problem, it might require additional tooling to address the repetitive nature of the changes. It could also suggest that this configuration might benefit from more centralized control, where a single repository can manage the package version for all connected repositories.
Retrofitting
Before diving into what I would consider a better approach, I would like to discuss how we can retrofit the easy solution (always install and use the latest package version in CI pipelines):
- Commit to backward compatibility: Avoid breaking changes at all costs.
- Support previous x versions:
- Maintain backward compatibility for the previous x versions.
- Notify users of required upgrades without breaking their current setup for a reasonable period.
- Provide upgrade support: Assist repositories in adapting before releasing breaking changes and updating the package version after new releases.
Solution
The solution I propose is to pin a specific package version in CI and upgrade only when necessary. To address the issues, I would also propose the improvement items mentioned in the discussion section:
- To deal with the troublesome manual updates:
- Create codemod-like tools or scripts to automate the process.
- Revert the usage model to more centralized control, where a single repository can configure the package version and the repositories that will use this package in the CI pipeline.
- To deal with cross-team, multi-repo changes:
- Find out the established process for proposing and getting support for cross-repo changes, which may involve sharing the proposal in a forum/meeting, getting the owners' support, and then proceeding with the changes with known assigned liaisons for each repository.