Dealing with Maven dependencies when switching to Git
そう、私たちは Git への移行を進めていて、git-flow が気に入っている。さて、次は？何もかもテストしてみよう！私のチームはすごい。彼らは Confluence にある開発ワークフローのヒットリスト、つまりこれまでのチームの開発をもとにしたすべてのワークフローと、将来取り組まなければならない可能性があると考える特殊なワークフローすべてをかき集めました。そして、私たちのプロジェクトをミラーリングしたプロジェクト構成（ただし、コードはなしで 、pom.xml だけ）を使って、すべてのワークフローを試してみました。
Maven dependencies were about to prove themselves to be our biggest problem in all of this.
Maven Build Numbering Maven produces
1.0.0-SNAPSHOTbuilds until you release. When you release,
-SNAPSHOTis removed and your version is
1.0.0. Your build process needs to be able to support incrementing your minor version up after the fact so that subsequent work on your next effort will produce builds like
1.1.0-SNAPSHOT. You're not tied to three digits -- you can define this when you start a project, but we use three. Anyway, the -SNAPSHOT part is really important to understand. This is always going to represent the latest pre-release cut of a project.
See, our big concern in all of these workflows was how we were going to ensure that our project versions and inter-project dependencies were properly managed.
ビルドで Maven の依存関係が検出されるたびに、既定では、その依存関係を Ye Olde Internet(e)TM からダウンロードします。これらのアーティファクトはローカルに保存されるため、その後のビルドはより速く実行されます。この手順にかかる手間を少し省く 1 つの方法が、ローカル ネットワーク上のアーティファクト リポジトリを外部依存関係用のキャッシュとして使用するやり方です。最も速い CDN からダウンロードするより、LAN による取得の方が大体いつも速くなります。私たちは Artifactory Pro をアーティファクト用リポジトリとして使用しています。また、マルチモジュール構成であるため、チームのビルド アーティファクトも Artifactory に保存しています。共通のパッケージの 1 つをビルドする時、その特定のバージョンを maven dependency resolution を介してダウンロードし、アーティファクト リポジトリーからすぐにアーティファクトを取得できます。
このやり方はとてもうまく機能します。また、Artifactory はインスタンス間でアーティファクトを同期してくれるため、たとえば Artifactory を使用して本番デプロイ用にリリース リポジトリをデータセンターに複製したい場合、別のプロセスをビルドすることなしにこの作業が実行できます。
Maven dependencies, feature branches and pull requests
All of our builds go into Artifactory. With SVN, we had been using a snapshot repository for keeping the latest 2 snapshot builds, a staging repository for any release builds not yet approved, and a release repository only for the builds blessed to go into production. These builds are numbered like I described earlier, and are retrievable by a predictable URL pattern based on repository and version.
As I said before we have multiple layers of dependency between our projects. There's a very good reason for this - both historically and strategically - for our products. We've considered alternate architectures that would eliminate this problem, but they'd introduce others. We can make our lives easier (and we did, but that's for a later post), but for now it's strategic for us to keep our structure as it is.
So developer A, let's call her Angela, starts work on a feature in Jira. This requires two branches: one from our common project and one from product X. The version for common is 2.1.0-SNAPSHOT. The version for productX is 2.4.0-SNAPSHOT. She works locally for a while and then finally pushes back up to Bitbucket Server. Bamboo picks up these changes, builds the common package and uploads common-2.1.0-SNAPSHOT to Artifactory, then builds productX with a dependency on common-2.1.0-SNAPSHOT, uploading productX-2.4.0-SNAPSHOT as well. Unit tests pass!
Developer B, let's call him Bruce, starts work on another feature in Jira, for a different product: productY. This also requires two branches: one from our common project and one from productY. The version for common is, as above, 2.1.0-SNAPSHOT. The version of product Y is 2.7.0-SNAPSHOT. He works locally for a while and then finally pushes his changes up to Bitbucket Server. Bamboo picks up these changes, builds the common package and uploads common-2.1.0-SNAPSHOT to Artifactory, then builds productX with a dependency on common-2.1.0-SNAPSHOT, uploading productX-2.4.0-SNAPSHOT as well. Unit tests pass!
Angela, meanwhile, finds a small bug in her productX code and writes a unit test to validate her fix. She runs it locally and it passes. She pushes her changes to Bitbucket Server, and Bamboo picks up the change and builds productX. The build succeeds, but some of her unit tests fail. It's not the new ones she wrote, but the first ones from her initial changes to the feature. Somehow the Bamboo build has found a regression that her local build didn't? How is that possible?
Because her common dependency, the one Bamboo pulled in when it built productX, was no longer her copy. Bruce over-wrote common-2.1.0-SNAPSHOT in artifactory when his feature build completed. There was no source code conflict - both developers were working in isolation on their own branches, but the source of truth for Maven's artifact retrieval was corrupted.
For about a month after we discovered this problem we tried everything to get around this. Through our TAM, we talked to people on the Bamboo team who use git-flow, and we talked to the developer who maintains git-flow, a java implementation of git-flow. They were all super helpful, but short of a process that required a list of manual steps for each developer every time they worked on a feature, we couldn't find a resolution that was tolerable.
If you're curious what we considered, here's everything we tried:
- We can do this with
mvn jgitflow:feature-startto create the branch.
- Bitbucket Server のフック、またはローカルの githook を使用できます。
- We can manually set with
mvn version:set-versionafter we create the branch.
- We can automate the change with the [maven-external-version] plugin.
- We can do this with
- We can do this with
mvn jgitflow:feature-finishto finish the branch.
- Use a git merge driver to handle pom conflicts.
- Use an asynchronous post-receive hook in Bitbucket Server
- We can do this with
Each one of these options had some sort of negative side-effect. Chiefly, manual steps for a developer each and every time they needed a feature branch. And we wanted them to create feature branches all the time. Also in most cases we could not effectively use pull requests, which was a deal-breaker.
1 人から 2 人のメンバーがほぼ 2 か月間費やした後、 私たちがこの問題に間違った方向から取り組んでいた（ショッキングな）理由が明らかになりました。
Hindsight being 20/20, I can clearly see that our biggest mistake was that we were focusing our attention on the git-flow tools rather than using the tools we already had to implement the workflow we wanted. We had:
- Jira Software
- Bamboo Server
- Artifactory Pro
Turns out, those were all of the tools we needed.
One of our engineers got the very bright idea that since the problem wasn't the build management itself but rather the artifacts being over-written, that we should fix Artifactory instead. His idea was to use a Maven property to set the snapshot repository URL to a custom URL which included the Jira issue ID, and then write out its artifacts to a dynamically-created repository in Artifactory with a custom template. Maven’s dependency resolver will find artifacts in the develop snapshot repository if we haven’t needed to branch them, for instance if we’re only working on a product and not also common.
We set that handy little property variable in our build settings file, and wrote a Maven plugin to populate it during the earliest part of maven’s build lifecycle. On paper, this sounded incredible and re-invigorated the team to work harder to solve this problem. Trouble was that we couldn't actually do this. The earliest stage of the maven lifecycle is 'validate'. By the time plugins bound to validate have been run, the repository URLs were already resolved. Because of this, our variable never populated and the URL is not branch-named after all. Even though we had been using a layout in a separate repository from our develop snapshots, it wouldn’t be isolated for parallel development.
“Here’s to beer: the cause of, and solution to, all of life’s problems.” - Homer Simpson
Extensions, like plugins, give you a whole host of power to enhance your Maven workflow, however they are executed before lifecycle goals, and have greater access to the Maven internals. By utilizing the RepositoryUtils package, we forced Maven to re-evaluate its URLs using a custom parser and then re-set them using our updated values.
Extension in place and tested, we started knocking off pre-migration tasks one after another, going from "this is never going to happen" to "this IS going to happen Monday... so now I need to write ten pages of documentation by tomorrow". I'll write more soon about how the tools work together to achieve our new development workflow, and some of the lessons we learned about the process.
: One downside here was that I had to use a script I wrote to hit the Artifactory REST API to "promote" builds from staging to release. It's fast enough, but begging for more automation.
: Technical Account Manager. More information here.
: After the initial development efforts, we found that we had to do even more to make this work 100% of the time, like when a snapshot is newer in Artifactory (from another engineer) than your local snapshot, Maven grabs the remote artifact because hey, it's NEWER, so it must be BETTER, right?