How to do incremental Jenkins Maven builds on large Multi-Module Project
up vote
0
down vote
favorite
I have a large multi-module project which takes about an hour to build from clean in Jenkins.
I am using Maven CI-friendly versioning using $revision for all module versions in the multi-module project, and I'm passing in something like -Drevision=$BUILD_NUMBER from Jenkins.
This works fine, but it is so inefficient when deployed across a team of 20-ish, with a new build for every Pull Request. Who wants to wait an hour every time a single line of code changes?!?
How can I get incremental build working with my project structure?
The obvious problem with the 'incremental build' tickbox is that when I build the changed modules (e.g. version 5) they will all depend on unchanged modules also with version 5, yet the last version that was actually built may be 1,2,3 or 4.
So $revision across the whole multi-module project will break incremental building
My other issue is that I am using multiple slaves, meaning I cannot simply rely on the file system being unchanged since last build (same Maven Local Repo, same working tree, etc).
In order to share unchanged artifacts between builds on different slaves I'd have to either use the Maven Repo (e.g. Nexus), or have some kind of file copy over SSH (e.g. each build transfers back up to Master and then down to slaves)
maven jenkins incremental-build
asked Nov 11 at 8:16
Jebonicus
1
add a comment |
up vote
0
down vote
favorite
I have a large multi-module project which takes about an hour to build from clean in Jenkins.
I am using Maven CI-friendly versioning using $revision for all module versions in the multi-module project, and I'm passing in something like -Drevision=$BUILD_NUMBER from Jenkins.
This works fine, but it is so inefficient when deployed across a team of 20-ish, with a new build for every Pull Request. Who wants to wait an hour every time a single line of code changes?!?
How can I get incremental build working with my project structure?
The obvious problem with the 'incremental build' tickbox is that when I build the changed modules (e.g. version 5) they will all depend on unchanged modules also with version 5, yet the last version that was actually built may be 1,2,3 or 4.
So $revision across the whole multi-module project will break incremental building
My other issue is that I am using multiple slaves, meaning I cannot simply rely on the file system being unchanged since last build (same Maven Local Repo, same working tree, etc).
In order to share unchanged artifacts between builds on different slaves I'd have to either use the Maven Repo (e.g. Nexus), or have some kind of file copy over SSH (e.g. each build transfers back up to Master and then down to slaves)
maven jenkins incremental-build
asked Nov 11 at 8:16
Jebonicus
1
How many modules do you have? How many tests do you have? How long does the build take? How long do the tests take? Furthermore do you use-T 4or similar? If you have a incremental checkbox in your jenkins you are using an extreme old version of Jenkins...
– khmarbaise
Nov 11 at 8:25
About 400 modules, 2000 tests. Parallel builds don't work because we have a load of horrible C/C++/JNI/Swig-wrapper projects built with combinations of ant/make wrapped in maven poms. The C++ stuff is parallel within individual modules, so that's not too bad. The root cause of our problem is that years ago the project was structured as one massive monolithic multi-module project rather than separate projects decoupled via Nexus artifact dependencies.
– Jebonicus
Nov 11 at 9:13
I'm leaning towards using the [Partial Build Plugin] (github.com/lesfurets/partial-build-plugin) to do the actual module building based on detected Git changes. Instead of global $revision versioning we would have to have a global dependencyManagement for each of the 400 modules. Every time one of the modules is built, its version gets updated. So a fairly complex solution, I was wondering if anyone could think of something simpler!
– Jebonicus
Nov 11 at 9:23
I haven't tried it but maybe it is worth looking at danielflower.github.io/multi-module-maven-release-plugin/…
– JF Meier
Nov 11 at 10:03
400 modules is not that much ...I've had projects with ca. 1000 modules with -T 5 which builds in 6 minutes..but needed some improvements of hardware and infrastructure...and 2000 tests is not much ...so this means those tests are not unit tests etc. one problem...combinations with Ant/Make is the issues...
– khmarbaise
Nov 11 at 10:04
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have a large multi-module project which takes about an hour to build from clean in Jenkins.
I am using Maven CI-friendly versioning using $revision for all module versions in the multi-module project, and I'm passing in something like -Drevision=$BUILD_NUMBER from Jenkins.
This works fine, but it is so inefficient when deployed across a team of 20-ish, with a new build for every Pull Request. Who wants to wait an hour every time a single line of code changes?!?
How can I get incremental build working with my project structure?
The obvious problem with the 'incremental build' tickbox is that when I build the changed modules (e.g. version 5) they will all depend on unchanged modules also with version 5, yet the last version that was actually built may be 1,2,3 or 4.
So $revision across the whole multi-module project will break incremental building
My other issue is that I am using multiple slaves, meaning I cannot simply rely on the file system being unchanged since last build (same Maven Local Repo, same working tree, etc).
In order to share unchanged artifacts between builds on different slaves I'd have to either use the Maven Repo (e.g. Nexus), or have some kind of file copy over SSH (e.g. each build transfers back up to Master and then down to slaves)
maven jenkins incremental-build
asked Nov 11 at 8:16
Jebonicus
1
I have a large multi-module project which takes about an hour to build from clean in Jenkins.
I am using Maven CI-friendly versioning using $revision for all module versions in the multi-module project, and I'm passing in something like -Drevision=$BUILD_NUMBER from Jenkins.
This works fine, but it is so inefficient when deployed across a team of 20-ish, with a new build for every Pull Request. Who wants to wait an hour every time a single line of code changes?!?
How can I get incremental build working with my project structure?
The obvious problem with the 'incremental build' tickbox is that when I build the changed modules (e.g. version 5) they will all depend on unchanged modules also with version 5, yet the last version that was actually built may be 1,2,3 or 4.
So $revision across the whole multi-module project will break incremental building
My other issue is that I am using multiple slaves, meaning I cannot simply rely on the file system being unchanged since last build (same Maven Local Repo, same working tree, etc).
In order to share unchanged artifacts between builds on different slaves I'd have to either use the Maven Repo (e.g. Nexus), or have some kind of file copy over SSH (e.g. each build transfers back up to Master and then down to slaves)
maven jenkins incremental-build
maven jenkins incremental-build
asked Nov 11 at 8:16
Jebonicus
1
asked Nov 11 at 8:16
Jebonicus
1
asked Nov 11 at 8:16
Jebonicus
1
asked Nov 11 at 8:16
Jebonicus
1
asked Nov 11 at 8:16
Jebonicus
1
1
How many modules do you have? How many tests do you have? How long does the build take? How long do the tests take? Furthermore do you use-T 4or similar? If you have a incremental checkbox in your jenkins you are using an extreme old version of Jenkins...
– khmarbaise
Nov 11 at 8:25
About 400 modules, 2000 tests. Parallel builds don't work because we have a load of horrible C/C++/JNI/Swig-wrapper projects built with combinations of ant/make wrapped in maven poms. The C++ stuff is parallel within individual modules, so that's not too bad. The root cause of our problem is that years ago the project was structured as one massive monolithic multi-module project rather than separate projects decoupled via Nexus artifact dependencies.
– Jebonicus
Nov 11 at 9:13
I'm leaning towards using the [Partial Build Plugin] (github.com/lesfurets/partial-build-plugin) to do the actual module building based on detected Git changes. Instead of global $revision versioning we would have to have a global dependencyManagement for each of the 400 modules. Every time one of the modules is built, its version gets updated. So a fairly complex solution, I was wondering if anyone could think of something simpler!
– Jebonicus
Nov 11 at 9:23
I haven't tried it but maybe it is worth looking at danielflower.github.io/multi-module-maven-release-plugin/…
– JF Meier
Nov 11 at 10:03
400 modules is not that much ...I've had projects with ca. 1000 modules with -T 5 which builds in 6 minutes..but needed some improvements of hardware and infrastructure...and 2000 tests is not much ...so this means those tests are not unit tests etc. one problem...combinations with Ant/Make is the issues...
– khmarbaise
Nov 11 at 10:04
add a comment |
How many modules do you have? How many tests do you have? How long does the build take? How long do the tests take? Furthermore do you use-T 4or similar? If you have a incremental checkbox in your jenkins you are using an extreme old version of Jenkins...
– khmarbaise
Nov 11 at 8:25
About 400 modules, 2000 tests. Parallel builds don't work because we have a load of horrible C/C++/JNI/Swig-wrapper projects built with combinations of ant/make wrapped in maven poms. The C++ stuff is parallel within individual modules, so that's not too bad. The root cause of our problem is that years ago the project was structured as one massive monolithic multi-module project rather than separate projects decoupled via Nexus artifact dependencies.
– Jebonicus
Nov 11 at 9:13
I'm leaning towards using the [Partial Build Plugin] (github.com/lesfurets/partial-build-plugin) to do the actual module building based on detected Git changes. Instead of global $revision versioning we would have to have a global dependencyManagement for each of the 400 modules. Every time one of the modules is built, its version gets updated. So a fairly complex solution, I was wondering if anyone could think of something simpler!
– Jebonicus
Nov 11 at 9:23
I haven't tried it but maybe it is worth looking at danielflower.github.io/multi-module-maven-release-plugin/…
– JF Meier
Nov 11 at 10:03
400 modules is not that much ...I've had projects with ca. 1000 modules with -T 5 which builds in 6 minutes..but needed some improvements of hardware and infrastructure...and 2000 tests is not much ...so this means those tests are not unit tests etc. one problem...combinations with Ant/Make is the issues...
– khmarbaise
Nov 11 at 10:04
How many modules do you have? How many tests do you have? How long does the build take? How long do the tests take? Furthermore do you use
-T 4 or similar? If you have a incremental checkbox in your jenkins you are using an extreme old version of Jenkins...– khmarbaise
Nov 11 at 8:25
How many modules do you have? How many tests do you have? How long does the build take? How long do the tests take? Furthermore do you use
-T 4 or similar? If you have a incremental checkbox in your jenkins you are using an extreme old version of Jenkins...– khmarbaise
Nov 11 at 8:25
About 400 modules, 2000 tests. Parallel builds don't work because we have a load of horrible C/C++/JNI/Swig-wrapper projects built with combinations of ant/make wrapped in maven poms. The C++ stuff is parallel within individual modules, so that's not too bad. The root cause of our problem is that years ago the project was structured as one massive monolithic multi-module project rather than separate projects decoupled via Nexus artifact dependencies.
– Jebonicus
Nov 11 at 9:13
About 400 modules, 2000 tests. Parallel builds don't work because we have a load of horrible C/C++/JNI/Swig-wrapper projects built with combinations of ant/make wrapped in maven poms. The C++ stuff is parallel within individual modules, so that's not too bad. The root cause of our problem is that years ago the project was structured as one massive monolithic multi-module project rather than separate projects decoupled via Nexus artifact dependencies.
– Jebonicus
Nov 11 at 9:13
I'm leaning towards using the [Partial Build Plugin] (github.com/lesfurets/partial-build-plugin) to do the actual module building based on detected Git changes. Instead of global $revision versioning we would have to have a global dependencyManagement for each of the 400 modules. Every time one of the modules is built, its version gets updated. So a fairly complex solution, I was wondering if anyone could think of something simpler!
– Jebonicus
Nov 11 at 9:23
I'm leaning towards using the [Partial Build Plugin] (github.com/lesfurets/partial-build-plugin) to do the actual module building based on detected Git changes. Instead of global $revision versioning we would have to have a global dependencyManagement for each of the 400 modules. Every time one of the modules is built, its version gets updated. So a fairly complex solution, I was wondering if anyone could think of something simpler!
– Jebonicus
Nov 11 at 9:23
I haven't tried it but maybe it is worth looking at danielflower.github.io/multi-module-maven-release-plugin/…
– JF Meier
Nov 11 at 10:03
I haven't tried it but maybe it is worth looking at danielflower.github.io/multi-module-maven-release-plugin/…
– JF Meier
Nov 11 at 10:03
400 modules is not that much ...I've had projects with ca. 1000 modules with -T 5 which builds in 6 minutes..but needed some improvements of hardware and infrastructure...and 2000 tests is not much ...so this means those tests are not unit tests etc. one problem...combinations with Ant/Make is the issues...
– khmarbaise
Nov 11 at 10:04
400 modules is not that much ...I've had projects with ca. 1000 modules with -T 5 which builds in 6 minutes..but needed some improvements of hardware and infrastructure...and 2000 tests is not much ...so this means those tests are not unit tests etc. one problem...combinations with Ant/Make is the issues...
– khmarbaise
Nov 11 at 10:04
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53246960%2fhow-to-do-incremental-jenkins-maven-builds-on-large-multi-module-project%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
How many modules do you have? How many tests do you have? How long does the build take? How long do the tests take? Furthermore do you use
-T 4or similar? If you have a incremental checkbox in your jenkins you are using an extreme old version of Jenkins...– khmarbaise
Nov 11 at 8:25
About 400 modules, 2000 tests. Parallel builds don't work because we have a load of horrible C/C++/JNI/Swig-wrapper projects built with combinations of ant/make wrapped in maven poms. The C++ stuff is parallel within individual modules, so that's not too bad. The root cause of our problem is that years ago the project was structured as one massive monolithic multi-module project rather than separate projects decoupled via Nexus artifact dependencies.
– Jebonicus
Nov 11 at 9:13
I'm leaning towards using the [Partial Build Plugin] (github.com/lesfurets/partial-build-plugin) to do the actual module building based on detected Git changes. Instead of global $revision versioning we would have to have a global dependencyManagement for each of the 400 modules. Every time one of the modules is built, its version gets updated. So a fairly complex solution, I was wondering if anyone could think of something simpler!
– Jebonicus
Nov 11 at 9:23
I haven't tried it but maybe it is worth looking at danielflower.github.io/multi-module-maven-release-plugin/…
– JF Meier
Nov 11 at 10:03
400 modules is not that much ...I've had projects with ca. 1000 modules with -T 5 which builds in 6 minutes..but needed some improvements of hardware and infrastructure...and 2000 tests is not much ...so this means those tests are not unit tests etc. one problem...combinations with Ant/Make is the issues...
– khmarbaise
Nov 11 at 10:04