Continuous Integration Systems are one of the most important tools for agile software development. They automate the process of building and testing. A lot of people seem to have realized their importance and there are quite a few products in this arena. I already used Hudson and CruiseControl, but for OpenMRS we need to find one which is best suited to our needs. So I started out about 3 weeks back to create this roundup of continuous integration servers. This should be a useful roundup for any project with similar requirements.
Why does OpenMRS need a Continuous Integration System?
Any software development effort needs to take care that regression doesn’t happen with new code changes. Often a change in the API/module core results in breaking of modules dependent on an earlier method. A Continuous Integration System will rebuild OpenMRS after a change is committed and provide information on how that change is affecting related code.
OpenMRS would also benefit from an easy to understand UI that Continuous Integration Systems provide for number of failing tests. The number of passing or failing Unit Tests will indicate the quality of a build and help implementers/testers realize the stability of a build. We can also set some goals on how many test methods we need to write before an API method can be finalized or deprecated.
The work done in different branches and modules can be monitored and looked at easily by the community.
Building of OpenMRS Installer using NBI can be automated and new users can directly test with the latest build of OpenMRS using the cross-platform installer.
Thus to summarize, a continuous integration system will bring better release quality, more transparency, quicker bug finding and fixing, simplicity and TDD frame-of-mind.
Disadvantages for OpenMRS in using Continuous Integration System
- Additional Load on Servers
- Not every developer is motivated to write unit tests ;-)
Features that OpenMRS needs (Not exhaustive)
- Easy to monitor tests and easy to understand dashboard
- Support for SVN and Ant
- Dependency integration
- Email/RSS/IRC notifications when a build fails or bad code is committed
- Warning flags when a committed code doesn’t follow coding rules (naming, newline format,etc.)
- Allow code committers to modify build and test parameters from the GUI
- Optimal Performance
- Price & Open-source development
Comparison of Continuous Integration Systems:
| CruiseControl | Continuum | TeamCity | Bamboo | Hudson | |
| Monitoring UI | Dashboard introduced since v2.7 is not intuitive | Dashboard only shows tests | Advanced UI & dashboard. | Advanced UI, Detailed reporting out-of-the-box, Intuitive | Simple Dashboard, Plugins enhance reporting, Intuitive, somewhat detailed |
| SCM Support | All Support SVN | ||||
| Dependency Integration | Scripts need to be written for each new dependency. Tracking different versions of dependency jars is very complex | Easy for Maven2.0+ projects, but not so easy for other types of projects | Dependency can be managed easily. Advanced UI for dependency management | Dependency management is easy and intuitive. Different versions of same library not automated. Creating test/build plans allows dependency of different versions | Dependency management is easy to configure. file fingerprinting simplifies identification of different versions. Automatically can detect and build project dependencies |
| Email/RSS/IRC | Emails. Plugins - RSS, blog, IM with Jabber | Email, IRC, IM with Jabber, MSN | Email, Jabber, RSS, external HTML widget | Emails, RSS, IM Notification using Jabber or OpenFire | Plugin – Emails, RSS, IRC, Jabber, Google Calender, Twitter |
| Code Quality and Patterns | Not very easy to define | Could not find a way | Can be defined with plugin for IntelliJ IDEA | Managed through test plans. Manual test plans have to be created | Plugin provides UI. Test plans can be created manually out-of-the-box |
| Security and User Management | Easy to configure with different roles | Roles can be easily defined | Roles can be easily defined | Simplistic UI for user build plan management. Easy integration with JIRA | Easy to configure roles for users |
| Performance | Fast | Fast | Somewhat slower in comparison, but includes a lot of features | Fast in build and integration. Slightly slower in reports. Includes lots of features that may not be used. Distributed builds with slaves speeds up performance | Comparatively lightweight out-of-the-box, but requires plugins. Distributed builds with slaves speeds up performance |
| Pricing | Free & OpenSource. Paid version called Cruise available. | Free & OpenSource | Professional version is free, but enterprise is paid | Free for opensource projects | Free & OpenSource |
7 comments:
Cruise, the commercial CI tool, is a very different beast from CruiseControl and has significantly more features.
It is by no means simply a paid version of the old Cruise.
And yes, as you pointed out, it's free for Open Source projects.
Nicely done, Sunny. Thanks for this due diligence. Looking at the table, there doesn't seem to be one that stands out from the crowd. I'm still leaning toward Bamboo, since we are migrating toward other Atlassian products (like Confluence, Crowd, and Crucible). Are there any other aspects of these tools that would set them apart or outweigh the benefit of using Bamboo if we're adopting other Atlassian tools?
This is very helpful. Strong work!
Burke, from the testing I realized that Bamboo and Hudson are pretty close to what we would want to implement. Bamboo is better out-of-the-box, while Hudson is really about extensibility with plugins... If we don't have the inclination to customize with our own plugins, it seems like Bamboo should do us good.
Hudson is great, but if you have alot of unit tests then it is a big memory hog due to the way it parses the junit xml output to provide reports. I can't use it currently because my build machine runs out of memory. Hopefully, that will be fixed.
From my experience, Atlassian products (specifically JIRA and Confluence) tend to be way slow from developing-world connections. For example, TRAC on openmrs.org works waaaay better for me than JIRA on the pih server.... though maybe the pih server is just slower.
I am with Atlassian and think that Bamboo would be a great fit for the OpenMRS project providing both powerful out-of-the-box features as well as a great plugin architecture. Let me know if you want to discuss further..
Post a Comment