Acceptance Testing and Continuous Integration

Company X currently runs a very agile development team, developing a system in java, very TDD, continuous integration (cruise) ... you get the picture.

They also have a dedicated test team, very impressive. Of that team, one group called "QA" who are essentially responsible for black-box testing and are very knowledgeable in the business process - they are very close to the user-base. In fact they report directly to the business, in contrast to the developers who report to Technology.

Another test group is responsible for white-box testing. These guys are trained developers but their sole purpose in life (thats going a bit far) is to break the system from the perspective of getting under the bonnet and tinkering, pouring oil in the radiator and mistakenly jamming a potato in the exhaust pipe. Then sit back, wait, see what blows up. Then run up to the developers and say "Come take a look at this!!!".

I hope you appreciate me painting the picture here, it will mean something later.

The main message behind this post is a nagging problem with our continuous integration, running on cruise, I will refer to it as Cruise for simplicity although this problem should occur on other CI systems. As I arrived on the project when it was already in full swing (I estimate 18 mths had already elapsed), a lot of the architecture, toolset, build scripts, tests are already well established - perhaps looking a bit shabby now, but were probably a great idea 2 yrs ago. One of the most annoying problems with Cruise is the length of time to build - ages it seems and nearing unacceptable. The system architecture is split as Common, Management, Capture and is built as such. In total I estimate 70mins to do a full build of all 3 subsytems.

The question I pose here is should the black-box and white-box tests be run on every Cruise build cycle? My gut says, no. I think it is always a case-by-case basis depending on the running time and urgency. For this site I describe I think they should run at least at the end of the day.

I would be interested in hearing comments.