Improvements in java based test framework to reduce execution time

Question

Let me give a bit of background on the problem.

We have a java application which has MQ integration and an Integration Test pack which is also written in java.

The test pack runs all scenarios (positive and negative) by running the test cases (methods) and changing the parameters depending on the scenario. The way it does the assertions is to put a message on the input queues of the application and expects a message in the output queues of the application. The message body is checked because it differs depending on the test scenario.

There is only 1 input queue and 1 output queue in the main application. The number of test cases (methods) are about 300 and due to running all the known scenarios, the number of test cases actually executed are about 2500. Due to this, the test pack takes around 6 hours to execute fully.

Since there is only 1 queue on which the test pack asserts the output message, we have to clear the queue each time before we run the next test case. Otherwise it may lead to mixing of the output messages. For e.g. method call#2 might end up consuming output message for method call#1. Hence we can't run the test cases in parrallel too.

I want to improve this process and would be highly grateful for any suggestions to optimize the test pack. Right now I am thinking to separate out the input and output queues of the main application as per business function and thus assert on multiple queues in parallel. But this approach means I have to change the original architecture so that my Test cases run faster and this is not easy since there are already clients using this application and interacting using the 2 queues and they will have to change their applications too.

Apologies for the long post but I would love to hear inputs of so many creative people in SO community.

Thanks!

Answer 1

Filters on message queues exist. You can throw them into the same message queue, and then assign a listener for messages based on a specific criteria. This criteria can be anything, to thread id to an arbitrary message name.

The end result is that messages not having that specific criteria don't get thrown away, they merely wait to be processed by handlers without that criteria, effectively allowing you to perform checks on specific aspects pertaining to specific messages without removing messages other than the ones you wish to check.

Answer 2

One of the things that used to drive me nuts was when eclipse would run my tests in parallel when I hadn't expected it. This vexed me because I'm an old school tester who still thinks writing to the console is useful. Two tests would write to the console and sometimes everything was fine. Sometimes one's text would be mixed halfway into the others.

Then I realized how simple the solution was. Just let each test build it's own string and output that one string at one time. Once I did that I could let the tests run in parallel just fine.

If you don't have sequential dependencies, shared memory, or multiple outputs from each job to keep in order I really don't see why you can't let them run in parallel.

Answer 3

Since there is only 1 queue on which the test pack asserts the output message, we have to clear the queue each time before we run the next test case. Otherwise it may lead to mixing of the output messages. For e.g. method call#2 might end up consuming output message for method call#1. Hence we can't run the test cases in parallel too.

Assign unique ids to the messages and listen for message with expected id in output queue with an upper time limit. This should get you testing parallel executions.

You can wrap this publish/listen logic in a generic request/response style implementation. From there on, use something like RxJava to compose scenarios of multiple such calls.

Answer 4

So the solution we have come up with is this. We will Dockerize the test pack so that we have multiple containers running multiple servers and multiple message queues. We will utilize the maven profiles to deploy the same war but run different test packs in the separate containers. The test pack would be modularized in such a way that the tests within each module do no overlap each other in terms of the queues/messages. Thanks to everyone for their time and their valuable comments.