Writing custom DSLs

Test execution steps logging with AspectJ

When we talk about custom reports, the most common question is what should be inside? What information could be useful for team or management? I saw different reports' implementations with a good level of details, but often people forget to put one important thing - execution steps.

Let's imaging some hand-written test case. It contains id / name, description, steps and some expected result. If we open any good bug report, we would see the same detailed description with steps, actual / expected results, etc. Such kind of details level has a positive impact on issue's root cause analysis, as well as on its fixing time.

When we develop tests, such frameworks as TestNG or JUnit put test name and assertion results into final report. But what's about test execution steps? When we look at report's body, who can say what was exactly tested in particular test? We can only assume real scenario. That's why I've started looking at a way of implementing such functionality. Of course someone can point that Thucydides already has such functionality, but as I've mentioned in my previous article, sometimes it's very expensive to switch to another framework only because it has a good reporting engine.

So how we can log our test execution steps? It could be done via proxy or AOP. In this article I'll talk about test execution steps logging with AspectJ.

To use AspectJ functionality, you need to include the following dependency and plugin:

Let's assume we're using PageObject pattern. In such case test steps would be PageObject's methods. So the main goal is to intercept PageObject's methods calls and save their names in some container to get further access, when we need to print steps into report. That can be easily done with aspects. But as you know, PageObjects may contain some useless for test results report methods (getters, helpers, etc.). So let's assume we want to filter methods to print only real test logic without any redundant calls. This can be achieved via custom annotation.

We can use this annotation for marking methods we want to publish into report.

Finally, we're ready to create a simple aspect that will intercept all the methods calls annotated with Publish annotation inside particular package:

After finishing test execution, you can easily print intercepted steps, that were saved inside ThreadLocal container. Let's see how could it look like in test results report:

You can find a source code example on GitHub.

Custom reporting engine with ReportNG

I've seen a lot of threads with the same question: how to create a good, reliable test results report, which would be useful for automation team, as well as for developers, manual QAs, management? This question is very popular nowadays, as common JUnit / TestNG reports are pretty ugly, and it doesn't make sense to extend them, as it would be definitely easier to create a new one from scratch. But what if you don't have appropriate skills to do that? In such case people start looking at existing solutions. There's a good one, which is a part of Thusydides framework. It's pretty nice and straightforward, but what if you already have your own framework? It's nonsense moving to another framework only because it contains a good report. Well, it would be also very expensive. In such case we need some external module, which is also compatible with TestNG or JUnit. In this article I'll talk about ReportNG, which is a simple HTML reporting plugin for TestNG framework.

Some of you may ask: why do we need to talk about custom reporting in a ReportNG context? Just add it into project and use! Well, it's a good point, but what if we want to add some specific info into our report? What if we want to change its structure? In such case it wouldn't be enough to add it into project. We would also need to understand its source code to push necessary changes. In this article I'll help you to understand its common components.

As I've mentioned above, ReportNG is a TestNG plugin, so to understand its structure, first you need to figure out with TestNG. To add some visual customization or inject your own custom data, you also have to read about Velocity templates syntax. And of course by default you should be familiar with Java, as it's a ReportNG host language.

To get started, first you need to pull ReportNG sources. Just create your fork and copy appropriate URL. I prefer using IntelliJ IDEA, sorry to Eclipse fans. IntelliJ allows you to clone sources directly from GitHub by provided URL:

After finishing cloning, IDE prompts you to create a new project. We will skip this step. Let's look at project's structure:

Generally, you would only need HTMLReporter / AbstractReporter and ReportNGUtils classes from main package. First one is intended for report's context preparation / generation. It sets different custom properties into Velocity context. It also extends AbstractReporter class, where you have to pay attention to createContext() method. It's very important for further customization.

All the properties we set via VelocityContext are acceptable within velocity templates, that represent report's body. It means that if we override this method, we can pass any custom properties / data into velocity templates. Next question is: how to override it? It can be easily done by creating our own TestNG listener, that extends HTMLReporter class. Such kind of extension allows us to move though inheritance chain and reach AbstractReporter class, which contains needed method with protected modifier.

As you see, I set custom properties and data into VelocityContext from overrriden createContext() method inside BaseHTMLReporter listener.

To force TestNG using our custom listener, just add it via maven-surefire-plugin.

Let's move on. ReportNGUtils provides us a set of useful helpers, that can be also accessed within velocity templates. As you saw from original and overridden createContext() methods, both use UTILS_KEY to set ReportNGUtils object into context. BaseHTMLReporter implementation uses extended version of ReportNGUtils class with overridden getTestOutput method to inject screenshot link into test results output.

Note that this is only a part of code. It's up to you to take care about creating screenshot and setting appropriate link into results context, as it's out of this guide's scope.

Now let's take a look at ReportNG resources. First, you should pay attention to reportng.properties. It contains all the messages displayed in test results report. You'll find a way of how to access them within velocity templates: $messages.getString(key); - where messages is a ResourceBundle object, that is set into VelocityContext inside AbstractReporter class.

Our last part is related to velocity templates. I won't pay attention to details, as it's up to you how you're going to customize your report's body. But let's take a quick look at their structure.

All the listed in html folder vm templates (their html representations) will appear in your results folder after tests execution is finished. So you can easily compare final result's body with its template, if you're familiar with HTML / JS / CSS. So to apply your custom styles or structure, you'll need to modify templates, rebuild plugin and update dependency in pom.xml.

As I've mentioned before, you can access all your custom properties or objects within velocity templates, if you previously set them via overridden createContext() method. For example, if I set testCasesInfo object, I can use it the following way:

By default IntelliJ doesn't highlight vm templates' syntax. But it can be easily configured. Just follow this guide.

Cloned project already contains Ant build.xml. So you can easily create your custom ReportNG version and install it into local maven repository. Note that ReportNG doesn't use latest TestNG and Velocity versions. So you should take care about updating them manually. Besides adding your updated ReportNG dependency, you should also add Velocity and Guice dependencies into pom.xml.

And here are some examples of how could ReportNG be customized:

You can find some sample sources on GitHub.