RESTful SikuliX

As you may know, new version of SikuliX should be released soon. I had a chance to participate in a development process. So in this article I'd like to share some notes about new feature - remote SikuliX client / server, implemented via REST.

The idea of using SikuliX in a remote context appeared due to automation architecture chosen for the project I'm currently working on. As we scale our tests between number of environments, there was a necessity in resolving some complicated tasks, that couldn't be done with common libraries like Selenium, on remote VMs. SikuliX had everything we needed, except remote platform we could use in a context of existing architecture.

After number of efforts, I've finally found approach that would fit all project needs. There was built a RESTful client-server platform, that included latest SikuliX API and several useful utility classes for interacting with remote file system. After some period of time I've realized that this approach could be useful for others, so now I'm happy to anounce new experimental features, that are already pushed into SikuliX repository - RemoteServer and RESTClient.

These modules are not fully tested yet, but you can give it a try and report all issues you'll find or push appropriate pull requests. There're some tests written for Windows / Linux OS, that could be a start point for you.

And now let's take a look at provided functionality.

RemoteServer module is based on Grizzly Http Sever and contains the following endpoints:

• http://ip:port/sikuli/cmd/execute - uses apache commons exec library for running command line on a remote VM.
• http://ip:port/sikuli/file/upload - uploads a list of provided files to the given remote path.
• http://ip:port/sikuli/file/download - downloads a single file from remote path (multiple file download feature is not implemented yet).
• http://ip:port/sikuli/file/delete - removes file or directory by a given remote path (quick note: file system operations were implemented using apache commons io library).
• http://ip:port/sikuli/file/exists - checks a list of inputs, if appropriate files or directories exist.
• http://ip:port/sikuli/file/createFolder - creates directory by a given remote path.
• http://ip:port/sikuli/file/cleanFolder- removes content of a given remote directory.
• http://ip:port/sikuli/file/copyFolder- copies 1 folder's content to another.
• http://ip:port/sikuli/image/click - uses SikuliX API for clicking provided image with a given wait timeout on a remote VM.
• http://ip:port/sikuli/image/setText - uses SikuliX API to print some text into appropriate control with a given wait timeout on a remote VM.
• http://ip:port/sikuli/image/exists - uses SikuliX API for checking if image is present on a screen or not on a remote VM.
• http://ip:port/sikuli/image/dragAndDrop - uses SikuliX API for dragging and dropping objects on remote VM.

If you take a look at code, you'll find it pretty straightforward, e.g. here's a delete processor:

    @POST
    @Path("/delete")
    public Response delete(@QueryParam("path") final String path) {
        return Response.status(FileUtility.delete(path) ?  
               Response.Status.OK : Response.Status.NOT_FOUND) 
               .build();
    }

As you may saw above, common SikuliX APIs provides us a way of setting wait timeout. This functionality is implemented via observers mechanism, that uses onAppear event to process requested actions:

    private RemoteDesktop onAppear(final Pattern element,  
            final SikuliAction action, final String text) {
        desktop.onAppear(element, new ObserverCallBack() {
            public void appeared(ObserveEvent e) {
                switch (action) {
                    case CLICK:
                        e.getMatch().click();
                        break;
                    case TYPE:
                        e.getMatch().click();
                        e.getMatch().type(text);
                        break;
                }

                desktop.stopObserver();
            }
        });

        return this;
    }

Before moving to client's part, let's take a look at one more interesting block, related to remote command line execution. As I've mentioned above, we use apache commons exec library for this purpose. And I must say, it provides a fantastic feature, that I struggled with for a while - delayed exit from main thread by timeout. You may know that common java command line executor will stuck forever, if started process waits for user input or it's just a simple server application. Let's look what commons exec can provide for this particular case:

    public static int executeCommandLine(final Command command) {
        if (command.getProcess() == null) {
            CMD_LOGGER.severe("There's nothing to execute.");
            return -1;
        }

        CMD_LOGGER.info("Processing the following command: " + 
                command.getProcess() + (command.getArgs() != null ? 
                " " + command.getArgs() : ""));

        final long timeout = (command.getTimeout() > 0 ? 
                command.getTimeout() : 0) * 1000;
        final CommandLine commandLine = new CommandLine( 
                separatorsToSystem(quoteArgument( 
                    command.getProcess())));

        if (command.getArgs() != null) {
            for (String arg : command.getArgs()) {
                commandLine.addArgument(quoteArgument(arg));
            }
        }

        final ExecutionResultsHandler resultHandler = 
                new ExecutionResultsHandler();
        final PumpStreamHandler streamHandler = 
                new PumpStreamHandler( 
                    new ExecutionLogger(CMD_LOGGER, Level.INFO), 
                    new ExecutionLogger(CMD_LOGGER, Level.SEVERE));
        final DefaultExecutor executor = new DefaultExecutor();

        executor.setStreamHandler(streamHandler);
        executor.setProcessDestroyer( 
                new ShutdownHookProcessDestroyer());

        try {
            executor.execute(commandLine, resultHandler);
            resultHandler.waitFor(timeout);
        } catch (InterruptedException | IOException e) {
            CMD_LOGGER.severe("Command execution failed: " 
                + e.getMessage());
            return -1;
        }

        return resultHandler.getExitValue();
    }

ExecutionResultsHandler will let process be released by a given timeout.

That's almost pretty much related to server side. To build remote server, use the following command:

mvn clean install

It will create a jar with all necessary dependencies in your target folder.

To start server, use the following command:

java -jar sikulixremoteserver-1.1.0-jar-with-dependencies.jar port

Port is optional. You can skip it, if you want to use default one - 4041.

Now it's time to look at client side, that is located inside RESTClient module.

There's nothing specific. Code is pretty straightforward, as it only takes care about sending necessary objects to listed above endpoints. Client implements SikuliX interface. Besides that, you may find some other interfaces used as an input methods' arguments. We decided to leave them in project to allow users overriding client's methods and common sending containers. It was done for number of reasons. One of them is incompatible Jersey 1.x and 2.x versions. If your project uses Jersey 1.x dependencies, you won't be able to use new SikuliX REST client, as it's based on Jersey 2.x. In such case you will need to implement your own client using SikuliX remote interfaces.

As a simple example of REST call implementation, let's take a look at multiple files upload API:

    public void uploadFile(final List filesPath, 
            final String saveToPath) {
        final MultiPart multiPart = 
            new MultiPart(MediaType.MULTIPART_FORM_DATA_TYPE);

        for (String path : filesPath) {
            multiPart.bodyPart(new FileDataBodyPart("file", 
                new File(separatorsToSystem(path)), 
                MediaType.APPLICATION_OCTET_STREAM_TYPE));
        }

        final Response response = service.path("file")
                .path("upload")
                .queryParam("saveTo", separatorsToSystem(saveToPath))
                .request(MediaType.APPLICATION_JSON_TYPE)
                .post(Entity.entity(multiPart, 
                    multiPart.getMediaType()));

        if (response.getStatus() == 
                Response.Status.OK.getStatusCode()) {
            CLIENT_LOGGER.info("File(-s) " + filesPath + 
                " has been saved to " + 
                separatorsToSystem(saveToPath) + " on " + ip);
        } else {
            CLIENT_LOGGER.severe("Unable to save file(-s) " + 
                filesPath + " to " + separatorsToSystem(saveToPath) + 
                " on " + ip);
        }

        response.close();
    }

As you see, we can pass a list of files' paths for uploading. It's pretty useful when we need to copy expected images we want to allow SikuliX interact with on a remote VM.

Provided tests were created for Windows OS and haven't been tested on Unix or Mac yet. If you're going to give it a try, you'll need to install and start remote server first. By default all the tests are disabled to avoid build failures, as such verifications are very platform and configuration specific. To enable them, just change the following option in a pom.xml:

skipTests=false

To choose classes to be included into test run, you need to modify suites.xml located in resources folder. Actually, you should carefully explore resources before execution. Batches' extensions should be renamed to .bat. And you may also need to provide your own images, as they are very OS specific.

When you finish with resources, you'll need to update BaseTest configuration:

• SIKULIX_SERVER_IP field must refer your newly raised remote server IP address.
• WAIT_TIMEOUT will tell SikuliX to wait until expected image is appeared on a screen.
• SIMILARITY level will be used while images comparison.

As we've already mentioned file upload scenario, let's take a look at appropriate test:

    @Test
    public void uploadFilesToServer() {
        getClient().uploadFile(Arrays.asList( 
                BATCH_RUNNER_SCRIPT.getPath(), 
                BATCH_SAMPLE_SCRIPT.getPath()), 
                SERVER_PATH.getPath());

        assertTrue(getClient().exists(Arrays.asList( 
                SERVER_PATH.getPath() + "/" + 
                    BATCH_RUNNER_SCRIPT.getName(), 
                SERVER_PATH.getPath() + "/" + 
                    BATCH_SAMPLE_SCRIPT.getName())));
    }

To perform common SikuliX actions, you can use the following example:

    @Test
    public void callCommandLineFromStartMenu() {
        getClient().click(new ImageBox( 
                getResource(RESOURCE_BUTTON_START_IMAGE).getPath(), 
                SIMILARITY), WAIT_TIMEOUT);

        getClient().setText(new ImageBox( 
                getResource(RESOURCE_INPUT_FIND_FILES_IMAGE)
                    .getPath(), SIMILARITY), 
                "cmd" + Key.ENTER, WAIT_TIMEOUT);

        assertTrue(getClient().exists(new ImageBox( 
                getResource(RESOURCE_INPUT_CMD_IMAGE).getPath(), 
                SIMILARITY), WAIT_TIMEOUT));
    }

You can find more examples in official SikuliX2 GitHub repository.

Take your time and let me or Raimund Hocke know, if you find these new features useful and what can be done better.