Search In A Box With Docker, Elastic Search, Spring Boot, and Selenium

Overview

This tutorial will show you how to build a search service in a box. It'll have a graphical front end to search for some funny jokes (well, they made me laugh). Notably, you'll be able to build, test, and package it into a container with a push of a button.

We'll learn about:

As always, you can checkout the code on Github.

Prerequisites

You'll need to have installed:

  • Maven,
  • Java (version 7 or later),
  • Docker (or, if you're on a Mac: Boot2Docker).

Preparation

Lets create a pom.xml file for our project:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>search-in-a-box</groupId>
    <artifactId>search-in-a-box</artifactId>
    <version>1.0.0-SNAPSHOT</version>

</project>

Create The Basic Search Query Page

We're going to build this using Acceptance Test Driven Development (ATDD). As this will be a web application, we're going to use Selenide, a simple DSL that works with Selenium WebDriver to test web apps. Add these dependencies to get started:

<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.12</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.codeborne</groupId>
    <artifactId>selenide</artifactId>
    <version>2.16</version>
    <scope>test</scope>
</dependency>

We'll create a test that verifies the homepage shows a search box:

package searchinabox;

import org.junit.Test;
import org.openqa.selenium.By;

import static com.codeborne.selenide.Condition.exist;
import static com.codeborne.selenide.Selenide.$;
import static com.codeborne.selenide.Selenide.open;

public class AppIT {

    @Test
    public void homepageShowsSearchBox() throws Exception {
        open("/");
        $(By.cssSelector("input[name='query']")).should(exist);
    }
}

Run this test and you'll find it is red. We need to get our search page running!

We're going to create a Spring Boot application. Add these lines to your pom.xml:

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.2.1.RELEASE</version>
</parent>

This will sort out our dependency versions for us. Next, we'll be using Spring Boot with Thyme-leaf for templates. Add the Spring Boot dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-thymeleaf</artifactId>
</dependency>

Next, create an application class, this will

  1. Expose the home page,
  2. Boot up the application.
@SpringBootApplication
@Controller
public class App {

    @RequestMapping("/")
    public String home() {
        return "home";
    }

    public static void main(String[] args) {
        SpringApplication.run(App.class, args);
    }
}

We need some HTML for the home page, so make src/main/resources/templates/home.html:

<!DOCTYPE HTML>
<html xmlns:th="http://www.thymeleaf.org">
<head>
    <title>Search In A Box</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<form method="post">
    <input name="query"/>
    <input type="submit"/>
</form>
</body>
</html>

When you run the application, you can also run the test and you'll see that it is green. You can also look at the search form manually at http://localhost:8080.

Searching For Jokes

Sticking with ATDD, create the test first:

@Test
public void searchingForBearsFindsTheNorthPollJoke() throws Exception {

    open("/");
    $(By.name("query")).sendKeys("bears");
    $(By.cssSelector("input[type='submit']")).submit();

    $$(By.tagName("td")).find(exactText("The North Poll!")).should(exist);
}

We need to speak to a client, so create the following Spring Java configuration file:

@Configuration
public class Config {

    @Bean(destroyMethod = "close")
    public Node node() {
        return NodeBuilder.nodeBuilder().node();
    }

    @Bean
    public Client client() {
        return node().client();
    }
}

Elastic Search creates directory called data in the root of our project. We want this to be cleaned so our test always start with a fresh instance. set-up the Maven Clean Plugin to do so:

<plugin>
    <artifactId>maven-clean-plugin</artifactId>
    <configuration>
        <filesets>
            <fileset>
                <directory>data</directory>
            </fileset>
        </filesets>
    </configuration>
</plugin>

We want to make a service for indexing and searching jokes. There's will be quite a bit going on here, so let's break it down:

  1. On start-up create an index called "jokes" to store our jokes in,
  2. Then store two jokes, each with unique IDs,
  3. Finally, provide a method to search for jokes.
@Service
public class JokeSearchService {

    @Autowired
    private Client client;

    @PostConstruct
    public void indexJokes() throws Exception {
        // create an index name "jokes" to store the jokes in
        try {
            client.admin().indices().prepareCreate("jokes").get();
        } catch (IndexAlreadyExistsException ignored) {
        }

        storeJoke(1, "Why are teddy bears never hungry? ", "They are always stuffed!");
        storeJoke(2, "Where do polar bears vote? ", "The North Poll!");
    }

    private void storeJoke(int id, String question, String answer) throws IOException {
        // index a document ID  of type "joke" in the "jokes" index
        client.prepareIndex("jokes", "joke", String.valueOf(id))
                .setSource(
                        XContentFactory.jsonBuilder()
                                .startObject()
                                .field("question", question)
                                .field("answer", answer)
                                .endObject()
                )
                .get();
    }

    public SearchHit[] search(String query) {
        return client.prepareSearch("jokes")
                .setTypes("joke")
                .setQuery(QueryBuilders.multiMatchQuery(query, "question", "answer"))
                .get().getHits().getHits();
    }
}

Update the App and add the following lines so the we have can accept POST requests and display results:

@Autowired
private JokeSearchService jokeSearchService;

@RequestMapping(value = "/", method = RequestMethod.POST)
public String search(@RequestParam("query") String query, Model model) {
    model.addAttribute("results", jokeSearchService.search(query));
    return "results";
}

Next, we need a results page so create src/main/resources/templates/results.html:

<!DOCTYPE HTML>
<html xmlns:th="http://www.thymeleaf.org">
<head>
    <title>Search In A Box - Results</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
</head>
<body>
<table>
    <tr th:each="result : ${results}">
        <td th:text="${result.source.question}"/>
        <td th:text="${result.source.answer}"/>
    </tr>
</table>
</body>
</html>

Restart the application, run the test, and you'll see it's now green.

Putting The Application Into A Container

We want to put the application into a Docker container. Spring Boot can create a standalone jar to put it into a container, so add this plugin to the pom.xml:

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
</plugin>

You can see this at work by creating a package:

$ mvn package
...
spring-boot-maven-plugin:1.2.1.RELEASE:repackage 

This replaces the original JAR, with a standalone version.

Next, we'll use a plugin to build the container:

<plugin>
    <groupId>com.alexecollins.docker</groupId>
    <artifactId>docker-maven-plugin</artifactId>
    <version>2.3.1</version>
    <dependencies>
        <!-- only needed if you are using Boot2Docker -->
        <dependency>
            <groupId>com.alexecollins.docker</groupId>
            <artifactId>docker-java-orchestration-plugin-boot2docker</artifactId>
            <version>2.3.1</version>
        </dependency>
    </dependencies>
</plugin>

The plugin needs some files to create the app. Each directory in src/main/docker in treated as a container, so in src/main/docker/searchinabox create a Dockerfile that:

  1. Adds the JAR,
  2. Adds a configuration file,
  3. Exposes the ports - both our app on 8080, and Elastic Search on 9200 and 9300 (so it can join a cluster),
  4. Sets the start-up command.
FROM dockerfile/java:oracle-java7

EXPOSE 8080
EXPOSE 9200
EXPOSE 9300

ADD ${project.build.finalName}.jar .

CMD java -jar /${project.build.finalName}.jar

We need a conf.yml file in the same directory, this:

  1. Indicates that we want to add the JAR as part of the Docker image,
  2. States the ports it should expose on the host,
  3. A health check URL we can use to smoke test the container,
  4. Finally, a tag for the container so we can easily identify it:
packaging:
  add:
    - target/${project.build.finalName}.jar
ports:
  - 8080
  - 9200
  - 9300
healthChecks:
  pings:
    - url: http://localhost:9200/
    - url: http://localhost:8080/
tag:
    searchinabox/searchinabox:${project.version}

Package this and start-up the container:

mvn docker:start

You should see this:

[INFO] Starting searchinabox
...
[INFO] BUILD SUCCESS

The container will be listed by the docker command

$ docker ps
CONTAINER ID        IMAGE                             COMMAND                CREATED             STATUS              PORTS                                                                    NAMES
f673731a9489        searchinabox/searchinabox:1.0.0-SNAPSHOT   "/bin/sh -c 'java  -   6 seconds ago       Up 4 seconds        0.0.0.0:8080->8080/tcp, 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   search-in-a-box_app  

We can check we can access Elastic Search by opening the http://localhost:9200 URL, and our application by opening http://localhost:8080.

Tips

Packing containers can go wrong. I find it helpful to print/tail the logs of the last started container with this command:

docker logs -f $(docker ps -qa|head -n1)

We often want to start the container up with a shell to debug it, for example I often get the start command wrong, so here's what I'd do:

docker run -t -i  searchinabox/searchinabox:1.0.0-SNAPSHOT bash

Continuous Integration

To complete the picture we want Maven to start the containers and run our acceptance tests. We'll use the Maven Failsafe Plugin to run the tests. Add this plugin as follows:

<plugin>
    <artifactId>maven-failsafe-plugin</artifactId>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>

We need to tell docker-maven-plugin to start and stop the container, so add these lines to it:

<executions>
    <execution>
        <goals>
            <goal>clean</goal>
            <goal>start</goal>
            <goal>stop</goal>
        </goals>
    </execution>
</executions>

Finally, test it.

mvn clean verify
...
[info] BUILD SUCCESSFUL

Conclusion

We seen how to create a search service in a box with Elastic Search, Spring Boot and Docker. We've seen how to create a build using Docker Maven Plugin. I hope you enjoyed this. Here are some exercises for you:

  • The UI is pretty drab, how about an attractive Bootstrap front end?
  • We've let Elastic Code leak all over the place. Perhaps we should refactor it behind an abstraction?
  • We might want to re-index out data. Use Spring to schedule the re-indexing.
  • We're not indexing a lot of things? Should we some new indexers for other items?

Related

  1. Geb, Selenium, Cucumber & Maven Tutorial
  2. Visual Testing With Selenium WebDriver
  3. Migrating to CircleCI
  4. Tutorial: Integration Testing with Selenium - Part 1
  5. Tutorial: Integration Testing with Selenium - Part 2