Integration Tests with External Dependencies in Go

Integration tests are tests which use or depend on some external software like a database, cache layer, or any service. While unit tests validate smaller pieces of software logic, it is always good to also run integration tests in an as-close-as-possible to real environment, and preferably also have them automated. This post will go over a way to easily include and automate the use of external dependencies in your integration tests and CI/CD environment.

We will run our dependencies in containers, and the tests will interact with these containers when they run. An extremely helpful library to help us do this is testcontainers-go. Testcontainers for Go makes it simple to create and clean up container-based dependencies for automated integration tests. The clean, easy-to-use API enables developers to programmatically define containers that should be run as part of a test and clean up those resources when the test is done.

Integration Test Declaration

It is important to distinguish between unit and integration tests so that they can be run independently of each other. Unlike integration, unit tests should not have external dependencies and run quickly.

An effective way to declare integration tests is to name their function declarations ending in Integration and skip them if the short flag is passed when running.

func TestSomeNameHereIntegration(t *testing.T) {
	if testing.Short() {
		t.Skip()
	}
    ...
}

Example

The following is an example of starting up a DynamoDB container in Docker using testcontainers-go and running a test against this DynamoDB instance. Please refer to the documentation for help with setting up your local container environment (Docker, colima, or Podman) to work with testcontainers-go.

Setup Container

The SetupDynamoDBContainer function can be called during the setup phase of the integration tests. It will start an amazon/dynamodb-local:1.20.0 container that tests can reach at the specified port.

type DynamoDBContainer struct {
	testcontainers.Container
}

func SetupDynamoDBContainer(ctx context.Context, port string) (*DynamoDBContainer, error) {
	exposedPort := fmt.Sprintf("%s:8000/tcp", port)
	req := testcontainers.ContainerRequest{
		Image:        "amazon/dynamodb-local:1.20.0",
		Cmd:          []string{"-jar", "DynamoDBLocal.jar", "-inMemory"},
		ExposedPorts: []string{exposedPort},
		// Block until DynamoDB has started
		WaitingFor: wait.ForExposedPort(),
	}
	ddbContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: req,
		Started:          true,
	})
	if err != nil {
		return nil, err
	}
	return &DynamoDBContainer{Container: ddbContainer}, nil
}

Test

The tests use TestMain to do some setup before running the actual tests. This is where we call our SetupDynamoDBContainer function and perform any additional setup steps.

var dynamoDBRepo *DynamoDBRepo

func TestMain(m *testing.M) {
	flag.Parse()
	if !testing.Short() {
		port := "8000"
		ctx := context.Background()
		ddbContainer, err := test.SetupDynamoDBContainer(ctx, port)
		if err != nil {
			panic(err)
		}
		// Stop the container when tests exit
		defer func() {
			if err := ddbContainer.Terminate(ctx); err != nil {
				fmt.Println(err)
			}
		}()
        // Get created container's host. For some CI/CD environemntes it will not be localhost.
		host, err := ddbContainer.Host(ctx)
		if err != nil {
			panic(err)
		}
        // Initialize the use of the DynamoDB instance
		dynamoDBRepo = NewDynamoDBRepo(true, WithHost(host), WithPort(port))
	}
	os.Exit(m.Run())
}

func TestDynamoDBRepoIntegration(t *testing.T) {
	if testing.Short() {
		t.Skip()
	}
    ... test code with dynamoDBRepo  ...
}

Makefile

An effective way to run our tests locally or in a CI/CD environment is with a Makefile. Here are some example Makefile targets that can run our unit, integration, or all tests. The PKG_LIST variable is evaluated earlier in the Makefile and lists all go packages that should be tested, excluding some packages containing generated code. You can remove it to test against all packages in your project.

.PHONY: test
test: ## Run all tests
	@echo "Running all unit and integration tests"
	@go test -v ${PKG_LIST}

test_unit: ## Run unit tests
	@echo "Running all unit tests"
	@go test -short -v ${PKG_LIST}

test_integration: ## Run integration tests
	@echo "Running all integration tests"
	@go test -run=".Integration" -v ${PKG_LIST}

GitHub Actions Integration

Running the tests in GitHub Actions CI/CD environment requires running the code in a Docker container and providing the container access to a Docker socket so that testcontainers-go can use it to spin up new dependency containers. Example of a test step running lint, unit and integration tests:

  lint-and-test:
    ...
    container:
      image: golangci/golangci-lint:v1.52
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
    steps:
      - uses: actions/checkout@v2
        if: startsWith(github.ref, 'refs/tags/') == false

      - name: Lint and Test
        id: lint-and-test
        run: |
          go mod download all
          make clean && make generate
          make lint
          make test

Stefan Gloutnikov