Increased security through Kubernetes: practical tips for securing container networks

Security plays an especially important role when using containers. The open-source system Kubernetes has become the standard for automating, deploying, scaling and managing container applications and therefore requires particular attention to security.

In this article we've put together a short guide on how you can operate Kubernetes and which aspects need to be considered.

Security scanners: what types actually exist?

Security scanner is umbrella term for programs that carry out checks in the security domain. Generally divided into three categories.

Vulnerability scanner

Vulnerability scanners are perhaps the best-known type of security scanner. They check for known vulnerabilities and, in the best case, provide further information about the vulnerability and whether a newer version with a fix already exists. These scanners can typically scan container images or also running operating systems. Vulnerability scanners classically use operating system package information as their data source, but more and more scanners can also recognise and use programming-language-specific installation systems such as npm or Go libraries.

Secret Scanner

Secret scanners are meant to prevent API keys or passwords being checked into a repository in plain text. This is mostly used in the area of infrastructure as code (IaC) data, or also with classic development data. This use case in particular makes clear that this kind of scan should be used as early as possible in the development process, to keep the effort of removing secrets from a repo as low as possible.

Best-practice scanner

Here the purpose is clearly focused on IAC files. The aim is to detect as early as possible whether best practices are being followed and to avoid misconfiguration. What does this look like in practice? A simple example is checking Dockerfiles for whether a user other than root is used, or in the Kubernetes environment whether the versions of the images used are pinned, or whether the tag latest, or no tag at all, is used.

Why does it make sense to use security scanners?

The use of the various scanners is essentially always aimed at two scenarios. For existing infrastructures and repositories, you can check whether everything is currently in order, or whether improvements are advisable, e.g. because API keys were checked into the repository. At the same time, the same scanners can also be used to integrate into the existing development process, ensuring that such problems can no longer arise in future. This shift, also known as "shift left", increases the quality of both the process and the results. For container images, it's also advisable to check for vulnerabilities before pushing to a registry, so that delivery starts from the best possible state without known vulnerabilities. Every vulnerability and every problem that never gets rolled out saves the effort of fixing it afterwards. Of course, existing configurations or running images also need to be checked cyclically, since new vulnerabilities may have become known in the meantime, or manual interventions, e.g. configuration changes in a Kubernetes cluster, may have occurred. 

In summary, scanners help bring innovations into production under the best possible conditions and continuously improve and monitor the live environment.

Which scanner do we use?

We opted for trivy, an open source scanner from Aqua Security. For us, the advantages clearly lie in two areas. First, it's an open source solution currently being developed very actively with an engaged community. Second, this tool covers all the use cases described, so we only need to use and master one tool. The documentation is extensive and easy to follow, which was another plus for us.

How do we use trivy?

We mainly use trivy in our CI/CD pipelines and via the Starboard Operator in our Kubernetes environments. This combination lets us do the approach described above — both checking what's due to be deployed and monitoring at runtime whether new vulnerabilities have arisen in the running containers, for example. By using the Starboard Operator we can feed metrics from the various scans directly into Prometheus and get informed or alerted about changes.

Examples

Vulnerability scanner for container images

We use trivy as a vulnerability scanner when building our own container images. In doing so, we not only check the software packages but also create an overview file of all versions and licences in a "Software Bill of Materials" sbom file. 

Before_script:

  # Determine current version of trivy, then output and download it

- export TRIVY_VERSION=$(wget -qO - "https://api.github.com/repos/aquasecurity/trivy/releases/latest" | grep '"tag_name":' | sed -E 's/.*"v([^"]+)".*/\1/')

  - echo $TRIVY_VERSION

  - wget --no-verbose https://github.com/aquasecurity/trivy/releases/download/v${TRIVY_VERSION}/trivy_${TRIVY_VERSION}_Linux-64bit.tar.gz -O - | tar -zxvf -

Function deploy_with_trivy_check() {

  docker login -u "$USERNAME" -p "$PASSWORD" $REGISTRY

  DOCKER_BUILDKIT=1 docker build --pull --no-cache -t $NAME:$TAG

  docker tag $NAME:$TAG $NAME:latest

  # Generate GitLab report

  ./trivy image --exit-code 0 --format template --template "@contrib/gitlab.tpl" -o gl-container-scanning-report.json $NAME:$TAG

  # Generate HTML report

  ./trivy image --exit-code 0 --format template --template "@contrib/html.tpl" -o $TAG-scanning-report.html $NAME:$TAG

  # Output of a report with vulnerabilities classified as high criticality

  ./trivy image --exit-code 0 --severity HIGH --ignore-unfixed $NAME:$TAG

  # Abort on critical vulnerabilities with an existing patch

  ./trivy image --exit-code 1 --ignore-unfixed --severity CRITICAL $NAME:$TAG

  # Creating sbom file

  ./trivy image --format cyclonedx -o sbom-$NAME.json $NAME:$TAG

  docker push --all-tags $1

}

Usage via CLI for testing

Misconfiguration check using a Dockerfile example

$ trivy config .

2022-08-18T15:23:27.481+0200 INFO Misconfiguration scanning is enabled

2022-08-18T15:23:27.682+0200 INFO Detected config files: 1

Dockerfile (dockerfile)

Tests: 22 (SUCCESSES: 21, FAILURES: 1, EXCEPTIONS: 0)

Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

HIGH: Specify at least 1 USER command in Dockerfile with non-root user as argument

Running containers with 'root' user can lead to a container escape situation. It is a best practice to run containers as non-root users, which can be done by adding a 'USER' statement to the Dockerfile.

See https://avd.aquasec.com/misconfig/ds002

Scanning of an image, considering only vulnerabilities for which a fix exists

Conclusion

With trivy we have a universal programme that not only contributes significantly to the security of operated workloads as an automated part of CI/CD pipelines and Kubernetes environments. On, say, the admin's or developer's workstation, its ease of use also makes it a good option for one's own toolbox. The earlier and more often security scanners are used, the less effort it takes should findings actually occur.