Vulnerability management: 3 best practices and tips for image building and scanning

As enterprises adopt containers, microservices, and Kubernetes for cloud-native applications, vulnerability management is crucial to improve the security posture of containerized workloads throughout build, deploy, and runtime. Securing your build artifacts and deployment pipeline, especially when it comes to images, is extremely important. By following best practices for image building and scanning throughout the application development and deployment process, you can help ensure the security of the containers and workloads in your environment.

Let’s look at some of the nuances of choosing a base image, hardening your container image, and container image scanning, including tips on choosing an appropriate scanning solution and tackling privacy concerns.

Choose an appropriate base image

It’s important to choose a base image that reduces the attack surface of your container. I recommend using a distroless or scratch image because they contain only the application and its runtime dependencies. Both types of images improve your security posture by reducing the attack surface and exposure to vulnerabilities.

If for some reason you can’t use a distroless or scratch image, choose a minimal distro. Modern immutable Linux distributions, such as Bottlerocket and Flatcar Container Linux, can be used as base images for containers, as can minimal versions of traditional Linux distributions (i.e. Ubuntu, Red Hat, Alpine, etc.). While a minimal image will work, keep in mind that this type of image does not protect against vulnerabilities in operating system (OS) packages in the OS.

Harden your container image

Container image hardening adds defensive layers that allow you to run applications securely within a container, while also reducing security weaknesses and the attack surface. Building hardened container images for workloads is important because nonhardened images open up your workloads to a range of risks, including disclosure of information and privilege escalation to the container host.

Here are a few ways to harden container images.

• Use base images from trusted sources only (e.g. official Ubuntu or Red Hat release channel).
• Double-check an image’s hash against released information, even if it’s from a trusted source. It’s not that difficult for an attacker to embed malicious code in a base image, and then make the image available on a repository (e.g. Docker Hub).
• Minimize your base images. They should only contain an application’s runtime dependencies.
• Run containers with the minimum necessary permissions, following the principle of least privilege (e.g. run containers as non-root when possible). This way, it will be more difficult for attackers to escape the container, like what happened with the CVE-2020-15257 vulnerability.
• Verify the image using container image signing. While Kubernetes does not natively include a container image verification tool, you can use Docker Notary to sign images, and verify an image signature using the Kubernetes admission controller.

If using a Docker image, I recommend the following:

• Do not use tags – Tags (e.g. mutable tags like latest or master) can cause application stability issues if a dependent library is moved or changed. They can also cause issues with image scanning in your CI/CD pipeline, since they are regularly updated with features and fixes. Instead of using tags, pin the base image version in the Dockerfile (e.g. ubuntu:20.08).
• Compress layers into one single layer – Layers show development history and can reveal sensitive information. Because container images built with Docker tend to include multiple layers, you should use a multistage build or option --squash(an experimental Docker daemon available in Docker API 1.25+) to compress existing layers.

Scan your container image

Container image scanning helps determine if there are vulnerable components present in an image by examining a container’s filesystem and metadata, and then comparing the collected data against vulnerability information from various trusted sources (e.g. National Vulnerability Database or private intelligence sources). There are plenty of scanning tools available, both open source and commercial.

You want to make sure that your image scanning tool scans all OS packages in your container image, and that it understands the language(s) used by your application so that it can scan application dependencies. A good container image scanning tool should also:

• Detect sensitive files, such as certificates and passwords, that exist in the filesystem
• Scan binaries (e.g. .elf or .exe)
• Integrate into your CI/CD system
• Have a low false-positive rate (note that both false positives and negatives are to be expected during container image scanning; your application and security teams will need to analyze these and assess risk)

While image and container scanning services are offered by many public cloud providers and container registry service providers, a lot of them don’t scan application dependencies and are limited in the OS versions they support.

Since information about security vulnerabilities in your product is sensitive data that would become a big liability in the wrong hands, don’t forget to address data security and privacy concerns before choosing a scanning tool. It’s important to understand what data a scanning tool will collect, and how, in order to understand the risk of data exposure. For example: Does it collect package metadata only, or will it upload your container image to its SaaS service?

You also need to determine whether the tool will store collected data on-premises or in the cloud as part of a SaaS, to ensure the tool meets guidelines set by your security and/or compliance teams. Be sure to check the contract (if using a commercial tool) to understand what kind of clauses are in place for damages in the case of a data breach, or the documentation (if using an open-source tool) to understand the risk of a data leak.

Summary

The tips and best practices included in this article serve as a good starting point for vulnerability management. I recommend reading chapter 3 of Kubernetes security and observability: A holistic approach to securing containers and cloud-native applications, an O’Reilly book I helped author, to learn about these best practices in further detail and to discover additional best practices for image building and scanning.

To learn more about securing containers and Kubernetes at the infrastructure level, read this free O’Reilly ebook.

To find out how Tigera’s CNAPP can improve the security of your cloud-native workloads, read about Calico’s image assurance feature.