Repository navigation
Empty directories inside a .dockerignore-excluded directory are still added to the build context #2701
Description
Location (docker-java/docker-java @ default branch 002b6eb):
if (files.length != 0) {
for (File f : files) {
if (f.isDirectory()) {
addFilesInDirectory(f);
} else if (effectiveMatchingIgnorePattern(f) == null) {
filesToAdd.add(f);
}
}
// base directory should at least contains Dockerfile, but better check
} else if (!isBaseDirectory(directory)) {
// add empty directory
filesToAdd.add(directory);
}Problem
With a .dockerignore that excludes a directory, an empty subdirectory inside that excluded directory is still returned in Dockerfile.ScannedResult.filesToAdd, and therefore ends up in the build-context tar (filesToAdd is what the build command archives).
project/
Dockerfile
.dockerignore # contains: ignored
included-file.txt
ignored/
empty-subdir/
In ScannedResult.addFilesInDirectory, regular files are checked with effectiveMatchingIgnorePattern(f), but directories are always recursed into, and a directory with no entries is added unconditionally. So ignored/empty-subdir bypasses the ignore check.
ignored/empty-subdir should not be part of the context. Docker's documentation for .dockerignore files says:
Files and directories that match patterns are removed from the build context before it is sent to the builder.
Empty directories that are not excluded should still be included as today, and ! exception patterns should keep working.
Reproduction
Test in docker-java/src/test/java/com/github/dockerjava/core/dockerfile/ using the public Dockerfile.parse() API:
DockerfileExcludedDirectoryTest.java
package com.github.dockerjava.core.dockerfile;
import com.google.common.base.Function;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.util.Collection;
import static com.google.common.collect.Collections2.transform;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.containsInAnyOrder;
public class DockerfileExcludedDirectoryTest {
private static final Function<File, String> TO_FILE_NAMES = File::getName;
@Rule
public TemporaryFolder temporaryFolder = new TemporaryFolder();
/**
* An empty directory inside a directory that matches an exclusion pattern
* in .dockerignore should not be included in the build context.
*
* Currently, addFilesInDirectory always recurses into every directory
* without checking if the directory itself matches an exclusion pattern.
* As a result, empty subdirectories inside excluded directories are
* incorrectly added to filesToAdd.
*/
@Test
public void emptyDirInsideExcludedDirShouldBeExcluded() throws Exception {
File baseDir = temporaryFolder.newFolder("buildContext");
// Create Dockerfile
writeFile(new File(baseDir, "Dockerfile"), "FROM scratch\n");
// Create .dockerignore that excludes "ignored"
writeFile(new File(baseDir, ".dockerignore"), "ignored\n");
// Create the excluded directory with an empty subdirectory inside
File ignoredDir = new File(baseDir, "ignored");
ignoredDir.mkdir();
File emptySubdir = new File(ignoredDir, "empty-subdir");
emptySubdir.mkdir();
// Create a file that should be included
writeFile(new File(baseDir, "included-file.txt"), "content");
// Parse and get files to add
Dockerfile dockerfile = new Dockerfile(new File(baseDir, "Dockerfile"), baseDir);
Dockerfile.ScannedResult result = dockerfile.parse();
Collection<String> fileNames = transform(result.filesToAdd, TO_FILE_NAMES);
// empty-subdir should NOT be in the result because it's inside the
// excluded 'ignored/' directory
assertThat(fileNames,
containsInAnyOrder("Dockerfile", ".dockerignore", "included-file.txt"));
}
private void writeFile(File file, String content) throws IOException {
Files.write(file.toPath(), content.getBytes(StandardCharsets.UTF_8));
}
}mvn -pl docker-java -am test -Dtest=DockerfileExcludedDirectoryTest -DfailIfNoTests=falseFailing output:
Expected: iterable with items ["Dockerfile", ".dockerignore", "included-file.txt"] in any order
but: no match for: "empty-subdir"
at com.github.dockerjava.core.dockerfile.DockerfileExcludedDirectoryTest.emptyDirInsideExcludedDirShouldBeExcluded(DockerfileExcludedDirectoryTest.java:61)
Environment:
- Commit:
002b6eb7aa2b892be729c7c2a5e95f782924db38(main) - Toolchain: OpenJDK 21.0.13, Apache Maven 3.9.16
Already fixed in docker/docker-py
PR: docker/docker-py#863 — "Don't descend into ignored directories when building context" (merged 2015-12-16T16:59:12Z, merge commit 371a17bc20c). Each directory now goes through should_include before it is added to the context, and excluded directories are not descended into unless exception rules exist. docker-py is another client implementation of the same .dockerignore behaviour, so this is precedent, not an obligation; the Docker documentation quoted above is the primary basis.
How docker-py fixed it:
--- a/docker/utils/utils.py
+++ b/docker/utils/utils.py
@@ -107,38 +107,68 @@ def exclude_paths(root, patterns, dockerfile=None):
...
+def should_include(path, exclude_patterns, include_patterns):
+ """
+ Given a path, a list of exclude patterns, and a list of inclusion patterns:
+
+ 1. Returns True if the path doesn't match any exclusion pattern
+ 2. Returns False if the path matches an exclusion pattern and doesn't match
+ an inclusion pattern
+ 3. Returns true if the path matches an exclusion pattern and matches an
+ inclusion pattern
+ """
+ for pattern in exclude_patterns:
+ if match_path(path, pattern):
+ for pattern in include_patterns:
+ if match_path(path, pattern):
+ return True
+ return False
+ return True
...
+ if not has_exceptions:
+
+ # Remove excluded patterns from the list of directories to traverse
+ # by mutating the dirs we're iterating over.
+ # This looks strange, but is considered the correct way to skip
+ # traversal. See https://docs.python.org/2/library/os.html#os.walk
+
+ dirs[:] = [d for d in dirs if
+ should_include(os.path.join(parent, d),
+ exclude_patterns, include_patterns)]
+
for path in dirs:
- paths.append(os.path.join(parent, path))
+ if should_include(os.path.join(parent, path),
+ exclude_patterns, include_patterns):
+ paths.append(os.path.join(parent, path))
+
for path in files:
- paths.append(os.path.join(parent, path))
+ if should_include(os.path.join(parent, path),
+ exclude_patterns, include_patterns):
+ paths.append(os.path.join(parent, path))Suggested fix
Run the .dockerignore check on the empty-directory branch of ScannedResult.addFilesInDirectory as well, so an empty directory is only added when no pattern excludes it, the same way regular files are already filtered (! exception patterns keep working because effectiveMatchingIgnorePattern applies them):
diff --git a/docker-java-core/src/main/java/com/github/dockerjava/core/dockerfile/Dockerfile.java b/docker-java-core/src/main/java/com/github/dockerjava/core/dockerfile/Dockerfile.java
--- a/docker-java-core/src/main/java/com/github/dockerjava/core/dockerfile/Dockerfile.java
+++ b/docker-java-core/src/main/java/com/github/dockerjava/core/dockerfile/Dockerfile.java
@@ -210,8 +210,8 @@ public class Dockerfile {
}
}
// base directory should at least contains Dockerfile, but better check
- } else if (!isBaseDirectory(directory)) {
- // add empty directory
+ } else if (!isBaseDirectory(directory) && effectiveMatchingIgnorePattern(directory) == null) {
+ // add empty directory unless .dockerignore excludes it
filesToAdd.add(directory);
}
}Verified on 002b6eb (OpenJDK 21.0.2, Maven 3.9.16): the test above fails without this patch and passes with it; no other test results change (mvn -pl docker-java test, which excludes the integration tests that need a Docker daemon).
I can submit a PR for this.
Checked: #2700 (leading / in .dockerignore patterns) is a related area with a different root cause; #1409 / #1411 / #1412 (scan performance, pattern compilation) and #1441 (large subdirectory test) do not cover excluded empty directories. No hits for dockerignore empty directory / "empty-subdir".
Activity