Repository navigation
Build context scan feels slow when directory contains large file tree #1409
Description
Environment: IntelliJ IDEA 2020.2 EAP Build #IU-202.5103.13 with bundled Docker plugin 202.5103.13. Docker plugin must use docker-java, connection to Docker Desktop for Windows via npipe. Cannot find out the exact version of docker-java that is used.
Given the following minimal project setup:
node_modules with +67k files
Dockerfile with single line FROM hello-world
.dockerignore with single line node_modules
Then, run docker build . in the root directory.
Expected behaviour: The image is built instantly, because build context contains only the Dockerfile (large node_modules directory is ignored), and there are almost no instructions in the Dockerfile itself.
Observed behaviour: For +30 seconds, the screen shows "Building image..." , and after that only one file is added to the build context archive.
I assume, the build traverses all +67k files in the node_modules directory, only to find none of them have to be included. This would be very inefficient. When there are no exceptions, that allow a file below node_modules, it might skip the whole directory.
Update:
Unit test in docker-java-core, testdir has the large folder as child, .dockerignore absent:
@Test
public void can_parse_dockerfile() throws IOException {
File dockerFile = new File(".\\testdir\\Dockerfile");
File baseDirectory = new File(".\\testdir");
Dockerfile dockerfile = new Dockerfile(dockerFile, baseDirectory);
Stopwatch stopwatch = Stopwatch.createStarted();
Dockerfile.ScannedResult parse = dockerfile.parse();
System.out.println("Parse took " + stopwatch.elapsed(TimeUnit.MILLISECONDS) + "ms");
System.out.println(parse.filesToAdd.size() + " files added");
}Output:
Parse took 28008ms
4 files added
Activity
KostyaSha commentedon Jun 7, 2020
Try
RUN lsif there are files then they were sent to build context into daemon.Please fill issue against IDEA first, dockefile parsing was updated few times in different versions.
viatra2 commentedon Jun 8, 2020
IDEA plugin overrides the
com.github.dockerjava.core.dockerfile.Dockerfilein order to provide some feedback in the build log during potentially long archive creation.However, the reported issue comes from the recursive call
ScannedResult # addFilesInDirectoryhere.viatra2 commentedon Jun 8, 2020
More specifically, this recursion always goes deep into the folder even if the current
directoryis ignored AND .dockerignore does not have exclusions for any paths below it (AFAIU this exclusion would be something like!directory/evenDeeper/doNotIgnoreMe*entry in the .dockerignore file)borlander commentedon Jun 8, 2020
AFAIU the Docker CLI code also walks the whole folder structure without further optimizations here
swiedenfeld commentedon Jun 8, 2020
@borlander Docker CLI is written in Go, and this is written in Java. I don't think they can be compared. Looking at the code,
ScannedResultusesjava.io, and it might be a bottleneck traversing the large directory. Maybejava.niocould improve processing?swiedenfeld commentedon Jun 12, 2020
I did a bit of research on this.
At first glance, the idea of @viatra2 in #1409 (comment) sounds correct and promising.
Following, I implemented a file walk with
java.nio.file.Files.walkFileTree, but it did not traverse the tree considerably faster.Then, I did a short circuit that prevents the
node_modulesto be traversed further if it is ignored. But the exception rules!rules bit me. There is a lot of combinations of exclude and include lines possible, plus their order in.dockerignorematters. It may thus be inevitable to visit each file and check their effective inclusion/exclusion.Nevertheless, one possible improvement I found is #1412.
Memoize file match patterns (#1412)
stale commentedon Jul 12, 2020
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[-]Possible performance issue in docker build step[/-][+]Build context scan feels slow when directory contains large file tree[/+]