Support BuildKit for building images

I gave it a quick go and I came up with working solution.

Main problem is that the API responses have a different format with BuildKit enabled ("version: 2"):
{"id":"moby.buildkit.trace","aux":"CncKR3NoYTI1NjpmMWM5ZDIzNTNhOWRlZjgwOTI2OGRmMjhhMjU4YzExNTA2ODcxMWE0ZmFiYzM1MjNkOGJhZDIxOTgyMDc3MjFjGhJleHBvcnRpbmcgdG8gaW1hZ2UqCwiwrO24BhCejrRrMgsIsKztuAYQ2pz6dA=="}
{"id":"moby.image.id","aux":{"ID":"sha256:ee85f5f73e52d5361ae55c326c7f5d61aee1972a80f7f06b880bccd5cef15b14"}}

If I can get some guidance on what would be the preferred way to fix the deserialization issue, I'd be happy to clean this up and make a real PR for this.

Let me know if this is something that you would like to get implemented.

Sorry I messed up and created another issue while submitting the PR :-(

gRPC API must be used in order to get full access to buildkit features.

@eddumelendez

@eddumelendez Will you be adding BuildKit support? I've tried to implement this myself and got stuck: docker/docs#21944

For what it's worth. I got this working in https://github.com/cowwoc/docker

thanks for sharing

@cowwoc

would be nice to adapt some code from your repo to docker-java and provide an additional implementation to

docker-java/docker-java-core/src/main/java/com/github/dockerjava/core/AbstractDockerCmdExecFactory.java

Lines 390 to 392 in 97579e5

    
           public BuildImageCmd.Exec createBuildImageCmdExec() { 
        
               return new BuildImageCmdExec(getBaseResource(), getDockerClientConfig()); 
        
           }

called createBuildKitImageCmdExec() (not the perfect name, but suggestions are welcome)

@cowwoc, would you be interested to contribute?

@eddumelendez

@eddumelendez I prefer focusing on the new API for now. It is also worth noting that there are some outstanding issues with invoking BuildKit using the REST API. For example, there are scenarios that cause "ERROR: no active sessions" to be returned and moby/buildkit#5309 (comment) does not inspire any confidence that this will be fixed.

I will try filing a bug report against the REST API for this when I get the time and will CC you for extra transparency.

Sorry if I misunderstod but here I thought you got it working with the new API (gRPC).

It is also worth noting that there are some outstanding issues with invoking BuildKit using the REST API.

Yes, aware of it and indeed we would like to integrate with new API.

I will try filing a bug report against the REST API for this when I get the time and will CC you for extra transparency.

We don't want to integrate with REST API due to limitations. See #2362 (comment)

@eddumelendez

@eddumelendez I am getting a lot of mixed messages from the Docker team. The BuildKit guys keep on saying that the REST API is deprecated and they only support interaction through the gRPC API. But I've yet to hear from @thaJeztah whether this is the case. Otherwise, why bother support BuildKit builds through the /build endpoint?

To answer your question: I only half-integrated gRPC. I am using the REST API to trigger a build using BuildKit, and using their gRPC API to parse the response. I plan to remove the use of the REST API altogether and only use gRPC for interacting with BuildKit but I got pulled in 3 different directions in the past month so I haven't had time to jump on it yet.

I hope to get back to it in the coming weeks.

For what it's worth. I got this working in https://github.com/cowwoc/docker

There's also still https://github.com/batect/docker-client, which sadly is not maintained anymore, but follows the very interesting approach of embedding the official Golang Docker client libraries.

I ended up giving up on using the gRPC API. It's possible to get it working, but the API is too low-level the the lack of documentation makes it a waste of time in my eyes. At this point, I'm planning to just wrap the CLI. This is much easier to implement and reduces the dependencies substantially.

Anyone see a downside of doing so?

Anyone see a downside of doing so?

Well, that wouldn't work on systems that have the Docker daemon but not the client installed, I guess. But I'm unsure how likely that scenario is.

Somewhat related, I just came across https://github.com/fussybeaver/bollard, which could probably be used via JNI.

@sschuberth

@sschuberth Good find. Do you have any idea how well their library actually works? The other downside to consider is that they doubled-down on an async model whereas the Java world has doubled-down on a sync model using virtual threads.

Do you have any idea how well their library actually works?

No idea. I just wanted to share it for reference.

I run into the same problem and tried to do a quick implementation (changes are on this branch) in combination with testscontainers.

Here are my findings:

aux (when it's a "trace") now contains a base64 encoded string with the progress:

Gsha256:3e19286f464521d06cd9210b2617df88e5505b87c5a11671c61de69a81253df3�Gsha256:b8f82b7c90dca2cf7b0bfa89de29dea8e6dadab8e03c685fe2c2751c85b67798�N[build-env 4/9] RUN mkdir /build     && chown limitedbuild:limitedbuild /build �*�����������2�����������
��

Related "serverside" Docker API: https://github.com/moby/moby/blob/9a9cade0dfa872fafa98ab15fe51ee152d13649d/api/types/build/build.go
For to some reason the output option is not working correctly:
According to the API docs setting version=2 uses Buildkit.
However
```
BuildImageV2Cmd buildImageCmd = dockerClient.buildImageV2Cmd(in)
				.withOutputs("[{\"type\": \"docker\"}]")
				.withVersion("2");
```
results in Status 500: {"message":"exporter \"docker\" could not be found"}.
This is odd because the normal docker build works and buildkit is active ("BuildKit is the default builder for users on Docker Desktop and Docker Engine v23.0 and later.").
It looks like the default/non-buildkit worker - and not the one from buildkit - is used because I was able to verify that the moby exporter exists which is only present in the non-buildkit variant.

Let DockerBuildImage build images with BuildKit

Navigation Menu

Repository navigation

Support BuildKit for building images #2361

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	public BuildImageCmd.Exec createBuildImageCmdExec() {
	return new BuildImageCmdExec(getBaseResource(), getDockerClientConfig());
	}

Support BuildKit for building images #2361

Description

Activity

markusreil commented on Oct 25, 2024

eddumelendez commented on Dec 16, 2024

cowwoc commented on Jan 31, 2025

cowwoc commented on Feb 19, 2025

eddumelendez commented on Mar 7, 2025

eddumelendez commented on Mar 25, 2025

cowwoc commented on Mar 26, 2025

eddumelendez commented on Mar 26, 2025

cowwoc commented on Mar 28, 2025

sschuberth commented on May 14, 2025

cowwoc commented on May 14, 2025

sschuberth commented on May 14, 2025

sschuberth commented on May 22, 2025

cowwoc commented on May 22, 2025

sschuberth commented on May 22, 2025

AB-xdev commented on Jun 16, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions