issues with apache httpclient 5.1.x #1789

Activity

1. peterdnight commented on Jan 21, 2022

   

   peterdnight

   on Jan 21, 2022

   More actions

   Same issue - thanks for taking time to report (although i still burned many hours isolating JVM crashes)

   	<!-- 
   		- DockerJava integration
   		-  5.1.2 hangs,  5.0.4 is the workaround pending next release 3.2.13 ?
   	 -->
   	<dependency>
   		<groupId>org.apache.httpcomponents.client5</groupId>
   		<artifactId>httpclient5</artifactId>
   		<version>5.0.4</version> 
   	</dependency>
   	
   	<dependency>
   		<groupId>com.github.docker-java</groupId>
   		<artifactId>docker-java-transport-httpclient5</artifactId>
   		<version>3.2.12</version>
   		<exclusions>
   			<exclusion>
   				<groupId>org.apache.httpcomponents.client5</groupId>
   				<artifactId>httpclient5</artifactId>
   			</exclusion>
   		</exclusions>
   	</dependency>
   
   
   	<dependency>
   		<groupId>com.github.docker-java</groupId>
   		<artifactId>docker-java</artifactId>
   		<version>3.2.12</version>
   		<exclusions>
   			<exclusion>
   				<groupId>com.github.docker-java</groupId>
   				<artifactId>
   					docker-java-transport-jersey
   				</artifactId>
   			</exclusion>
   			<exclusion>
   				<groupId>com.github.docker-java</groupId>
   				<artifactId>docker-java-transport-netty</artifactId>
   			</exclusion>
   		</exclusions>
   	</dependency>
   
2. dimovelev commented on Jan 21, 2022

   

   dimovelev

   on Jan 21, 2022

   Author

   More actions

   @peterdnight i switched to the zerodep client - it seems to be a shaded httpclient5 so it does not clash with the version i have in the remaining part of my application. I guess that is the safer bet for future updates and i will stick to it.
3. dimovelev commented on Jan 21, 2022

   

   dimovelev

   on Jan 21, 2022

   Author

   More actions

   After reading through the other issue I have a short reproducer:

   package com.example;
   
   import java.io.IOException;
   import java.util.List;
   
   import com.github.dockerjava.api.DockerClient;
   import com.github.dockerjava.api.model.Image;
   import com.github.dockerjava.core.DefaultDockerClientConfig;
   import com.github.dockerjava.core.DockerClientImpl;
   import com.github.dockerjava.httpclient5.ApacheDockerHttpClient;
   import com.github.dockerjava.transport.DockerHttpClient;
   import org.slf4j.Logger;
   
   import static org.slf4j.LoggerFactory.getLogger;
   
   public class Issue {
       private static final Logger LOGGER = getLogger(Issue.class);
   
       public static void main(String[] args) throws IOException, InterruptedException {
           final var config = DefaultDockerClientConfig.createDefaultConfigBuilder() //
               .withDockerHost("npipe:////./pipe/docker_engine") //
               .build();
           final DockerHttpClient httpClient = new ApacheDockerHttpClient.Builder() //
               .dockerHost(config.getDockerHost()) //
               .build();
           final DockerClient client = DockerClientImpl.getInstance(config, httpClient);
           try {
               LOGGER.info("Listing images");
               final List<Image> result1 = client.listImagesCmd().exec();
               LOGGER.info("Found {} images", result1.size());
               Thread.sleep(2_000);
               LOGGER.info("Listing images");
               final List<Image> result2 = client.listImagesCmd().exec();
               LOGGER.info("Found {} images", result2.size());
           } finally {
               client.close();
               httpClient.close();
           }
       }
   }
   

   The second list images hangs forever.
   The crucial part is sleeping for at least 2 seconds (1 second is fine). With that and going through the apache httpclient commits I would guess that the issue is this one - apache/httpcomponents-client@10e2deb
4. dimovelev commented on Jan 21, 2022

   

   dimovelev

   on Jan 21, 2022

   Author

   More actions

   So the connection staleness check in org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager seems to block forever. It is triggered (with default configuration) after 2 seconds of inactivity. If you change that to -1 the check is disabled and the code above does not block. you can call this method after creating the client to change the value:

       private static void changeValidateAfterInactivity(final DockerHttpClient httpClient) throws NoSuchFieldException,
           IllegalAccessException {
           final Field httpClientField = httpClient.getClass().getSuperclass().getDeclaredField("httpClient");
           httpClientField.setAccessible(true);
           final Object apacheClient = httpClientField.get(httpClient);
           final Field connManagerField = apacheClient.getClass().getDeclaredField("connManager");
           connManagerField.setAccessible(true);
           final PoolingHttpClientConnectionManager connManager =
               (PoolingHttpClientConnectionManager) connManagerField.get(apacheClient);
           connManager.setValidateAfterInactivity(TimeValue.NEG_ONE_MILLISECOND);
       }
   

   So I guess a change in com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl will be required to set this value.
5. dimovelev commented on Jan 21, 2022

   

   dimovelev

   on Jan 21, 2022

   Last edited by dimovelev

   Author

   More actions

   The connection staleness check in apache httpclient seems to rely on modifying the socket timeout to 1 ms then trying to read something and resetting the timeout back to the original value. I guess that this fails in our case - the timeout is not reset and thus the read blocks forever (as the original timeout was 0).
6. dimovelev commented on Jan 21, 2022

   

   dimovelev

   on Jan 21, 2022

   Last edited by dimovelev

   Author

   More actions

   that was a big waste of time. the issue has been fixed (well the timeout set to -1 as i figured out above) in November 2021 - see #1727 . The latest release of docker-java does not include the fix.

   @bsideup when is the next released planned?
7. peterdnight commented on Jan 24, 2022

   

   peterdnight

   on Jan 24, 2022

   More actions

   @dimovelev - i feel your pain. fyi - if you are curious regarding my use case, check out my docker tutorial:
   https://www.youtube.com/playlist?list=PLETtnz3MXNMUBFq_kSSXrXm1lnp10rt0I
8. bsideup commented on Feb 15, 2022

   

   bsideup

   on Feb 15, 2022

   Member

   More actions

   Should be fixed in 3.2.13
9. bsideup

   closed this as completedon Feb 15, 2022