Repository navigation
async support & httpx-based python-gitlab #1025
Description
I've completely reworked python-gitlab to be async-compatible: https://github.com/vishes-shell/python-gitlab :
- code it fully async compatible via awesome
httpx - async tests are rewritten with
pytest,pytest-asyncioandrespx - functional tests are also rewritten to be async
You can see changes in pull request vishes-shell#1 , i've tried to stick to project rules, but without pytest it would be harder...
I've seen some work that's done in #998 , and at the beginning i started on top of it, but found it hard to continue, due to some incomplete changes that are done, and trying to do sync stuff via run_until_complete would not work.
I would like to release it under python-gitlab-async module.
How i see it:
- i would hear your thoughts on this
- make some changes that you require (maybe drop
clisupport sincepython-gitlabdo it good) - update documentation
- transfer my fork under your
python-gitlabnamespace aspython-gitlab-async, become a member of it - release it in pypi.org
- take care of it
I think it's not that hard to keep up with changes from master branch and try to release as python-gitlab with same versions.
It was a pleasure to work with this brilliantly written library, and the same time you have a lot of defs to become async defs.:)
Activity
[-]`httpx` based python-gitlab a.k.a `python-gitlab-async`[/-][+]httpx based python-gitlab a.k.a python-gitlab-async[/+][-]httpx based python-gitlab a.k.a python-gitlab-async[/-][+]httpx based python-gitlab[/+]nejch commentedon Feb 22, 2020
Why not try to have the changes merged into this project? :) The maintainer has already signaled in #998 that he was willing to merge httpx-based work, they were just waiting with the other contributor for a stable httpx 1.0 to be released in the next month or so.
I totally agree with migrating to pytest though. All existing unit tests can already be run via pytest (with 2 lines of code changed); it would make proper testcases possible in functional tests both for the API and CLI, and both could then reuse fixtures, run entire suites independently of individual failures etc.
vishes-shell commentedon Feb 22, 2020
@nejch as far as I know it's really bad practice to implement every methods' sync functions through
So without sync methods that would break any sync projects that rely on this. Maybe I'm not aware of a good way to make async/sync compatible library.
nejch commentedon Feb 22, 2020
Ah sorry, the async/sync mess :/ I'm not sure either. It seems like some people are trying though, if this helps with future discussion:
https://stackoverflow.com/questions/55152952/duplication-of-code-for-synchronous-and-asynchronous-implementations
vishes-shell commentedon Feb 22, 2020
@nejch yeah, as i was saying it's not that simple to being able to create async/sync library (thats why we have ton of sync and async library separate), as it can require rebuilding the way the library works, as my personal perspective the library is brilliantly written and does it's work, and does it well.
So i am for new library, that would be not that hard to maintain since we have async version, maybe sometime in the future we would never require sync version, and just continue develop async version, but until it's not happen, it's probably better to have two libraries.:)
vishes-shell commentedon Feb 24, 2020
@bufferoverflow @max-wittig i've seen that you liked the comment that is about maintaining only one library (the current one) and having async/sync interface. That's a so much different that i've suggested, but still achievable. The downsides as i see it:
defhttpx, another way is to move completelyhttpxsince it supports sync operationsasyncsiblingAs i was saying, there is not much libraries which support sync and async interface, most of them have some sort of library clone, that operates only in
asyncway. The one, that i am aware of ishttpx, and the public interface is fully doubled, you can see https://github.com/encode/httpx/blob/master/httpx/_client.py .Can i hear your thoughts on this?
Thanks you!
khvn26 commentedon Feb 25, 2020
Async creep is inevitable when code is too dependent on I/O calls. Using
asyncio.runorloop.run_until_completeis not acceptable since it adds way too much unneeded overhead for purely sync consumers.If we want to support both sync in async in one codebase and avoid problems mentioned by @vishes-shell, we'll probably have to refactor some of the higher-level code.
I'd suggest decoupling serialization logic from transport code, providing a single interface for HTTP requests in concrete
SyncGitlabandAsyncGitlabsubclasses. Most of the job would be rewritingRESTObjectsubclasses to remove HTTP calls, perhaps by introducing some proxyLazyGitlabResponseobject representing how HTTP response needs to be deserialized.bufferoverflow commentedon Feb 25, 2020
@vishes-shell I'm still unsure about the real benefits of async support within python-gitlab, could you elaborate a bit on why this is useful and required? How many projects, people do you have on your GitLab? What's the improvement in regard of performance when using async within your environment?
How do you handle http error code 429 (too many requests) with the async lib?
nejch commentedon Feb 25, 2020
I think that something similar to what @khvn26 is suggesting for Sync/Async classes is done in Kenneth Reithz's requests-html (admittedly a much smaller wrapper library):
https://github.com/psf/requests-html/blob/master/requests_html.py#L787-L843
vishes-shell commentedon Feb 25, 2020
@bufferoverflow i believe it crutial to have async library if you want to build an application based on async features, more and more applications begin to be developed with async feature in mind, thanks to
asyncsince Python 3.5, we can see the growth of interest in async web frameworks: Starlette, FastAPI, django channels.Having async application and blocking gitlab http client would kill all features that are gained with async. I've seen some timeout errors in gitlab with 30 seconds, that's definitely an overkill. So basically if you want to build something async you want async in your I/O.
Projects, users? Thousands. Does it matter?
429 errors? There is functional tests in library which are perfectly passing. We just replace
time.sleep(wait_time)withasyncio.sleep(wait_time).max-wittig commentedon Feb 25, 2020
@vishes-shell Have you found an example of other API wrapper libraries using async already? I haven't found any, but I can imagine how web frameworks benefit from asyncio. Not sure about this wrapper library though.
vishes-shell commentedon Feb 25, 2020
@max-wittig
I can search for more. More of the api wrapper libraries are still sync just because of they are built a long time ago and
asyncis rather new.max-wittig commentedon Feb 25, 2020
@vishes-shell I guess it would make sense to use a separate library for this, but I don't see how I can maintain two libraries around python-gitlab instead of one, so feel free to publish python-gitlab-async but I may not be able to help much.
Additionally it would be nice, if you could
backportyour pytest changes to python-gitlabvishes-shell commentedon Feb 25, 2020
@max-wittig i've realized how much work it will require to maintain two libraries. I totally ok with supporting the async one, but to have two up-to-date versions of mostly same project would require x2 issues/MRs.
I will try to dig into different implementation that would require less changes for new features and keep everything under same project. Will keep you updated.
9 remaining items
graingert commentedon Jun 21, 2021
httpx (httpcore really) itself uses unasync for single source async/sync support: https://github.com/encode/httpcore/blob/c8d93d07369dab11d5c03f11de25607be1d21faf/unasync.py
nejch commentedon Jun 26, 2021
@graingert cool! That seems better to avoid duplication. The httpcore implementation doesn't make it obvious but there is a package for this that plugs into the build process and doesn't need the duplicated sync code committed: https://unasync.readthedocs.io/en/latest/ (edit: oh, I guess you're probably involved in that project somehow 😁 )
I just randomly found it here: https://github.com/johtso/httpx-caching/blob/master/gen_sync.py
Looks like doing some reading up on Sans IO would also help here.. https://sans-io.readthedocs.io/
CarliJoy commentedon Dec 28, 2021
Any progress here?
Btw: There is already a separate library released: https://github.com/pan-net-security/aio-gitlab
[-]httpx based python-gitlab[/-][+]async support & httpx-based python-gitlab[/+]yairsimantov20 commentedon May 31, 2023
Any updates
would love to use async client @vishes-shell
nikableh commentedon Nov 24, 2024
@nejch, greetings. Huge thank you for this library 🩵 It is awesome, I am very thankful it exists.
CarliJoy commentedon Nov 24, 2024
@synalice:
I have on idea on 1.
The answer for 2. depends on your usecase.
If you mainly use async because you working with Async Webframework like FastAPI, the easiest way is create sync functions that do all the work related to the gitlab api (multiple API calls).
This way you ensure things are not blocking and you don't spawm not required threads that could impact performance.
On the other hand if you want to use async calls for non blocking parallelism (i.e.
asyncio.TaskGrouporasyncio.gather, you need indeed to wrap all toasyncio.to_thread).nejch commentedon Nov 25, 2024
@synalice for GraphQL I'll be adding an async client soon if that helps you already, as we already base that client on httpx and it'll be pretty easy to do.
For REST it's a little trickier as the library has evolved a lot since the initial proposal so we'll need to touch a lot of modules to fully cover sync/async one-to-one and I'd need to really get back into this.
Thanks for providing your input as well @CarliJoy!
graingert commentedon Nov 25, 2024
For the async support please test on trio and asyncio using the pytest plug-in built into anyio
nejch commentedon Nov 25, 2024
@synalice I've added a little async client for GraphQL here: #3045
Pending review this should make it into the next release in a few days, but if you'd like to give it a try and have any feedback already would be appreciated 🙇 Not sure though if graphql covers the endpoints & attributes that you're looking for.
github-actions commentedon Jul 9, 2025
This issue was marked stale because it has been open 60 days with
no activity. Please remove the stale label or comment on this
issue. Otherwise, it will be closed in 15 days.
As an open-source project, we rely on community contributions to
address many of the reported issues. Without a proposed fix or
active work towards a solution it is our policy to close inactive
issues. This is documented in CONTRIBUTING.rst
How to keep this issue open:
investigate a fix, please comment and let us know.
solution, that would be fantastic.
will also prevent closure.
We value your input. If you can help provide a fix, we'd be happy
to keep this issue open and support your efforts.
This is documented in CONTRIBUTING.rst
https://github.com/python-gitlab/python-gitlab/blob/main/CONTRIBUTING.rst
github-actions commentedon Jul 25, 2025
This issue was closed because it has been marked stale for 15 days
with no activity.
This open-source project relies on community contributions, and
while we value all feedback, we have a limited capacity to address
every issue without a clear path forward.
Currently, this issue hasn't received a proposed fix, and there
hasn't been recent active discussion indicating someone is planning
to work on it. To maintain a manageable backlog and focus our
efforts, we will be closing this issue for now.
This doesn't mean the issue isn't valid or important. If you or
anyone else in the community is willing to investigate and propose
a solution (e.g., by submitting a pull request), please do.
We believe that those who feel a bug is important enough to fix
should ideally be part of the solution. Your contributions are
highly welcome.
Thank you for your understanding and potential future
contributions.
This is documented in CONTRIBUTING.rst
https://github.com/python-gitlab/python-gitlab/blob/main/CONTRIBUTING.rst
Zohair-coder commentedon Mar 8, 2026
Would love to see this library support async... reading this issue's history, it is still unclear why @vishes-shell MR (#1036) wasn't merged. Curious if we can add support for async.