Skip to content
This repository was archived by the owner on Jan 13, 2021. It is now read-only.
This repository was archived by the owner on Jan 13, 2021. It is now read-only.

Proper HTTP/1.1 parser #98

Closed
Closed
Proper HTTP/1.1 parser#98
Labels
Long-Term Goal

Description

@Lukasa
Lukasa
opened on Mar 13, 2015

Parsing HTTP/1.1 is substantially trickier than parsing HTTP/2. I'm happy with my HTTP/2 parsing code, but I'm not even close to happy with the HTTP/1.1 parsing code in #92 (that will eventually make it into master).

I think the ad-hoc parsing that's there right now is totally acceptable for an initial release, but as time passes we should really tighten it up.

There are some options here:

  1. Handwrite a parser in pure-Python. This idea terrifies me: the HTTPHeaderMap already has some unfortunate performance problems in CPython, and writing a parser that has to individually examine every byte of the HTTP header is going to be terribly slow in CPython.
  2. As (1), but use zero buffer to speed stuff up. It's not clear to me what performance improvement this would give me (particularly over using memoryviews, which I already use) over (1).
  3. Embed a parser written in another language that I distribute with the code. An obvious choice is picohttpparser. This is likely to be the fastest approach to the problem.

In an ideal world I'd like to do (3). However, the idea of distributing C code along with my library gives me pause. On Windows and OS X it should be not too terrible, because I can presumably bundle the dynamic library with a wheel. With Linux this becomes much more of an extraordinary pain in the ass, because I need to get the code to build at install time.

Additionally, it causes me to incur much larger development overheads. I'll need Windows boxes to build wheels for Windows, OS X boxes to build wheels for OS X...it's all kind of a nightmare.

Activity

  1. Lukasa commented on Mar 13, 2015

    @Lukasa
    Lukasa
    on Mar 13, 2015
    MemberAuthor

    Actually, for Linux it might not be too bad, because picohttpparser doesn't have to link against Python itself. That means the .so should be fairly platform independent (he said, hopefully).

    I think the way to think about this is to see if I can build a good cffi-based Python wrapper for picohttpparser. If I can, then I can bring that in as a dependency and solve this problem neatly.

  2. Lukasa commented on Mar 13, 2015

    @Lukasa
    Lukasa
    on Mar 13, 2015
    MemberAuthor

    Btw, if @dstufft could weigh in with insights about packaging this stuff that'd be really helpful. I already know that @dstufft wants me to do (3) in some form. =D

  3. Lukasa
    added
    Long-Term Goal
    on Mar 13, 2015

  4. Lukasa commented on Mar 13, 2015

    @Lukasa
    Lukasa
    on Mar 13, 2015
    MemberAuthor

    Some notes from an extremely helpful conversation with @reaperhulk and @dstufft (so I don't forget!):

    <lukasa> What's the recommended story for bundling a .so with a python package, and calling into it with CFFI?
<lukasa> Wheels for Win and OS X, then src for Linux?
<Alex_Gaynor> lukasa: works fine; for cryptography we ship wheels for windows which statistically link openssl
<lukasa> Mm, I glanced through what cryptography was doing
<lukasa> Couldn't entirely convince myself that it worked because some of the magic is done on Jenkins
<Alex_Gaynor> reaperhulk: can tell you what jenkins does.
<Alex_Gaynor> I'm pretty sure it's just `pip wheel cryptography` though
<reaperhulk> lukasa: Yeah we just pip wheel cryptography
<reaperhulk> The entirety of the jenkins batch file is: https://gist.github.com/reaperhulk/a580c72e58f99e99f254
<reaperhulk> there's also one for windows64 but that's just to point it at the 64-bit binaries/libraries
<lukasa> However, pip install wheel/pip wheel do bundle the .dll?
<reaperhulk> We statically link it so there's no bundling required
<lukasa> Ah, that makes more sense
<lukasa> Actually makes way more sense, that's a far better plan
<lukasa> Ok, time to go back to your CFFI invocation and work out how you do that. ;)
<reaperhulk> It works pretty well on Windows, although it does require us to ship a new cryptography whenever openssl gets revv'd
<lukasa> Thanks! Would like to distribute a C library with my Python module without it requiring everyone to compile everything all the time
<reaperhulk> incidentally if you need a wheel job for windows we can potentially set it up on our jenkins. It's a pain to maintain this infra so you're welcome to piggyback on it
<lukasa> reaperhulk: Thanks! Won't happen in the next few days or anything, but you might come to regret that offer. ;)
<reaperhulk> haha, well we build cffi wheels already
<reaperhulk> https://jenkins.cryptography.io :)
<lukasa> Shiny
<lukasa> So, follow-on question
<lukasa> Can that static-link approach work on Linux as well? Or is there some pain there I'm not familiar with?
<dstufft> it works on Linux-ish, you still have to link to Python, and Python might have different API incompatabilities
<dstufft> but PyPI doesn't allow linux wheel uploads
<reaperhulk> so dstufft's got some thoughts there, but in general you'll have to statically link everything so you're going to pull in glibc and such (and if you're on a distro that uses a different libc is that okay) + python linking
<reaperhulk> oop dstufft already responded
<lukasa> Hmm
<lukasa> So, can I distribute the .c file and get that compiled at install time with an sdist?
<lukasa> (This is a whole world of Python packaging I've never had to think about before)
<dstufft> yea
<dstufft> https://caremad.io/2014/11/distributing-a-cffi-project/
<lukasa> Shiny

    This makes it seem like building a python wrapper for picohttpparser as a PyPI module and then depending on that might actually work.

  5. Lukasa commented on Mar 14, 2015

    @Lukasa
    Lukasa
    on Mar 14, 2015
    MemberAuthor

    @dstufft In a hypothetical world where hyper ends up in urllib3, how much difficulty does a cffi dependency represent for pip?

  6. dstufft commented on Mar 14, 2015

    @dstufft
    dstufft
    on Mar 14, 2015

    Ideally you'd have a pure Python fallback, pip can't have a C dependency because of the bootstrapping problem. I recognize that's more work but an optional C dependency is most likely the best way to handle it, not just for pip but for anyone trying to use requests in a situation where C deps are less ideal.

    On Mar 14, 2015, at 10:44 AM, Cory Benfield [email protected] wrote:

    @dstufft In a hypothetical world where hyper ends up in urllib3, how much difficulty does a cffi dependency represent for pip?

    —
    Reply to this email directly or view it on GitHub.

  7. Lukasa commented on Mar 27, 2015

    @Lukasa
    Lukasa
    on Mar 27, 2015
    MemberAuthor

    Ok, this is done and merged into #92: hyper will use pycohttpparser if it's available, and if not it'll fall back to using the pure-Python bundled version.

  8. Lukasa
    closed this as completedon Mar 27, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Long-Term Goal

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions