推荐学习书目
› Learn Python the Hard Way
Python Sites
› PyPI - Python Package Index
› http://diveintopython.org/toc/index.html
› Pocoo
值得关注的项目
› PyPy
› Celery
› Jinja2
› Read the Docs
› gevent
› pyenv
› virtualenv
› Stackless Python
› Beautiful Soup
› 结巴中文分词
› Green Unicorn
› Sentry
› Shovel
› Pyflakes
› pytest
Python 编程
› pep8 Checker
Styles
› PEP 8
› Google Python Style Guide
› Code Style from The Hitchhiker's Guide
annoygaga
V2EX  ›  Python

requests 为什么慢 urllib3 如此之多?

  •  1
     
  •   annoygaga · Nov 26, 2023 · 4352 views
    This topic created in 1077 days ago, the information mentioned may be changed or developed.

    目前使用 requests 库去进行一些 api 的访问,惊讶的发现 requests 库速度相较于 urllib3 慢非常多

    $ url = "https://httpbin.org"
    
    $ %timeit r = requests.get(url)
    
    119 ms ± 42.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
    
    $ %timeit r = httpx.get(url)
    
    %117 ms ± 17.6 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
    
    $ http = urllib3.PoolManager()
    $ %timeit response = http.request("GET", url)
    The slowest run took 6.87 times longer than the fastest. This could mean that an intermediate result is being cached.
    10.6 ms ± 6.91 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
    
    

    其中还尝试了 session 等方法,貌似也没有变快,上面的是多次的测试,我还做了一次请求的测试,看上去也是很慢

    看了看源码和原理,想象不出来这个速度差来自哪里?

    需求就是请求网址需要快,但不一定需要多次请求

    24 replies  •  2023-12-04 11:34:27 +08:00
    annoygaga
    annoygaga
    OP
    Nov 26, 2023
    1
    这里也试了带 json 数据的 post 方法,结果也是一样的
    WildCat
    WildCat
    Nov 26, 2023
    2
    感觉 urllib3.PoolManager() 是一个 connection pool ?
    有 re-use 现成的 TCP 链接
    ysc3839
    ysc3839
    Nov 26, 2023 via Android
    3
    控制变量都没做好,requests 是直接 get 的,utllib3 怎么就用了 PoolManager ?
    annoygaga
    annoygaga
    OP
    Nov 26, 2023
    4
    @WildCat 我试过 session ,也是一样的结果
    annoygaga
    annoygaga
    OP
    Nov 26, 2023
    5
    @ysc3839 我试过 session 和 httpx 的 withclient ,也是一样的结果
    annoygaga
    annoygaga
    OP
    Nov 26, 2023
    6
    而且单次请求的速度,试了几次,也是 urllib3 快一些
    annoygaga
    annoygaga
    OP
    Nov 26, 2023
    7
    ```
    %timeit r = sess.get(url)
    The slowest run took 16.83 times longer than the fastest. This could mean that an intermediate result is being cached.
    20.8 ms ± 21.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
    ```

    补一下 session 的结果,会快一些

    但是我的需求其实是单次请求快(不求复用链接。。。
    009694
    009694
    Nov 26, 2023
    8
    requests 是 urllib3 的封装。 你怎么可能期望更高层的封装比底层调用快?
    annoygaga
    annoygaga
    OP
    Nov 26, 2023
    9
    @009694 我看了源码,我也不解呀
    justFxxk2060
    justFxxk2060
    Nov 26, 2023
    10
    import requests
    session = requests.Session()
    url = "https://httpbin.org"
    %timeit session.get(url)

    这样试试

    @ysc3839 是对的
    justFxxk2060
    justFxxk2060
    Nov 26, 2023
    11
    Session 单次虽然没有意义,但是也能复用底层 TCP 连接

    比如你先运行
    http = urllib3.PoolManager()
    然后运行
    session.get(url)

    指不定也会更快一些
    zeusho871
    zeusho871
    Nov 27, 2023
    12
    119 117 10.6ms?相差 10 倍 这么离谱的吗
    F281M6Dh8DXpD1g2
    F281M6Dh8DXpD1g2
    Nov 27, 2023
    13
    "The slowest run took 6.87 times longer than the fastest. This could mean that an intermediate result is being cached."
    这几行字是看不见么?
    annoygaga
    annoygaga
    OP
    Nov 27, 2023
    14
    @zeusho871 你可以试试
    annoygaga
    annoygaga
    OP
    Nov 27, 2023
    15
    @liprais 我已经在上面回复关于链接池的测试了,而且我的需求是一次请求,非多次,一次请求下也是类似的效果
    u823tg
    u823tg
    Nov 27, 2023
    16
    不都用 httpx 了吗还是我好久没写 python 了,又转回到 requests 风向了
    annoygaga
    annoygaga
    OP
    Nov 27, 2023
    17
    @u823tg httpx 也测了,参考上面,with client 下和 requests 的 session 差不多
    mengzhuo
    mengzhuo
    Nov 27, 2023
    18
    大概率是 cache ,每次请求带个随机数参数试试?
    linhua
    linhua
    Nov 27, 2023
    19
    单次时间,那就不要用 timeit 测量,timeit 是 多次请求 平均的时间。
    annoygaga
    annoygaga
    OP
    Nov 27, 2023
    21
    @mengzhuo 也是 urllib3 快,你可以试试
    annoygaga
    annoygaga
    OP
    Nov 27, 2023
    22
    @linhua 我测算了单次的速度,也是 urllib3 块
    louisxxx
    louisxxx
    Nov 28, 2023 via iPhone
    23
    本身如此啊,你得为方便牺牲性能
    bbxiong
    bbxiong
    Dec 4, 2023
    24
    关注
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Privacy   ·   Solana   ·   814 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 46ms · UTC 19:05 · PVG 03:05 · LAX 11:05 · JFK 14:05
    ♥ Do have faith in what you're doing.