Python之Requests编程
1.Requests是什么
Requests是用python语言基于urllib编写的，采用的是Apache2 Licensed开源协议的HTTP库 如果你看过上篇文章关于urllib库的使用，你会发现，其实urllib还是非常不方便的，
而Requests它会比urllib更加方便，可以节约我们大量的工作。（用了requests之后，你基本都不愿意用urllib了）一句话，requests是python实现的最简单易用的HTTP库，建议
爬虫使用requests库。 默认安装好python之后，是没有安装requests模块的，需要单独通过pip3安装
2.Requests安装
pip3 install requests
3.Requests公共方法
在使用requests时，无论我们使用的请求方法是get也好，post也罢，都会给我们返回一个响应的对象，而针对这个对象有一些特定的方法，能够让你进一步解析响应从而得到你想要的数据，如下：
response.json() # 以json的形式返回响应内容，对象格式为dict response.content # 以二进制的形式返回响应内容，对象格式为bytes response.text # 以字符串的形式返回响应内容，对象格式为str response.url # 返回请求的url response.status_code # 返回本次请求的状态码 response.reason # 返回状态码对应的原因 response.headers # 返回响应头 response.cookies # 返回cookice信息 response.raw # 返回原始响应体 response.encoding # 返回编码格式
响应数据?般使?response.text来获取，如果响应数据是JSON格式的，那么使?response.json()
4.Requests的基本应用
4.1GET请求方法
4.1.1不带参数的GET请求
import requests r=requests.get(url='http://47.95.142.233:8000/login/auth/') #输出协议状态码 print("协议状态码:",r.status_code) #输出响应数据 print("返回字符串:",r.text) print("返回二进制的内容:",r.content) print("返回的是字典数据类型:",r.json()) #输出响应头 print("响应头:",r.headers) #输出请求地址 print("请求地址:",r.url)
输出结果：
协议状态码: 200 返回字符串: {"status":0,"msg":"ok"} 返回二进制的内容: b'{"status":0,"msg":"ok"}' 返回的是字典数据类型: {'status': 0, 'msg': 'ok'} 响应头: {'Date': 'Wed, 19 Jan 2022 13:43:23 GMT', 'Server': 'WSGIServer/0.2 CPython/3.7.1', 'Content-Type': 'application/json', 'Vary': 'Accept, Origin, Cookie', 'Allow': 'GET, POST, HEAD, OPTIONS', 'X-Frame-Options': 'DENY', 'Content-Length': '23', 'X-Content-Type-Options': 'nosniff', 'Referrer-Policy': 'same-origin'} 请求地址: http://47.95.142.233:8000/login/auth/
4.1.2带参数的GET请求
import requests r=requests.get( url="http://ws.webxml.com.cn/WebServices/MobileCodeWS.asmx/getMobileCodeInfo", params={"mobileCode":"15227132986","userID":""} ) print("请求地址:",r.url) print("协议状态码:",r.status_code) print("响应数据:",r.text) print("请求头：",r.headers)
输出结果：
请求地址: http://ws.webxml.com.cn/WebServices/MobileCodeWS.asmx/getMobileCodeInfo?mobileCode=15227132986&userID= 协议状态码: 200 响应数据: <?xml version="1.0" encoding="utf-8"?>"http://WebXml.com.cn/">15227132986：河北 石家庄 河北移动全球通卡 请求头： {'Cache-Control': 'private, max-age=0', 'Content-Type': 'text/xml; charset=utf-8', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding', 'Server': 'Microsoft-IIS/7.5', 'X-AspNet-Version': '2.0.50727', 'X-Powered-By': 'ASP.NET', 'Date': 'Wed, 19 Jan 2022 13:43:27 GMT', 'Content-Length': '276'}
4.2POST请求方法
请求参数分为data和json
什么时候使?json？
当请求参数是JSON的时候使?json的参数
当请求参数是json，但是要使?data的参数，那么请求参数要进?序列化的处理
什么时候使?data？
当请求参数是表单的时候使?data
当请求参数是JSON格式的时候，那么请求参数要进?序列化的处理
params和data两个参数的区别？ 1. 发送get请求时，由于get请求没有请求体，请求参数只能跟在url地址后的，而且服务器也只能通过解析url获得请求的参数，因此get()方法发送get请求时只能使用params参数，它会把请求的参数默认追加到url地址后面 2. 通常情况下用户需要提交某些数据时，发送的请求一般都为post请求，post请求会提交一个form表单，那么我们就可以构造一个字典格式的数据，使用data参数传递，由于post请求是有请求体的，而且请求参数就存放在请求
体中，服务器也只能通过解析请求体中内容而获得请求的参数，所以post请求不能使用params传递接口参数，只能使用data，json，file等, data参数会把请求参数放到请求体中
4.2.1请求参数为JSON格式
import requests r=requests.post( url="http://47.95.142.233:8000/login/auth/", json={"username":"******","password":"*****"}, headers={"content-type":"application/json"} ) print(json.dumps(r.json(),indent=True,ensure_ascii=False))
输出结果：
{ "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyaWQiOiJRZVh2MlJLZFE4eVF1eWIyNFdSbVY5IiwiZXhwIjoxNjQyNjg2Nzk1fQ.2Ivo0snIc5n5btGoMCy5XMRtxbetc9WkUq1T6t_GYW0", "user": { "uid": "QeXv2RKdQ8yQuyb24WRmV9", "telephone": "*******", "username": "特昂糖", "email": null, "avator": "", "date_joined": "2022-01-09T11:12:25.772118+08:00", "is_active": true } }
4.2.2请求参数为表单格式
4.2.2.1简单的表单数据格式
import requests r=requests.post( url="http://ws.webxml.com.cn/WebServices/MobileCodeWS.asmx/getMobileCodeInfo", data={"mobileCode":"15117132986","userID":""}, headers={"Content-Type": "application/x-www-form-urlencoded"}) print("请求地址:",r.url) print("协议状态码:",r.status_code) print("响应数据:",r.text) print("请求头：",r.headers)
输出结果：
请求地址: http://ws.webxml.com.cn/WebServices/MobileCodeWS.asmx/getMobileCodeInfo 协议状态码: 200 响应数据: <?xml version="1.0" encoding="utf-8"?>"http://WebXml.com.cn/">15227132986：河北 石家庄 河北移动全球通卡 请求头： {'Cache-Control': 'private, max-age=0', 'Content-Type': 'text/xml; charset=utf-8', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding', 'Server': 'Microsoft-IIS/7.5', 'X-AspNet-Version': '2.0.50727', 'X-Powered-By': 'ASP.NET', 'Date': 'Wed, 19 Jan 2022 14:02:44 GMT', 'Content-Length': '276'}
4.2.2复杂的表单数据格式
import requests r=requests.post( url="https://www.lagou.com/jobs/v2/positionAjax.json", data={ 'data':'iwbM%2B9rJjjjZLZZf5Ay5FfhdbfRqy7iwF7fgWIjdYWFhFrrvE4Exs07vujnDDIyRDWNrUpzXBLetYm2yF%2BDMcfVr2%2FqsXvpfqzc%2BZeh6W2c98Evm%2BkmPI0K4dM%2Bb%2FCbrnlZU374VSXGg3ozLH%2Fyt80VxiJ1KoFbJhYOkfT5k7jC48uSFXkXO2EabccvJMJelMRlv8LrckLtP5V41t4Dgg8el7JbwNUrP%2BB3Me4UpVJM33qCHDNYupNNaz7vKSeY%2B'}, headers={ "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36", "cookie":"RECOMMEND_TIP=true; user_trace_token=20211117101925-1096ffb3-de88-4236-9d5d-c95a70da86a8; LGUID=20211117101925-a1bc98f2-aea5-4d8c-9a6b-67b399c9c2dc; _ga=GA1.2.998045054.1637115565; LG_LOGIN_USER_ID=190b05f6c061b57510c80df1c63fb9ed9c0a8b92feb796b0466e114e8907ab76; LG_HAS_LOGIN=1; JSESSIONID=ABAAAECABIEACCAC3B39946DA9B8D092597B4708E35EBDC; WEBTJ-ID=20220119145206-17e711b5ebd11c6-0589cb7dfcd43e-f791b31-2073600-17e711b5ebebf9; PRE_UTM=; PRE_HOST=; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2F; LGSID=20220119145206-3f757167-28bb-471b-9737-f8159fc78b3d; PRE_SITE=https%3A%2F%2Fwww.lagou.com; _gat=1; LGRID=20220119145206-6ccb9037-ee44-4677-b633-ef71b294fa85; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1641350117,1641371443,1641479206,1642575127; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1642575127; sensorsdata2015session=%7B%7D; _gid=GA1.2.1644369567.1642575127; privacyPolicyPopup=false; index_location_city=%E5%85%A8%E5%9B%BD; X_HTTP_TOKEN=11f935550cdc0cbf6315752461de168e2c7f9b53f9; X_MIDDLE_TOKEN=605e0097ee8113cc90034447d5bfe984; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%2222603975%22%2C%22first_id%22%3A%2217d2bb113615b9-02ad440ad4e4b5-b7a1438-2073600-17d2bb11362da6%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%2C%22%24os%22%3A%22Windows%22%2C%22%24browser%22%3A%22Chrome%22%2C%22%24browser_version%22%3A%2297.0.4692.71%22%7D%2C%22%24device_id%22%3A%2217d2bb113615b9-02ad440ad4e4b5-b7a1438-2073600-17d2bb11362da6%22%7D", "referer":"https://www.lagou.com/wn/jobs?labelWords=sug&fromSearch=true&suginput=%E8%87%AA%E5%8A%A8%E5%8C%96%E6%B5%8B%E8%AF%95&kd=%25E8%2587%25AA%25E5%258A%25A8%25E5%258C%2596%25E6%25B5%258B%25E8%25AF%2595%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588", "conten-type":"application/x-www-form-urlencoded; charset=UTF-8" }) print(r.status_code) print(json.dumps(r.json(),indent=True,ensure_ascii=False))
5.动态参数解决方法
在Python中，我们使用的解决方法是return返回值的思想，比如我们登录产品或系统拿到返会的token，我们进行下一步操作要提供这个token
import requests def login(): r=requests.post( url="http://47.95.142.233:8000/login/auth/", json={"username":"15117132986","password":"123456"}, headers={"content-type":"application/json"} ) return r.json()["token"] def index(): r=requests.get( url="http://47.95.142.233:8000/interface/index", headers={"Authorization":"JWT {0}".format(login())}) print(r.text) assert r.json()["count"]["api"]==4 index()
6.Requests常用的参数详解
def request(method, url, **kwargs): """Constructs and sends a :class:`Request`. :param method: method for the new :class:`Request` object: ``GET``, ``OPTIONS``, ``HEAD``, ``POST``, ``PUT``, ``PATCH``, or ``DELETE``. :param url: URL for the new :class:`Request` object. :param params: (optional) Dictionary, list of tuples or bytes to send in the query string for the :class:`Request`. :param data: (optional) Dictionary, list of tuples, bytes, or file-like object to send in the body of the :class:`Request`. :param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`. :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`. :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`. :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload. ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')`` or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content-type'`` is a string defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers to add for the file. :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth. :param timeout: (optional) How many seconds to wait for the server to send data before giving up, as a float, or a :ref:`(connect timeout, read timeout) ` tuple. :type timeout: float or tuple :param allow_redirects: (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to ``True``. :type allow_redirects: bool :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy. :param verify: (optional) Either a boolean, in which case it controls whether we verify the server's TLS certificate, or a string, in which case it must be a path to a CA bundle to use. Defaults to ``True``. :param stream: (optional) if ``False``, the response content will be immediately downloaded. :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
6.1 headers
??填写请求头的信息，请求头的信息都是字典的数据类型，如以下：headers={ "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36", "cookie":"RECOMMEND_TIP=true; user_trace_token=20211117101925-1096ffb3-de88-4236-9d5d-c95a70da86a8; LGUID=20211117101925-a1bc98f2-aea5-4d8c-9a6b-67b399c9c2dc; _ga=GA1.2.998045054.1637115565; LG_LOGIN_USER_ID=190b05f6c061b57510c80df1c63fb9ed9c0a8b92feb796b0466e114e8907ab76; LG_HAS_LOGIN=1; JSESSIONID=ABAAAECABIEACCAC3B39946DA9B8D092597B4708E35EBDC; WEBTJ-ID=20220119145206-17e711b5ebd11c6-0589cb7dfcd43e-f791b31-2073600-17e711b5ebebf9; PRE_UTM=; PRE_HOST=; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2F; LGSID=20220119145206-3f757167-28bb-471b-9737-f8159fc78b3d; PRE_SITE=https%3A%2F%2Fwww.lagou.com; _gat=1; LGRID=20220119145206-6ccb9037-ee44-4677-b633-ef71b294fa85; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1641350117,1641371443,1641479206,1642575127; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1642575127; sensorsdata2015session=%7B%7D; _gid=GA1.2.1644369567.1642575127; privacyPolicyPopup=false; index_location_city=%E5%85%A8%E5%9B%BD; X_HTTP_TOKEN=11f935550cdc0cbf6315752461de168e2c7f9b53f9; X_MIDDLE_TOKEN=605e0097ee8113cc90034447d5bfe984; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%2222603975%22%2C%22first_id%22%3A%2217d2bb113615b9-02ad440ad4e4b5-b7a1438-2073600-17d2bb11362da6%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%2C%22%24os%22%3A%22Windows%22%2C%22%24browser%22%3A%22Chrome%22%2C%22%24browser_version%22%3A%2297.0.4692.71%22%7D%2C%22%24device_id%22%3A%2217d2bb113615b9-02ad440ad4e4b5-b7a1438-2073600-17d2bb11362da6%22%7D", "referer":"https://www.lagou.com/wn/jobs?labelWords=sug&fromSearch=true&suginput=%E8%87%AA%E5%8A%A8%E5%8C%96%E6%B5%8B%E8%AF%95&kd=%25E8%2587%25AA%25E5%258A%25A8%25E5%258C%2596%25E6%25B5%258B%25E8%25AF%2595%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588", "conten-type":"application/x-www-form-urlencoded; charset=UTF-8" }
6.2 cookies
在请求头??需要带上服务端的响应头中返回来的set-cookie的值import requests def login(): r=requests.post( url="https://home.51cto.com/index?reback=https%3A%2F%2Fwww.51cto.com%2F&iframe=0&is_go_to_user_set_mobile=1", data={"_csrf":"TG9wWTd4bTB9CxJrQjtYXjtfRTNeTDpVCAwcNkZPJXceBRhrBDEpZQ==", "LoginForm[username]":"******", "LoginForm[password]":"*******", "show_qr":0}, headers={"Content-Type":"application/x-www-form-urlencoded", "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36", "Referer":"https//home.51cto.com/index", "Cookie":"_ga=GA1.1.284464699.1636946048; gr_user_id=1c6ca82c-b1b9-4da8-a0f7-1d826248a570; ssxmod_itna=YqRhAK0KGIx+OxIxBcR=1E7+jbD7FDUgDD53+N4QD/CtIDnqD=GFDK40EESpRXoq/QB0vb+7am+q441OlqLIe72OiRetfKxBPDHxY=DUPO0mbTDeWtD5xGoDPxDeDADYo2DAqiOD7qDdRh5sZr5Dbxi3RxiaDGeDeo5kDY5DhxDCuDKDwx0CRZY7TUyIgP2wtBGCD7HvDlcxU/hCH/8jseBwvI8=qY4DUYnXtl25VbiKpGYdjx0k=q0OZ6vG+6gXc=AsYKn4jQ04TDnDA1GdrBio1l2KhvD5YTDqWQie5BdqPG2/HlxDirjdmDQexxD=; ssxmod_itna2=YqRhAK0KGIx+OxIxBcR=1E7+jbD7FDUgDD53+N4G9F5bnDBLmYx7P5vwYrkwbre+g7G=D6BAHI7dtzDvYI5X6n2rL5/eLYbkrF8eWreF6iQHjvhaIifrdi59u7ofroCm5U9t05/Ob2ByDpjHQupWCtl2rq07GCcem5oOkQ9aGHFe7FZ7WK6OCPrc58aWeK+fmramv2CoGr1oaFzPaplSrPfTUKPkXM/oHV/KpKFC2x8msSCQ+U+WhKPhmB+efGbcx2DfsCeycghvRqnmrx1l+gPQytY143Zg=GKNyfP2Icbv+529FHo3k0AjlX=PPcpwBneDcIk0UTOYYtoMQeX7A5OwqBGCpXCZdKaNQhfp0AeWAYj6bUE6cPreR+Gt0neBxUIYx0aAxQR9WQ3UN+DUvBNmlA18XQ4te8PpZe+h3KR62=RNpup86r9rigPlehYmd+C3X9P1RIN6WkjDhAcEILfctXlxLWDczFyZcBSP7lNBaAPi0pStq6nNCwHYdsGDkeeS8q3RYqQ3fa2poBg1DqtejxvpiAwmBDgD=T2iTofQGaDG24=qBuw/0YT4a9G5KRbo0bon=ex8iB62rUrMLL=tl=rfw=3hFW5NF8HwDw8iAPb4d5GRGYQBKCDDFqD+EpNY+PjD8lfn3L=lfHl3WfzCZ8dca9qzCOlq3+YYD===; _ga_WVMQZ6MDKJ=GS1.1.1636946048.1.1.1636946565.0; www51cto=7542F7BAF18A5D73ABBB74E69A92AF49CazY; _uab_collina=164205853999128799574466; _ourplusFirstTime=122-1-13-15-22-20; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%2217e5250e5332de-0b88265ed2dbf8-f791b31-2073600-17e5250e5348ee%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%7D%2C%22%24device_id%22%3A%2217e5250e5332de-0b88265ed2dbf8-f791b31-2073600-17e5250e5348ee%22%7D; pub_cookietime=2592000; once_p=ba1831; pub_wechatopen=aG0wVVNRBVcCBgcEUg; Hm_lvt_110fc9b2e1cae4d110b7959ee4f27e3b=1642058538,1642512417,1642581243; acw_tc=276077b616425812449616986eb2391c039194a363028f3b99d72db4f54af1; PHPSESSID=7cies2j5gdrd05tvjti3a0lfj1; Hm_lpvt_110fc9b2e1cae4d110b7959ee4f27e3b=1642581246; _csrf=fbb4d59e04c7a11e0cd0aa9f7a9814ef26a7e7655b5544bb7017baf469a3bd0ba%3A2%3A%7Bi%3A0%3Bs%3A5%3A%22_csrf%22%3Bi%3A1%3Bs%3A32%3A%221db2uC5nw05ji4WeDcloq7HGRjh23IDU%22%3B%7D; _ourplusReturnCount=31; _ourplusReturnTime=122-1-19-16-34-11; login_from=www.51cto.com; reg_from=www.51cto.com; Hm_lvt_844390da7774b6a92b34d40f8e16f5ac=1642058597,1642512428,1642512534,1642581252; Hm_lpvt_844390da7774b6a92b34d40f8e16f5ac=1642581252"}) return r.cookies def profile(): r=requests.get( url="https://edu.51cto.com/center/user/info/get-user-info", #通过请求头??的cookie把返回来的sessionID发送给服务端进?身份验证 cookies=login() ) print(r.status_code) print(r.text)
profile()
6.3 files
?件上传的参数6.4 auth
权限验证import requests from requests.auth import HTTPBasicAuth
r=requests.get( url='http://101.43.158.84:5000/v1/api/books', auth=HTTPBasicAuth(username="admin",password='admin')) print(r.status_code) print(r.json())
6.5 timeout
主要指的是服务端响应数据很慢，但有时候很快，所以这个时候建议加timeout还有?种就是当?络请求出现requests.exceptions.ConnectTimeout的错误时候，那么就意味着该参数需要上场了import requests from requests.auth import HTTPBasicAuth r=requests.get( url='http://101.43.158.84:5000/v1/api/books', auth=HTTPBasicAuth(username="admin",password='admin'), timeout=180)
6.6verify
当我们的代码没任何的问题，但是请求就是返回ssl 以及403的问题，那么就意味着需要设置verify的参数 如返回错误：requests.exceptions.SSLError。那么就需要带上该参数import requests from requests.auth import HTTPBasicAuth r=requests.get( url='https://101.43.158.84:5000/v1/api/books', auth=HTTPBasicAuth(username="admin",password='admin'), timeout=180,verify=False) print(r.status_code) print(r.json())