|
| 1 | +.. WebOb Toolkit documentation master file, created by |
| 2 | + sphinx-quickstart on Sat Mar 8 09:00:00 2014. |
| 3 | + You can adapt this file completely to your liking, but it should at least |
| 4 | + contain the root `toctree` directive. |
| 5 | +
|
| 6 | +WebOb Toolkit: Requests for Aliens! |
| 7 | +=================================== |
| 8 | + |
| 9 | +WebOb Toolkit is a tool kit for building HTTP_ clients using WebOb_ |
| 10 | +request and response objects and `wsgi middleware`_. |
| 11 | + |
| 12 | +You may already be familiar with Webob_ request and response objects |
| 13 | +if you have experience with a web framework that uses them such as |
| 14 | +pyramid_ or WebTest_ . |
| 15 | + |
| 16 | + |
| 17 | +Usage |
| 18 | +----- |
| 19 | + |
| 20 | +Here are some examples of how WebOb Toolkit can be used. |
| 21 | + |
| 22 | + |
| 23 | +A Very Simple HTTP Client |
| 24 | +~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 25 | + |
| 26 | +If you didn't know, WebOb's Request object grew a "get_response_" method for |
| 27 | +sending an http request to either a wsgi application, or a url and |
| 28 | +returning the response. :: |
| 29 | + |
| 30 | + >>> from webob import Request |
| 31 | + >>> str(Request.blank("https://google.com").get_response()) |
| 32 | + '301 Moved Permanently\nLocation: https://www.google.com/\nContent-Type: text/html; charset=UTF-8\nDate: Sat, 08 Mar 2014 14:58:59 GMT\nExpires: Mon, 07 Apr 2014 14:58:59 GMT\nCache-Control: public, max-age=2592000\nServer: gws\nContent-Length: 220\nX-XSS-Protection: 1; mode=block\nX-Frame-Options: SAMEORIGIN\nAlternate-Protocol: 443:quic\n\n<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">\n<TITLE>301 Moved</TITLE></HEAD><BODY>\n<H1>301 Moved</H1>\nThe document has moved\n<A HREF="https://www.google.com/">here</A>.\r\n</BODY></HTML>\r\n' |
| 33 | + >>> |
| 34 | + |
| 35 | + |
| 36 | +That is a pretty neat trick, but as http clients go, typically you |
| 37 | +need other functionality like handling `http compression`_ for example |
| 38 | +or `handling cookies`_. |
| 39 | + |
| 40 | +WebOb Toolkit provides this additional functionality as `wsgi |
| 41 | +middleware`_ which allows you to compose your own solutions in much |
| 42 | +the same way as you compose wsgi applications to be used as HTTP_ |
| 43 | +Servers. |
| 44 | + |
| 45 | +Handling Compression |
| 46 | +~~~~~~~~~~~~~~~~~~~~ |
| 47 | + |
| 48 | +According to the `http compression`_ article on wikipedia, an http |
| 49 | +client can request a compressed response by including an |
| 50 | +"Accept-Encoding" header with the request. In WebOb_ you would |
| 51 | +do... :: |
| 52 | + |
| 53 | + >>> from webob import Request |
| 54 | + >>> Request.blank("https://github.com", headers={"Accept-Encoding": "gzip, deflate"}).get_response().headers.get("Content-Encoding", "Content was not encoded") |
| 55 | + 'gzip' |
| 56 | + >>> |
| 57 | + |
| 58 | +As you can see, we requested gzipped content from github, and it |
| 59 | +responded nicely. However, if we were to do anything with the body of |
| 60 | +the response we would have to uncompress it first. So, it seems that |
| 61 | +the rules for compression is to a header with each request and |
| 62 | +uncompress the body of each response if the response includes a |
| 63 | +"Content-Encoding" header. |
| 64 | + |
| 65 | + |
| 66 | +The next example, uses webobtoolkit's decode_filter to handle |
| 67 | +compressed responses. :: |
| 68 | + |
| 69 | + >>> from webob import Request |
| 70 | + >>> from webob.client import send_request_app as app |
| 71 | + >>> from webobtoolkit.filters import decode_filter |
| 72 | + >>> app_gzip = decode_filter(app) |
| 73 | + >>> Request.blank("https://github.com", headers={"Accept-Encoding": "gzip"}).get_response(app).body[:100] |
| 74 | + '\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\xdd[\xdbr\xe4\xc6\x91}\xf7W\x94\x9b\x1b\xa3\xdd0\xd1\xf7\x1b9M:\xe6B\xd9\xb3\xb6%\xda\x1cY\x92\x1d\x0eF5Ph`\x08\xa0 \\H\xf6\xfc\x98\xdf\xf7\xcb|\xb2\xaa\x00\x14\xd0M69Kk\xbdV\x84\xa6\x9b@]\xb32O\x9e\xcc\xac^\xfd\xf2\xfd\xb7\xef>\xfexy\xc1\x82"\x8e\xce\x7f' |
| 75 | + >>> Request.blank("https://github.com", headers={"Accept-Encoding": "gzip"}).get_response(app_gzip).body[:100] |
| 76 | + '<!DOCTYPE html>\n<html>\n <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb# object: http:/' |
| 77 | + >>> |
| 78 | + |
| 79 | + |
| 80 | + |
| 81 | +From the above example you will notice a couple of differences from |
| 82 | +previous examples. Firstly we are importing WebOb's send_request_app_ |
| 83 | +and some `wsgi middleware`_ from webobtoolkit for handling compressed |
| 84 | +responses. We wrap webob's app with the decode_filter to create an app |
| 85 | +that will decompress any response we may encounter. |
| 86 | + |
| 87 | +The first call to github is through webobs app. And as you can see, |
| 88 | +the response is compressed just like we asked. The second call is |
| 89 | +through the new app we created by wrapping webob's app with |
| 90 | +webobtoolkits decode_filter, and as you can see the response has been |
| 91 | +decompressed. |
| 92 | + |
| 93 | +The name "filter" is being used here to differentiate between `wsgi |
| 94 | +middleware`_ which is usually used in the context of servers and `wsgi |
| 95 | +middleware`_ that is intended for use with clients. Filters and |
| 96 | +middleware are identical in regards to how they are implemented. |
| 97 | + |
| 98 | +Here is how the decode_filter is implemented. As you can see, it |
| 99 | +doesn't take much to write a filter. |
| 100 | + |
| 101 | +.. literalinclude:: ../../../webobtoolkit/webobtoolkit/filters.py |
| 102 | + :pyobject: decode_filter |
| 103 | + |
| 104 | + |
| 105 | +A More Robust HTTP Client |
| 106 | +~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 107 | + |
| 108 | +requests_ is a good example of a more useful http client. According to |
| 109 | +the docs it includes a lot of useful things that one would want for in |
| 110 | +an http client. Some of the things it includes are. |
| 111 | + |
| 112 | +* handling compression |
| 113 | +* handling cookies |
| 114 | +* handling redirects |
| 115 | +* guessing, handling charset decoding |
| 116 | +* connection pooling |
| 117 | +* ssl verification |
| 118 | +* form posts |
| 119 | +* file uploads |
| 120 | + |
| 121 | +And probably much more. |
| 122 | + |
| 123 | +A lot of this functionality can be written on top of webob and a |
| 124 | +series of filters. We've already seen how one might handle |
| 125 | +compression. WebOb toolkit provides filters for handling cookies, |
| 126 | +redirects and handling unspecified charsets. :: |
| 127 | + |
| 128 | + >>> from webob import Request |
| 129 | + >>> from webob.client import send_request_app |
| 130 | + >>> from webobtoolkit import filters |
| 131 | + >>> requests_app = filters.auto_redirect_filter(filters.cookie_filter(filters.decode_filter(filters.charset_filter(send_request_app)))) |
| 132 | + >>> Request.blank("https://google.com").get_response(requests_app) |
| 133 | + >>> str(Request.blank("https://google.com").get_response(requests_app))[:500] |
| 134 | + '200 OK\nDate: Sat, 08 Mar 2014 17:35:40 GMT\nExpires: -1\nCache-Control: private, max-age=0\nContent-Type: text/html; charset=ISO-8859-1\nServer: gws\nX-XSS-Protection: 1; mode=block\nX-Frame-Options: SAMEORIGIN\n\n<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"><head><meta content="Search the world\'s information, including webpages, images, videos and more. Google has many special features to help you find exactly what you\'re looking for." name="description"><meta content="noodp" ' |
| 135 | + |
| 136 | + |
| 137 | +We construct requests_app out of a number of filters that for handling requests and responses. |
| 138 | + |
| 139 | +* if a charset is not specified on the response(which sometimes |
| 140 | + happens), a safe default is chosen in order to reduce the cahnces |
| 141 | + for decoding errors |
| 142 | + |
| 143 | +* the client advertise support for gzip encoding and decompress the response if necessary |
| 144 | +* cookies will be persisted for each response and sent for each request |
| 145 | +* if a redirect is encountered follow it automatically. |
| 146 | +* form posts are handled by webob already, though some might prefer a better syntax. |
| 147 | + |
| 148 | + |
| 149 | +Todo |
| 150 | +~~~~ |
| 151 | +* for connection pooling, urllib3 provides an implementation that could be easily used to construct an alternate send_request_app (see: webob.client.SendRequest) |
| 152 | +* ssl verification is also provided by urllib3 |
| 153 | + |
| 154 | + |
| 155 | +.. _webob: http://webob.org |
| 156 | +.. _wsgi: http://wsgi.org |
| 157 | +.. _http: http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol |
| 158 | +.. _pyramid: http://docs.pylonsproject.org/en/latest/ |
| 159 | +.. _webtest: http://webtest.readthedocs.org/en/latest/ |
| 160 | +.. _get_response: http://docs.webob.org/en/latest/modules/webob.html#webob.request.BaseRequest.get_response |
| 161 | +.. _`http compression`: http://en.wikipedia.org/wiki/HTTP_compression |
| 162 | +.. _`handling cookies`: http://en.wikipedia.org/wiki/HTTP_cookie |
| 163 | +.. _`wsgi middleware`: http://be.groovie.org/2005/10/07/wsgi_and_wsgi_middleware_is_easy.html |
| 164 | +.. _send_request_app: http://docs.webob.org/en/latest/modules/client.html#webob.client.send_request_app |
| 165 | +.. _requests: http://docs.python-requests.org/en/latest/ |
0 commit comments