commit | 2813d1873947bdb73c3fc0dec372b5ea7e36baa6 | [log] [tgz] |
---|---|---|
author | Jay Doane <jaydoane@apache.org> | Mon Apr 08 23:32:52 2019 -0700 |
committer | Robert Newson <robert.newson@uk.ibm.com> | Fri Oct 09 18:22:09 2020 +0100 |
tree | ae854136553e15e4b068352f8edbc3a0af824d2d | |
parent | e6a0c366fc8fc982d7981087c1e9d1e28e2a235a [diff] |
Get a new lb pid if existing pid is not alive It's possible for the connection process associated with a pid in the ibrowse_lb ets table to die, yet remain in the table, in which case subsequent requests to the corresponding {Host, Port} will result in an error like the following: (node1@127.0.0.1)9> ibrowse:send_req("http://localhost:15984", [], get). ** exception exit: {noproc, {gen_server,call, [<0.2451.0>, {spawn_connection, {url,"http://localhost:15984","localhost",15984, undefined,undefined,"/",http,hostname}, 10,10, {[],false}, []}]}} in function gen_server:call/2 (gen_server.erl, line 215) in call from ibrowse:try_routing_request/14 (src/ibrowse.erl, line 377) This checks whether the pid about to be returned from the table is alive, and if not, the entry is deleted, and a new pid is obtained.
ibrowse is a HTTP client written in erlang.
License: ibrowse is available under two different licenses. LGPL or the BSD license.
Comments to: chandrashekhar.mullaparthi@gmail.com
Current Version: 4.4.2
Latest Version: git://github.com/cmullaparthi/ibrowse.git
Remember to start ibrowse first:
5> ibrowse:start(). {ok,<0.94.0>}
A simple GET
request:
6> ibrowse:send_req("http://intranet/messenger/", [], get). {ok,"200", [{"Server","Microsoft-IIS/5.0"}, {"Content-Location","http://intranet/messenger/index.html"}, {"Date","Fri, 17 Dec 2004 15:16:19 GMT"}, {"Content-Type","text/html"}, {"Accept-Ranges","bytes"}, {"Last-Modified","Fri, 17 Dec 2004 08:38:21 GMT"}, {"Etag","\"aa7c9dc313e4c41:d77\""}, {"Content-Length","953"}], "<html>...</html>"}
A GET
using a proxy:
7> ibrowse:send_req("http://www.google.com/", [], get, [], [{proxy_user, "XXXXX"}, {proxy_password, "XXXXX"}, {proxy_host, "proxy"}, {proxy_port, 8080}], 1000). {ok,"302", [{"Date","Fri, 17 Dec 2004 15:22:56 GMT"}, {"Content-Length","217"}, {"Content-Type","text/html"}, {"Set-Cookie", "PREF=ID=f58155c797f9..."}, {"Server","GWS/2.1"}, {"Location", "http://www.google.co.uk/cxfer?c=PREF%3D:TM%3D110329..."}, {"Via","1.1 netapp01 (NetCache NetApp/5.5R2)"}], "<HTML>...</HTML>\r\n"}
A GET
response saved to file. A temporary file is created and the filename returned. The response will only be saved to file if the status code is in the 200
range. The directory to download to can be set using the application env var download_dir
- the default is the current working directory:
8> ibrowse:send_req("http://www.erlang.se/", [], get, [], [{proxy_user, "XXXXX"}, {proxy_password, "XXXXX"}, {proxy_host, "proxy"}, {proxy_port, 8080}, {save_response_to_file, true}], 1000). {error,req_timedout} 9> ibrowse:send_req("http://www.erlang.se/", [], get, [], [{proxy_user, "XXXXX"}, {proxy_password, "XXXXX"}, {proxy_host, "proxy"}, {proxy_port, 8080}, {save_response_to_file, true}], 5000). {ok,"200", [{"Transfer-Encoding","chunked"}, {"Date","Fri, 17 Dec 2004 15:24:36 GMT"}, {"Content-Type","text/html"}, {"Server","Apache/1.3.9 (Unix)"}, {"Via","1.1 netapp01 (NetCache NetApp/5.5R2)"}], {file,"/Users/chandru/code/ibrowse/src/ibrowse_tmp_file_1103297041125854"}}
Setting the size of the connection pool and pipeline. This sets the number of maximum connections to the specified server to 10
and the pipeline size to 1
. Connections are assumed to be already setup.
11> ibrowse:set_dest("www.hotmail.com", 80, [{max_sessions, 10}, {max_pipeline_size, 1}]). ok
Example using the HEAD
method:
56> ibrowse:send_req("http://www.erlang.org", [], head). {ok,"200", [{"Date","Mon, 28 Feb 2005 04:40:53 GMT"}, {"Server","Apache/1.3.9 (Unix)"}, {"Last-Modified","Thu, 10 Feb 2005 09:31:23 GMT"}, {"Etag","\"8d71d-1efa-420b29eb\""}, {"Accept-ranges","bytes"}, {"Content-Length","7930"}, {"Content-Type","text/html"}], []}
Example using the OPTIONS
method:
62> ibrowse:send_req("http://www.sun.com", [], options). {ok,"200", [{"Server","Sun Java System Web Server 6.1"}, {"Date","Mon, 28 Feb 2005 04:44:39 GMT"}, {"Content-Length","0"}, {"P3p", "policyref=\"http://www.sun.com/p3p/Sun_P3P_Policy.xml\", CP=\"CAO DSP COR CUR ADMa DEVa TAIa PSAa PSDa CONi TELi OUR SAMi PUBi IND PHY ONL PUR COM NAV INT DEM CNT STA POL PRE GOV\""}, {"Set-Cookie", "SUN_ID=X.X.X.X:169191109565879; EXPIRES=Wednesday, 31-Dec-2025 23:59:59 GMT; DOMAIN=.sun.com; PATH=/"}, {"Allow", "HEAD, GET, PUT, POST, DELETE, TRACE, OPTIONS, MOVE, INDEX, MKDIR, RMDIR"}], []}
Example of an asynchronous GET
request:
18> ibrowse:send_req("http://www.google.com", [], get, [], [{proxy_user, "XXXXX"}, {proxy_password, "XXXXX"}, {proxy_host, "proxy"}, {proxy_port, 8080}, {stream_to, self()}]). {ibrowse_req_id,{1115,327256,389608}} 19> flush(). Shell got {ibrowse_async_headers,{1115,327256,389608}, "302", [{"Date","Thu, 05 May 2005 21:06:41 GMT"}, {"Content-Length","217"}, {"Content-Type","text/html"}, {"Set-Cookie", "PREF=ID=b601f16bfa32f071:CR=1:TM=1115327201:LM=1115327201:S=OX5hSB525AMjUUu7; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com"}, {"Server","GWS/2.1"}, {"Location", "http://www.google.co.uk/cxfer?c=PREF%3D:TM%3D1115327201:S%3DDS9pDJ4IHcAuZ_AS&prev=/"}, {"Via", "1.1 hatproxy01 (NetCache NetApp/5.6.2)"}]} Shell got {ibrowse_async_response,{1115,327256,389608}, "<HTML>...</HTML>\r\n"} Shell got {ibrowse_async_response_end,{1115,327256,389608}} ok
Another asynchronous GET
request:
24> ibrowse:send_req("http://yaws.hyber.org/simple_ex2.yaws", [], get, [], [{proxy_user, "XXXXX"}, {proxy_password, "XXXXX"}, {proxy_host, "proxy"}, {proxy_port, 8080}, {stream_to, self()}]). {ibrowse_req_id,{1115,327430,512314}} 25> flush(). Shell got {ibrowse_async_headers,{1115,327430,512314}, "200", [{"Date","Thu, 05 May 2005 20:58:08 GMT"}, {"Content-Length","64"}, {"Content-Type","text/html;charset="}, {"Server", "Yaws/1.54 Yet Another Web Server"}, {"Via", "1.1 hatproxy01 (NetCache NetApp/5.6.2)"}]} Shell got {ibrowse_async_response,{1115,327430,512314}, "<html>...</html>\n"} Shell got {ibrowse_async_response_end,{1115,327430,512314}}
Example of request which fails when using the async option. Here the {ibrowse_req_id, ReqId}
is not returned. Instead the error code is returned.
68> ibrowse:send_req("http://www.earlyriser.org", [], get, [], [{stream_to, self()}]). {error,conn_failed}
Example of request using both Proxy-Authorization and authorization by the final webserver:
17> ibrowse:send_req("http://www.erlang.se/lic_area/protected/patches/erl_756_otp_beam.README", [], get, [], [{proxy_user, "XXXXX"}, {proxy_password, "XXXXX"}, {proxy_host, "proxy"}, {proxy_port, 8080}, {basic_auth, {"XXXXX", "XXXXXX"}}]). {ok,"200", [{"Accept-Ranges","bytes"}, {"Date","Thu, 05 May 2005 21:02:09 GMT"}, {"Content-Length","2088"}, {"Content-Type","text/plain"}, {"Server","Apache/1.3.9 (Unix)"}, {"Last-Modified","Tue, 03 May 2005 15:08:18 GMT"}, {"ETag","\"1384c8-828-427793e2\""}, {"Via","1.1 hatproxy01 (NetCache NetApp/5.6.2)"}], "Patch Id:\t\terl_756_otp_beam\n..."}
Example of a TRACE
request. Very interesting! yaws.hyber.org didn't support this. Nor did www.google.com. But good old BBC supports this:
37> ibrowse:send_req("http://www.bbc.co.uk/", [], trace, [], [{proxy_user, "XXXXX"}, {proxy_password, "XXXXX"}, {proxy_host, "proxy"}, {proxy_port, 8080}]). {ok,"200", [{"Transfer-Encoding","chunked"}, {"Date","Thu, 05 May 2005 21:40:27 GMT"}, {"Content-Type","message/http"}, {"Server","Apache/2.0.51 (Unix)"}, {"Set-Cookie", "BBC-UID=7452e72a..."}, {"Set-Cookie", "BBC-UID=7452e72a..."}, {"Via","1.1 hatproxy01 (NetCache NetApp/5.6.2)"}], "TRACE / HTTP/1.1\r\nHost: www.bbc.co.uk\r\nConnection: keep-alive\r\nX-Forwarded-For: 172.24.28.29\r\nVia: 1.1 hatproxy01 (NetCache NetApp/5.6.2)\r\nCookie: BBC-UID=7452e...\r\n\r\n"}
A GET
using a socks5:
ibrowse:send_req("http://google.com", [], get, [], [{socks5_host, "127.0.0.1"}, {socks5_port, 5335}]). ibrowse:send_req("http://google.com", [], get, [], [{socks5_host, "127.0.0.1"}, {socks5_port, 5335}, {socks5_user, "user4321"}, {socks5_password, "pass7654"}]).