Class: WIKK::WebBrowser
- Inherits:
-
Object
- Object
- WIKK::WebBrowser
- Defined in:
- lib/wikk_webbrowser.rb
Overview
WIKK WebBrowser class under MIT Lic. https://github.com/wikarekare. Wrapper around ruby’s http classes WIKK_WebBrowser.new.https_session(host: ‘www.blah.com’) do |session| response = get_page(query: ,’/’) end
Defined Under Namespace
Classes: Error
Constant Summary collapse
- VERSION =
'0.9.8'
Instance Attribute Summary collapse
-
#cookies ⇒ Object
Returns the value of attribute cookies.
-
#debug ⇒ Object
Returns the value of attribute debug.
-
#host ⇒ Object
readonly
Returns the value of attribute host.
-
#page ⇒ Object
readonly
Returns the value of attribute page.
-
#port ⇒ Object
Returns the value of attribute port.
-
#referer ⇒ Object
Returns the value of attribute referer.
-
#response ⇒ Object
Returns the value of attribute response.
-
#session ⇒ Object
Returns the value of attribute session.
-
#use_ssl ⇒ Object
Returns the value of attribute use_ssl.
-
#verify_cert ⇒ Object
Returns the value of attribute verify_cert.
Class Method Summary collapse
-
.http_session(host:, port: nil, debug: false, cookies: {}) {|the| ... } ⇒ Object
Create a WIKK_WebBrowser instance, connect to the host via http, and yield the WIKK_WebBrowser instance.
-
.https_session(host:, port: nil, verify_cert: true, cookies: {}, debug: false) {|the| ... } ⇒ Object
Create a WIKK_WebBrowser instance, connect to the host via https, and yield the WIKK_WebBrowser instance.
Instance Method Summary collapse
-
#add_cookies(cookies) ⇒ Object
Add additional cookies.
-
#basic_authorization(user:, password:) ⇒ String
Web basic authentication (not exactly secure).
-
#bearer_authorization(token:) ⇒ String
Dropbox style token authentication.
-
#cookies_to_s ⇒ Object
Convert @cookies to ; separated strings.
-
#delete_req(query:, authorization: nil, form_values: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String
(also: #delete)
send a DELETE query to the server and return the response.
-
#extract_input_fields(body) ⇒ Hash
Extract form field values from the html body.
-
#extract_link_fields(body) ⇒ Hash
Extract links from the html body.
-
#form_values_to_s(form_values = nil, has_q = false) ⇒ String
Take a hash of the params to the post and generate a safe URL string.
-
#get_page(query:, form_values: nil, authorization: nil, extra_headers: {}, extra_cookies: {}) ⇒ String
(also: #get)
send a GET query to the web server using an http get, and returns the response.
-
#header_value(key:) ⇒ String
Get a header value, from the last response.
-
#http_session ⇒ Object
Creating a session for http connection attached block would then call get or post NET::HTTP calls.
-
#initialize(host:, port: nil, use_ssl: false, cookies: {}, verify_cert: true, debug: false) ⇒ WIKK_WebBrowser
constructor
Create a WIKK_WebBrowser instance.
-
#patch_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String
(also: #patch)
send a PATCH query to the server and return the response.
-
#post_page(query:, authorization: nil, content_type: 'application/x-www-form-urlencoded', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String
(also: #post)
send a POST query to the server and return the response.
-
#put_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String
(also: #put)
send a PUT query to the server and return the response.
-
#save_cookies(response) ⇒ Object
Save cookies returned by last html get/post.
Constructor Details
#initialize(host:, port: nil, use_ssl: false, cookies: {}, verify_cert: true, debug: false) ⇒ WIKK_WebBrowser
Create a WIKK_WebBrowser instance
47 48 49 50 51 52 53 54 55 |
# File 'lib/wikk_webbrowser.rb', line 47 def initialize(host:, port: nil, use_ssl: false, cookies: {}, verify_cert: true, debug: false) @host = host # Need to do this, as passing nil is different to passing nothing to initialize! @cookies = .nil? ? {} : @debug = debug @use_ssl = use_ssl @port = port.nil? ? ( use_ssl ? 443 : 80 ) : port @verify_cert = verify_cert @response = nil end |
Instance Attribute Details
#cookies ⇒ Object
Returns the value of attribute cookies.
30 31 32 |
# File 'lib/wikk_webbrowser.rb', line 30 def @cookies end |
#debug ⇒ Object
Returns the value of attribute debug.
33 34 35 |
# File 'lib/wikk_webbrowser.rb', line 33 def debug @debug end |
#host ⇒ Object (readonly)
Returns the value of attribute host.
28 29 30 |
# File 'lib/wikk_webbrowser.rb', line 28 def host @host end |
#page ⇒ Object (readonly)
Returns the value of attribute page.
31 32 33 |
# File 'lib/wikk_webbrowser.rb', line 31 def page @page end |
#port ⇒ Object
Returns the value of attribute port.
35 36 37 |
# File 'lib/wikk_webbrowser.rb', line 35 def port @port end |
#referer ⇒ Object
Returns the value of attribute referer.
32 33 34 |
# File 'lib/wikk_webbrowser.rb', line 32 def referer @referer end |
#response ⇒ Object
Returns the value of attribute response.
37 38 39 |
# File 'lib/wikk_webbrowser.rb', line 37 def response @response end |
#session ⇒ Object
Returns the value of attribute session.
29 30 31 |
# File 'lib/wikk_webbrowser.rb', line 29 def session @session end |
#use_ssl ⇒ Object
Returns the value of attribute use_ssl.
36 37 38 |
# File 'lib/wikk_webbrowser.rb', line 36 def use_ssl @use_ssl end |
#verify_cert ⇒ Object
Returns the value of attribute verify_cert.
34 35 36 |
# File 'lib/wikk_webbrowser.rb', line 34 def verify_cert @verify_cert end |
Class Method Details
.http_session(host:, port: nil, debug: false, cookies: {}) {|the| ... } ⇒ Object
Create a WIKK_WebBrowser instance, connect to the host via http, and yield the WIKK_WebBrowser instance. Automatically closes the http session on returning from the block passed to it.
68 69 70 71 72 73 |
# File 'lib/wikk_webbrowser.rb', line 68 def self.http_session(host:, port: nil, debug: false, cookies: {}) wb = self.new(host: host, port: port, debug: debug, use_ssl: false, cookies: ) wb.http_session do yield wb end end |
.https_session(host:, port: nil, verify_cert: true, cookies: {}, debug: false) {|the| ... } ⇒ Object
Create a WIKK_WebBrowser instance, connect to the host via https, and yield the WIKK_WebBrowser instance. Automatically closes the http session on returning from the block passed to it.
82 83 84 85 86 87 |
# File 'lib/wikk_webbrowser.rb', line 82 def self.https_session(host:, port: nil, verify_cert: true, cookies: {}, debug: false) wb = self.new(host: host, port: port, cookies: , use_ssl: true, verify_cert: verify_cert, debug: debug) wb.http_session do yield wb end end |
Instance Method Details
#add_cookies(cookies) ⇒ Object
Add additional cookies
124 125 126 |
# File 'lib/wikk_webbrowser.rb', line 124 def () .each { |, | @cookies[] = } end |
#basic_authorization(user:, password:) ⇒ String
Web basic authentication (not exactly secure)
110 111 112 113 |
# File 'lib/wikk_webbrowser.rb', line 110 def (user:, password:) # req.basic_auth( user, password) if user != nil 'Basic ' + Base64.encode64( "#{user}:#{password}" ) end |
#bearer_authorization(token:) ⇒ String
Dropbox style token authentication
118 119 120 |
# File 'lib/wikk_webbrowser.rb', line 118 def (token:) 'Bearer ' + token end |
#cookies_to_s ⇒ Object
Convert @cookies to ; separated strings
142 143 144 |
# File 'lib/wikk_webbrowser.rb', line 142 def @cookies.to_a.map { |v| v.join('=') }.join('; ') end |
#delete_req(query:, authorization: nil, form_values: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: delete
send a DELETE query to the server and return the response.
284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 |
# File 'lib/wikk_webbrowser.rb', line 284 def delete_req(query:, authorization: nil, form_values: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) $stderr.puts 'Debugging On' if @debug query += form_values_to_s(form_values, query.index('?') != nil) url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query.gsub(/^\//, '')}") req = Net::HTTP::Delete.new(query) header = { 'HOST' => @host } () header['Cookie'] = if @cookies.length > 0 header['Authorization'] = if != nil extra_headers.each do |k, v| header[k] = v end req.initialize_http_header( header ) if data.nil? req.body = '' elsif data.instance_of?(Hash) if content_type =~ /application\/octet-stream/ req.set_form_data(data, '&') else req.set_form_data.to_j end else req.body = data # If json as a string or raw string end begin @response = @session.request(req) (@response) if @response.code.to_i >= 300 raise "#{url} : #{@response.code} #{@response.}" end return @response.body rescue StandardError => e puts "#{e}" return nil end end |
#extract_input_fields(body) ⇒ Hash
Extract form field values from the html body.
455 456 457 458 459 460 461 |
# File 'lib/wikk_webbrowser.rb', line 455 def extract_input_fields(body) @inputs = {} doc = Nokogiri::HTML(body) doc.xpath('//form/input').each do |f| @inputs[f.get_attribute('name')] = f.get_attribute('value') end end |
#extract_link_fields(body) ⇒ Hash
Extract links from the html body.
466 467 468 469 470 471 472 473 |
# File 'lib/wikk_webbrowser.rb', line 466 def extract_link_fields(body) @inputs = {} doc = Nokogiri::HTML(body) doc.xpath('//a').each do |f| return URI.parse( f.get_attribute('href') ).path if f.get_attribute('name') == 'URL$1' end return nil end |
#form_values_to_s(form_values = nil, has_q = false) ⇒ String
Take a hash of the params to the post and generate a safe URL string.
479 480 481 482 483 484 485 486 487 488 489 490 |
# File 'lib/wikk_webbrowser.rb', line 479 def form_values_to_s(form_values = nil, has_q = false) return '' if form_values.nil? || form_values.length == 0 s = (has_q == true ? '' : '?') first = true form_values.each do |key, value| s += '&' unless first s += "#{CGI.escape(key.to_s)}=#{CGI.escape(value.to_s)}" first = false end return s end |
#get_page(query:, form_values: nil, authorization: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: get
send a GET query to the web server using an http get, and returns the response. Cookies in the response get preserved in @cookies, so they will be sent along with subsequent calls We are currently ignoring redirects from the PDU’s we are querying.
163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 |
# File 'lib/wikk_webbrowser.rb', line 163 def get_page(query:, form_values: nil, authorization: nil, extra_headers: {}, extra_cookies: {}) $stderr.puts 'Debugging On' if @debug query += form_values_to_s(form_values, query.index('?') != nil) # Should be using req.set_form_data, but it seems to by stripping the leading / and then the query fails. url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query.gsub(/^\//, '')}") $stderr.puts url if @debug req = Net::HTTP::Get.new(url.request_uri) header = { 'HOST' => @host } header['Accept'] = '*/*' header['Accept-Encoding'] = 'gzip, deflate, br' header['Accept-Language'] = 'en-US,en;q=0.5' header['Connection'] = 'keep-alive' header['User-Agent'] = 'Mozilla/5.0' header['Content-Type'] = 'application/x-www-form-urlencoded' () header['Cookie'] = if @cookies.length > 0 header['DNT'] = '1' header['Authorization'] = if != nil extra_headers.each do |k, v| header[k] = v end req.initialize_http_header( header ) @response = @session.request(req) (@response) $stderr.puts @response.code.to_i if @debug if @response.code.to_i >= 300 if @response.code.to_i == 302 # ignore the redirects. # $stderr.puts "302" # @response.each {|key, val| $stderr.printf "%s = %s\n", key, val } #Location seems to have cgi params removed. End up with .../cginame?& # $stderr.puts "Redirect to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?& # $stderr.puts return elsif @response.code.to_i >= 400 && @response.code.to_i < 500 return @response.body end raise Error.new(web_return_code: @response.code.to_i, message: "#{@response.code} #{@response.} #{query} #{form_values} #{@response.body}") end return @response.body end |
#header_value(key:) ⇒ String
Get a header value, from the last response
150 151 152 |
# File 'lib/wikk_webbrowser.rb', line 150 def header_value(key:) @response.header[key] end |
#http_session ⇒ Object
Creating a session for http connection attached block would then call get or post NET::HTTP calls
95 96 97 98 99 100 101 102 103 104 |
# File 'lib/wikk_webbrowser.rb', line 95 def http_session @http = Net::HTTP.new(@host, @port) @http.set_debug_output($stdout) if @debug @http.use_ssl = @use_ssl @http.verify_mode = OpenSSL::SSL::VERIFY_NONE if ! @use_ssl || ! @verify_cert @http.start do |session| # ensure we close the session after the block @session = session yield end end |
#patch_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: patch
send a PATCH query to the server and return the response.
399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 |
# File 'lib/wikk_webbrowser.rb', line 399 def patch_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query}") req = Net::HTTP::Patch.new(url.path) header = { 'HOST' => @host } header['Accept'] = '*/*' header['Accept-Encoding'] = 'gzip, deflate, br' header['Accept-Language'] = 'en-US,en;q=0.5' header['Connection'] = 'keep-alive' header['User-Agent'] = 'Mozilla/5.0' header['Content-Type'] = content_type () header['Cookie'] = if @cookies.length > 0 header['DNT'] = '1' header['Authorization'] = if != nil extra_headers.each do |k, v| header[k] = v end req.initialize_http_header( header ) if data.nil? req.body = '' elsif data.instance_of?(Hash) if content_type =~ /application\/octet-stream/ req.set_form_data(data, '&') else req.set_form_data.to_j end else req.body = data # If json as a string or raw string end @response = @session.request(req) (@response) if @response.code.to_i >= 300 if @response.code.to_i == 302 # ignore the redirects. # puts "302" # @response.each {|key, val| printf "%s = %s\n", key, val } #Location seems to have cgi params removed. End up with .../cginame?& # puts "Redirect of Post to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?& return end raise Error.new(web_return_code: @response.code, message: "#{@response.code} #{@response.} #{query} #{data} #{@response.body}") end return @response.body end |
#post_page(query:, authorization: nil, content_type: 'application/x-www-form-urlencoded', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: post
send a POST query to the server and return the response.
222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 |
# File 'lib/wikk_webbrowser.rb', line 222 def post_page(query:, authorization: nil, content_type: 'application/x-www-form-urlencoded', data: nil, extra_headers: {}, extra_cookies: {}) url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query}") req = Net::HTTP::Post.new(url.path) header = { 'HOST' => @host } header['Accept'] = '*/*' header['Accept-Encoding'] = 'gzip, deflate, br' header['Accept-Language'] = 'en-US,en;q=0.5' header['Connection'] = 'keep-alive' header['User-Agent'] = 'Mozilla/5.0' header['Content-Type'] = content_type () header['Cookie'] = if @cookies.length > 0 header['DNT'] = '1' header['Authorization'] = if != nil extra_headers.each do |k, v| header[k] = v end req.initialize_http_header( header ) if data.nil? req.body = '' elsif data.instance_of?(String) req.body = data # If we are given a String, just use it as is elsif content_type =~ /application\/json/ req.body = data.to_j # Encode data as json. It wasn't already a String. elsif content_type =~ /application\/octet-stream/ && (data.instance_of?(Hash) || data.instance_of?(Array)) req.set_form_data(data, '&') # Should handle Array as multiple entries with the same key, and Hash else # Assuming data can become a string that makes sense to the other end req.body = data.to_s # raw string end @response = @session.request(req) (@response) if @response.code.to_i >= 300 if @response.code.to_i == 302 # ignore the redirects. # puts "302" # @response.each {|key, val| printf "%s = %s\n", key, val } #Location seems to have cgi params removed. End up with .../cginame?& # puts "Redirect of Post to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?& return end raise Error.new(web_return_code: @response.code, message: "#{@response.code} #{@response.} #{query} #{data} #{@response.body}") end return @response.body end |
#put_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: put
send a PUT query to the server and return the response.
338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 |
# File 'lib/wikk_webbrowser.rb', line 338 def put_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query}") req = Net::HTTP::Put.new(url.path) header = { 'HOST' => @host } header['Accept'] = '*/*' header['Accept-Encoding'] = 'gzip, deflate, br' header['Accept-Language'] = 'en-US,en;q=0.5' header['Connection'] = 'keep-alive' header['User-Agent'] = 'Mozilla/5.0' header['Content-Type'] = content_type () header['Cookie'] = if @cookies.length > 0 header['DNT'] = '1' header['Authorization'] = if != nil extra_headers.each do |k, v| header[k] = v end req.initialize_http_header( header ) if data.nil? req.body = '' elsif data.instance_of?(Hash) if content_type =~ /application\/octet-stream/ req.set_form_data(data, '&') else req.set_form_data.to_j end else req.body = data # If json as a string or raw string end @response = @session.request(req) (@response) if @response.code.to_i >= 300 if @response.code.to_i == 302 # ignore the redirects. # puts "302" # @response.each {|key, val| printf "%s = %s\n", key, val } #Location seems to have cgi params removed. End up with .../cginame?& # puts "Redirect of Post to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?& return end raise Error.new(web_return_code: @response.code, message: "#{@response.code} #{@response.} #{query} #{data} #{@response.body}") end return @response.body end |
#save_cookies(response) ⇒ Object
Save cookies returned by last html get/post. Removes previous cookies.
131 132 133 134 135 136 137 138 |
# File 'lib/wikk_webbrowser.rb', line 131 def (response) if ( = response.get_fields('set-cookie')) != nil .each do || = .split('; ').map { |v| v.split('=') } .each { |c| @cookies[c[0]] = c[1] } end end end |