Class: WIKK::WebBrowser

Inherits:
Object
  • Object
show all
Defined in:
lib/wikk_webbrowser.rb

Overview

WIKK WebBrowser class under MIT Lic. https://github.com/wikarekare. Wrapper around ruby’s http classes WIKK_WebBrowser.new.https_session(host: ‘www.blah.com’) do |session| response = get_page(query: ,’/’) end

Defined Under Namespace

Classes: Error

Constant Summary collapse

VERSION =
'0.9.8'

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(host:, port: nil, use_ssl: false, cookies: {}, verify_cert: true, debug: false) ⇒ WIKK_WebBrowser

Create a WIKK_WebBrowser instance

Parameters:

  • host (String)

    the host we want to connect to

  • port (Fixnum) (defaults to: nil)

    Optional http server port

  • use_ssl (Boolean) (defaults to: false)

    Use https, if true

  • verify_cert (Boolean) (defaults to: true)

    Validate certificate if true (Nb lots of embedded devices have self signed certs, so verify will fail)



47
48
49
50
51
52
53
54
55
# File 'lib/wikk_webbrowser.rb', line 47

def initialize(host:, port: nil, use_ssl: false, cookies: {}, verify_cert: true, debug: false)
  @host = host # Need to do this, as passing nil is different to passing nothing to initialize!
  @cookies = cookies.nil? ? {} : cookies
  @debug = debug
  @use_ssl = use_ssl
  @port = port.nil? ? ( use_ssl ? 443 : 80 ) : port
  @verify_cert = verify_cert
  @response = nil
end

Instance Attribute Details

#cookiesObject

Returns the value of attribute cookies.



30
31
32
# File 'lib/wikk_webbrowser.rb', line 30

def cookies
  @cookies
end

#debugObject

Returns the value of attribute debug.



33
34
35
# File 'lib/wikk_webbrowser.rb', line 33

def debug
  @debug
end

#hostObject (readonly)

Returns the value of attribute host.



28
29
30
# File 'lib/wikk_webbrowser.rb', line 28

def host
  @host
end

#pageObject (readonly)

Returns the value of attribute page.



31
32
33
# File 'lib/wikk_webbrowser.rb', line 31

def page
  @page
end

#portObject

Returns the value of attribute port.



35
36
37
# File 'lib/wikk_webbrowser.rb', line 35

def port
  @port
end

#refererObject

Returns the value of attribute referer.



32
33
34
# File 'lib/wikk_webbrowser.rb', line 32

def referer
  @referer
end

#responseObject

Returns the value of attribute response.



37
38
39
# File 'lib/wikk_webbrowser.rb', line 37

def response
  @response
end

#sessionObject

Returns the value of attribute session.



29
30
31
# File 'lib/wikk_webbrowser.rb', line 29

def session
  @session
end

#use_sslObject

Returns the value of attribute use_ssl.



36
37
38
# File 'lib/wikk_webbrowser.rb', line 36

def use_ssl
  @use_ssl
end

#verify_certObject

Returns the value of attribute verify_cert.



34
35
36
# File 'lib/wikk_webbrowser.rb', line 34

def verify_cert
  @verify_cert
end

Class Method Details

.http_session(host:, port: nil, debug: false, cookies: {}) {|the| ... } ⇒ Object

Create a WIKK_WebBrowser instance, connect to the host via http, and yield the WIKK_WebBrowser instance. Automatically closes the http session on returning from the block passed to it.

Parameters:

  • host (String)

    the host we want to connect to

  • port (Fixnum) (defaults to: nil)

    (80) the port the remote web server is running on

  • block (Proc)

Yield Parameters:

  • the (WIKK_WebBrowser)

    session descriptor for further calls.



68
69
70
71
72
73
# File 'lib/wikk_webbrowser.rb', line 68

def self.http_session(host:, port: nil, debug: false, cookies: {})
  wb = self.new(host: host, port: port, debug: debug, use_ssl: false, cookies: cookies)
  wb.http_session do
    yield wb
  end
end

.https_session(host:, port: nil, verify_cert: true, cookies: {}, debug: false) {|the| ... } ⇒ Object

Create a WIKK_WebBrowser instance, connect to the host via https, and yield the WIKK_WebBrowser instance. Automatically closes the http session on returning from the block passed to it.

Parameters:

  • host (String)

    the host we want to connect to

  • port (Fixnum) (defaults to: nil)

    (443) the port the remote web server is running on

  • verify_cert (Boolean) (defaults to: true)

    Validate certificate if true (Nb lots of embedded devices have self signed certs, so verify will fail)

  • block (Proc)

Yield Parameters:

  • the (WIKK_WebBrowser)

    session descriptor for further calls.



82
83
84
85
86
87
# File 'lib/wikk_webbrowser.rb', line 82

def self.https_session(host:, port: nil, verify_cert: true, cookies: {}, debug: false)
  wb = self.new(host: host, port: port, cookies: cookies, use_ssl: true, verify_cert: verify_cert, debug: debug)
  wb.http_session do
    yield wb
  end
end

Instance Method Details

#add_cookies(cookies) ⇒ Object

Add additional cookies

Parameters:

  • cookies (Hash)

    cookie_name => cookie_value



124
125
126
# File 'lib/wikk_webbrowser.rb', line 124

def add_cookies(cookies)
  cookies.each { |cookie_name, cookie_value| @cookies[cookie_name] = cookie_value }
end

#basic_authorization(user:, password:) ⇒ String

Web basic authentication (not exactly secure)

Parameters:

  • user (String)

    Account name

  • password (String)

    Accounts password

Returns:

  • (String)

    Base64 encoded concatentation of user + ‘:’ + password



110
111
112
113
# File 'lib/wikk_webbrowser.rb', line 110

def basic_authorization(user:, password:)
  # req.basic_auth( user, password) if  user != nil
  'Basic ' + Base64.encode64( "#{user}:#{password}" )
end

#bearer_authorization(token:) ⇒ String

Dropbox style token authentication

Parameters:

  • token (String)

    Token, as issued by dropbox

Returns:

  • (String)

    Concatenation of ‘Bearer ‘ + token



118
119
120
# File 'lib/wikk_webbrowser.rb', line 118

def bearer_authorization(token:)
  'Bearer ' + token
end

#cookies_to_sObject

Convert @cookies to ; separated strings

Returns:

  • cookies string



142
143
144
# File 'lib/wikk_webbrowser.rb', line 142

def cookies_to_s
  @cookies.to_a.map { |v| v.join('=') }.join('; ')
end

#delete_req(query:, authorization: nil, form_values: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: delete

send a DELETE query to the server and return the response.

Parameters:

  • query (String)

    URL, less the ‘http://host/’ part

  • authorization (String) (defaults to: nil)

    If present, add Authorization header, using this string

  • form_values (Hash{String=>Object-with-to_s}) (defaults to: nil)

    The parameter passed to the web server eg. ?key1=value1&key2=value2…

  • content_type (String) (defaults to: '"application/octet-stream"')

    Posted content type. Only meaningful if there is data

  • data (String) (defaults to: nil)

    Text to add to body of DELETE to the web server. Not always supported at the server end.

  • extra_headers (Hash) (defaults to: {})

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {})

    Add these to standard cookies

Returns:

  • (String)

    The Net::HTTPResponse.body text response from the web server



284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
# File 'lib/wikk_webbrowser.rb', line 284

def delete_req(query:, authorization: nil, form_values: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {})
  $stderr.puts 'Debugging On' if @debug

  query += form_values_to_s(form_values, query.index('?') != nil)
  url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query.gsub(/^\//, '')}")
  req = Net::HTTP::Delete.new(query)

  header = { 'HOST' => @host }
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k, v|
    header[k] = v
  end
  req.initialize_http_header( header )

  if data.nil?
    req.body = ''
  elsif data.instance_of?(Hash)
    if content_type =~ /application\/octet-stream/
      req.set_form_data(data, '&')
    else
      req.set_form_data.to_j
    end
  else
    req.body = data # If json as a string or raw string
  end

  begin
    @response = @session.request(req)
    save_cookies(@response)

    if @response.code.to_i >= 300
      raise "#{url} : #{@response.code} #{@response.message}"
    end

    return @response.body
  rescue StandardError => e
    puts "#{e}"
    return nil
  end
end

#extract_input_fields(body) ⇒ Hash

Extract form field values from the html body.

Parameters:

  • body (String)

    The html response body

Returns:

  • (Hash)

    Keys are the field names, values are the field values



455
456
457
458
459
460
461
# File 'lib/wikk_webbrowser.rb', line 455

def extract_input_fields(body)
  @inputs = {}
  doc = Nokogiri::HTML(body)
  doc.xpath('//form/input').each do |f|
    @inputs[f.get_attribute('name')] = f.get_attribute('value')
  end
end

Extract links from the html body.

Parameters:

  • body (String)

    The html response body

Returns:

  • (Hash)

    Keys are the link text, values are the html links



466
467
468
469
470
471
472
473
# File 'lib/wikk_webbrowser.rb', line 466

def extract_link_fields(body)
  @inputs = {}
  doc = Nokogiri::HTML(body)
  doc.xpath('//a').each do |f|
    return URI.parse( f.get_attribute('href') ).path if f.get_attribute('name') == 'URL$1'
  end
  return nil
end

#form_values_to_s(form_values = nil, has_q = false) ⇒ String

Take a hash of the params to the post and generate a safe URL string.

Parameters:

  • form_values (Hash) (defaults to: nil)

    Keys are the field names, values are the field values

  • has_q (Boolean) (defaults to: false)

    We have a leading ? for the html get, so don’t need to add one.

Returns:

  • (String)

    The ‘safe’ text for fields the get or post query to the web server



479
480
481
482
483
484
485
486
487
488
489
490
# File 'lib/wikk_webbrowser.rb', line 479

def form_values_to_s(form_values = nil, has_q = false)
  return '' if form_values.nil? || form_values.length == 0

  s = (has_q == true ? '' : '?')
  first = true
  form_values.each do |key, value|
    s += '&' unless first
    s += "#{CGI.escape(key.to_s)}=#{CGI.escape(value.to_s)}"
    first = false
  end
  return s
end

#get_page(query:, form_values: nil, authorization: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: get

send a GET query to the web server using an http get, and returns the response. Cookies in the response get preserved in @cookies, so they will be sent along with subsequent calls We are currently ignoring redirects from the PDU’s we are querying.

Parameters:

  • query (String)

    The URL after the http://host/ bit and not usually not including parameters, if form_values are passed in

  • form_values (Hash{String=>Object-with-to_s}) (defaults to: nil)

    The parameter passed to the web server eg. ?key1=value1&key2=value2…

  • authorization (String) (defaults to: nil)

    If present, add Authorization header, using this string

  • extra_headers (Hash) (defaults to: {})

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {})

    Add these to standard cookies

Returns:

  • (String)

    The Net::HTTPResponse.body text response from the web server



163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
# File 'lib/wikk_webbrowser.rb', line 163

def get_page(query:, form_values: nil, authorization: nil, extra_headers: {}, extra_cookies: {})
  $stderr.puts 'Debugging On' if @debug
  query += form_values_to_s(form_values, query.index('?') != nil) # Should be using req.set_form_data, but it seems to by stripping the leading / and then the query fails.
  url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query.gsub(/^\//, '')}")
  $stderr.puts url if @debug

  req = Net::HTTP::Get.new(url.request_uri)

  header = { 'HOST' => @host }
  header['Accept'] = '*/*'
  header['Accept-Encoding'] = 'gzip, deflate, br'
  header['Accept-Language'] = 'en-US,en;q=0.5'
  header['Connection'] = 'keep-alive'
  header['User-Agent'] = 'Mozilla/5.0'
  header['Content-Type'] = 'application/x-www-form-urlencoded'
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['DNT'] = '1'
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k, v|
    header[k] = v
  end

  req.initialize_http_header( header )

  @response = @session.request(req)
  save_cookies(@response)

  $stderr.puts @response.code.to_i if @debug

  if @response.code.to_i >= 300
    if @response.code.to_i == 302
      # ignore the redirects.
      # $stderr.puts "302"
      # @response.each {|key, val| $stderr.printf "%s = %s\n", key, val }  #Location seems to have cgi params removed. End up with .../cginame?&
      # $stderr.puts "Redirect to #{@response['location']}"   #Location seems to have cgi params removed. End up with .../cginame?&
      # $stderr.puts
      return
    elsif @response.code.to_i >= 400 && @response.code.to_i < 500
      return @response.body
    end

    raise Error.new(web_return_code: @response.code.to_i, message: "#{@response.code} #{@response.message} #{query} #{form_values} #{@response.body}")
  end

  return @response.body
end

#header_value(key:) ⇒ String

Get a header value, from the last response

Parameters:

  • key (String)

    header key

Returns:

  • (String)

    header value, for the given key.



150
151
152
# File 'lib/wikk_webbrowser.rb', line 150

def header_value(key:)
  @response.header[key]
end

#http_sessionObject

Creating a session for http connection attached block would then call get or post NET::HTTP calls

Parameters:

  • port (Fixnum)

    Optional http server port

  • use_ssl (Boolean)

    Use https, if true

  • verify_cert (Boolean)

    Validate certificate if true (Nb lots of embedded devices have self signed certs, so verify will fail)

  • block (Proc)


95
96
97
98
99
100
101
102
103
104
# File 'lib/wikk_webbrowser.rb', line 95

def http_session
  @http = Net::HTTP.new(@host, @port)
  @http.set_debug_output($stdout) if @debug
  @http.use_ssl = @use_ssl
  @http.verify_mode = OpenSSL::SSL::VERIFY_NONE if ! @use_ssl || ! @verify_cert
  @http.start do |session| # ensure we close the session after the block
    @session = session
    yield
  end
end

#patch_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: patch

send a PATCH query to the server and return the response.

Parameters:

  • query (String)

    URL, less the ‘http://host/’ part

  • authorization (String) (defaults to: nil)

    If present, add Authorization header, using this string

  • content_type (String) (defaults to: '"application/octet-stream"')

    Posted content type

  • data (String) (defaults to: nil)

    Text to add to body of post to the web server

  • extra_headers (Hash) (defaults to: {})

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {})

    Add these to standard cookies

Returns:

  • (String)

    The Net::HTTPResponse.body text response from the web server



399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
# File 'lib/wikk_webbrowser.rb', line 399

def patch_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {})
  url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query}")
  req = Net::HTTP::Patch.new(url.path)

  header = { 'HOST' => @host }
  header['Accept'] = '*/*'
  header['Accept-Encoding'] = 'gzip, deflate, br'
  header['Accept-Language'] = 'en-US,en;q=0.5'
  header['Connection'] = 'keep-alive'
  header['User-Agent'] = 'Mozilla/5.0'
  header['Content-Type'] = content_type
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['DNT'] = '1'
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k, v|
    header[k] = v
  end
  req.initialize_http_header( header )

  if data.nil?
    req.body = ''
  elsif data.instance_of?(Hash)
    if content_type =~ /application\/octet-stream/
      req.set_form_data(data, '&')
    else
      req.set_form_data.to_j
    end
  else
    req.body = data # If json as a string or raw string
  end

  @response = @session.request(req)
  save_cookies(@response)

  if @response.code.to_i >= 300
    if @response.code.to_i == 302
      # ignore the redirects.
      # puts "302"
      # @response.each {|key, val| printf "%s = %s\n", key, val }  #Location seems to have cgi params removed. End up with .../cginame?&
      # puts "Redirect of Post to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?&
      return
    end

    raise Error.new(web_return_code: @response.code, message: "#{@response.code} #{@response.message} #{query} #{data} #{@response.body}")
  end

  return @response.body
end

#post_page(query:, authorization: nil, content_type: 'application/x-www-form-urlencoded', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: post

send a POST query to the server and return the response.

Parameters:

  • query (String)

    URL, less the ‘http://host/’ part

  • authorization (String) (defaults to: nil)

    If present, add Authorization header, using this string

  • content_type (String) (defaults to: 'application/x-www-form-urlencoded')

    Posted content type

  • data (String) (defaults to: nil)

    Text to add to body of post to the web server

  • extra_headers (Hash) (defaults to: {})

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {})

    Add these to standard cookies

Returns:

  • (String)

    The Net::HTTPResponse.body text response from the web server



222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
# File 'lib/wikk_webbrowser.rb', line 222

def post_page(query:, authorization: nil, content_type: 'application/x-www-form-urlencoded', data: nil, extra_headers: {}, extra_cookies: {})
  url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query}")
  req = Net::HTTP::Post.new(url.path)

  header = { 'HOST' => @host }
  header['Accept'] = '*/*'
  header['Accept-Encoding'] = 'gzip, deflate, br'
  header['Accept-Language'] = 'en-US,en;q=0.5'
  header['Connection'] = 'keep-alive'
  header['User-Agent'] = 'Mozilla/5.0'
  header['Content-Type'] = content_type
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['DNT'] = '1'
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k, v|
    header[k] = v
  end
  req.initialize_http_header( header )

  if data.nil?
    req.body = ''
  elsif data.instance_of?(String)
    req.body = data # If we are given a String, just use it as is
  elsif content_type =~ /application\/json/
    req.body = data.to_j # Encode data as json. It wasn't already a String.
  elsif content_type =~ /application\/octet-stream/ && (data.instance_of?(Hash) || data.instance_of?(Array))
    req.set_form_data(data, '&') # Should handle Array as multiple entries with the same key, and Hash
  else # Assuming data can become a string that makes sense to the other end
    req.body = data.to_s # raw string
  end

  @response = @session.request(req)
  save_cookies(@response)

  if @response.code.to_i >= 300
    if @response.code.to_i == 302
      # ignore the redirects.
      # puts "302"
      # @response.each {|key, val| printf "%s = %s\n", key, val }  #Location seems to have cgi params removed. End up with .../cginame?&
      # puts "Redirect of Post to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?&
      return
    end

    raise Error.new(web_return_code: @response.code, message: "#{@response.code} #{@response.message} #{query} #{data} #{@response.body}")
  end

  return @response.body
end

#put_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: put

send a PUT query to the server and return the response.

Parameters:

  • query (String)

    URL, less the ‘http://host/’ part

  • authorization (String) (defaults to: nil)

    If present, add Authorization header, using this string

  • content_type (String) (defaults to: '"application/octet-stream"')

    Posted content type

  • data (String) (defaults to: nil)

    Text to add to body of post to the web server

  • extra_headers (Hash) (defaults to: {})

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {})

    Add these to standard cookies

Returns:

  • (String)

    The Net::HTTPResponse.body text response from the web server



338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
# File 'lib/wikk_webbrowser.rb', line 338

def put_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {})
  url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query}")
  req = Net::HTTP::Put.new(url.path)

  header = { 'HOST' => @host }
  header['Accept'] = '*/*'
  header['Accept-Encoding'] = 'gzip, deflate, br'
  header['Accept-Language'] = 'en-US,en;q=0.5'
  header['Connection'] = 'keep-alive'
  header['User-Agent'] = 'Mozilla/5.0'
  header['Content-Type'] = content_type
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['DNT'] = '1'
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k, v|
    header[k] = v
  end
  req.initialize_http_header( header )

  if data.nil?
    req.body = ''
  elsif data.instance_of?(Hash)
    if content_type =~ /application\/octet-stream/
      req.set_form_data(data, '&')
    else
      req.set_form_data.to_j
    end
  else
    req.body = data # If json as a string or raw string
  end

  @response = @session.request(req)
  save_cookies(@response)

  if @response.code.to_i >= 300
    if @response.code.to_i == 302
      # ignore the redirects.
      # puts "302"
      # @response.each {|key, val| printf "%s = %s\n", key, val }  #Location seems to have cgi params removed. End up with .../cginame?&
      # puts "Redirect of Post to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?&
      return
    end

    raise Error.new(web_return_code: @response.code, message: "#{@response.code} #{@response.message} #{query} #{data} #{@response.body}")
  end

  return @response.body
end

#save_cookies(response) ⇒ Object

Save cookies returned by last html get/post. Removes previous cookies.

Parameters:

  • response (Net::HTTPResponse)

    result from HTTP calls



131
132
133
134
135
136
137
138
# File 'lib/wikk_webbrowser.rb', line 131

def save_cookies(response)
  if (cookie_lines = response.get_fields('set-cookie')) != nil
    cookie_lines.each do |cookie_line|
      cookies = cookie_line.split('; ').map { |v| v.split('=') }
      cookies.each { |c| @cookies[c[0]] = c[1] }
    end
  end
end