code-quality

Scrapping kaymu products using NOKOGIRI

scrapping kaymu products

Engineering Manager · Product Owner · Varicon

Jun 18, 2016·2 min read

It’s saturday morning and I have nothing interesting to do. So I thought of scraping Kaymu and list out products and their price using ruby. I targeted raspberry pi for this exercise.

Libraries I used are:

<a href=https://github.com/jnunemaker/httparty rel=“nofollow” target=“_blank”>httparty to make http request
<a href=https://github.com/sparklemotion/nokogiri rel=“nofollow” target=“_blank”>nokogiri to parse html
<a href=https://github.com/pry/pry rel=“nofollow” target=“_blank”>pry for debugging

Here is the commented script:

# requiring dependencies
require 'nokogiri' 
require 'httparty' 
require 'pry'
 
# kaymu url to search raspberry pi and list them in ascending order of price
SITE = "http://www.kaymu.com.np/catalog/?q=raspberry+pi&sort=price&dir=asc"
 
# hitting the url to get data
page = HTTParty.get(SITE)
 
parse_page = Nokogiri::HTML(page)
 
products_array = []
 
# acquiring required data using Nokogiri's css methods
parse_page.css('.row-products .product').each do |product|
  products_array.push({item: product.css('h3').text, price: product.css('.price').text})
end
 
p products_array

Output:

[
{:item => "Raspberry Pi Lcd 3.5 inch", :price => "Rs 3,500 "}, 
{:item => "Raspberry Pi 3.5” TFT with Touch Screen", :price => "Rs 3,970 "},
{:item => "Raspberry Pi - 2 Camera", :price => "Rs 4,669 "}, 
{:item => "Raspberry PI Model B+", :price => "Rs 5,900 "}, 
{:item => "Raspberry Pi Model B+", :price => "Rs 6,330 "}, 
{:item => "Raspberry Pi-2 B Plus with Casing", :price => "Rs 6,899 "}, 
{:item => "5 inch HDMI Touch Screen for Raspberry Pi", :price => "Rs 7,200 "}, 
{:item => "Raspberry Pi 3", :price => "Rs 7,300 "}, 
{:item => "Raspberry Pi 3 (Made in UK)", :price => "Rs 8,299 "}, 
{:item => "Raspberry Pi 3 Model B", :price => "Rs 8,500 "}
]

If you want to learn scrapping yourself, <a href=https://www.distilled.net/resources/web-scraping-with-ruby-and-nokogiri-for-beginners/ rel=“nofollow” target=“_blank”>you can follow better tutorial here.

About the author

Prakash Poudel Sharma

Engineering Manager · Product Owner · Varicon

Engineering Manager at Varicon, leading the Onboarding squad as Product Owner. Eleven years of building software — first as a programmer, then as a founder, now sharpening the product craft from the inside of a focused team.

Follow on LinkedIn More essays

Keep reading