Web Scraping

Commands:
$ pip install scrapy
$ pip install autopep8
$ pip install ipython
$ scrapy shell
$ scrapy startproject worldometers
$ cd worldometers
$ scrapy genspider countries www.worldometers.info/world-population/population-by-country
# Modify spider
$ scrapy crawl countries

$ scrapy shell
> url = "https://www.worldometers.info/world-population/population-by-country"
> r = scrapy.Request(url)
> fetch(r)
> view(response)
> title = response.xpath('//h1/text()').get()
> countries = response.xpath('//td/a/text()').getall()
> title = response.css('h1::text').get()
> countries = response.css('td a::text').getall()