{"id":13517,"date":"2020-01-26T15:07:22","date_gmt":"2020-01-26T22:07:22","guid":{"rendered":"http:\/\/macblaze.ca\/?p=13517"},"modified":"2021-12-10T12:15:29","modified_gmt":"2021-12-10T19:15:29","slug":"web-scraping-python-code","status":"publish","type":"post","link":"https:\/\/macblaze.ca\/?p=13517","title":{"rendered":"Web scraping Python code"},"content":{"rendered":"

In my previous post<\/a> I explained that I was looking for a way to use web scraping to extract data from my Calibre-Web shelves and automatically post them to my Books Read<\/a> page here on my site. In this post I will step through my final Python script to explain to my future self what I did and why.<\/p>\n

Warning: Security<\/h3>\n

A heads up. I have no guarantees that this code is secure enough to use in a production environment. In fact I would guess it isn’t. But my Web-Calibre webserver is local to my home network and I trust that my hosted server (macblaze.ca) is secure enough. But since you are passing passwords etc. back and forth I wouldn’t count on any of this to be secure without a lot more effort than I am willing to put in. <\/p>\n

The code in bits<\/h2>\n

# import various libraries<\/h3>\n
import requests\nfrom bs4 import BeautifulSoup\nimport re<\/code><\/pre>\n
This loads the various libraries the script uses. Requests<\/em> is a http library that allows you to send requests to websites, BeautifulSoup<\/em> is a library to pull data from html and re<\/em> is a regex library to allow you to do custom searches.<\/p>\n
# set variables<\/h3>\n
# set header to avoid being labeled a bot\nheaders = {\n    'user-agent': 'Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36'\n}\n\n# set base url\nurlpath='http:\/\/urlpath'\n\n# website login data\nlogin_data = {\n    'next': '\/',\n    'username': 'username',\n    'password': 'password',\n    'remember_me': 'on',\n}\n\n# set path to export as markdown file \npath_folder=\"\/Volumes\/www\/books\/\"\nfile = open(path_folder+\"filename.md\",\"w\")<\/code><\/pre>\n
This sets up various variables used for login including a header to try and avoid being labeled a bot, the base url of the Calibre-web installation, login data and specifies a location and name for the resulting markdown<\/a> file. The open command is marked with a ‘w’ switch to indicate the script will write a new file every time it is executed, overwriting the old one.<\/p>\n
# log in and open http session<\/h3>\n
with requests.Session() as sess:\n    url = urlpath+'\/login'\n    res = sess.get(url, headers=headers)\n    res = sess.post(url, data=login_data)<\/code><\/pre>\n
Then, using Requests<\/em>, I opened a session on the webserver and log in using the variables.<\/p>\n
Writing the File<\/h3>\n
Note:<\/strong> The code has matching file.write()<\/code> and print()<\/code> statements throughout. The print()<\/code> statements just write to the terminal app and allow me to see what is being written to the actual file using file.write()<\/code>. They are completely unnecessary.<\/p>\n
# Set Title<\/h3>\n
file.write(\"# Books Read\\n\")\nprint(\"# Books Read\\n\")<\/code><\/pre>\n
Pretty basic: write the words Books Read<\/em> followed by a carriage return, tagged with a # to indicate it is a h1 head. This will become the actual page name.<\/p>\n
# find list of shelves<\/h3>\n
shelfhtml = sess.get(urlpath)\nsoup = BeautifulSoup(shelfhtml.text, \"html.parser\")\nshelflist = soup.find_all('a', href=re.compile('\/shelf\/[1-9]'))\nprint (shelflist)<\/code><\/pre>\n
So now we set the variable shelfhtml<\/em> to the session we opened earlier. Using BeautifulSoup we grab all the html code and search for all a<\/strong> links that have an href<\/strong> that contain the regex expression ‘\/shelf\/[1-9]’. (Hopefully I won’t have more than 9 shelves or I will have to redo this bit.) The variable now contains list of all the links that match that pattern and looks like this:<\/p>\n
[<a href=\"\/shelf\/3\"><span class=\"glyphicon glyphicon-list private_shelf\"><\/span>2018<\/a>, <a href=\"\/shelf\/2\"><span class=\"glyphicon glyphicon-list private_shelf\"><\/span>2019<\/a>, <a href=\"\/shelf\/1\"><span class=\"glyphicon glyphicon-list private_shelf\"><\/span>2020<\/a>]<\/code><\/pre>\n
This as you can see, contains the links to all three of my current Year shelves, displayed in ascending numerical order.
\n <\/p>\n
#reverse order of urllist<\/h3>\n
dateshelflist=(get_newshelflist())\ndateshelflist.reverse()\nprint (dateshelflist)<\/code><\/pre>\n
I wanted to display my book lists from newest to oldest so I used python to reverse the items in the list.<\/p>\n
First loop: the shelves<\/h2>\n
The first loop loops through all the shelves (in this case 3 of them) and starts the process of building a book list for each.<\/p>\n
# loop through sorted shelves<\/h3>\n
for shelf in dateshelflist:\n    #set shelf page url\n    res = sess.get(urlpath+shelf.get('href'))\n    soup = BeautifulSoup(res.text, \"html.parser\")\n\n    # find year from shelflist and format\n    shelfyear = soup.find('h2')\n    year = re.search(\"([0-9]{4})\", shelfyear.text)\n    year.group()\n    file.write(\"### {}\\n\".format(year.group()))\n    print(\"### {}\\n\".format(year.group()))<\/code><\/pre>\n
In the first iteration of the loop, the script goes to the actual shelf page using the base url and then adding an href<\/strong> extracted from the list by using a get<\/em> command and then accesses the html<\/em> from the resulting webpage. Then the script finds the year info, which is a H2<\/strong>, extracts the 4-digit year with the regex ([0-9]{4})<\/em> and writes it to the file, formatted as an H3<\/strong> header and followed by a line break.<\/p>\n
# find all books<\/h3>\n
books = soup.find_all('div', class_='col-sm-3 col-lg-2 col-xs-6 book')<\/code><\/pre>\n
Using BeautifulSoup we extract the list of books from the page knowing they are all marked with a div<\/strong> in the class col-sm-3 col-lg-2 col-xs-6 book<\/strong>.<\/p>\n
Second loop: the books<\/h2>\n
#loop though books. Each book is a new BeautifulSoup object.<\/h3>\n
for book in books:\n        title = book.find('p', class_='title')\n        author = book.find('a', class_='author-name')\n        seriesname = book.find('p', class_='series')\n        pubdate = book.find('p', class_='publishing-date')\n        coverlink = book.find('div', class_='cover')\n        if None in (title, author, coverlink, seriesname, pubdate):\n            continue\n        # extract year from pubdate\n        pubyear = re.search(\"([0-9]{4})\", pubdate.text)\n        pubyear.group()<\/code><\/pre>\n
This is the beginning of the second loop. For each book we use soup to extract the title, author, series, pubdate and cover (which I don’t end up using). Each search is based on the class assigned to it in the original html code. Because I only want the pub year and not pub date, I again use a regex to extract the 4-digit year. The if None…<\/em> statement is there just in case one of the fields is empty and prevents the script from hanging.<\/p>\n
# construct line using markdown<\/h3>\n
newstring = \"* ***{}*** \u2014 {} ({})\\{} \u2013 ebook\\n\".format(title.text, author.text, pubyear.group(), seriesname.text)\nfile.write(newstring)\nprint (newstring)<\/code><\/pre>\n
Next we construct the book entry based on how we want it to appear on the web page. In my case I want each entry to be an li<\/strong> and end up looking like this:<\/p>\n