One of the recent updates to Calibre Web<\/a> (I am on v0.6.14 now) introduced some security to the login in the form of a csfr token<\/em>. I had no idea why my script wouldn’t work anymore until I started taking it all apart.<\/p>\n What I ended up having to do was use Beautiful Soup to load the html content of the login page, find the Just a note, if any of you have found this page wanting to scrape Calibre-Web, if you search the site for the tag calibre<\/strong> you find the whole journey<\/p>\ncsfr_token<\/code> and then add it to the login data. I also added in the cookies, although frankly by that time I wasn’t sure if it was necessary or not. But it works so I left it.<\/p>\nThe current python script<\/h2>\n
#! \/usr\/local\/bin\/python3\n# coding: utf-8\n\n# python script to import shelfs of books from Calibre-Web\n# and save them as a markdown file\n\n# import various libraries\nimport requests\nfrom bs4 import BeautifulSoup\nimport re\nimport os\n# enable sys.exit()\nimport sys\n\n# set header to avoid being labeled a bot\nheaders = {\n 'user-agent': 'Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36'\n}\n\n# set base url\nurlpath = 'https:\/\/MY_CALIBREWEB_URL'\n\n# calibre-web login data\nlogin_data = {\n 'next': '\/',\n 'username': 'MY_USERNAME',\n 'password': 'MY_PASSWORD',\n 'remember_me': 'on',\n}\n\n# login as a session\nwith requests.Session() as sess:\n res = sess.get(urlpath + '\/login', headers=headers)\n login = BeautifulSoup(res._content, 'html.parser')\n\n # find security token and extract value\n csrf_token = login.find('input', {'name': 'csrf_token'})\n csrf_token = (csrf_token.attrs['value'])\n # print(csrf_token)\n\n # append securtiy token to login dictionary\n login_data['csrf_token'] = csrf_token\n\n # login with token\n res = sess.post(urlpath + '\/login', data=login_data, cookies=res.cookies)\n # print(res.text)\n\n\n# set path to export as markdown file\npath_folder = \"FOLDER_PATH_MOUNTED_BY_SHELL_SCRIPT\"\n\n# check ifserver is already mounted and change path if so\nif os.path.isdir(path_folder):\n print('valid drive')\nelif os.path.isdir(\"\/Volumes\/www\/home\/books\/\"):\n path_folder = \"\/Volumes\/www\/home\/books\/\"\nelse:\n path_folder = \"\/Volumes\/www-1\/home\/books\/\"\n\n# open the file\nfile = open(path_folder+\"bookfile.md\", \"w\")\n\n# Set Title\nfile.write(\"# Books Read since 2012\\n\")\n# print(\"# Books Read\\n\")\n\n# Set intro blurb\nfile.write(\"This is an automatically generated list of books scraped from my Calibre ebook library ( Making A \u201cBooks Read\u201d Page <\/a> ). As a result it does not include any paper books I may have read as they do not exist in that library.\\n\\nI update it regularly and finally went back and added all the previous years. Links to previous years\u2019 book count posts: \\n- 2012 <\/a> (85)\\n- 2013 <\/a> (95)\\n- 2014 <\/a> (106)\\n- 2015 <\/a> (92)\\n- 2016 <\/a> (101)\\n- 2017 <\/a> (120)\\n- 2018 <\/a> (142)\\n- 2019 <\/a> (123)\\n- 2020 <\/a> (112)\\n\")\n\n\n# find list of shelves\nshelfhtml = sess.get(urlpath)\nsoup = BeautifulSoup(shelfhtml.text, \"html.parser\")\nshelflist = soup.find_all('a', href=re.compile('\/shelf\/[1-9]'))\nprint(shelflist)\n\n# reverse order of urllist\ndateshelflist = (shelflist)\ndateshelflist.reverse()\nprint(dateshelflist)\n\n# loop through sorted shelves\nfor shelf in dateshelflist:\n # set shelf page url\n res = sess.get(urlpath+shelf.get('href'))\n soup = BeautifulSoup(res.text, \"html.parser\")\n\n # find year from shelflist and format\n shelfyear = soup.find('h2')\n year = re.search(\"([0-9]{4})\", shelfyear.text)\n year.group()\n file.write(\"\\n### {}\\n\".format(year.group()))\n # print(\"### {}\\n\".format(year.group()))\n\n # find all books\n books = soup.find_all('div', class_='col-sm-3 col-lg-2 col-xs-6 book')\n\n # loop though books. Each book is a new BeautifulSoup object.\n for book in books:\n title = book.find('p', class_='title')\n # print(title)\n author = book.find('a', class_='author-name')\n #print (author)\n seriesnamea = book.find('p', id='series') # I have to manually add this id to the shelf.html template\n seriesname = (seriesnamea.text if seriesnamea else \"\").replace(\n \" \", \"\").replace(\"(\", \" Book \").replace(\")\", \"\").replace(\"\\n\", \"\")\n if (seriesname != \"\"):\n seriesname = \"*\" + seriesname + \"*\"\n #print (seriesname)\n# NOTE: pubdate is custom added by me to \/templates\/shelf and won't work in a standard install\n pubdate = book.find('p', class_='series', id='pubdate')\n if pubdate:\n #print (pubdate)\n # extract year from pubdate\n pubyear = re.search(\"([0-9]{4})\", pubdate.text)\n pubyear.group()\n pubyear = pubyear.group()\n else:\n pubyear = \"n\/a\"\n # construct line using markdown\n newstring = \"* ***{}*** \u2014 {} ({})\\n{} \u2013 ebook\\n\".format(\n title.text, author.text, pubyear.group(), seriesname)\n #print (newstring)\n file.write(newstring)\n\nfile.close()\n<\/code><\/pre>\n