Ceci est une ancienne révision du document !
As many of you who have been reading this column for a while might know, one of my hobbies is cooking. Since I'm the one in the family who cooks every night, I'd have to say that it's my favorite hobby. My eighth article for Full Circle back in FCM#34 (February 2010) was about creating a very small and generic cookbook database. The fact that it is 10 years to the month since I first wrote about the database program hasn't escaped me.
Anyway, I've started re-writing the program, pretty much from scratch, and again using Page as the GUI designer. I wanted to give it a newer, sleeker look, with a nicer interface, and there have been many things that i've wanted to add for years, but just never got around to doing. Things like adding a way to have a picture of the finished product, a way to grab a recipe from one of the many recipe websites I search, and more. While I'm still in the process of development and the UI is still somewhat in flux, I'm pretty proud of the new look and feel. Here's a quick screenshot of the main form of the program.
As I said, one of the things that I was both excited and worried about was the webpage scraper. I've tried writing a generic scraper before with limited success, but never could wrap my head around it properly. Part of the problem was that other things with a higher priority level would come up just as I was starting to be comfortable with the process, and I would have to put the project on hold. By the time I got around to revisiting the project, I had to spend a good while trying to remember what I was doing and how I had done it. I ended up so frustrated I started searching the web for some tips and tricks that others posted that might give me a leg up on my learning process.
I stumbled upon a really nice project called “recipe-scrapers” that seemed to be created just for my issue. It was a free and open source library for Python that provides custom scrapers for many cooking recipe websites, one of which is Allrecipes.com, which I end up on at least twice a week when I'm looking for new ideas for dinner recipes. At the time, there were about 20 sites that they supported and it seemed that they were very active in adding more sites. As of this writing, there are 41 different sites that are supported. Let's look at how to install the library and utilize it.
The repository is located at https://github.com/hhursev/recipe-scrapers. The main page provides a list of supported recipe sites as well as a short example of how to use the library. As always, you have to install the library before you can use it. You can use pip to do that… $ pip install recipe-scrapers You can also clone or download the repository and, once you have it on your machine, go to the main folder (recipe-scrapers) and use pip to install it directly from the source… $ pip install -e .
This is a good way to install it if you are interested in how the program works and if you want to write your own scrapers. Now, open your favorite IDE or editor and create a new file. Let's call it “scrapertest.py”. Of course, the first thing we need to do is to import the library… from recipe_scrapers import scrape_me Next, you will need a recipe page that you want to scrape. You need to find one that is a single recipe, not a category page. For this tutorial, we will use a page from Allrecipes.com that provides the recipe for Asian Orange Chicken https://www.allrecipes.com/recipe/61024/asian-orange-chicken/ . Below left is a quick glance of what that page looks like.
The next thing that we should do is create a variable to hold the URL of the site page… site = 'https://www.allrecipes.com/recipe/61024/asian-orange-chicken/' Now we create an instance of the scraper and provide it with the URL. scraper = scrape_me(site) Once we have that done, we can start grabbing some of the information that the scraper comes back with. Each bit of information is handled by a separate method. Note: Some scrapers may provide more or less information depending on the site and if the author of the scraper included it.
From the above code, we will be able to get the recipe title, the total amount of time that the recipe will take to prepare, the number of servings it will make (yields), a list of the ingredients, the instructions, and a URL of the image, if one is available. In this case, we have set the data for each into a variable. Now (bottom right), let's print the data in the terminal. When we run the program, the output looks like that shown on the next page, top.
It’s obvious that the ingredients come back as a Python list, so let's change the program a little bit to make the data a bit more readable. Comment out the line that prints the ingredients as a “glob” and replace it with… # print(f'Ingredients: {ingredients}') print('Ingredients:\n') for ing in ingredients: print(f' {ing}') With this small change, our output now looks like that shown far right.
Now, let’s make the program a bit more usable by allowing the user to enter the URL at runtime, rather than hard coding it. Comment out the line that assigns the URL (site = ‘’) and replace it with… site = input('Please enter the website URL to scrape (blank line to quit) →') if site != '': # site = 'https://www.allrecipes.com/recipe/61024/asian-orange-chicken/' scraper = scrape_me(site) Be sure to indent the rest of the code so that it all falls under the if statement.
For this run, we’ll use a different known good recipe page, again from Allrecipes. https://www.allrecipes.com/recipe/8849/baked-chicken-nuggets/ … Now when you run the program with the new URL, your output looks like this: With a bit more cleanup of the output portion of the code, it will be pretty nice. However, what happens when you enter a website that is not one of the sites that is supported by the library? Let’s look by trying a site that I know is not supported (below) https://www.jennycancook.com/recipes/custard-filled-paczki/
This error is easy to avoid. All of the sites that are supported are stored in a dictionary named SCRAPERS. What we will want to do is grab the domain from the URL and see if it is in the SCRAPERS dictionary. We can do that by importing the urlparse library… from urllib.parse import urlparse Be sure to place this at the top of the file, just under the other import statement. The existing code will be shown here as ‘not bold’ and the new code as ‘bold’ (top right). Again, be sure to match the indentation level of the rest of the code. Finally, at the very end of the code, add the following two lines (below).
Now when you run the program, using the unsupported URL, you’ll see the following… Please enter the website URL to scrape (blank line to quit) →https://www.jennycancook.com/recipes/custard-filled-paczki/ Sorry, that website is not currently supported. That’s it.This base code can be easily worked into a GUI form as well. Here’s a shot of what my GUI scraper form looks like.
As I usually do, I’ve put the code up on Pastebin at https://pastebin.com/t0t8a0fm Until next time, Keep coding!