Building a Python Web Scraper with Beautiful Soup

Web scraping is a powerful technique for collecting data from websites. In this tutorial, we’ll build a simple web scraper using Python and Beautiful Soup.

Setting Up Your Environment

First, let’s install the necessary packages:

pip install requests beautifulsoup4

Basic Web Scraper

Here’s a simple script to scrape headlines from a news website:

import requests
from bs4 import BeautifulSoup

# Send a GET request to the website
url = 'https://example-news-site.com'
response = requests.get(url)

# Parse the HTML content
soup = BeautifulSoup(response.text, 'html.parser')

# Find all headline elements
headlines = soup.find_all('h2', class_='headline')

# Extract and print the text from each headline
for headline in headlines:
    print(headline.text.strip())
            

Handling Pagination

Many websites split their content across multiple pages. Here’s how to handle pagination:

import requests
from bs4 import BeautifulSoup

def scrape_page(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    headlines = soup.find_all('h2', class_='headline')

    results = []
    for headline in headlines:
        results.append(headline.text.strip())

    return results

# Scrape multiple pages
base_url = 'https://example-news-site.com/page/'
all_headlines = []

for page_num in range(1, 6):  # Scrape pages 1-5
    page_url = base_url + str(page_num)
    page_headlines = scrape_page(page_url)
    all_headlines.extend(page_headlines)
    print(f"Scraped page {page_num}, found {len(page_headlines)} headlines")

print(f"Total headlines scraped: {len(all_headlines)}")
            

Ethical Considerations

When scraping websites, always:

• Check the website’s robots.txt file
• Include delays between requests to avoid overloading the server
• Identify your scraper with a user agent
• Respect the website’s terms of service

In the next tutorial, we’ll explore how to store the scraped data in a database and create a simple API to access it.