Beautiful Soup

Beautiful Soup is a Python library used to parse HTML and XML documents for web scraping and data extraction.

It provides a simple interface for navigating page structures, searching for elements, and extracting or modifying content based on tags, attributes, and CSS selectors.

Also known as: BS4, Beautiful Soup 4.

Beautiful Soup vs. Scrapy

Beautiful Soup and Scrapy are both used in web scraping, but they serve different purposes.

Beautiful Soup is mainly a parsing library. It is lightweight and well suited to small or medium-sized projects where developers need to process HTML or XML content.

Scrapy is a complete web scraping framework that includes crawling, request scheduling, pipelines, middleware, and other tools for building larger scraping systems.

In simple terms:

• Beautiful Soup: Focuses on parsing and extracting page content.
• Scrapy: Provides a complete framework for crawling and scraping websites.

Beautiful Soup vs. Selenium

Beautiful Soup works best with HTML that has already been returned by a server.

Selenium, by contrast, controls a real browser and can interact with JavaScript-powered pages, click buttons, fill forms, scroll, and wait for dynamic content to load.

Beautiful Soup is usually faster and simpler for static pages, while Selenium is more suitable when browser interaction or JavaScript rendering is required.

Advantages of Beautiful Soup

• Easy to learn: Provides a straightforward Python API for parsing web pages.
• Lightweight: Works well for simple scraping and data extraction tasks.
• Flexible parser support: Can work with parsers such as html.parser and lxml.
• Powerful element selection: Supports searching by tag names, attributes, text, and CSS selectors.
• Useful for messy HTML: Can handle many imperfect or poorly structured HTML documents.

Limitations of Beautiful Soup

• No built-in crawler: It does not manage large-scale crawling by itself.
• No JavaScript execution: It cannot directly render content generated by JavaScript.
• Less suitable for large scraping systems: Bigger projects may require a framework such as Scrapy.
• Requires another HTTP client: Libraries such as requests or httpx are typically used to download pages before Beautiful Soup parses them.

Beautiful Soup and Proxies

Beautiful Soup itself does not send web requests or manage proxies.

A common scraping workflow combines Beautiful Soup with an HTTP client such as Python's requests library. The HTTP client retrieves the webpage, while Beautiful Soup parses the returned HTML.

When proxy routing is required, it is normally configured in the HTTP client rather than in Beautiful Soup.

A typical workflow may include:

1. Send an HTTP request through a proxy.
2. Receive the HTML response.
3. Pass the HTML to Beautiful Soup.
4. Locate the required elements.
5. Extract and store the data.

This separation makes Beautiful Soup easy to integrate into proxy-based scraping workflows.

Example

The following example retrieves a webpage with requests and uses Beautiful Soup to extract article titles:

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
response = requests.get(url)

soup = BeautifulSoup(response.text, "html.parser")

titles = soup.find_all("h2", class_="article-title")

for title in titles:
    print(title.get_text(strip=True))