CoderVortex
codervortex5@codervortex.com

Building A Basic Web Scraper With Online XPath Evaluators

A web scraper can collect structured information from pages that do not offer a convenient API. For a beginner, the hardest part is often not writing Python code. It is identifying the exact HTML elements that contain the title, price, address or publication date you need.

An online XPath evaluator makes that first step easier. You can inspect a page, enter a path such as //h1/text(), and see which nodes match before adding the selector to a script. This shortens the trial-and-error process and helps you understand how a document is organised.

The same method works for Australian business directories, property listings, event pages and technology websites. Whether you are comparing Melbourne cafés, tracking Sydney rental listings or collecting product data from an online shop, XPath provides a precise way to select page content.

Choosing A Target Page

Start with a page that contains information in a consistent layout. A news archive might place every headline inside an h2 element, while a product catalogue may use a repeated article or div element with a distinctive class. Static HTML is easier to scrape than a page that loads all content through JavaScript.

Check the website’s terms of service and robots.txt file before sending requests. Publicly visible data is not automatically free from restrictions. Avoid collecting personal information, bypassing access controls or creating a request load that affects the site. A small delay between requests is a sensible practice for Australian sites operating across different time zones, including pages managed from Perth, Brisbane or Sydney.

For market research, a business directory can provide useful context alongside scraped data. CoderVortex’s business resources may also help when the project involves comparing industries, companies or commercial trends rather than extracting page content alone.

Understanding XPath Selectors

XPath describes a route through an HTML or XML document. The expression //h1 selects every level-one heading, while //article//a/@href finds links inside article elements and returns their destination attributes. A slash moves to a direct child; a double slash searches descendants at any depth.

Attributes make selectors more specific. For example, //div[@class="product-card"] selects a div whose class is exactly product-card. A more flexible expression is //div[contains(@class, "product-card")], which can still match when the element has several class names.

Text matching is useful when the markup has no reliable class. //a[contains(normalize-space(), "Read more")] searches links whose visible text includes those words. When several results are returned, position can narrow the selection, as in (//h2)[1]. Use positional selectors carefully because a small design change can move the item you want.

Testing Queries In A Browser Tool

Open the target page in an XPath testing utility and begin with broad queries. Test //h1, //p or //a first, then inspect the matching nodes. Once you know where the content lives, refine the expression with classes, attributes or parent-child relationships.

A useful workflow is to test one field at a time. Find the headline, then the summary, then the date and link. For a listing page, identify the repeated container first, such as //article, and then select fields relative to that container: .//h2/text() for a title or .//a/@href for a link.

Some online evaluators request the page directly, while others require pasted HTML. A live evaluator may fail because of browser security rules, regional restrictions or a site’s bot protection. If that happens, save a small HTML sample and test it locally or paste the relevant markup into the tool.

Writing The Scraper

Once the XPath expressions work, Python can fetch the page and parse its HTML. The requests and lxml packages are enough for many static pages:

import requests
from lxml import html

url = "https://example.com/news"
headers = {"User-Agent": "Mozilla/5.0 (compatible; ResearchBot/1.0)"}

response = requests.get(url, headers=headers, timeout=15)
response.raise_for_status()

document = html.fromstring(response.content)

for card in document.xpath("//article"):
    title = " ".join(card.xpath(".//h2//text()")).strip()
    link = card.xpath(".//a/@href")
    print(title, link[0] if link else "")

The dot in .//h2 is important. It limits the search to the current article rather than selecting every h2 on the whole document. Joining text nodes also handles headings that contain nested tags, such as a link or a highlighted word.

For pages that load data after the initial request, requests may return only a shell of the page. In that case, inspect the browser’s network panel for an underlying JSON endpoint, or use a browser automation tool such as Playwright. Do not assume that adding random delays or headers will reproduce content generated by JavaScript.

Handling Australian Web Data

Australian pages often display prices with dollar signs, GST references, suburb names and state abbreviations such as NSW, VIC and QLD. Keep the original text during extraction, then normalise it in a separate step. This preserves the source value while allowing you to convert $1,250 including GST into a clean numeric field later.

Dates can also require care. An event listed as 12/04/2026 may be interpreted differently by software configured for the United States. Store Australian dates as day-month-year when displaying them, and convert them explicitly to ISO format for databases. Time-sensitive projects should record the relevant zone, such as AEST or AEDT, rather than treating every timestamp as UTC.

Local language and geography affect matching too. A page may use “arvo” in an event description, list a venue in the western suburbs of Melbourne, or distinguish Sydney CBD from nearby suburbs. Exact text filters can miss these variations, so combine XPath extraction with later search rules that account for aliases, abbreviations and state names.

Practical Checks Before Running

A working selector is only one part of a dependable scraper. Pages change, network requests fail and empty fields are common. Save a small sample of the extracted output so you can compare future runs and notice when a website redesign has broken the parser.

Keep requests modest and record errors instead of silently discarding them. If the project collects business information for an Australian market report, document the source page, access date and any transformations applied to the data. This makes the results easier to audit and reduces confusion when a listing changes.

An online XPath evaluator is best viewed as a selector workshop, not a complete scraping platform. It helps you discover the structure of a page quickly; Python or another programming language handles retrieval, cleaning, storage and scheduling. Starting with careful inspection produces smaller, clearer scripts and makes future maintenance far less frustrating.