How to Use HTMLParser in Python

A Python script that reads page titles or links needs a way to pick those values out of HTML. HTMLParser reads supplied HTML and calls functions you define when it reaches tags and text.

In this piece, I’ll walk through collecting a title and link destinations with HTMLParser, then explain when a parser that builds a tree makes later lookups simpler.

TL;DR: Use HTMLParser for callback-based extraction

Python’s html.parser.HTMLParser reads supplied markup and calls methods you define for tags and text.

• Override callbacks to collect the fields your task needs.
• Send decoded text to feed() and finish with close().
• Choose a tree parser when you need selectors or later navigation.

What is HTMLParser in Python?

The class lives in Python’s standard library. Subclassing lets you supply only the callbacks the task needs.

HTMLParser is an event parser. It reads markup and calls methods on your subclass as it encounters start tags, end tags, and text. The handler decides what to save, so the class does not create a navigable document tree for you.

That distinction affects the shape of your code. A callback is a function invoked during parsing, and its arguments describe the event just read. The handler must retain any earlier value needed to interpret a later event.

Before writing callbacks, name the output fields and the markup that supplies each one. A title comes from text between title tags, while a link destination comes from an anchor attribute. This small map keeps the handler focused on the requested result.

Import the class from the module. Python includes it, so this example needs no third-party install.

How to extract page data with HTMLParser step by step

The example stores title text and anchor destinations from a fixed HTML string. The callbacks keep parsing separate from the later code that reads the collected values.

Step 1: Define callbacks for the tags and text

The start-tag callback receives a tag and attribute pairs, while the data callback handles text. An end-tag event lets the handler stop collecting title content.

from html.parser import HTMLParser

class ArticleParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.in_title = False
        self.title_parts = []
        self.links = []

    def handle_starttag(self, tag, attrs):
        if tag == "title":
            self.in_title = True
        if tag == "a":
            href = dict(attrs).get("href")
            if href:
                self.links.append(href)

    def handle_endtag(self, tag):
        if tag == "title":
            self.in_title = False

    def handle_data(self, data):
        if self.in_title:
            self.title_parts.append(data.strip())

html = '<html><head><title>Docs &amp; Guides</title></head><body><a href="/guide">Read</a></body></html>'
parser = ArticleParser()
parser.feed(html)
parser.close()
print("Title:", "".join(parser.title_parts))
print("Links:", parser.links)

Run python3 parser_demo.py. The captured output is shown below.

Title: Docs & Guides
Links: ['/guide']

Link destinations can be saved without retaining anchor text. The title flag ties the opening and closing events to the title fragments.

Step 2: Handle attributes, nested text, and empty values

Attribute names are normalized to lower case, and a valueless attribute can have the value None. Test for that value before calling string methods or joining it with other text.

Nested elements create separate text events. For a heading containing emphasis, append each relevant piece to a list and join the list when the heading ends. Replacing the saved value on every callback would keep only the final fragment.

The attribute list contains pairs, not a dictionary. Converting it with dict(attrs) is convenient when each attribute name appears once, as with href in this example. HTMLParser lowercases attribute names, so look up href rather than preserving the source spelling.

A boolean-style attribute may have None as its value. For example, a disabled attribute can arrive without a value after the equals sign. Check for None before using string operations, and decide whether an empty string and a missing value mean different things in your output.

Text callbacks can split a visible phrase around nested tags. For markup such as a title containing an em element, the parser can report text before the child, the child’s text, and text after it as separate events. Appending fragments and joining them at the closing title tag preserves the whole phrase.

Whitespace needs a deliberate policy. Calling strip() on each fragment removes spaces at the edges of every piece, which can join words that were separated in the original text. For general text, preserve the fragments and normalize the joined result once, or collect text from only the elements relevant to the field.

Step 3: Feed decoded input and finish the parse

Read bytes as text before passing them to the parser. If a file declares an encoding, decode it using that declaration instead of assuming a character set. For a large document, pass text chunks to one parser instance.

At end-of-input, call close() so any buffered markup is processed. Treat the parser instance as belonging to one document and create a new instance before parsing another independent page.

What should you know about HTMLParser’s limits?

A callback handler needs state to connect related elements. Repeated structural questions become harder to express as that state grows.

For a file, decode bytes before calling feed(). The method accepts text, not a byte stream. Use the declared file encoding and handle decoding errors before parsing.

Calling feed() more than once lets one parser instance receive text in chunks. The parser buffers unfinished markup between calls, which is useful when input arrives incrementally. Call close() once at the end so pending data is processed, then create a fresh handler for another document.

The Python documentation notes that the parser accepts invalid markup. It does not check whether end tags match start tags. It also omits callbacks for some elements browsers close implicitly.

Because HTMLParser does not validate a balanced tree, an end-tag callback is not a reliable signal that every element has closed. The documentation also notes that elements browsers close implicitly do not always produce a matching callback. Handlers should define their own reset rules for the specific markup they consume.

Character references are converted by default outside elements such as script and style. Text inside those elements is reported as data rather than interpreted as child markup. Filter these sections if they are irrelevant to the field being extracted.

The parser is a markup reader, not a network client. Fetch a response separately, check the response and content type, then decode its body before parsing. A page whose useful content is inserted by JavaScript needs a browser-rendering step because the original response may not contain those elements.

HTMLParser works on the text given to it. It does not retrieve a URL or execute JavaScript, so a page that fills its content in the browser needs a separate fetch or rendering step before extraction.

Beautiful Soup is a separate package that presents parsed markup as a tree of Python objects, with search and navigation methods for related elements. lxml.html provides HTML elements and XPath. Choose the query API your code needs rather than an assumed speed difference.

For a fetch-then-parse workflow, see the Python Requests guide. For a tree-oriented example, see Beautiful Soup in Python.

A title is a simple example because it has one start and one end tag. Production pages also contain repeated records, optional attributes and nested sections, so the output structure should guide what the handler stores.

When a tree becomes simpler

A tree API represents parent and child relationships so later lookups can start from an element instead of callback-maintained state.

A page with nested cards gives each link an enclosing card and heading. A handler can save every destination, but matching one destination to its card requires state for the current card and its heading. An element tree keeps those parent and child relationships available for later queries.

In Beautiful Soup or lxml.html, locate a heading, move to its parent, then inspect a child link. That operation uses the tree directly instead of a custom stack of flags and temporary lists.

Beautiful Soup can use different parser backends, so installation alone does not determine parsing behavior. When behavior on malformed input matters, select and document the backend rather than assuming every parser repairs the same markup in the same way.

lxml.html exposes elements and XPath for queries such as selecting nodes by structure. This can fit a pipeline that already uses lxml. For a small script whose only job is counting anchors, adding that dependency may bring no benefit.

Do not choose from speed claims copied from a different document or machine. Performance depends on input, parser configuration and the operation being timed. Measure the whole task with representative input if throughput becomes a requirement.

A browser-rendering path is a separate choice when the desired content appears only after scripts run. The parser handles markup text, not browser execution.

• Use HTMLParser for a fixed set of fields from supplied markup.
• Use Beautiful Soup for readable navigation and search across a tree.
• Use lxml.html when XPath or element operations fit the surrounding code.

Keep fetching, parsing and extraction as separate functions. That boundary lets you replace the parser without changing the code that obtains the document.

For a one-off field, a callback that appends a value to a list is usually a small, testable unit. For a record with several related descendants, first decide which ancestor identifies the record and which descendants belong to it. If the handler needs nested flags for each possible path, a tree query is often clearer.

Selection should follow the next operation, not a general claim that one parser is faster. For a small set of values, callback state can be compact and direct. If the extraction keeps asking which child belongs to which parent, tree navigation gives the code a closer match to the question.

Choose a handler state model that follows the document. The title example needs one boolean because only one title is collected. A repeated list of articles would need a current-record object, plus a place to append completed records when the matching container ends.

Nested containers make a single boolean less dependable when cards can contain other cards. Keep a stack for open containers, pushing state at a relevant start tag and popping it at the matching end event. Define a recovery rule for malformed input.

Keep callbacks small by moving output cleanup into a separate function. The parser can collect fragments and attributes, while the cleanup function converts them into the shape the caller expects. This separation also makes it possible to test extraction logic using a short markup string without downloading a page.

For links, decide whether the output should contain raw href values or absolute URLs. The parser returns the attribute as written. A relative value such as /guide needs a base URL and URL-joining logic before it can be used as a standalone address.

Do not treat extracted markup as trusted text for an HTML page or a shell command. Parsing identifies values, but it does not sanitize them for another output context. Escape or validate each value according to the system that will consume it.

The practical decision is about how you will use the result. A one-pass handler is a good fit when the required fields are known and the output can be assembled as events arrive. A tree library is a better fit when the code needs to revisit ancestors, filter sibling elements or try several selectors.

Conclusion: Choose callbacks or a parse tree

Match the API to the next operation: handlers fit a fixed set of fields, while element queries fit repeated structural lookups.

For callback details and documented limitations, read the Python html.parser documentation.

FAQ

These answers cover installation and the parser’s output structure.