Pyjo.DOM - Minimalistic HTML/XML DOM parser with CSS selectors

from Pyjo.DOM import DOM

# Parse
dom = DOM('<div><p id="a">Test</p><p id="b">123</p></div>')

# Find
print(dom.at('#b').text)
print(dom.find('p').map('text').join("\n"))
print(dom.find('[id]').map('attr', 'id').join("\n"))

# Or import the module, as in Google Python Style Guide

import Pyjo.DOM

# Parse
dom = Pyjo.DOM.new('<div><p id="a">Test</p><p id="b">123</p></div>')

# Find
print(dom.at('#b').text)
print(dom.find('p').map('text').join("\n"))
print(dom.find('[id]').map('attr', 'id').join("\n"))

# Iterate
dom.find('p[id]').reverse().each(lambda e, num: print(e['id']))

# Loop
for e in dom.find('p[id]').each():
    print(e['id'], ':', e.text)

# Modify
dom.find('div p').last().append('<p id="c">456</p>')
dom.at('#c').prepend(dom.new_tag('p', '789', id='d'))
dom.find(':not(p)').map('strip')

# Render
print(dom)

Pyjo.DOM is a minimalistic and relaxed HTML/XML DOM parser with CSS selector support. It will even try to interpret broken HTML and XML, so you should not use it for validation or sanitization. Its interpretation of broken markup can differ from that of a web browser.

Nodes and elements

When we parse an HTML/XML fragment, it gets turned into a tree of nodes.

<!DOCTYPE html>
<html>
  <head><title>Hello</title></head>
  <body>World!</body>
</html>

There are currently eight different kinds of nodes, cdata, comment, doctype, pi, raw, root, tag and text. Elements are nodes of the type tag.

root
|- doctype (html)
+- tag (html)
   |- tag (head)
   |  +- tag (title)
   |     +- text (Hello)
   +- tag (body)
      +- text (World!)

While all node types are represented as Pyjo.DOM objects, some methods like attr() and namespace only apply to elements.

HTML and XML

Pyjo.DOM defaults to HTML semantics, that means all tags and attribute names are lowercased and selectors need to be lowercase as well.

# HTML semantics
dom = Pyjo.DOM.new('<P ID="greeting">Hi!</P>')
print(dom.at('p[id]').text)

If the document starts with an XML declaration, the parser will automatically switch into XML mode and everything becomes case-sensitive.

# XML semantics
dom = Pyjo.DOM.new('<?xml version="1.0"?><P ID="greeting">Hi!</P>')
print(dom.at('P[ID]').text)

HTML or XML semantics can also be forced with the xml attribute.

# Force HTML semantics
dom = Pyjo.DOM.new().set(xml=False).parse('<P ID="greeting">Hi!</P>')
print(dom.at('p[id]').text)

# Force XML semantics
dom = Pyjo.DOM.new().set(xml=True).parse('<P ID="greeting">Hi!</P>')
print(dom.at('P[ID]').text)

Bytes are a whole document, as a file or the body of a response, whose encoding is found as the HTML Living Standard sniffs it: a byte order mark, the encoding of an XML declaration, a <meta charset> or <meta http-equiv="Content-Type"> in the first 1024 bytes, and UTF-8 by default.

# Bytes of a page in windows-1250
dom = Pyjo.DOM.new('<meta charset="windows-1250"><p>Zażółć gęślą jaźń</p>'.encode('windows-1250'))
print(dom.at('p').text)

See also

Pyjoyment, https://mojolicious.org.

Ported from Mojo::DOM of Mojolicious 9.51.

Classes

class Pyjo.DOM.Pyjo_DOM(html: Anything = None)[source]

Pyjo.DOM inherits all methods from Pyjo.Base and Pyjo.String.Mixin and implements the following new ones.

__bool__() → bool[source]
boolean = bool(dom)

Always true.

__contains__(name: Anything) → bool[source]
boolean = 'id' in dom

Check if this element has the attribute.

__delitem__(name: str) → None[source]
del dom['id']

Remove an attribute of this element.

__getitem__(key: str) → str | None[source]
__getitem__(key: int) → Pyjo_DOM
__getitem__(key: slice) → Pyjo_Collection[Pyjo_DOM]
value = dom['id']
node = dom[0]

An attribute of this element, as attr() gives it, or a child node, as child_nodes has it.

# "test"
dom.parse('<div id="test">Test</div>').at('div')['id']

# "<!-- Test -->"
dom.parse('<!-- Test --><b>123</b>')[0]
__init__(html: Anything = None) → None[source]
dom = Pyjo.DOM.new()
dom = Pyjo.DOM.new('<foo bar="baz">I ♥ Pyjoyment!</foo>')

Construct a new Pyjo.DOM object and parse() HTML/XML fragment if necessary.

__iter__() → Iterator[Pyjo_DOM][source]
for node in dom:
    ...

Iterate the child nodes, as child_nodes has them.

__len__() → int[source]
size = len(dom)

Number of child nodes.

__setitem__(name: str, value: str | None) → None[source]
dom['id'] = 'test'

Set an attribute of this element, as attr() does.

property all_text: str[source]
text = dom.all_text

Extract text content from all descendant nodes of this element. For HTML documents script and style elements are excluded.

# "foo\nbarbaz\n"
dom.parse("<div>foo\n<p>bar</p>baz\n</div>").at('div').all_text
ancestors(css: Anything = None) → Pyjo_Collection[Pyjo_DOM][source]
collection = dom.ancestors()
collection = dom.ancestors('div ~ p')

Find all ancestor elements of this node matching the CSS selector and return a Pyjo.Collection object containing these elements as Pyjo.DOM objects. All selectors from Pyjo.DOM.CSS are supported.

# List tag names of ancestor elements
print(dom.ancestors().map('tag').join("\n"))
append(new: Anything) → Self[source]
dom = dom.append('<p>I ♥ Pyjoyment!</p>')
dom = dom.append(Pyjo.DOM.new())

Append HTML/XML fragment to this node (for all node types other than root).

# "<div><h1>Test</h1><h2>123</h2></div>"
dom.parse('<div><h1>Test</h1></div>').at('h1').append('<h2>123</h2>').root

# "<p>Test 123</p>"
dom.parse('<p>Test</p>').at('p').child_nodes.first().append(' 123').root
append_content(new: Anything) → Self[source]
dom = dom.append_content('<p>I ♥ Pyjoyment!</p>')
dom = dom.append_content(Pyjo.DOM.new())

Append HTML/XML fragment (for root and tag nodes) or raw content to this node’s content.

# "<div><h1>Test123</h1></div>"
dom.parse('<div><h1>Test</h1></div>').at('h1').append_content('123').root

# "<!-- Test 123 --><br>"
dom.parse('<!-- Test --><br>').child_nodes.first().append_content('123 ').root

# "<p>Test<i>123</i></p>"
dom.parse('<p>Test</p>').at('p').append_content('<i>123</i>').root
at(css: Anything, **ns: str) → Pyjo_DOM | None[source]
result = dom.at('div ~ p')
result = dom.at('svg|line', svg='http://www.w3.org/2000/svg')

Find first descendant element of this element matching the CSS selector and return it as a Pyjo.DOM object, or None if none could be found. All selectors from Pyjo.DOM.CSS are supported.

# Find first element with "svg" namespace definition
namespace = dom.at('[xmlns\:svg]')['xmlns:svg']

Trailing keyword arguments can be used to declare xml namespace aliases.

# "<rect></rect>"
dom.parse('<svg xmlns="http://www.w3.org/2000/svg"><rect /></svg>').at(
    'svg|rect', svg='http://www.w3.org/2000/svg'
)
attr() → dict[str, str | None][source]
attr(name: str, /) → str | None
attr(name: str, value: str | None, /) → Self
attr(values: dict[str, Any], /) → Self
attr(**values: Any) → Self
attrs = dom.attr()
foo = dom.attr('foo')
dom = dom.attr({'foo': 'bar'})
dom = dom.attr('foo', 'bar')
dom = dom.attr(foo='bar')

This element’s attributes.

# Remove an attribute
del dom.attr()['id']

# Attribute without value
dom.attr(selected=None)

# List id attributes
print(dom.find('*').map('attr', 'id').compact().join("\n"))
property child_nodes: Pyjo_Collection[Pyjo_DOM][source]
collection = dom.child_nodes

Return a Pyjo.Collection object containing all child nodes of this element as Pyjo.DOM objects.

# "<p><b>123</b></p>"
dom.parse('<p>Test<b>123</b></p>').at('p').child_nodes.first().remove()

# "<!DOCTYPE html>"
dom.parse('<!DOCTYPE html><b>123</b>').child_nodes.first()

# " Test "
dom.parse('<b>123</b><!-- Test -->').child_nodes.last().content
children(css: Anything = None) → Pyjo_Collection[Pyjo_DOM][source]
collection = dom.children()
collection = dom.children('div ~ p')

Find all child elements of this element matching the CSS selector and return a Pyjo.Collection object containing these elements as Pyjo.DOM objects. All selectors from Pyjo.DOM.CSS are supported.

# Show tag name of random child element
print(dom.children().shuffle().first().tag)
property content: str[source]
string = dom.content
dom.content = '<p>I ♥ Pyjoyment!</p>'
dom.content = Pyjo.DOM.new()

Return this node’s content or replace it with HTML/XML fragment (for root and tag nodes) or raw content.

# "<b>Test</b>"
dom.parse('<div><b>Test</b></div>').at('div').content

# "<div><h1>123</h1></div>"
dom.parse('<div><h1>Test</h1></div>').at('h1').set(content='123').root

# "<p><i>123</i></p>"
dom.parse('<p>Test</p>').at('p').set(content='<i>123</i>').root

# "<div><h1></h1></div>"
dom.parse('<div><h1>Test</h1></div>').at('h1').set(content='').root

# " Test "
dom.parse('<!-- Test --><br>').child_nodes.first().content

# "<div><!-- 123 -->456</div>"
dom.parse('<div><!-- Test -->456</div>').at('div').child_nodes.first().set(content=' 123 ').root
property descendant_nodes: Pyjo_Collection[Pyjo_DOM][source]
collection = dom.descendant_nodes

Return a Pyjo.Collection object containing all descendant nodes of this element as Pyjo.DOM objects.

# "<p><b>123</b></p>"
dom.parse('<p><!-- Test --><b>123<!-- 456 --></b></p>').descendant_nodes.grep(
    lambda e: e.type == 'comment'
).map('remove').first()

# "<p><b>test</b>test</p>"
dom.parse('<p><b>123</b>456</p>').at('p').descendant_nodes.grep(
    lambda e: e.type == 'text'
).map('content', 'test').first().root
find(css: Anything, **ns: str) → Pyjo_Collection[Pyjo_DOM][source]
collection = dom.find('div ~ p')
collection = dom.find('svg|line', svg='http://www.w3.org/2000/svg')

Find all descendant elements of this element matching the CSS selector and return a Pyjo.Collection object containing these elements as Pyjo.DOM objects. All selectors from Pyjo.DOM.CSS are supported.

# Find a specific element and extract information
id = dom.find('div')[23]['id']

# Extract information from multiple elements
headers = dom.find('h1, h2, h3').map('text').each()

# Count all the different tags
counts = dom.find('*').reduce(lambda a, b: a | {b.tag: a.get(b.tag, 0) + 1}, {})

# Find elements with a class that contains dots
divs = dom.find('div.foo\.bar').each()

Trailing keyword arguments can be used to declare xml namespace aliases.

# "<rect></rect>"
dom.parse('<svg xmlns="http://www.w3.org/2000/svg"><rect /></svg>').find(
    'svg|rect', svg='http://www.w3.org/2000/svg'
).first()
following(css: Anything = None) → Pyjo_Collection[Pyjo_DOM][source]
collection = dom.following()
collection = dom.following('div ~ p')

Find all sibling elements after this node matching the CSS selector and return a Pyjo.Collection object containing these elements as Pyjo.DOM objects. All selectors from Pyjo.DOM.CSS are supported.

# List tags of sibling elements after this node
print(dom.following().map('tag').join("\n"))
property following_nodes: Pyjo_Collection[Pyjo_DOM][source]
collection = dom.following_nodes

Return a Pyjo.Collection object containing all sibling nodes after this node as Pyjo.DOM objects.

# "C"
dom.parse('<p>A</p><!-- B -->C').at('p').following_nodes.last().content
iterate(css: Anything, **ns: str) → Pyjo.Iterator.object[Pyjo_DOM][source]
iterator = dom.iterate('div ~ p')
iterator = dom.iterate('svg|line', svg='http://www.w3.org/2000/svg')

Same as find(), but returns a lazy Pyjo.Iterator, which finds the descendant elements as they are taken, in the order of the document, so that the first one is processed before the last one is found.

# Print the text of each paragraph as soon as it is found
dom.iterate('p').map('all_text').each(lambda text, num: print(text))

# Find no more elements after the first one with a link
link = dom.iterate('p').first(lambda p: p.at('a[href]'))
matches(css: Anything, **ns: str) → bool | None[source]
boolean = dom.matches('div ~ p')
boolean = dom.matches('svg|line', svg='http://www.w3.org/2000/svg')

Check if this element matches the CSS selector. All selectors from Pyjo.DOM.CSS are supported.

# True
dom.parse('<p class="a">A</p>').at('p').matches('.a')
dom.parse('<p class="a">A</p>').at('p').matches('p[class]')

# False
dom.parse('<p class="a">A</p>').at('p').matches('.b')
dom.parse('<p class="a">A</p>').at('p').matches('p[id]')

Trailing keyword arguments can be used to declare xml namespace aliases.

# True
dom.parse('<svg xmlns="http://www.w3.org/2000/svg"><rect /></svg>').matches(
    'svg|rect', svg='http://www.w3.org/2000/svg'
)
property namespace: str | None[source]
namespace = dom.namespace

Find this element’s namespace, or return None if none could be found.

# "http://www.w3.org/2000/svg"
Pyjo.DOM.new('<svg xmlns:svg="http://www.w3.org/2000/svg"><svg:circle>3.14</svg:circle></svg>').at(
    'svg\:circle'
).namespace

# Find namespace for an element with namespace prefix
namespace = dom.at('svg > svg\:circle').namespace

# Find namespace for an element that may or may not have a namespace prefix
namespace = dom.at('svg > circle').namespace
new_tag(**kwargs: Any) → R[source]
tag = Pyjo.DOM.object.new_tag('div')
tag = dom.new_tag('div')
tag = dom.new_tag('div', id='foo', hidden=None)
tag = dom.new_tag('div', 'safe content')
tag = dom.new_tag('div', 'safe content', id='foo')
tag = dom.new_tag('div', 'safe content', data={'pyjo': 'rocks'})
tag = dom.new_tag('div', lambda: 'unsafe content', id='foo')

Construct a new Pyjo.DOM object for an HTML/XML tag with or without attributes and content. The data attribute may contain a dict with key/value pairs to generate attributes from.

# "<br>"
dom.new_tag('br')

# "<div></div>"
dom.new_tag('div')

# "<div hidden id="foo"></div>"
dom.new_tag('div', id='foo', hidden=None)

# "<div>test &amp; 123</div>"
dom.new_tag('div', 'test & 123')

# "<div id="foo">test &amp; 123</div>"
dom.new_tag('div', 'test & 123', id='foo')

# "<div data-bar="test" data-foo="1">test &amp; 123</div>"
dom.new_tag('div', 'test & 123', data={'foo': 1, 'Bar': 'test'})

# "<div id="foo">test & 123</div>"
dom.new_tag('div', lambda: 'test & 123', id='foo')

# "<div>Hello<b>Pyjo!</b></div>"
dom.parse('<div>Hello</div>').at('div').append_content(dom.new_tag('b', 'Pyjo!')).root
property next: Pyjo_DOM | None[source]
sibling = dom.next

Return Pyjo.DOM object for next sibling element, or None if there are no more siblings.

# "<h2>123</h2>"
dom.parse('<div><h1>Test</h1><h2>123</h2></div>').at('h1').next
property next_node: Pyjo_DOM | None[source]
sibling = dom.next_node

Return Pyjo.DOM object for next sibling node, or None if there are no more siblings.

# "456"
dom.parse('<p><b>123</b><!-- Test -->456</p>').at('b').next_node.next_node

# " Test "
dom.parse('<p><b>123</b><!-- Test -->456</p>').at('b').next_node.content
property parent: Pyjo_DOM | None[source]
parent = dom.parent

Return Pyjo.DOM object for parent of this node, or None if this node has no parent.

# "<b><i>Test</i></b>"
dom.parse('<p><b><i>Test</i></b></p>').at('i').parent
parse(html: Anything) → Self[source]
dom = dom.parse('<foo bar="baz">I ♥ Pyjoyment!</foo>')

Parse HTML/XML fragment with Pyjo.DOM.HTML.

# Parse XML
dom = Pyjo.DOM.new().set(xml=True).parse('<foo>I ♥ Pyjoyment!</foo>')
preceding(css: Anything = None) → Pyjo_Collection[Pyjo_DOM][source]
collection = dom.preceding()
collection = dom.preceding('div ~ p')

Find all sibling elements before this node matching the CSS selector and return a Pyjo.Collection object containing these elements as Pyjo.DOM objects. All selectors from Pyjo.DOM.CSS are supported.

# List tags of sibling elements before this node
print(dom.preceding().map('tag').join("\n"))
property preceding_nodes: Pyjo_Collection[Pyjo_DOM][source]
collection = dom.preceding_nodes

Return a Pyjo.Collection object containing all sibling nodes before this node as Pyjo.DOM objects.

# "A"
dom.parse('A<!-- B --><p>C</p>').at('p').preceding_nodes.first().content
prepend(new: Anything) → Self[source]
dom = dom.prepend('<p>I ♥ Pyjoyment!</p>')
dom = dom.prepend(Pyjo.DOM.new())

Prepend HTML/XML fragment to this node (for all node types other than root).

# "<div><h1>Test</h1><h2>123</h2></div>"
dom.parse('<div><h2>123</h2></div>').at('h2').prepend('<h1>Test</h1>').root

# "<p>Test 123</p>"
dom.parse('<p>123</p>').at('p').child_nodes.first().prepend('Test ').root
prepend_content(new: Anything) → Self[source]
dom = dom.prepend_content('<p>I ♥ Pyjoyment!</p>')
dom = dom.prepend_content(Pyjo.DOM.new())

Prepend HTML/XML fragment (for root and tag nodes) or raw content to this node’s content.

# "<div><h2>Test123</h2></div>"
dom.parse('<div><h2>123</h2></div>').at('h2').prepend_content('Test').root

# "<!-- Test 123 --><br>"
dom.parse('<!-- 123 --><br>').child_nodes.first().prepend_content(' Test').root

# "<p><i>123</i>Test</p>"
dom.parse('<p>Test</p>').at('p').prepend_content('<i>123</i>').root
property previous: Pyjo_DOM | None[source]
sibling = dom.previous

Return Pyjo.DOM object for previous sibling element, or None if there are no more siblings.

# "<h1>Test</h1>"
dom.parse('<div><h1>Test</h1><h2>123</h2></div>').at('h2').previous
property previous_node: Pyjo_DOM | None[source]
sibling = dom.previous_node

Return Pyjo.DOM object for previous sibling node, or None if there are no more siblings.

# "123"
dom.parse('<p>123<!-- Test --><b>456</b></p>').at('b').previous_node.previous_node

# " Test "
dom.parse('<p>123<!-- Test --><b>456</b></p>').at('b').previous_node.content
remove() → Pyjo_DOM | None[source]
parent = dom.remove()

Remove this node and return root (for root nodes) or parent.

# "<div></div>"
dom.parse('<div><h1>Test</h1></div>').at('h1').remove()

# "<p><b>456</b></p>"
dom.parse('<p>123<b>456</b></p>').at('p').child_nodes.first().remove().root
replace(new: Anything) → Pyjo_DOM | None[source]
parent = dom.replace('<div>I ♥ Pyjoyment!</div>')
parent = dom.replace(Pyjo.DOM.new())

Replace this node with HTML/XML fragment and return root (for root nodes) or parent.

# "<div><h2>123</h2></div>"
dom.parse('<div><h1>Test</h1></div>').at('h1').replace('<h2>123</h2>')

# "<p><b>123</b></p>"
dom.parse('<p>Test</p>').at('p').child_nodes[0].replace('<b>123</b>').root
property root: Pyjo_DOM[source]
root = dom.root

Return Pyjo.DOM object for root node.

property selector: str | None[source]
selector = dom.selector

Get a unique CSS selector for this element.

# "ul:nth-child(1) > li:nth-child(2)"
dom.parse('<ul><li>Test</li><li>123</li></ul>').find('li').last().selector

# "p:nth-child(1) > b:nth-child(1) > i:nth-child(1)"
dom.parse('<p><b><i>Test</i></b></p>').at('i').selector
strip() → Pyjo_DOM | None[source]
parent = dom.strip()

Remove this element while preserving its content and return parent.

# "<div>Test</div>"
dom.parse('<div><h1>Test</h1></div>').at('h1').strip()
property tag: str | None[source]
tag = dom.tag
dom.tag = 'div'

This element’s tag name.

# List tag names of child elements
print(dom.children().map('tag').join("\n"))
property text: str[source]
text = dom.text

Extract text content from this element only (not including child elements).

# "bar"
dom.parse("<div>foo<p>bar</p>baz</div>").at('p').text

# "foo\nbaz\n"
dom.parse("<div>foo\n<p>bar</p>baz\n</div>").at('div').text

To extract text content from all descendant nodes see all_text.

to_str() → str[source]
string = dom.to_str()

Render this node and its content to HTML/XML.

# "<b>Test</b>"
dom.parse('<div><b>Test</b></div>').at('div b').to_str()
property tree: Tree[source]
tree = dom.tree
dom.tree = ['root']

Document Object Model. Note that this structure should only be used very carefully since it is very dynamic.

property type: str[source]
type = dom.type

This node’s type, usually cdata, comment, doctype, pi, raw, root, tag or text.

# "cdata"
dom.parse('<![CDATA[Test]]>').child_nodes.first().type

# "comment"
dom.parse('<!-- Test -->').child_nodes.first().type

# "doctype"
dom.parse('<!DOCTYPE html>').child_nodes.first().type

# "pi"
dom.parse('<?xml version="1.0"?>').child_nodes.first().type

# "raw"
dom.parse('<script>Test</script>').at('script').child_nodes.first().type

# "root"
dom.parse('<p>Test</p>').type

# "tag"
dom.parse('<p>Test</p>').at('p').type

# "text"
dom.parse('<p>Test</p>').at('p').child_nodes.first().type
property val: str | list[str] | None[source]
value = dom.val

Extract value from form element (such as button, input, option, select and textarea), or return None if this element has no value. In the case of select with multiple attribute, find option elements with selected attribute and return a list with all values, or None if none could be found.

# "a"
dom.parse('<input name=test value=a>').at('input').val

# "b"
dom.parse('<textarea>b</textarea>').at('textarea').val

# "c"
dom.parse('<option value="c">Test</option>').at('option').val

# "d"
dom.parse('<select><option selected>d</option></select>').at('select').val

# "e"
dom.parse('<select multiple><option selected>e</option></select>').at('select').val[0]

# "on"
dom.parse('<input name=test type=checkbox>').at('input').val
wrap(new: Anything) → Self[source]
dom = dom.wrap('<div></div>')
dom = dom.wrap(Pyjo.DOM.new())

Wrap HTML/XML fragment around this node (for all node types other than root), placing it as the last child of the first innermost element.

# "<p>123<b>Test</b></p>"
dom.parse('<b>Test</b>').at('b').wrap('<p>123</p>').root

# "<div><p><b>Test</b></p>123</div>"
dom.parse('<b>Test</b>').at('b').wrap('<div><p></p>123</div>').root

# "<p><b>Test</b></p><p>123</p>"
dom.parse('<b>Test</b>').at('b').wrap('<p></p><p>123</p>').root

# "<p><b>Test</b></p>"
dom.parse('<p>Test</p>').at('p').child_nodes.first().wrap('<b>').root
wrap_content(new: Anything) → Self[source]
dom = dom.wrap_content('<div></div>')
dom = dom.wrap_content(Pyjo.DOM.new())

Wrap HTML/XML fragment around this node’s content (for root and tag nodes), placing it as the last children of the first innermost element.

# "<p><b>123Test</b></p><p></p>"
dom.parse('<p>Test<p>').at('p').wrap_content('<b>123</b>').root

# "<p><b>Test</b></p><p>123</p>"
dom.parse('<b>Test</b>').wrap_content('<p></p><p>123</p>')
property xml: bool | None[source]
boolean = dom.xml
dom.xml = boolean

Disable HTML semantics in parser and activate case-sensitivity, defaults to auto-detection based on XML declarations.