Pyjo.DOM - Minimalistic HTML/XML DOM parser with CSS selectors¶
from Pyjo.DOM import DOM
# Parse
dom = DOM('<div><p id="a">Test</p><p id="b">123</p></div>')
# Find
print(dom.at('#b').text)
print(dom.find('p').map('text').join("\n"))
print(dom.find('[id]').map('attr', 'id').join("\n"))
# Or import the module, as in Google Python Style Guide
import Pyjo.DOM
# Parse
dom = Pyjo.DOM.new('<div><p id="a">Test</p><p id="b">123</p></div>')
# Find
print(dom.at('#b').text)
print(dom.find('p').map('text').join("\n"))
print(dom.find('[id]').map('attr', 'id').join("\n"))
# Iterate
dom.find('p[id]').reverse().each(lambda e, num: print(e['id']))
# Loop
for e in dom.find('p[id]').each():
print(e['id'], ':', e.text)
# Modify
dom.find('div p').last().append('<p id="c">456</p>')
dom.at('#c').prepend(dom.new_tag('p', '789', id='d'))
dom.find(':not(p)').map('strip')
# Render
print(dom)
Pyjo.DOM is a minimalistic and relaxed HTML/XML DOM parser with CSS
selector support. It will even try to interpret broken HTML and XML, so you
should not use it for validation or sanitization. Its interpretation of broken
markup can differ from that of a web browser.
Nodes and elements¶
When we parse an HTML/XML fragment, it gets turned into a tree of nodes.
<!DOCTYPE html>
<html>
<head><title>Hello</title></head>
<body>World!</body>
</html>
There are currently eight different kinds of nodes, cdata, comment,
doctype, pi, raw, root, tag and text. Elements are
nodes of the type tag.
root
|- doctype (html)
+- tag (html)
|- tag (head)
| +- tag (title)
| +- text (Hello)
+- tag (body)
+- text (World!)
While all node types are represented as Pyjo.DOM objects, some methods
like attr() and namespace only apply to
elements.
HTML and XML¶
Pyjo.DOM defaults to HTML semantics, that means all tags and attribute
names are lowercased and selectors need to be lowercase as well.
# HTML semantics
dom = Pyjo.DOM.new('<P ID="greeting">Hi!</P>')
print(dom.at('p[id]').text)
If the document starts with an XML declaration, the parser will automatically switch into XML mode and everything becomes case-sensitive.
# XML semantics
dom = Pyjo.DOM.new('<?xml version="1.0"?><P ID="greeting">Hi!</P>')
print(dom.at('P[ID]').text)
HTML or XML semantics can also be forced with the xml
attribute.
# Force HTML semantics
dom = Pyjo.DOM.new().set(xml=False).parse('<P ID="greeting">Hi!</P>')
print(dom.at('p[id]').text)
# Force XML semantics
dom = Pyjo.DOM.new().set(xml=True).parse('<P ID="greeting">Hi!</P>')
print(dom.at('P[ID]').text)
Bytes are a whole document, as a file or the body of a response, whose
encoding is found as the HTML Living Standard sniffs it: a byte order mark,
the encoding of an XML declaration, a <meta charset> or
<meta http-equiv="Content-Type"> in the first 1024 bytes, and UTF-8
by default.
# Bytes of a page in windows-1250
dom = Pyjo.DOM.new('<meta charset="windows-1250"><p>Zażółć gęślą jaźń</p>'.encode('windows-1250'))
print(dom.at('p').text)
See also¶
Pyjoyment, https://mojolicious.org.
Ported from Mojo::DOM of Mojolicious 9.51.
Classes¶
- class Pyjo.DOM.Pyjo_DOM(html: Anything = None)[source]¶
Pyjo.DOMinherits all methods fromPyjo.BaseandPyjo.String.Mixinand implements the following new ones.- __contains__(name: Anything) bool[source]¶
boolean = 'id' in dom
Check if this element has the attribute.
- __getitem__(key: str) str | None[source]¶
- __getitem__(key: int) Pyjo_DOM
- __getitem__(key: slice) Pyjo_Collection[Pyjo_DOM]
value = dom['id'] node = dom[0]
An attribute of this element, as
attr()gives it, or a child node, aschild_nodeshas it.# "test" dom.parse('<div id="test">Test</div>').at('div')['id'] # "<!-- Test -->" dom.parse('<!-- Test --><b>123</b>')[0]
- __init__(html: Anything = None) None[source]¶
dom = Pyjo.DOM.new() dom = Pyjo.DOM.new('<foo bar="baz">I ♥ Pyjoyment!</foo>')
Construct a new
Pyjo.DOMobject andparse()HTML/XML fragment if necessary.
- __iter__() Iterator[Pyjo_DOM][source]¶
for node in dom: ...
Iterate the child nodes, as
child_nodeshas them.
- __setitem__(name: str, value: str | None) None[source]¶
dom['id'] = 'test'
Set an attribute of this element, as
attr()does.
- property all_text: str[source]¶
text = dom.all_text
Extract text content from all descendant nodes of this element. For HTML documents
scriptandstyleelements are excluded.# "foo\nbarbaz\n" dom.parse("<div>foo\n<p>bar</p>baz\n</div>").at('div').all_text
- ancestors(css: Anything = None) Pyjo_Collection[Pyjo_DOM][source]¶
collection = dom.ancestors() collection = dom.ancestors('div ~ p')
Find all ancestor elements of this node matching the CSS selector and return a
Pyjo.Collectionobject containing these elements asPyjo.DOMobjects. All selectors fromPyjo.DOM.CSSare supported.# List tag names of ancestor elements print(dom.ancestors().map('tag').join("\n"))
- append(new: Anything) Self[source]¶
dom = dom.append('<p>I ♥ Pyjoyment!</p>') dom = dom.append(Pyjo.DOM.new())
Append HTML/XML fragment to this node (for all node types other than
root).# "<div><h1>Test</h1><h2>123</h2></div>" dom.parse('<div><h1>Test</h1></div>').at('h1').append('<h2>123</h2>').root # "<p>Test 123</p>" dom.parse('<p>Test</p>').at('p').child_nodes.first().append(' 123').root
- append_content(new: Anything) Self[source]¶
dom = dom.append_content('<p>I ♥ Pyjoyment!</p>') dom = dom.append_content(Pyjo.DOM.new())
Append HTML/XML fragment (for
rootandtagnodes) or raw content to this node’s content.# "<div><h1>Test123</h1></div>" dom.parse('<div><h1>Test</h1></div>').at('h1').append_content('123').root # "<!-- Test 123 --><br>" dom.parse('<!-- Test --><br>').child_nodes.first().append_content('123 ').root # "<p>Test<i>123</i></p>" dom.parse('<p>Test</p>').at('p').append_content('<i>123</i>').root
- at(css: Anything, **ns: str) Pyjo_DOM | None[source]¶
result = dom.at('div ~ p') result = dom.at('svg|line', svg='http://www.w3.org/2000/svg')
Find first descendant element of this element matching the CSS selector and return it as a
Pyjo.DOMobject, orNoneif none could be found. All selectors fromPyjo.DOM.CSSare supported.# Find first element with "svg" namespace definition namespace = dom.at('[xmlns\:svg]')['xmlns:svg']
Trailing keyword arguments can be used to declare xml namespace aliases.
# "<rect></rect>" dom.parse('<svg xmlns="http://www.w3.org/2000/svg"><rect /></svg>').at( 'svg|rect', svg='http://www.w3.org/2000/svg' )
- attr() dict[str, str | None][source]¶
- attr(name: str, /) str | None
- attr(name: str, value: str | None, /) Self
- attr(values: dict[str, Any], /) Self
- attr(**values: Any) Self
attrs = dom.attr() foo = dom.attr('foo') dom = dom.attr({'foo': 'bar'}) dom = dom.attr('foo', 'bar') dom = dom.attr(foo='bar')
This element’s attributes.
# Remove an attribute del dom.attr()['id'] # Attribute without value dom.attr(selected=None) # List id attributes print(dom.find('*').map('attr', 'id').compact().join("\n"))
- property child_nodes: Pyjo_Collection[Pyjo_DOM][source]¶
collection = dom.child_nodes
Return a
Pyjo.Collectionobject containing all child nodes of this element asPyjo.DOMobjects.# "<p><b>123</b></p>" dom.parse('<p>Test<b>123</b></p>').at('p').child_nodes.first().remove() # "<!DOCTYPE html>" dom.parse('<!DOCTYPE html><b>123</b>').child_nodes.first() # " Test " dom.parse('<b>123</b><!-- Test -->').child_nodes.last().content
- children(css: Anything = None) Pyjo_Collection[Pyjo_DOM][source]¶
collection = dom.children() collection = dom.children('div ~ p')
Find all child elements of this element matching the CSS selector and return a
Pyjo.Collectionobject containing these elements asPyjo.DOMobjects. All selectors fromPyjo.DOM.CSSare supported.# Show tag name of random child element print(dom.children().shuffle().first().tag)
- property content: str[source]¶
string = dom.content dom.content = '<p>I ♥ Pyjoyment!</p>' dom.content = Pyjo.DOM.new()
Return this node’s content or replace it with HTML/XML fragment (for
rootandtagnodes) or raw content.# "<b>Test</b>" dom.parse('<div><b>Test</b></div>').at('div').content # "<div><h1>123</h1></div>" dom.parse('<div><h1>Test</h1></div>').at('h1').set(content='123').root # "<p><i>123</i></p>" dom.parse('<p>Test</p>').at('p').set(content='<i>123</i>').root # "<div><h1></h1></div>" dom.parse('<div><h1>Test</h1></div>').at('h1').set(content='').root # " Test " dom.parse('<!-- Test --><br>').child_nodes.first().content # "<div><!-- 123 -->456</div>" dom.parse('<div><!-- Test -->456</div>').at('div').child_nodes.first().set(content=' 123 ').root
- property descendant_nodes: Pyjo_Collection[Pyjo_DOM][source]¶
collection = dom.descendant_nodes
Return a
Pyjo.Collectionobject containing all descendant nodes of this element asPyjo.DOMobjects.# "<p><b>123</b></p>" dom.parse('<p><!-- Test --><b>123<!-- 456 --></b></p>').descendant_nodes.grep( lambda e: e.type == 'comment' ).map('remove').first() # "<p><b>test</b>test</p>" dom.parse('<p><b>123</b>456</p>').at('p').descendant_nodes.grep( lambda e: e.type == 'text' ).map('content', 'test').first().root
- find(css: Anything, **ns: str) Pyjo_Collection[Pyjo_DOM][source]¶
collection = dom.find('div ~ p') collection = dom.find('svg|line', svg='http://www.w3.org/2000/svg')
Find all descendant elements of this element matching the CSS selector and return a
Pyjo.Collectionobject containing these elements asPyjo.DOMobjects. All selectors fromPyjo.DOM.CSSare supported.# Find a specific element and extract information id = dom.find('div')[23]['id'] # Extract information from multiple elements headers = dom.find('h1, h2, h3').map('text').each() # Count all the different tags counts = dom.find('*').reduce(lambda a, b: a | {b.tag: a.get(b.tag, 0) + 1}, {}) # Find elements with a class that contains dots divs = dom.find('div.foo\.bar').each()
Trailing keyword arguments can be used to declare xml namespace aliases.
# "<rect></rect>" dom.parse('<svg xmlns="http://www.w3.org/2000/svg"><rect /></svg>').find( 'svg|rect', svg='http://www.w3.org/2000/svg' ).first()
- following(css: Anything = None) Pyjo_Collection[Pyjo_DOM][source]¶
collection = dom.following() collection = dom.following('div ~ p')
Find all sibling elements after this node matching the CSS selector and return a
Pyjo.Collectionobject containing these elements asPyjo.DOMobjects. All selectors fromPyjo.DOM.CSSare supported.# List tags of sibling elements after this node print(dom.following().map('tag').join("\n"))
- property following_nodes: Pyjo_Collection[Pyjo_DOM][source]¶
collection = dom.following_nodes
Return a
Pyjo.Collectionobject containing all sibling nodes after this node asPyjo.DOMobjects.# "C" dom.parse('<p>A</p><!-- B -->C').at('p').following_nodes.last().content
- iterate(css: Anything, **ns: str) Pyjo.Iterator.object[Pyjo_DOM][source]¶
iterator = dom.iterate('div ~ p') iterator = dom.iterate('svg|line', svg='http://www.w3.org/2000/svg')
Same as
find(), but returns a lazyPyjo.Iterator, which finds the descendant elements as they are taken, in the order of the document, so that the first one is processed before the last one is found.# Print the text of each paragraph as soon as it is found dom.iterate('p').map('all_text').each(lambda text, num: print(text)) # Find no more elements after the first one with a link link = dom.iterate('p').first(lambda p: p.at('a[href]'))
- matches(css: Anything, **ns: str) bool | None[source]¶
boolean = dom.matches('div ~ p') boolean = dom.matches('svg|line', svg='http://www.w3.org/2000/svg')
Check if this element matches the CSS selector. All selectors from
Pyjo.DOM.CSSare supported.# True dom.parse('<p class="a">A</p>').at('p').matches('.a') dom.parse('<p class="a">A</p>').at('p').matches('p[class]') # False dom.parse('<p class="a">A</p>').at('p').matches('.b') dom.parse('<p class="a">A</p>').at('p').matches('p[id]')
Trailing keyword arguments can be used to declare xml namespace aliases.
# True dom.parse('<svg xmlns="http://www.w3.org/2000/svg"><rect /></svg>').matches( 'svg|rect', svg='http://www.w3.org/2000/svg' )
- property namespace: str | None[source]¶
namespace = dom.namespace
Find this element’s namespace, or return
Noneif none could be found.# "http://www.w3.org/2000/svg" Pyjo.DOM.new('<svg xmlns:svg="http://www.w3.org/2000/svg"><svg:circle>3.14</svg:circle></svg>').at( 'svg\:circle' ).namespace # Find namespace for an element with namespace prefix namespace = dom.at('svg > svg\:circle').namespace # Find namespace for an element that may or may not have a namespace prefix namespace = dom.at('svg > circle').namespace
- new_tag(**kwargs: Any) R[source]¶
tag = Pyjo.DOM.object.new_tag('div') tag = dom.new_tag('div') tag = dom.new_tag('div', id='foo', hidden=None) tag = dom.new_tag('div', 'safe content') tag = dom.new_tag('div', 'safe content', id='foo') tag = dom.new_tag('div', 'safe content', data={'pyjo': 'rocks'}) tag = dom.new_tag('div', lambda: 'unsafe content', id='foo')
Construct a new
Pyjo.DOMobject for an HTML/XML tag with or without attributes and content. Thedataattribute may contain a dict with key/value pairs to generate attributes from.# "<br>" dom.new_tag('br') # "<div></div>" dom.new_tag('div') # "<div hidden id="foo"></div>" dom.new_tag('div', id='foo', hidden=None) # "<div>test & 123</div>" dom.new_tag('div', 'test & 123') # "<div id="foo">test & 123</div>" dom.new_tag('div', 'test & 123', id='foo') # "<div data-bar="test" data-foo="1">test & 123</div>" dom.new_tag('div', 'test & 123', data={'foo': 1, 'Bar': 'test'}) # "<div id="foo">test & 123</div>" dom.new_tag('div', lambda: 'test & 123', id='foo') # "<div>Hello<b>Pyjo!</b></div>" dom.parse('<div>Hello</div>').at('div').append_content(dom.new_tag('b', 'Pyjo!')).root
- property next: Pyjo_DOM | None[source]¶
sibling = dom.next
Return
Pyjo.DOMobject for next sibling element, orNoneif there are no more siblings.# "<h2>123</h2>" dom.parse('<div><h1>Test</h1><h2>123</h2></div>').at('h1').next
- property next_node: Pyjo_DOM | None[source]¶
sibling = dom.next_node
Return
Pyjo.DOMobject for next sibling node, orNoneif there are no more siblings.# "456" dom.parse('<p><b>123</b><!-- Test -->456</p>').at('b').next_node.next_node # " Test " dom.parse('<p><b>123</b><!-- Test -->456</p>').at('b').next_node.content
- property parent: Pyjo_DOM | None[source]¶
parent = dom.parent
Return
Pyjo.DOMobject for parent of this node, orNoneif this node has no parent.# "<b><i>Test</i></b>" dom.parse('<p><b><i>Test</i></b></p>').at('i').parent
- parse(html: Anything) Self[source]¶
dom = dom.parse('<foo bar="baz">I ♥ Pyjoyment!</foo>')
Parse HTML/XML fragment with
Pyjo.DOM.HTML.# Parse XML dom = Pyjo.DOM.new().set(xml=True).parse('<foo>I ♥ Pyjoyment!</foo>')
- preceding(css: Anything = None) Pyjo_Collection[Pyjo_DOM][source]¶
collection = dom.preceding() collection = dom.preceding('div ~ p')
Find all sibling elements before this node matching the CSS selector and return a
Pyjo.Collectionobject containing these elements asPyjo.DOMobjects. All selectors fromPyjo.DOM.CSSare supported.# List tags of sibling elements before this node print(dom.preceding().map('tag').join("\n"))
- property preceding_nodes: Pyjo_Collection[Pyjo_DOM][source]¶
collection = dom.preceding_nodes
Return a
Pyjo.Collectionobject containing all sibling nodes before this node asPyjo.DOMobjects.# "A" dom.parse('A<!-- B --><p>C</p>').at('p').preceding_nodes.first().content
- prepend(new: Anything) Self[source]¶
dom = dom.prepend('<p>I ♥ Pyjoyment!</p>') dom = dom.prepend(Pyjo.DOM.new())
Prepend HTML/XML fragment to this node (for all node types other than
root).# "<div><h1>Test</h1><h2>123</h2></div>" dom.parse('<div><h2>123</h2></div>').at('h2').prepend('<h1>Test</h1>').root # "<p>Test 123</p>" dom.parse('<p>123</p>').at('p').child_nodes.first().prepend('Test ').root
- prepend_content(new: Anything) Self[source]¶
dom = dom.prepend_content('<p>I ♥ Pyjoyment!</p>') dom = dom.prepend_content(Pyjo.DOM.new())
Prepend HTML/XML fragment (for
rootandtagnodes) or raw content to this node’s content.# "<div><h2>Test123</h2></div>" dom.parse('<div><h2>123</h2></div>').at('h2').prepend_content('Test').root # "<!-- Test 123 --><br>" dom.parse('<!-- 123 --><br>').child_nodes.first().prepend_content(' Test').root # "<p><i>123</i>Test</p>" dom.parse('<p>Test</p>').at('p').prepend_content('<i>123</i>').root
- property previous: Pyjo_DOM | None[source]¶
sibling = dom.previous
Return
Pyjo.DOMobject for previous sibling element, orNoneif there are no more siblings.# "<h1>Test</h1>" dom.parse('<div><h1>Test</h1><h2>123</h2></div>').at('h2').previous
- property previous_node: Pyjo_DOM | None[source]¶
sibling = dom.previous_node
Return
Pyjo.DOMobject for previous sibling node, orNoneif there are no more siblings.# "123" dom.parse('<p>123<!-- Test --><b>456</b></p>').at('b').previous_node.previous_node # " Test " dom.parse('<p>123<!-- Test --><b>456</b></p>').at('b').previous_node.content
- remove() Pyjo_DOM | None[source]¶
parent = dom.remove()
Remove this node and return
root(forrootnodes) orparent.# "<div></div>" dom.parse('<div><h1>Test</h1></div>').at('h1').remove() # "<p><b>456</b></p>" dom.parse('<p>123<b>456</b></p>').at('p').child_nodes.first().remove().root
- replace(new: Anything) Pyjo_DOM | None[source]¶
parent = dom.replace('<div>I ♥ Pyjoyment!</div>') parent = dom.replace(Pyjo.DOM.new())
Replace this node with HTML/XML fragment and return
root(forrootnodes) orparent.# "<div><h2>123</h2></div>" dom.parse('<div><h1>Test</h1></div>').at('h1').replace('<h2>123</h2>') # "<p><b>123</b></p>" dom.parse('<p>Test</p>').at('p').child_nodes[0].replace('<b>123</b>').root
- property selector: str | None[source]¶
selector = dom.selector
Get a unique CSS selector for this element.
# "ul:nth-child(1) > li:nth-child(2)" dom.parse('<ul><li>Test</li><li>123</li></ul>').find('li').last().selector # "p:nth-child(1) > b:nth-child(1) > i:nth-child(1)" dom.parse('<p><b><i>Test</i></b></p>').at('i').selector
- strip() Pyjo_DOM | None[source]¶
parent = dom.strip()
Remove this element while preserving its content and return
parent.# "<div>Test</div>" dom.parse('<div><h1>Test</h1></div>').at('h1').strip()
- property tag: str | None[source]¶
tag = dom.tag dom.tag = 'div'
This element’s tag name.
# List tag names of child elements print(dom.children().map('tag').join("\n"))
- property text: str[source]¶
text = dom.text
Extract text content from this element only (not including child elements).
# "bar" dom.parse("<div>foo<p>bar</p>baz</div>").at('p').text # "foo\nbaz\n" dom.parse("<div>foo\n<p>bar</p>baz\n</div>").at('div').text
To extract text content from all descendant nodes see
all_text.
- to_str() str[source]¶
string = dom.to_str()
Render this node and its content to HTML/XML.
# "<b>Test</b>" dom.parse('<div><b>Test</b></div>').at('div b').to_str()
- property tree: Tree[source]¶
tree = dom.tree dom.tree = ['root']
Document Object Model. Note that this structure should only be used very carefully since it is very dynamic.
- property type: str[source]¶
type = dom.type
This node’s type, usually
cdata,comment,doctype,pi,raw,root,tagortext.# "cdata" dom.parse('<![CDATA[Test]]>').child_nodes.first().type # "comment" dom.parse('<!-- Test -->').child_nodes.first().type # "doctype" dom.parse('<!DOCTYPE html>').child_nodes.first().type # "pi" dom.parse('<?xml version="1.0"?>').child_nodes.first().type # "raw" dom.parse('<script>Test</script>').at('script').child_nodes.first().type # "root" dom.parse('<p>Test</p>').type # "tag" dom.parse('<p>Test</p>').at('p').type # "text" dom.parse('<p>Test</p>').at('p').child_nodes.first().type
- property val: str | list[str] | None[source]¶
value = dom.val
Extract value from form element (such as
button,input,option,selectandtextarea), or returnNoneif this element has no value. In the case ofselectwithmultipleattribute, findoptionelements withselectedattribute and return a list with all values, orNoneif none could be found.# "a" dom.parse('<input name=test value=a>').at('input').val # "b" dom.parse('<textarea>b</textarea>').at('textarea').val # "c" dom.parse('<option value="c">Test</option>').at('option').val # "d" dom.parse('<select><option selected>d</option></select>').at('select').val # "e" dom.parse('<select multiple><option selected>e</option></select>').at('select').val[0] # "on" dom.parse('<input name=test type=checkbox>').at('input').val
- wrap(new: Anything) Self[source]¶
dom = dom.wrap('<div></div>') dom = dom.wrap(Pyjo.DOM.new())
Wrap HTML/XML fragment around this node (for all node types other than
root), placing it as the last child of the first innermost element.# "<p>123<b>Test</b></p>" dom.parse('<b>Test</b>').at('b').wrap('<p>123</p>').root # "<div><p><b>Test</b></p>123</div>" dom.parse('<b>Test</b>').at('b').wrap('<div><p></p>123</div>').root # "<p><b>Test</b></p><p>123</p>" dom.parse('<b>Test</b>').at('b').wrap('<p></p><p>123</p>').root # "<p><b>Test</b></p>" dom.parse('<p>Test</p>').at('p').child_nodes.first().wrap('<b>').root
- wrap_content(new: Anything) Self[source]¶
dom = dom.wrap_content('<div></div>') dom = dom.wrap_content(Pyjo.DOM.new())
Wrap HTML/XML fragment around this node’s content (for
rootandtagnodes), placing it as the last children of the first innermost element.# "<p><b>123Test</b></p><p></p>" dom.parse('<p>Test<p>').at('p').wrap_content('<b>123</b>').root # "<p><b>Test</b></p><p>123</p>" dom.parse('<b>Test</b>').wrap_content('<p></p><p>123</p>')