w3lib Package

encoding Module

Functions for handling encoding of web pages

w3lib.encoding.html_body_declared_encoding(html_body_str: str | bytes) → str | None[source]

Return the encoding specified in meta tags in the html body, or None if no suitable encoding was found

>>> import w3lib.encoding
>>> w3lib.encoding.html_body_declared_encoding(
... """<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
...      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
... <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
... <head>
...     <title>Some title</title>
...     <meta http-equiv="content-type" content="text/html;charset=utf-8" />
... </head>
... <body>
... ...
... </body>
... </html>""")
'utf-8'
>>>
w3lib.encoding.html_encoding(content_type_header: str | None, html_body_str: bytes, *, url: str | None = None) → str[source]

Return the encoding html_to_unicode() would decode html_body_str with, given the same url, without decoding it.

Added in version 2.6.0.

>>> import w3lib.encoding
>>> w3lib.encoding.html_encoding(None, b'<meta charset="latin1">')
'cp1252'
w3lib.encoding.html_to_unicode(content_type_header: str | None, html_body_str: bytes, default_encoding: str | None = None, auto_detect_fun: Callable[[bytes], str | None] | None = None, *, url: str | None = None) → tuple[str, str][source]

Convert raw html bytes to unicode

Added in version 2.6.0: The url parameter.

This finds the encoding of the html body the way a web browser does, trying in order:

  • BOM (byte-order mark)

  • http content type header

  • meta or xml tag declarations

  • UTF-8, if the body is valid UTF-8

  • the encoding guessed from the bytes of the body and the top-level domain of url, the URL of the body, as Firefox does

If a BOM is found, it will be stripped.

If content_type_header is not present, None can be passed signifying that the header was not present.

This method will not fail, if characters cannot be converted to unicode, \ufffd (the unicode replacement character) will be inserted instead.

Returns a tuple of (<encoding used>, <unicode_string>)

Examples:

>>> import w3lib.encoding
>>> w3lib.encoding.html_to_unicode(None,
... b"""<!DOCTYPE html>
... <head>
... <meta charset="UTF-8" />
... <meta name="viewport" content="width=device-width" />
... <title>Creative Commons France</title>
... <link rel='canonical' href='http://creativecommons.fr/' />
... <body>
... <p>Creative Commons est une organisation \xc3\xa0 but non lucratif
... qui a pour dessein de faciliter la diffusion et le partage des oeuvres
... tout en accompagnant les nouvelles pratiques de cr\xc3\xa9ation \xc3\xa0 l\xe2\x80\x99\xc3\xa8re numerique.</p>
... </body>
... </html>""")
('utf-8', '<!DOCTYPE html>\n<head>\n<meta charset="UTF-8" />\n<meta name="viewport" content="width=device-width" />\n<title>Creative Commons France</title>\n<link rel=\'canonical\' href=\'http://creativecommons.fr/\' />\n<body>\n<p>Creative Commons est une organisation \xe0 but non lucratif\nqui a pour dessein de faciliter la diffusion et le partage des oeuvres\ntout en accompagnant les nouvelles pratiques de cr\xe9ation \xe0 l\u2019\xe8re numerique.</p>\n</body>\n</html>')
>>>
w3lib.encoding.http_content_type_encoding(content_type: str | None) → str | None[source]

Extract the encoding in the content-type header

>>> import w3lib.encoding
>>> w3lib.encoding.http_content_type_encoding("Content-Type: text/html; charset=ISO-8859-4")
'iso8859-4'
w3lib.encoding.read_bom(data: bytes) → tuple[None, None] | tuple[str, bytes][source]

Read the byte order mark in the text, if present, and return the encoding represented by the BOM and the BOM.

If no BOM can be detected, (None, None) is returned.

As in browsers, a UTF-32LE BOM reads as a UTF-16LE one, and a UTF-16 body that starts with an XML declaration and no BOM gives an empty BOM.

>>> import w3lib.encoding
>>> w3lib.encoding.read_bom(b'\xfe\xff\x6c\x34')
('utf-16-be', b'\xfe\xff')
>>> w3lib.encoding.read_bom(b'\xff\xfe\x34\x6c')
('utf-16-le', b'\xff\xfe')
>>> w3lib.encoding.read_bom(b'\x01\x02\x03\x04')
(None, None)
>>>
w3lib.encoding.resolve_encoding(encoding_alias: str) → str | None[source]

Return the Python codec name of the encoding of the Encoding Standard that encoding_alias maps to, or None if there is none, e.g. for utf-32.

>>> import w3lib.encoding
>>> w3lib.encoding.resolve_encoding('latin1')
'cp1252'
>>> w3lib.encoding.resolve_encoding('gb_2312-80')
'gbk'
>>> w3lib.encoding.resolve_encoding('base64')
>>>
w3lib.encoding.to_unicode(data_str: bytes, encoding: str) → str[source]

Convert a str object to unicode using the encoding given

If encoding names an encoding of the Encoding Standard, data_str is decoded the way browsers decode it, e.g. utf-16 is UTF-16LE.

Characters that cannot be converted will be converted to \ufffd (the unicode replacement character).

html Module

Functions for dealing with markup text

w3lib.html.get_base_url(text: str | bytes, baseurl: str | bytes = '', encoding: str = 'utf-8', *, max_scan: int | None = None) → str[source]

Return the base URL of the HTML text, as a web browser finds it.

baseurl is the absolute URL of the document, which is also the result when text declares no usable base URL.

w3lib.html.get_meta_refresh(text: str | bytes, baseurl: str = '', encoding: str = 'utf-8', ignore_tags: Iterable[str] = ('script', 'noscript'), *, max_scan: int | None = None) → tuple[None, None] | tuple[float, str][source]

Return (interval, url) for the meta refresh of the HTML text that a web browser follows, or (None, None) if there is none.

interval is the delay in seconds, and url is the absolute URL to load, resolved against baseurl, the absolute URL of the document, and against a <base> element of text.

<noscript> content counts only if ignore_tags does not contain "noscript", as in a browser with JavaScript disabled.

w3lib.html.remove_comments(text: str | bytes, encoding: str | None = None) → str[source]

Remove HTML Comments.

>>> import w3lib.html
>>> w3lib.html.remove_comments(b"test <!--textcoment--> whatever")
'test  whatever'
>>>
w3lib.html.remove_tags(text: str | bytes, which_ones: Iterable[str] = (), keep: Iterable[str] = (), encoding: str | None = None) → str[source]

Remove HTML Tags only.

which_ones and keep are both tuples, there are four cases:

which_ones

keep

what it does

not empty

empty

remove all tags in which_ones

empty

not empty

remove all tags except the ones in keep

empty

empty

remove all tags

not empty

not empty

not allowed

Remove all tags:

>>> import w3lib.html
>>> doc = '<div><p><b>This is a link:</b> <a href="http://www.example.com">example</a></p></div>'
>>> w3lib.html.remove_tags(doc)
'This is a link: example'
>>>

Keep only some tags:

>>> w3lib.html.remove_tags(doc, keep=('div',))
'<div>This is a link: example</div>'
>>>

Remove only specific tags:

>>> w3lib.html.remove_tags(doc, which_ones=('a','b'))
'<div><p>This is a link: example</p></div>'
>>>

You can’t remove some and keep some:

>>> w3lib.html.remove_tags(doc, which_ones=('a',), keep=('p',))
Traceback (most recent call last):
    ...
ValueError: Cannot use both which_ones and keep
>>>
w3lib.html.remove_tags_with_content(text: str | bytes, which_ones: Iterable[str] = (), encoding: str | None = None) → str[source]

Remove tags and their content.

which_ones is a tuple of which tags to remove including their content. If is empty, returns the string unmodified.

>>> import w3lib.html
>>> doc = '<div><p><b>This is a link:</b> <a href="http://www.example.com">example</a></p></div>'
>>> w3lib.html.remove_tags_with_content(doc, which_ones=('b',))
'<div><p> <a href="http://www.example.com">example</a></p></div>'
>>>
w3lib.html.replace_entities(text: str | bytes, keep: Iterable[str] = (), remove_illegal: bool = True, encoding: str = 'utf-8') → str[source]

Remove entities from the given text by converting them to their corresponding unicode character.

text can be a unicode string or a byte string encoded in the given encoding (which defaults to ‘utf-8’).

If keep is passed (with a list of entity names) those entities will be kept (they won’t be removed).

It supports both numeric entities (&#nnnn; and &#hhhh;) and named entities (such as &nbsp; or &gt;).

If remove_illegal is True, entities that can’t be converted are removed. If remove_illegal is False, entities that can’t be converted are kept “as is”. For more information see the tests.

Always returns a unicode string (with the entities removed).

>>> import w3lib.html
>>> w3lib.html.replace_entities(b'Price: &pound;100')
'Price: \xa3100'
>>> print(w3lib.html.replace_entities(b'Price: &pound;100'))
Price: £100
>>>
w3lib.html.replace_escape_chars(text: str | bytes, which_ones: Iterable[str] = ('\n', '\t', '\r'), replace_by: str | bytes = '', encoding: str | None = None) → str[source]

Remove escape characters.

which_ones is a tuple of which escape characters we want to remove. By default removes \n, \t, \r.

replace_by is the string to replace the escape characters by. It defaults to '', meaning the escape characters are removed.

w3lib.html.replace_tags(text: str | bytes, token: str = '', encoding: str | None = None) → str[source]

Replace all markup tags found in the given text by the given token. By default token is an empty string so it just removes all tags.

text can be a unicode string or a regular string encoded as encoding (or 'utf-8' if encoding is not given.)

Always returns a unicode string.

Examples:

>>> import w3lib.html
>>> w3lib.html.replace_tags('This text contains <a>some tag</a>')
'This text contains some tag'
>>> w3lib.html.replace_tags('<p>Je ne parle pas <b>fran\xe7ais</b></p>', ' -- ', 'latin-1')
' -- Je ne parle pas  -- fran\xe7ais --  -- '
>>>
w3lib.html.strip_html5_whitespace(text: str) → str[source]

Strip all leading and trailing space characters (as defined in https://www.w3.org/TR/html5/infrastructure.html#space-character).

Such stripping is useful e.g. for processing HTML element attributes which contain URLs, like href, src or form action - HTML5 standard defines them as “valid URL potentially surrounded by spaces” or “valid non-empty URL potentially surrounded by spaces”.

>>> strip_html5_whitespace(' hello\n')
'hello'
w3lib.html.unquote_markup(text: str | bytes, keep: Iterable[str] = (), remove_illegal: bool = True, encoding: str | None = None) → str[source]

This function receives markup as a text (always a unicode string or a UTF-8 encoded string) and does the following:

  1. removes entities (except the ones in keep) from any part of it

    that is not inside a CDATA

  2. searches for CDATAs and extracts their text (if any) without modifying it.

  3. removes the found CDATAs

http Module

w3lib.http.basic_auth_header(username: str | bytes, password: str | bytes, encoding: str = 'ISO-8859-1') → bytes[source]

Return an Authorization header field value for HTTP Basic Access Authentication (RFC 2617)

>>> import w3lib.http
>>> w3lib.http.basic_auth_header('someuser', 'somepass')
b'Basic c29tZXVzZXI6c29tZXBhc3M='
w3lib.http.headers_dict_to_raw(headers_dict: Mapping[bytes, Any | Sequence[bytes]]) → bytes[source]
w3lib.http.headers_dict_to_raw(headers_dict: None) → None

Returns a raw HTTP headers representation of headers

For example:

>>> import w3lib.http
>>> w3lib.http.headers_dict_to_raw({b'Content-type': b'text/html', b'Accept': b'gzip'})
'Content-type: text/html\\r\\nAccept: gzip'
>>>

Note that keys and values must be bytes.

Argument is None (returns None):

>>> w3lib.http.headers_dict_to_raw(None)
>>>
w3lib.http.headers_raw_to_dict(headers_raw: bytes) → MutableMapping[bytes, list[bytes]][source]
w3lib.http.headers_raw_to_dict(headers_raw: None) → None

Convert raw headers (single multi-line bytestring) to a dictionary.

For example:

>>> import w3lib.http
>>> w3lib.http.headers_raw_to_dict(b"Content-type: text/html\n\rAccept: gzip\n\n")
{'Content-type': ['text/html'], 'Accept': ['gzip']}

Incorrect input:

>>> w3lib.http.headers_raw_to_dict(b"Content-typt gzip\n\n")
{}
>>>

Argument is None (return None):

>>> w3lib.http.headers_raw_to_dict(None)
>>>

url Module

This module contains general purpose URL functions not found in the standard library.

class w3lib.url.ParseDataURIResult(media_type: str, media_type_parameters: dict[str, str], data: bytes)[source]

Named tuple returned by parse_data_uri().

data: bytes

Data, decoded if it was encoded in base64 format.

media_type: str

MIME type type and subtype, separated by / (e.g. "text/plain").

media_type_parameters: dict[str, str]

MIME type parameters (e.g. {"charset": "US-ASCII"}).

w3lib.url.add_http_if_no_scheme(url: str) → str[source]

Add http as the default scheme if it is missing from url.

w3lib.url.add_or_replace_parameter(url: str, name: str, new_value: str, *, separator: str = '&') → str[source]

Add or remove a parameter to a given url

>>> import w3lib.url
>>> w3lib.url.add_or_replace_parameter('http://www.example.com/index.php', 'arg', 'v')
'http://www.example.com/index.php?arg=v'
>>> w3lib.url.add_or_replace_parameter('http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3', 'arg4', 'v4')
'http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3&arg4=v4'
>>> w3lib.url.add_or_replace_parameter('http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3', 'arg3', 'v3new')
'http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3new'
>>> w3lib.url.add_or_replace_parameter('http://www.example.com/index.php?arg1=v1;arg2=v2', 'arg3', 'v3', separator=';')
'http://www.example.com/index.php?arg1=v1;arg2=v2;arg3=v3'
>>>

separator is used both to split the existing query and to join the resulting one.

w3lib.url.add_or_replace_parameters(url: str, new_parameters: dict[str, str], *, separator: str = '&') → str[source]

Add or remove a parameters to a given url

>>> import w3lib.url
>>> w3lib.url.add_or_replace_parameters('http://www.example.com/index.php', {'arg': 'v'})
'http://www.example.com/index.php?arg=v'
>>> args = {'arg4': 'v4', 'arg3': 'v3new'}
>>> w3lib.url.add_or_replace_parameters('http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3', args)
'http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3new&arg4=v4'
>>>
w3lib.url.any_to_uri(uri_or_path: str) → str[source]

If given a path name, return its File URI, otherwise return it unmodified

w3lib.url.canonicalize_url(url: str | bytes | ParseResult, keep_blank_values: bool = True, keep_fragments: bool = False, encoding: str | None = None, *, query_separator: str = '&') → str[source]

Return a normalized form of url, to compare or deduplicate URLs.

Changed in version 2.5.0: Dot segments (. and ..) in the path are now resolved, and IPv6 addresses in the host are now normalized.

URLs that differ only in ways that servers usually ignore get the same canonical form. The canonical form may point to a different resource than url, so to send a request, use safe_url_string() on url instead.

The following changes are applied:

  • make the URL safe, as safe_url_string() does

  • lowercase the host

  • sort query arguments, first by key, then by value

  • normalize all spaces (in query arguments) ‘+’ (plus symbol)

  • normalize percent encodings case (%2f -> %2F)

  • remove query arguments with blank values (unless keep_blank_values is True)

  • remove fragments (unless keep_fragments is True)

  • resolve dot segments (. and ..) in the path

  • normalize IPv6 addresses in the host

>>> import w3lib.url
>>>
>>> # sorting query arguments
>>> w3lib.url.canonicalize_url('http://www.example.com/do?c=3&b=5&b=2&a=50')
'http://www.example.com/do?a=50&b=2&b=5&c=3'
>>>
>>> # UTF-8 conversion + percent-encoding of non-ASCII characters
>>> w3lib.url.canonicalize_url('http://www.example.com/r\u00e9sum\u00e9')
'http://www.example.com/r%C3%A9sum%C3%A9'
>>>
>>> # a query separator other than the default '&'
>>> w3lib.url.canonicalize_url('http://www.example.com/do?c=3;a=50', query_separator=';')
'http://www.example.com/do?a=50;c=3'
>>>
>>> # resolving dot segments and normalizing an IPv6 address
>>> w3lib.url.canonicalize_url('http://[::0:1]/a/../b')
'http://[::1]/b'
>>>

For more examples, see the tests in tests/test_url.py.

w3lib.url.file_uri_to_path(uri: str) → str[source]

Convert File URI to local filesystem path according to: http://en.wikipedia.org/wiki/File_URI_scheme

w3lib.url.parse_data_uri(uri: str | bytes) → ParseDataURIResult[source]

Parse a data: URI into ParseDataURIResult.

Base64 data is decoded the way browsers do it, i.e. ignoring ASCII whitespace and allowing missing = padding.

Raise ValueError for invalid data URIs.

w3lib.url.path_to_file_uri(path: str | PathLike[str]) → str[source]

Convert local filesystem path to legal File URIs as described in: http://en.wikipedia.org/wiki/File_URI_scheme

w3lib.url.safe_url_string(url: str | bytes, encoding: str = 'utf8', path_encoding: str = 'utf8', quote_path: bool = True) → str[source]

Return url with its syntax fixed, ready to be sent in a request.

Use it on URLs found in HTML or given by users before requesting them. To compare or deduplicate URLs, use canonicalize_url() instead.

url is parsed the way web browsers parse it, following the URL living standard, and during serialization additional characters are percent-encoded, so that the result is valid by all of the following URL standards known to be enforced by modern-day web browsers and web servers:

Only the syntax is fixed: any scheme (e.g. file or javascript), any host (e.g. localhost, a private IP address or a domain name that resolves to one) and relative URLs are allowed. If url comes from an untrusted source, it is up to you to check those.

If url is bytes, it is decoded with encoding. encoding is also used to percent-encode the query and the fragment, so it should be the encoding of the page where url was found. path_encoding is used to percent-encode the path, unless quote_path is False, in which case the path is left as is.

Calling this function on an already safe URL returns the URL unmodified.

w3lib.url.url_query_cleaner(url: str | bytes, parameterlist: str | Sequence[str] = (), sep: str = '&', kvsep: str = '=', remove: bool = False, unique: bool = True, keep_fragments: bool = False) → str[source]

Clean URL arguments leaving only those passed in the parameterlist keeping order

>>> import w3lib.url
>>> w3lib.url.url_query_cleaner("product.html?id=200&foo=bar&name=wired", ('id',))
'product.html?id=200'
>>> w3lib.url.url_query_cleaner("product.html?id=200&foo=bar&name=wired", ['id', 'name'])
'product.html?id=200&name=wired'
>>>

If unique is False, do not remove duplicated keys

>>> w3lib.url.url_query_cleaner("product.html?d=1&e=b&d=2&d=3&other=other", ['d'], unique=False)
'product.html?d=1&d=2&d=3'
>>>

If remove is True, leave only those not in parameterlist.

>>> w3lib.url.url_query_cleaner("product.html?id=200&foo=bar&name=wired", ['id'], remove=True)
'product.html?foo=bar&name=wired'
>>> w3lib.url.url_query_cleaner("product.html?id=2&foo=bar&name=wired", ['id', 'foo'], remove=True)
'product.html?name=wired'
>>>

By default, URL fragments are removed. If you need to preserve fragments, pass the keep_fragments argument as True.

>>> w3lib.url.url_query_cleaner('http://domain.tld/?bla=123#123123', ['bla'], remove=True, keep_fragments=True)
'http://domain.tld/#123123'
w3lib.url.url_query_parameter(url: str | bytes, parameter: str, default: None = None, keep_blank_values: bool | int = 0, *, separator: str = '&') → str | None[source]
w3lib.url.url_query_parameter(url: str | bytes, parameter: str, default: str, keep_blank_values: bool | int = 0, *, separator: str = '&') → str

Return the value of a url parameter, given the url and parameter name

If url contains multiple parameters, the first leftmost one is returned.

General case:

>>> import w3lib.url
>>> w3lib.url.url_query_parameter("product.html?id=200&foo=bar", "id")
'200'
>>>

Return a default value if the parameter is not found:

>>> w3lib.url.url_query_parameter("product.html?id=200&foo=bar", "notthere", "mydefault")
'mydefault'
>>>

Returns None if keep_blank_values not set or 0 (default):

>>> w3lib.url.url_query_parameter("product.html?id=", "id")
>>>

Returns an empty string if keep_blank_values set to 1:

>>> w3lib.url.url_query_parameter("product.html?id=", "id", keep_blank_values=1)
''
>>>