98 lines
4.3 KiB
Markdown
98 lines
4.3 KiB
Markdown
---
|
||
Title: Manipulate XML/HTML with Xidel
|
||
Date: 2023-10-29 22:00
|
||
Lang: en
|
||
Author: Fabrice
|
||
Category: software
|
||
Tags: xidel, html, xml, cli
|
||
Slug: xidel
|
||
table-of-contents: false
|
||
Header_Cover:
|
||
Summary: Some information I would have love to learn earlier in (Neo)Vim.
|
||
---
|
||
|
||
You may know [jq](https://jqlang.github.io/jq/) process
|
||
[json](https://www.json.org/json-en.html) files in command line. At some point I
|
||
was looking for the simplicity of such a swiss-knife tool for
|
||
[XML](https://www.w3.org/XML/)/[HTML](https://html.spec.whatwg.org/multipage/),
|
||
mostly for simple usages that don't require me to resort to a full-fledged
|
||
scripting language such as [python](https://python.org) or dabbing in [regular
|
||
expressions](https://en.wikipedia.org/wiki/Regular_expression) that will never
|
||
work because of a carriage return at an unexpected place, and guess what? It exists!
|
||
|
||
This tool is [xidel](https://www.videlibri.de/xidel.html). It is a bit more than
|
||
that as it also allows downloading files, which enables extra features such as
|
||
navigating a site following specific links. You can find more about it in the
|
||
[list of examples](https://www.videlibri.de/xidel.html#examples) given in the
|
||
project website, which is a nice introduction to the possibilities of the tool.
|
||
|
||
However, I mainly use it for simple cases, where I mix-and-match the best of
|
||
both worlds: a graphical client (such as
|
||
[firefox](https://www.mozilla.org/en-US/firefox/new/)), and a CLI tool, which in
|
||
this case is xidel.
|
||
|
||
To do this, we will see a simple use case, where filtering by hand can be a bit
|
||
tedious. Let us assume that we want to obtain the URL list of pdf versions of
|
||
Victor Hugo's novels in French from Wikisource if available.
|
||
|
||
We start from this page: <https://fr.wikisource.org/wiki/Auteur:Victor_Hugo>,
|
||
that lists which is available on <https://fr.wikisource.org>.
|
||
|
||
Now, we can simply select the “Romans” section as it is and copy it. Normally
|
||
you can check that you indeed have the html in your clipboard by typing
|
||
`wl-paste -t text/html` on wayland or `xclip -selection clipboard -o -t
|
||
text/html` on X11 if you have xclip installed. In the following we will assume a
|
||
Wayland environment with
|
||
[wl-clipboard](https://github.com/bugaevc/wl-clipboard), but it should also work
|
||
with `xclip` (not tested, please let me know how it behaves).
|
||
|
||
Now that's good, but we now need to filter and parse it, we can start with a
|
||
simple test:
|
||
|
||
```bash
|
||
wl-paste -t text/html | xidel -e '//a/@href'
|
||
```
|
||
|
||
Which will show us the target of each links in our selection. To explain the
|
||
syntax, the option `-e` tells `xidel` to extract the content that is passed as
|
||
input, which is either a
|
||
[template](https://benibela.de/documentation/internettools/extendedhtmlparser.THtmlTemplateParser.html)
|
||
or following the [XPath](https://en.wikipedia.org/wiki/XPath) syntax to parse
|
||
the [DOM](https://en.wikipedia.org/wiki/Document_Object_Model) tree. In the
|
||
above example we used the latter, to obtain every anchors (`//a`) and then their
|
||
`href` attribute with `@href`.
|
||
From there we can see that pdf versions contains the string… “pdf”.
|
||
Now, we can see another nice part of XPath, is that we can filter using
|
||
functions:
|
||
|
||
```bash
|
||
wl-paste -t text/html | xidel -e '//a/@href[contains(., "pdf")]'
|
||
```
|
||
|
||
The last magical part here, is the dot notation, which refers to the current
|
||
item “value”. I’m not the most familiar with the subtleties here, and you can
|
||
refer to this stackoverflow [short answer](https://stackoverflow.com/a/38240971)
|
||
or long answer just above for more details.
|
||
|
||
You can also edit the way the filtering is done, for instance if the anchors you
|
||
are targeting are named “Download”, you can obtain the links with:
|
||
```bash
|
||
wl-paste -t text/html | xidel -e '//a[contains(., "Download")]/@href'
|
||
```
|
||
|
||
If you want strict equality because there are “Download PDF” and “Download epub”
|
||
links for instance:
|
||
|
||
```bash
|
||
wl-paste -t text/html | xidel -e '//a[text()="Download PDF"]/@href'
|
||
```
|
||
|
||
To go further, you can also pass HTTP headers and cookies to `xidel` via the
|
||
`--header/-H` and `--load-cookies` options respectively. It is also possible to
|
||
use the `--follow/-f` command to hop in the pages that matches (using the same
|
||
syntax as above) to obtain a link from it… or event directly download it with
|
||
the `--download` option and so on.
|
||
|
||
In this blogpost we only look at a local version of pre-filtered content using
|
||
you web browser, but the possibilities are endless!
|