Behind a concert hall’s agenda, a shop’s catalogue or a shelter’s adoption page there’s often a list the site keeps somewhere, and many sites hand that list to anyone who asks for it.
We wanted to know how often, so we took the 51 list pages behind our test sheets (concerts and theatres in Madrid, cooking classes in Barcelona, dogs for adoption, exhibitions, festivals) and looked at each one without any model. Between them they showed 1,323 items.
| What the site served | Sites | Share of the items |
|---|---|---|
| The whole list as data (90% or more of what the page shows) | 24 | 66% |
| Part of it | 8 | 15% |
| Nothing, only the page’s text | 18 | 19% |
| It blocked us | 1 | 0% |
Two thirds of the items were sitting there as data. Having a model read them off the page’s text costs more and misses some of them.
Where the data is
None of these routes belongs to one site. They come with the platform the site is built on:
- WordPress’s own API (
/wp-json/wp/v2/…), on 8 sites. Each kind of post the site has (shows, dogs, courses) is a list you can read. - WooCommerce’s shop API, on 6. Cooking schools sell their classes as products, and some shelters list their dogs that way too. Name, price, photo and link.
- The Events Calendar’s API, on 4. Name, date, photo and link for every event.
- The JSON the page itself asks for while it loads, on 4. The richest route, and a different one on every site.
- Sitemaps on 4, with links only. Event data marked up for search engines on 2, iCal feeds on 2, and one RSS feed.
WordPress is behind 19 of the 24 sites that served everything. The 18 that served nothing have no platform in common, and many of them sell their tickets through someone else.
Two examples
Sala Riviera’s agenda shows 51 concerts. Its events API returns 106, because it also has events the agenda doesn’t show. So jurl doesn’t take the whole answer: it keeps the endpoint whose items link to the pages the agenda links to, and narrows it with the site’s own filters (a category every shown item has, or the date of the oldest one).
Sala Galileo Galilei puts its 75 concerts in its listing as structured event data, with name, date, photo, link and, for most of them, the price.
What jurl does with it
Since 19 October, when jurl reads a list it checks first whether the site runs on WordPress, and whether WooCommerce or The Events Calendar is on it. It can tell from the page itself: WordPress links its API in the page’s head, and the plugins leave their names in the markup. If so, it takes the list from there, with each item’s name, date, price, photo and link as the site keeps them. Reading the page’s text is what’s left for the sites that don’t publish anything.
The list isn’t the whole row. WordPress’s API almost never includes a date or a price, so those still come from each item’s own page.
It also changes what a sheet costs to keep. A model reading a hundred events off the page costs a few euros. Reading the same hundred from the site’s data costs cents, and gives the same result every time, which is what you want from something that runs every morning.