jurl asks every site for markdown first:
Accept: text/markdown, text/html;q=0.9, */*;q=0.5
Sites behind Cloudflare’s Markdown for Agents send text/markdown. Most others send HTML, which jurl takes apart itself. react.dev does something in between. It sends markdown and labels it plain text.
$ curl -s -D - -o /dev/null -H "Accept: text/markdown" https://react.dev/reference/react/useEffect | grep -i content-type
content-type: text/plain; charset=utf-8
The body starts with ---, title: useEffect, ---: markdown with front matter. Without that header, the same URL is text/html.
jurl 0.1.10 believed the label. Plain text isn’t markdown, so the body went to the HTML extractor, which found almost no tags in it and ran all the text together. This is what came back for one of our benchmark questions:
$ jurl -q "how do I clean up a subscription when the component unmounts?" react.dev/reference/react/useEffect
# Welcome to the {roomId} room!
<https://react.dev/reference/react/useEffect> · docs (1.00)
--- ## Reference {/*reference*/} ### `useEffect(setup, dependencies?)` {/*useeffect*/} Call `useEffect` at the top level of your component to declare an Effect: ```js import { useState, useEffect } from 'react'; import { createConnection } from './chat.js'; function ChatRoom({ roomId }) { const [ser…
We cut that last line short. It goes on for 5,758 characters. Every code fence, heading and list is in it, joined by spaces. And the title isn’t the page’s: <h1>Welcome to the {roomId} room!</h1> is a line of JSX in one of the page’s code examples, and the HTML parser took it for the page’s heading.
Our own benchmark scored it as right. The answer it checks for, return () =>, is in there word for word, on one line with everything else. A strict text check passed on code nobody could paste.
The fix in 0.1.11 is three lines in src/fetch.rs:
fn served_markdown(content_type: &str, body: &str) -> bool {
content_type.contains("markdown") || (content_type.starts_with("text/plain") && !body.trim_start().starts_with('<'))
}
Plain text that doesn’t start with < is now read as markdown. Real plain text read as markdown loses nothing. The same question now gives:
$ jurl -q "how do I clean up a subscription when the component unmounts?" react.dev/reference/react/useEffect
# useEffect
<https://react.dev/reference/react/useEffect> · docs (1.00)
### `useEffect(setup, dependencies?)` {/*useeffect*/}
```js
import { useState, useEffect } from 'react';
import { createConnection } from './chat.js';
function ChatRoom({ roomId }) {
const [serverUrl, setServerUrl] = useState('https://localhost:1234');
useEffect(() => {
const connection = createConnection(serverUrl, roomId);
connection.connect();
return () => {
connection.disconnect();
};
}, [serverUrl, roomId]);
// ...
}
```
It goes on with the setup parameter and two more examples, each cleanup intact.
There’s a cost. Read as HTML, the page was a few long blocks, cut short before Jev saw them: 5,415 tokens. Read as markdown, it’s every paragraph: 31,380 tokens, about $0.0013 instead of $0.0002. What the agent reads got shorter, though: 684 to 749 tokens in the benchmark’s runs, against 1,703 to 1,905.
More sites are starting to serve agents markdown. The headers haven’t caught up, so a client has to read the body as well as the label.