Repository navigation
[Bug]: After a streamed deep crawl is stopped early, the next deep crawl returns only the start page #2380
Description
crawl4ai version
0.9.5 (same on 0.9.0)
Expected Behavior
Every crawler.arun(url, config) with a deep_crawl_strategy and stream=True runs a deep crawl, including after a previous streamed deep crawl was stopped early.
Current Behavior
If the caller stops iterating a streamed deep crawl (break), the next deep crawl from the same task returns only the start page.
DeepCrawlDecorator sets the deep_crawl_active ContextVar to True and resets it in the finally of the stream wrapper. When the caller leaves the stream without closing it, that finally runs later, when the generator is garbage-collected, in another context. The flag stays True in the caller's context, so the next arun() is treated as a nested call and falls through to a single-page crawl. The same happens if the stream is consumed from another task (for example asyncio.create_task(anext(stream))): the reset only reaches that task's copy of the context.
Is this reproducible?
Yes
Code snippets
import asyncio
from aiohttp import web
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
PAGES = 5
async def handle(request):
links = "".join(f'<a href="/p{i}">p{i}</a>' for i in range(PAGES)) if request.path == "/" else ""
return web.Response(text=f"<html><body>{links}<p>{'text ' * 200}</p></body></html>", content_type="text/html")
async def deep_crawl(crawler, url, stop_after=None):
config = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1), stream=True)
pages = []
async for result in await crawler.arun(url, config=config):
pages.append(result.url)
if stop_after and len(pages) == stop_after:
break # e.g. the caller restarts the crawl with another browser config
return pages
async def main():
app = web.Application()
app.router.add_get("/{tail:.*}", handle)
runner = web.AppRunner(app)
await runner.setup()
await web.TCPSite(runner, "127.0.0.1", 8765).start()
url = "http://127.0.0.1:8765/"
async with AsyncWebCrawler() as crawler:
print("full crawl:", len(await deep_crawl(crawler, url))) # 6
print("stopped early:", len(await deep_crawl(crawler, url, stop_after=2))) # 2
print("next crawl:", len(await deep_crawl(crawler, url))) # 1, expected 6
await runner.cleanup()
asyncio.run(main())Suggested fix
Reset the flag as soon as deep_crawl_strategy.arun() has returned the stream, with the token from set(), instead of in the stream's finally. The strategies' own batches already run with deep_crawl_strategy=None, so the guard is not needed while the stream is consumed. Callers can work around it by closing the stream in their own task (async with contextlib.aclosing(stream)).
Activity
PiUnknown commentedon Oct 15, 2026
Taking this one. The cause is that DeepCrawlDecorator sets deep_crawl_active as a ContextVar and resets it in the generator's finally block, which runs late in a different context when the stream is stopped early - so the flag stays set and the next crawl exits after the start page. Plan: reset the flag using the token from set() (guarded so a late finally can't clobber a new crawl's state), plus a regression test built on the repro above (aiohttp server, break after the first result). PR coming shortly.
ntohidi commentedon Oct 26, 2026
Confirmed on develop, and your read of it is right.
DeepCrawlDecoratorsetsdeep_crawl_activebefore the crawl, but in stream mode it only clears it in the wrapper'sfinally. After abreak, thatfinallyruns later, when the generator gets cleaned up, and in another context, so the flag staysTruein your task. The nextarun()then thinks it's nested and does a single-page crawl.#2386 fixes it by clearing the flag at each
yield. I ran your repro against it and it passes, so we'll go with that one. Until it ships,contextlib.aclosing(stream)works around it.Merge pull request #2386 from PiUnknown/fix/deep-crawl-reset
ntohidi commentedon Oct 26, 2026
The fix is in the
developbranch