Repository navigation
[Bug]: CrawlerRunConfig.clone() drops fallback_fetch_function, so deep crawls never use it #2379
Description
crawl4ai version
0.9.5 (same on 0.9.0)
Expected Behavior
CrawlerRunConfig.clone() returns a copy with every field, fallback_fetch_function included. BFS, DFS and BestFirst deep crawls clone the run config for every batch (config.clone(deep_crawl_strategy=None, stream=...)), so a configured fallback should also apply to deep-crawled pages.
Current Behavior
clone() rebuilds the config from to_dict(), and to_dict() leaves fallback_fetch_function out, so the clone has fallback_fetch_function=None. In a deep crawl, a page whose fetch raises (or is blocked) never reaches the fallback, while the same page fetched with arun() / arun_many() does.
Is this reproducible?
Yes
Code snippets
from crawl4ai import CrawlerRunConfig
async def fallback(url: str) -> str:
return "<html>...</html>"
config = CrawlerRunConfig(fallback_fetch_function=fallback)
print(config.fallback_fetch_function) # <function fallback ...>
print(config.clone(stream=True).fallback_fetch_function) # NoneSuggested fix
Carry callables over in clone(), for example by copying fallback_fetch_function (and any other non-serializable field) from self onto the clone unless it is passed in kwargs. A subclass of CrawlerRunConfig whose clone() copies the field over works as a workaround.
Activity
siva23367 commentedon Oct 19, 2026
I would like to fix this issue, please assign it to me
fix: keep fallback_fetch_function and other fields in CrawlerRunConfi…
ntohidi commentedon Oct 26, 2026
Thanks for the clear report. The cause was in
to_dict().clone()builds the copy from it, and any field missing from that list was reset to its default. That's why deep crawls lostfallback_fetch_function.A few other fields had the same problem. #2385 fixed
fallback_fetch_functionand three more, and #2447 adds the last two (base_urlandforce_viewport_screenshot). It also adds a test, so a new field can't go missing again without someone noticing. Once #2447 is merged, it'll be in the next release.fix: keep base_url and force_viewport_screenshot in CrawlerRunConfig.…