[Bug]: PDF scrape thread outlives Docker crawl deadline; no total download/parse timeout #2460

Summary

Source-inspected cancellation/lifetime gap, observed in upstream main and v0.9.5; not yet a measured live exploit.

PDFContentScrapingStrategy.ascrap() calls await asyncio.to_thread(self.scrap, url, html, **kwargs). scrap() synchronously downloads and parses the PDF. The downloader's requests.Session.get(..., stream=True, timeout=(20, 600), allow_redirects=False) uses a 600-second per-read timeout, not a total elapsed deadline. It iterates chunks and checks max_pdf_bytes, but does not check a monotonic elapsed deadline.

The Docker /crawl endpoint wraps the coroutine in asyncio.wait_for(..., wall_clock_s). Cancelling that coroutine cannot forcibly stop a thread already running the synchronous download or PDF parser. A source that steadily streams below the configured byte cap may keep the worker active well beyond the API/bridge timeout; a stalled parser may similarly persist. Repeated operations can accumulate lingering work, defeating the expected isolation of a long-lived self-hosted Docker crawler.

Relevant locations (v0.9.5 and current main)

• crawl4ai/processors/pdf/__init__.py ascrap: offloads synchronous download + parse using asyncio.to_thread.
• _fetch_with_redirect_checks: timeout=(20, 600) per socket operation.
• _get_pdf_path: streamed byte cap checked but no elapsed-time cap.
• Docker API: asyncio.wait_for is coroutine-scoped, not killable-thread-scoped.

Expected

A configurable, enforced PDF total elapsed download + parse budget, with no lingering worker or temporary PDF after timeout/cancellation. This should coexist with the existing (valuable) download-byte/page caps, redirect validation, and peer-IP/egress protections.

Proposed focused regression tests

1. Local loopback HTTP fixture that streams one small chunk periodically, always within the 600-second per-read timeout, while exceeding a short total PDF budget. Once the budget expires, verify the worker has terminated and temporary file has been removed—not merely that the HTTP client received a 504.
2. Deliberately delayed PDF parsing on a local fixture; cancel the API crawl and verify no persistent processing thread/process or temporary file remains after the budget.
3. Repeat the timed-out request a few times and assert bounded worker count and resource cleanup. Verify normal small PDFs and byte/page/redirect limits still work.

An elapsed-time check inside the chunk loop plus a shorter socket read timeout helps the download path but does not guarantee prompt cancellation of an already-blocked read or synchronous parsing. If the parser cannot be interrupted safely in a thread, a killable subprocess or another lifecycle-bound worker may be needed. The aim is a narrow upstream fix, not general queue/supervisor infrastructure.

Context / related work

• The open #2236 adds page-specific crawl timeouts and cancellation-safe page cleanup, but its code path is for browser pages, not PDFContentScrapingStrategy synchronous worker threads.
• #2319 reports a different concurrent PDF page-numbering race; it is not a solution to cancellation.
• This was found when prototyping bounded native PDF extraction in a guarded personal MCP deployment. We closed the prototype unmerged because the current PDF processing lifetime cannot be bounded, even with 12 MiB / 100-page caps and 0 redirects.

Impact: potential long-running worker/resource exhaustion in self-hosted Docker research deployments. This report is based on source inspection; a live slow-stream reproduction has not yet been performed.