You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the Issue
The fetch scraper does not support proxy settings.
Expected Behavior
The fetch scraper should use the same proxy settings as the playwright-service:
PROXY_SERVER=
PROXY_USERNAME=
PROXY_PASSWORD=
Environment (please complete the following information):
N/A
Logs
worker-1 | 2025-01-03 16:30:37 info [ScrapeURL:]: Scraping via playwright...
worker-1 | 2025-01-03 16:30:43 info [ScrapeURL:]: An unexpected error happened while scraping with playwright.
worker-1 | 2025-01-03 16:30:43 info [ScrapeURL:]: Scraping via fetch...
worker-1 | 2025-01-03 16:30:43 info [ScrapeURL:]: Scrape via fetch deemed successful.
Describe the Issue
The
fetch
scraper does not support proxy settings.Expected Behavior
The
fetch
scraper should use the same proxy settings as theplaywright-service
:Environment (please complete the following information):
Logs
Configuration
N/A
Additional Context
This should be fixable by adding a ProxyAgent and passing it via the
dispatcher
parameter here:https://github.com/mendableai/firecrawl/blob/87757d9b8e6bacc658b48832deb47c51eaf7412a/apps/api/src/scraper/scrapeURL/engines/fetch/index.ts#L17C7-L20
The text was updated successfully, but these errors were encountered: