v1.1.0

[nickscamara]

Starting today we are going to be posting weekly releases here and on firecrawl.dev/changelog. This release is just a summary of all the improvements and fixes we pushed since v1 release here. Thank you all for the contributions!

v1.1.0

Changelog Highlights

Feature Enhancements

• New Features:
  • Geolocation, mobile scraping, 4x faster parsing, better webhooks,
  • Credit packs, auto-recharges and batch scraping support.
  • Iframe support and query parameter differentiation for URLs.
  • Similar URL deduplication.
  • Enhanced map ranking and sitemap fetching.

Performance Improvements

• Faster crawl status filtering and improved map ranking algorithm.
• Optimized Kubernetes setup and simplified build processes.
• Sitemap discoverability and performance improved

Bug Fixes

• Resolved issues:
  • Badly formatted JSON, scrolling actions, and encoding errors.
  • Crawl limits, relative URLs, and missing error handlers.
• Fixed self-hosted crawling inconsistencies and schema errors.

SDK Updates

• Added dynamic WebSocket imports with fallback support.
• Optional API keys for self-hosted instances.
• Improved error handling across SDKs.

Documentation Updates

• Improved API docs and examples.
• Updated self-hosting URLs and added Kubernetes optimizations.
• Added articles: mastering /scrape and /crawl.

Miscellaneous

• Added new Firecrawl examples
• Enhanced metadata handling for webhooks and improved sitemap fetching.
• Updated blocklist and streamlined error messages.

What's Changed

• Add docs to api spec example by @ericciarla in Add docs to api spec example #637
• [Docs] upgraded the path of the self-hosted documentation URL to /v1. by @shige in [Docs] upgraded the path of the self-hosted documentation URL to /v1. #635
• Removal of generic classnames/ids from onlyMainContent cleaning by @nickscamara in Removal of generic classnames/ids from onlyMainContent cleaning #638
• Improved team credits check and billing notifications by @nickscamara in Improved team credits check and billing notifications #640
• Fixed 500 errors when JSON is badly formatted by @nickscamara in Fixed 500 errors when JSON is badly formatted #648
• Better engine for wait + other params by @nickscamara in Better engine for wait + other params #649
• fix(py-sdk): removed asyncio package by @rafaelsideguide in fix(py-sdk): removed asyncio package #654
• perf(js-sdk): move dotenv and uuid to devDependencies, fix zod import by @MonsterDeveloper in perf(js-sdk): move dotenv and uuid to devDependencies, fix zod import #614
• build(js-sdk): simplify build process by @MonsterDeveloper in build(js-sdk): simplify build process #611
• fix(v0/crawl-status): don't crash on big crawls when requesting jobs from supa by @mogery in fix(v0/crawl-status): don't crash on big crawls when requesting jobs from supa #653
• Manual Rate Limiter for select team ids by @nickscamara in Manual Rate Limiter for select team ids #664
• O1 crawler example by @ericciarla in O1 crawler example #676
• [Bug] Fixed screenshot typo and added test for fullpage screenshot by @rafaelsideguide in [Bug] Fixed screenshot typo and added test for fullpage screenshot #677
• v1/map improvements + higher limits by @nickscamara in v1/map improvements + higher limits #674
• Remove print statement in map by @anjor in Remove print statement in map #612
• fix wrong link to self host documentation by @itasli in fix wrong link to self host documentation #623
• feat: kubernetes example optimization by @yekkhan in feat: kubernetes example optimization #639
• Rust SDK 1.0.0 by @mogery in Rust SDK 1.0.0 #689
• feat: Actions by @mogery in feat: Actions #682
• Fix the error message when trying search in v0 by @nickscamara in Fix the error message when trying search in v0 #690
• remove space in the examples/o1_web_crawler folder name by @h4r5h4 in remove space in the examples/o1_web_crawler folder name #679
• o1 job recommender example by @ericciarla in o1 job recommender example #707
• Move auth and check credits operations into an RPC by @mogery in Move auth and check credits operations into an RPC #704
• bugfix: using onlyIncludeTags and removeTags together by @skeptrunedev in bugfix: using onlyIncludeTags and removeTags together #685
• Concurrency limits by @mogery in Concurrency limits #721
• Docs: Remove wait_until_done from python-sdk example by @bytrangle in Docs: Remove wait_until_done from python-sdk example #728
• Improves error handler in Node SDK to return the status code by @nickscamara in Improves error handler in Node SDK to return the status code #727
• Fixes crawl failed and webhooks not working properly by @nickscamara in Fixes crawl failed and webhooks not working properly #731
• [BUG] Fixed URLs with params by @rafaelsideguide in [BUG] Fixed URLs with params #732
• Fixed the self host issues where methods don't work by @nickscamara in Fixed the self host issues where methods don't work #733
• Make sure the entrypoint script has the correct line endings by @busaud in Make sure the entrypoint script has the correct line endings #753
• Rm cluster mode + rm fly deployments by @nickscamara in Rm cluster mode + rm fly deployments #754
• Fixed Issue [Bug] timeout parameter not passed to playwright service #734 by @Harsh0707005 in Fixed Issue #734 #747
• bugfix: self-host crawling doesnt respect limit by @busaud in bugfix: self-host crawling doesnt respect limit #755
• [BUG] Fixed missing error handling in JS-SDK by @rafaelsideguide in [BUG] Fixed missing error handling in JS-SDK #759
• [SKD] Cancel Crawl by @rafaelsideguide in [SKD] Cancel Crawl #760
• fixed developer.notion special case by @rafaelsideguide in fixed developer.notion special case #762
• Spelling Corrections in README by @fadkeabhi in Spelling Corrections in README #763
• [RPC] Improvements to credit_usage rpc by @nickscamara in [RPC] Improvements to credit_usage rpc #767
• [BUG] filters failed and unknown jobs now by @rafaelsideguide in [BUG] filters failed and unknown jobs now #761
• [Doc] Better explained how includePaths and excludePaths work by @rafaelsideguide in [Doc] Better explained how includePaths and excludePaths work #766
• Update README.md by @busaud in Update README.md #757
• ADDED : Contributors and Back to top by @Ruhi14 in ADDED : Contributors and Back to top  #768
• Retries for ACUC RPC + Price credits fallback by @nickscamara in Retries for ACUC RPC + Price credits fallback #773
• [BUG] added check files on crawl by @rafaelsideguide in [BUG] added check files on crawl #779
• [Feat] Performance improvements crawl status filters by @rafaelsideguide in [Feat] Performance improvements crawl status filters #780
• Admin alerts for high usage by @nickscamara in Admin alerts for high usage #783
• Geolocation support for Firecrawl by @nickscamara in Geolocation support for Firecrawl #784
• Return all the website metadata by @nickscamara in Return all the website metadata #785
• Extractor options logging v1 fix by @nickscamara in Extractor options logging v1 fix #788
• Update requirements.txt by @rishi-raj-jain in Update requirements.txt #790
• Improved /map ranking algorithm for search queries by @nickscamara in Improved /map ranking algorithm for search queries #798
• Fix Typos and Grammar in SELF_HOST.md by @Mefisto04 in Fix Typos and Grammar in SELF_HOST.md #799
• [Bug] encoding error for special token by @rafaelsideguide in [Bug] encoding error for special token #793
• [BUG-SDK] missing error in response by @rafaelsideguide in [BUG-SDK] missing error in response #796
• examples: sales web crawler by @rishi-raj-jain in examples: sales web crawler #797
• feat: clear ACUC cache endpoint based on team ID by @mogery in feat: clear ACUC cache endpoint based on team ID #807
• feat: skipTlsVerification by @tomkosm in feat: skipTlsVerification #808
• feat: Batch Scrape by @mogery in feat: Batch Scrape #789
• feat: Auto Recharge Credits + Credit Packs by @nickscamara in feat: Auto Recharge Credits + Credit Packs #809
• Remove ph logs for single_urls by @nickscamara in Remove ph logs for single_urls #829
• Bump to gemini-1.5-pro-002 website_qa_with_gemini_caching.ipynb and add flash example by @s-smits in Bump to gemini-1.5-pro-002 website_qa_with_gemini_caching.ipynb and add flash example #739
• Add SearchApi as a Web Search Tool by @SebastjanPrachovskij in Add SearchApi as a Web Search Tool #628
• RM wait before interacting by @nickscamara in RM wait before interacting #838
• chore(README.md): use satisfies instead of as for ts example by @twlite in chore(README.md): use satisfies instead of as for ts example #831
• Geo-location rename to location by @nickscamara in Geo-location rename to location #830
• concurrency limit fix by @mogery in concurrency limit fix #824
• [feat] Iframe support by @tomkosm in [feat] Iframe support #855
• Fix go parser by @tomkosm in Fix go parser #856
• Support for the 2 new actions by @nickscamara in Support for the 2 new actions #858
• Adds support for mobile web scraping + mobile screenshot by @nickscamara in Adds support for mobile web scraping + mobile screenshot #847
• [Feat] Added remove base64 images options (true by default) by @rafaelsideguide in [Feat] Added remove base64 images options (true by default) #867
• [Fix] Prevent Python Firecrawl logger from interfering with loggers in client applications by @reasonmethis in [Fix] Prevent Python Firecrawl logger from interfering with loggers in client applications #613
• [BUG] Added trycatch and removed redundancy by @rafaelsideguide in [BUG] Added trycatch and removed redundancy #869
• Update CONTRIBUTING.md by @swyxio in Update CONTRIBUTING.md #849
• WebScraper refactor into scrapeURL by @mogery in WebScraper refactor into scrapeURL #714
• Exec js - actions by @nickscamara in Exec js - actions #872
• feat(crawl): Similar URL deduplication by @mogery in feat(crawl): Similar URL deduplication #878
• [SDK] Added next handler for python sdk (js is ok) by @rafaelsideguide in [SDK] Added next handler for python sdk (js is ok) #880
• [BUG] fixes scroll action by @rafaelsideguide in [BUG] fixes scroll action #881
• feat(crawl): add parameter to treat differing query parameters as different URLs by @mogery in feat(crawl): add parameter to treat differing query parameters as different URLs #892
• fix(crawler): relative URL handling on non-start pages by @mogery in fix(crawler): relative URL handling on non-start pages #893
• Redlock for sending email notifications by @nickscamara in Redlock for sending email notifications #895
• feat(v1/webhook): complex webhook object w/ headers by @mogery in feat(v1/webhook): complex webhook object w/ headers #899
• fix(sitemap): scrape with tlsclient by @mogery in fix(sitemap): scrape with tlsclient #891
• Allows /map to only return links present in the sitemap by @nickscamara in Allows /map to only return links present in the sitemap #901
• Node SDK : Add Mobile Scraping by @ad-angelo in Node SDK : Add Mobile Scraping #914
• Add notebook and markdown files for two articles: mastering /scrape and mastering /crawl by @BexTuychiev in Add notebook and markdown files for two articles: mastering /scrape and mastering /crawl #918
• Extract (beta) by @nickscamara in Extract (beta) #915
• Add a new project to examples that shows how to scrape Hacker News website by @BexTuychiev in Add a new project to examples that shows how to scrape Hacker News website #928
• Fix/while next loop by @rafaelsideguide in Fix/while next loop #939
• Crawl not respecting the limit when ignoreSitemap is false by @nickscamara in Crawl not respecting the limit when ignoreSitemap is false #940
• Crawl fixes: fixed the n-1 bug by @nickscamara in Crawl fixes: fixed the n-1 bug #941
• fixed keyerror for data on sdk by @rafaelsideguide in fixed keyerror for data on sdk #943
• Submitting two articles to the blog by @BexTuychiev in Submitting two articles to the blog #944
• removed microsoft from blocklist by @rafaelsideguide in removed microsoft from blocklist #958
• Add assets for the Automated Amazon Price Tracking article by @BexTuychiev in Add assets for the Automated Amazon Price Tracking article #946
• Fixed Prettier by @nickscamara in Fixed Prettier #965
• Fixes schema base model extract by @rafaelsideguide in Fixes schema base model extract #954
• Metadata for webhooks by @nickscamara in Metadata for webhooks #970
• Remove Block List by @ericciarla in Remove Block List #971
• feat(runWebScraper): retry a scrape max 3 times in a crawl if the status code is failure (FIR-293) by @mogery in feat(runWebScraper): retry a scrape max 3 times in a crawl if the status code is failure (FIR-293) #975
• feat: add scrapeOptions.fastMode (FIR-288) by @mogery in feat: add scrapeOptions.fastMode (FIR-288) #973
• fix: adjust Playwright service response to match API schema expectations by @AsyncArtisan in fix: adjust Playwright service response to match API schema expectations #977
• Timeout fixes on user defined timeouts by @nickscamara in Timeout fixes on user defined timeouts #978
• Default to pdf2md, if under 500 chars (indicating failure) use LlamaParse by @ericciarla in Default to pdf2md, if under 500 chars (indicating failure) use LlamaParse #984
• Revert to pdf parse by @ericciarla in Revert to pdf parse #987
• [BUG] fixed title extra info by @rafaelsideguide in [BUG] fixed title extra info #996
• feat-SDK/added crawl id to ws by @rafaelsideguide in feat-SDK/added crawl id to ws #994
• [SDK] Added try catch to ws message handler by @rafaelsideguide in [SDK] Added try catch to ws message handler #998
• Credit usage endpoint by @nickscamara in Credit usage endpoint #999
• Refactored Blocklist Error Messages by @nickscamara in Refactored Blocklist Error Messages #1001
• feat(js-sdk): Make API key optional for self-hosted instances by @RutamBhagat in feat(js-sdk): Make API key optional for self-hosted instances #989
• docs(CONTRIBUTING.md): Add Docker Compose setup instructions by @RutamBhagat in docs(CONTRIBUTING.md): Add Docker Compose setup instructions #1005
• feat: Support environments without ws by dynamically importing WebSocket module with error handling by @tomkosm in feat: Support environments without ws by dynamically importing WebSocket module with error handling #997
• [bug/JS-SDK]Added check for object and trycatch as workaround for 502s by @rafaelsideguide in [bug/JS-SDK]Added check for object and trycatch as workaround for 502s #974
• docs(credit-usage-api): add new endpoint documentation for credit usage by @RutamBhagat in docs(credit-usage-api): add new endpoint documentation for credit usage #1003
• Improves sitemap fetching by @nickscamara in Improves sitemap fetching #1015

New Contributors

• @shige made their first contribution in [Docs] upgraded the path of the self-hosted documentation URL to /v1. #635
• @MonsterDeveloper made their first contribution in perf(js-sdk): move dotenv and uuid to devDependencies, fix zod import #614
• @anjor made their first contribution in Remove print statement in map #612
• @itasli made their first contribution in fix wrong link to self host documentation #623
• @yekkhan made their first contribution in feat: kubernetes example optimization #639
• @h4r5h4 made their first contribution in remove space in the examples/o1_web_crawler folder name #679
• @skeptrunedev made their first contribution in bugfix: using onlyIncludeTags and removeTags together #685
• @bytrangle made their first contribution in Docs: Remove wait_until_done from python-sdk example #728
• @busaud made their first contribution in Make sure the entrypoint script has the correct line endings #753
• @Harsh0707005 made their first contribution in Fixed Issue #734 #747
• @fadkeabhi made their first contribution in Spelling Corrections in README #763
• @Ruhi14 made their first contribution in ADDED : Contributors and Back to top  #768
• @rishi-raj-jain made their first contribution in Update requirements.txt #790
• @Mefisto04 made their first contribution in Fix Typos and Grammar in SELF_HOST.md #799
• @s-smits made their first contribution in Bump to gemini-1.5-pro-002 website_qa_with_gemini_caching.ipynb and add flash example #739
• @SebastjanPrachovskij made their first contribution in Add SearchApi as a Web Search Tool #628
• @twlite made their first contribution in chore(README.md): use satisfies instead of as for ts example #831
• @reasonmethis made their first contribution in [Fix] Prevent Python Firecrawl logger from interfering with loggers in client applications #613
• @swyxio made their first contribution in Update CONTRIBUTING.md #849
• @ad-angelo made their first contribution in Node SDK : Add Mobile Scraping #914
• @BexTuychiev made their first contribution in Add notebook and markdown files for two articles: mastering /scrape and mastering /crawl #918
• @AsyncArtisan made their first contribution in fix: adjust Playwright service response to match API schema expectations #977
• @RutamBhagat made their first contribution in feat(js-sdk): Make API key optional for self-hosted instances #989

Full Changelog: v1.0.0...v1.1.0

We're gonna be updating the release every week now and will be updating it on firecrawl.dev/changelog too!

---

This discussion was created from the release v1.1.0.