Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Medium Article CloudFlare Blocking #49

Open
RishiRavula opened this issue Jan 13, 2025 · 0 comments
Open

Medium Article CloudFlare Blocking #49

RishiRavula opened this issue Jan 13, 2025 · 0 comments

Comments

@RishiRavula
Copy link

Describe the bug or technical issue
Seems when I run the scrapper, the output in qdrant is:

Just a moment...medium.comVerify you are human by completing the action below.medium.com needs to review the security of your connection before proceeding.Verification successfulWaiting for medium.com to respond...Enable JavaScript and cookies to continue Ray ID: 90122c119b67c434Performance & security by Cloudflare

To Reproduce
Steps to reproduce the behavior:
Simply running the web scrapper and letting it run through the pipeline

Expected behavior
Ideally free articles that are open and viewable without a sub or an account should work

Screenshots
image.

Desktop (please complete the following information):

  • Browser chromium
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant