Skip to content

Site, which shows information about the current scraping status. Built with Django and Scrapy.

Notifications You must be signed in to change notification settings

pozharliev/adatapro-internship-2021

Repository files navigation

Team Red



Site, which shows information about the current scraping status.



💻 About

Our project consists of a site that aims to help its users by providing them with up-to-date information on the availability of the PlayStation 5. The information is retrieved automatically through crawling. Once a day, registered users are sent an email with information about an available PS5, and users can set up the system to receive information only from the places they are interested in.


🎥 Interface

⌨️ How it works

Our work can be split into 4 different parts:

  1. The Site (Our Front-end)
    • Home - gives basic information about the site.
    • Information - gives more information about the site and how to use the functions of the site.
    • Registration and login - to create an account and log in.
    • Account - to control user settings and functions.
  2. The Django Application (Our Back-end)
    • Our Back-end is entirely written on py Django.
    • It is centered around views, which are in different 📁 folders.
    • It can be started by the 📁 manage.py script.
    • You can find in the in 📁 API/ folder.
    • It is hosted on localhost on port 8000, although you can change it.
  3. 📊 The Database
    • We are using mariadb MariaDB because of its worldwide popularity and its simplicity.
    • We have a few of tables, but the main ones are the auth_user, accountview_profilepreferences and historyview_historyscraping.
      • The auth_user stores all ®️ registered accounts.
      • The accountview_profilepreferences stores all the profile preferences related to which sites you want to see.
      • The historyview_historyscraping stores all the information from the scrapers.
    • All of the tables are generated from py Django models.
  4. The Scraper

⛏️ Installation

Step 1: Clone the repository

git clone https://github.com/ABPozharliev19/adatapro-internship-2021.git
cd adatapro-internship-2021/

Step 2: Install dependencies

You can do this by either using pipenv:

pip3 install pipenv # Install pipenv with pip
pipenv shell # Activate the environment
pipenv install # Install all dependencies in the Pipfile.lock file

Or by using pip:

pip3 install -r requirements.txt # Install all dependencies in the requirements.

Step 3: Rename env.example to .env

Step 4: Change the information in the .env file

SECRET_KEY = 'Your django secret key'
DB_NAME = 'Your database name'
DB_USER = 'Your database user'
DB_PASSWORD = 'Your database password'
DB_HOST = 'Your host'
DB_PORT = 3306 # The port you want to use
EMAIL_HOST_USER = 'Your email username' # Should be gmail
EMAIL_HOST_PASSWORD = 'Your email password'

Step 5: Make the migrations and migrate the models

python manage.py makemigrations
python manage.py migrate

Step 6: Run the server

python manage.py runserver

Or specify the port for the server

python manage.py runserver PORT

Step 7: Go to localhost:PORT and use the site!

🧒 Our Team

Atanas Pozharliev - ABPozharliev19 - Back-End Developer

Kostadin Taligadzhiev - KNTaligadzhiev19 - Front-End Developer

Igor Simeonov - GrandMaster-42 - Scraping

Velin Kurtev - Velin1234 - Scraping

Diyan Nedelchev - dido322 - Scraping

Kaloyan Totev - KaloyanTotev339 - Grafana & Docker Setup

About

Site, which shows information about the current scraping status. Built with Django and Scrapy.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •