3.2 KiB

Raw Blame History

coach-scraper

Caution! Be careful running this script.

We intentionally delay each batch of requests. Make sure any adjustments to this script appropriately rate-limit.

Overview

This is a simple web scraper for coaches listed on:

The program searches for coach usernames as well as specific information about each of them (their profile, recent activity, and stats). Data is streamed into a Postgres instance. Downloaded content is found in a newly created data directory with the following structure:

data
└── <site>
│   ├── coaches
│   │   ├── <username>
│   │   │   ├── <username>.html
│   │   │   └── ...
│   │   ├── ...
└── pages
    ├── <n>.txt
    ├── ...

Quickstart

Included in the development shell of this flake is a Postgres client (version 15.5). Generate an empty Postgres cluster at /db by running

$ pg_ctl -D db init

To start the database, run the following:

$ pg_ctl -D db -l db/logfile -o --unix_socket_directories=@scraper start

In the above command, @scraper refers to an abstract socket name. Rename to whatever is appropriate for your use case. To then connect to this database instance, run:

$ psql -h @scraper

To later shut the database down, run:

$ pg_ctl -D db stop

Initialize the table that scraped content will be streamed into:

$ psql -h @scraper -f sql/init.sql

If you have nix available, you can now run the scraper via

$ nix run . -- ...

Otherwise, ensure you have poetry on your machine and instead run the following:

$ poetry run python3 -m app ...

Development

nix is used for development. The included flakes.nix file automatically loads in Python (version 3.11.6) with packaging and dependency management handled by poetry (version 1.7.0). direnv can be used to a launch a dev shell upon entering this directory (refer to .envrc). Otherwise run via:

$ nix develop

Language Server

The python-lsp-server (version v1.9.0) is included in this flake, along with the python-lsp-black and pyls-isort plugins. Additionally, pylsp is expected to be configured to use:

Refer to your editor for configuration details.

Formatting

Formatting depends on the black (version 23.9.1) tool. A pre-commit hook is included in .githooks that can be used to format all *.py files prior to commit. Install via:

$ git config --local core.hooksPath .githooks/

If running direnv, this hook is installed automatically when entering the directory.

3.2 KiB Raw Blame History