Joshua Potter f2fd289225 | ||
---|---|---|
.githooks | ||
app | ||
sql | ||
.envrc | ||
.gitignore | ||
README.md | ||
default.nix | ||
flake.lock | ||
flake.nix | ||
poetry.lock | ||
pyproject.toml |
README.md
coach-scraper
Caution! Be careful running this script.
We intentionally delay each batch of requests. Make sure any adjustments to this script appropriately rate-limit.
Overview
This is a simple web scraper for coaches listed on:
The program searches for coach usernames as well as specific information about
each of them (their profile, recent activity, and stats). Data is streamed into
a Postgres instance. Downloaded content is found in a newly created data
directory with the following structure:
data
└── <site>
│ ├── coaches
│ │ ├── <username>
│ │ │ ├── <username>.html
│ │ │ └── ...
│ │ ├── ...
└── pages
├── <n>.txt
├── ...
Quickstart
Included in the development shell of this flake is a Postgres
client (version 15.5). Generate an empty Postgres cluster at /db
by running
$ pg_ctl -D db init
To start the database, run the following:
$ pg_ctl -D db -l db/logfile -o --unix_socket_directories=@scraper start
In the above command, @scraper
refers to an abstract socket name.
Rename to whatever is appropriate for your use case. To then connect to this
database instance, run:
$ psql -h @scraper
To later shut the database down, run:
$ pg_ctl -D db stop
Initialize the table that scraped content will be streamed into:
$ psql -h @scraper -f sql/init.sql
If you have nix available, you can now run the scraper via
$ nix run . -- ...
Otherwise, ensure you have poetry on your machine and instead run the following:
$ poetry run python3 -m app ...
Development
nix is used for development. The included flakes.nix
file automatically loads in Python (version 3.11.6) with packaging and
dependency management handled by poetry (version 1.7.0). direnv
can be used to a launch a dev shell upon entering this directory (refer to
.envrc
). Otherwise run via:
$ nix develop
Language Server
The python-lsp-server
(version v1.9.0) is included in this flake, along with the python-lsp-black
and pyls-isort plugins.
Additionally, pylsp
is expected to be configured to use:
- McCabe,
- pycodestyle, and
- pyflakes.
Refer to your editor for configuration details.
Formatting
Formatting depends on the black
(version 23.9.1) tool. A pre-commit
hook is included in .githooks
that can
be used to format all *.py
files prior to commit. Install via:
$ git config --local core.hooksPath .githooks/
If running direnv, this hook is installed automatically when entering the directory.