Commit Graph

23 Commits (1c0dc05b42dd6103a4014b6f1ad97fa1571f86b3)

Author SHA1 Message Date
Joshua Potter 1c0dc05b42 Separate initialization from loading. Prefer upserts. 2023-12-04 08:14:33 -07:00
Joshua Potter 9a5274f0dd Use more robust pre-commit. 2023-12-03 14:31:36 -07:00
Joshua Potter b4e42e6de2 Simplify pre-commit. 2023-12-03 11:30:14 -07:00
Joshua Potter 9b9e561e49 Apply pyls-isort. 2023-12-01 16:37:05 -07:00
Joshua Potter a4b1647e53 Allow specifying multiple sites in command line. 2023-12-01 16:36:22 -07:00
Joshua Potter 0c4e008b45
Rewrite export as NDJSON and include script to load result into postgres. (#3)
* Allow loading exported data into database.

* Explanation on E2E.
2023-12-01 10:30:44 -07:00
Joshua Potter 9b81105a5e Use lxml to speed up parsing. 2023-12-01 07:12:40 -07:00
Joshua Potter d549e5f5eb Export blitz and bullet ratings. 2023-12-01 07:10:58 -07:00
Joshua Potter 36d471e395 Export rapid ratings. 2023-11-30 20:35:20 -07:00
Joshua Potter e050d13aa7 Add class for wrapping around exports. 2023-11-30 17:30:28 -07:00
Joshua Potter bc2ffeae9d
Add a scraper for lichess. (#2) 2023-11-30 15:36:44 -07:00
Joshua Potter 10801b560c
Generalize in anticipation of merging the lichess scraper. (#1)
* Add a general `Scraper` class.

* Setup main as primary entrypoint.

* Abstract original scraper into scraper class.

* Add better logging and cleaner bash commands.

* Ensure exporting works.
2023-11-30 15:15:15 -07:00
Joshua Potter 3cc31f8f24 Add guard on failed page download. 2023-11-28 07:57:05 -07:00
Joshua Potter fe2e504de9 Package into app for `nix build`. 2023-11-28 05:53:09 -07:00
Joshua Potter 27577aa5b0 Add `mypy` and lock poetry dependencies. 2023-11-28 04:55:09 -07:00
Joshua Potter 99c89a3a6d Restructure and add documentation. Require specifying user-agent. 2023-11-27 20:06:42 -07:00
Joshua Potter 7308c47eb5 Have requests actually use the `chesscom_request` function. 2023-11-27 14:28:10 -07:00
Joshua Potter 97cdb3b9cd Add User-Agent to requests. 2023-11-27 14:13:56 -07:00
Joshua Potter 1a4199e95d Add README descriptions. 2023-11-27 14:13:46 -07:00
Joshua Potter 932d97bf3b Create intermediate directories and explicit f-strings. 2023-11-27 13:53:51 -07:00
Joshua Potter 659988355c Fix classname. 2023-11-27 13:41:06 -07:00
Joshua Potter 23aa622126 Break coach listing download per-page. 2023-11-27 13:32:56 -07:00
Joshua Potter 1710e1aefa Initial commit. 2023-11-27 13:09:40 -07:00