Joshua Potter
58815d3ae5
Add worker count and type database connections.
2023-12-09 16:57:36 -07:00
Joshua Potter
3ab4f893b7
Add logging around failed network requests.
2023-12-09 15:46:38 -07:00
Joshua Potter
811cf997d1
Update app name and version.
2023-12-08 17:55:49 -07:00
Joshua Potter
8f21cb64af
Add random position.
2023-12-07 08:08:53 -07:00
Joshua Potter
44a18fc59c
Add language detection for chesscom profiles.
2023-12-07 05:12:13 -07:00
Joshua Potter
f2fd289225
Remove activity download.
2023-12-06 20:11:54 -07:00
Joshua Potter
47e8d245c3
Scrape titles.
2023-12-06 19:52:40 -07:00
Joshua Potter
0b9a721368
Maintain order on languages.
2023-12-05 16:06:04 -07:00
Joshua Potter
f20fc76081
Load languages into the database.
2023-12-05 15:15:42 -07:00
Joshua Potter
ef5d296097
Scrape languages from lichess listing. ( #10 )
2023-12-05 14:20:46 -07:00
Joshua Potter
82dbef21b6
Fix all mypy warnings. ( #9 )
2023-12-05 12:54:12 -07:00
Joshua Potter
8d7f1e7c4a
Scrape content into an asynchronous pipeline. ( #8 )
2023-12-05 11:43:13 -07:00
Joshua Potter
63764a22c4
Transition to a CSV; Postgres can handle that better.
2023-12-04 15:08:17 -07:00
Joshua Potter
ec94a16140
Re-order consistently.
2023-12-04 08:17:24 -07:00
Joshua Potter
1c0dc05b42
Separate initialization from loading. Prefer upserts.
2023-12-04 08:14:33 -07:00
Joshua Potter
9a5274f0dd
Use more robust pre-commit.
2023-12-03 14:31:36 -07:00
Joshua Potter
b4e42e6de2
Simplify pre-commit.
2023-12-03 11:30:14 -07:00
Joshua Potter
9b9e561e49
Apply pyls-isort.
2023-12-01 16:37:05 -07:00
Joshua Potter
a4b1647e53
Allow specifying multiple sites in command line.
2023-12-01 16:36:22 -07:00
Joshua Potter
0c4e008b45
Rewrite export as NDJSON and include script to load result into postgres. ( #3 )
...
* Allow loading exported data into database.
* Explanation on E2E.
2023-12-01 10:30:44 -07:00
Joshua Potter
9b81105a5e
Use lxml to speed up parsing.
2023-12-01 07:12:40 -07:00
Joshua Potter
d549e5f5eb
Export blitz and bullet ratings.
2023-12-01 07:10:58 -07:00
Joshua Potter
36d471e395
Export rapid ratings.
2023-11-30 20:35:20 -07:00
Joshua Potter
e050d13aa7
Add class for wrapping around exports.
2023-11-30 17:30:28 -07:00
Joshua Potter
bc2ffeae9d
Add a scraper for lichess. ( #2 )
2023-11-30 15:36:44 -07:00
Joshua Potter
10801b560c
Generalize in anticipation of merging the lichess scraper. ( #1 )
...
* Add a general `Scraper` class.
* Setup main as primary entrypoint.
* Abstract original scraper into scraper class.
* Add better logging and cleaner bash commands.
* Ensure exporting works.
2023-11-30 15:15:15 -07:00
Joshua Potter
3cc31f8f24
Add guard on failed page download.
2023-11-28 07:57:05 -07:00
Joshua Potter
fe2e504de9
Package into app for `nix build`.
2023-11-28 05:53:09 -07:00
Joshua Potter
27577aa5b0
Add `mypy` and lock poetry dependencies.
2023-11-28 04:55:09 -07:00
Joshua Potter
99c89a3a6d
Restructure and add documentation. Require specifying user-agent.
2023-11-27 20:06:42 -07:00
Joshua Potter
7308c47eb5
Have requests actually use the `chesscom_request` function.
2023-11-27 14:28:10 -07:00
Joshua Potter
97cdb3b9cd
Add User-Agent to requests.
2023-11-27 14:13:56 -07:00
Joshua Potter
1a4199e95d
Add README descriptions.
2023-11-27 14:13:46 -07:00
Joshua Potter
932d97bf3b
Create intermediate directories and explicit f-strings.
2023-11-27 13:53:51 -07:00
Joshua Potter
659988355c
Fix classname.
2023-11-27 13:41:06 -07:00
Joshua Potter
23aa622126
Break coach listing download per-page.
2023-11-27 13:32:56 -07:00
Joshua Potter
1710e1aefa
Initial commit.
2023-11-27 13:09:40 -07:00
Joshua Potter