Commit graph

1742 commits

Author SHA1 Message Date
Samuel Clay
f552d6f2c7 Adding support for hyphens in youtube usernames. 2024-04-01 08:33:45 -04:00
Samuel Clay
65e51ac6ec Handling broken youtube channel 2024-03-04 09:24:23 -05:00
Samuel Clay
0181a2e4b0 Youtube username/title 2024-03-04 09:21:00 -05:00
Samuel Clay
7d08632e9c Handling youtube usernames that are actually handles. 2024-03-03 13:09:59 -05:00
Samuel Clay
a5222d9675 Updating youtube fetcher to use channels/playlists/users for everything, no longer relying on RSS/xml url. 2024-03-03 12:59:12 -05:00
Samuel Clay
ec44039cc9 Updating certs. 2024-03-02 08:37:20 -05:00
Samuel Clay
fe708196ee Tailing logs needs a bit more width for new task servers. 2024-02-18 08:23:41 -05:00
Samuel Clay
7bd7d4ba9d Tailing logs needs a bit more width for new task servers. 2024-02-18 08:23:21 -05:00
Samuel Clay
0e71bfd27a Rewriting tlnb command to handle naming servers. 2024-02-10 15:01:28 -05:00
Samuel Clay
f9ad93e336 Adding hetzner servers. 2024-01-07 10:00:39 -05:00
Samuel Clay
ea22256c59 Increasing apns payload size close to the max of 4kb. 2023-05-10 07:26:22 -04:00
Samuel Clay
8adc2035ea Also handling time out in feedparser. 2023-05-09 09:35:56 -04:00
Samuel Clay
792ba9d204 Removing fetch user agent on timeout as well as 400 errors. 2023-05-09 09:31:09 -04:00
Samuel Clay
3de497d57e Bumping log colors. 2023-04-30 18:12:11 -04:00
Samuel Clay
9c49e0d5fe New twitter auth error. 2023-04-04 09:36:40 -04:00
Samuel Clay
6095bd709d Fixing relative image urls to be absolute urls. Looks like a BeautifulSoup4 upgrade didn't catch the new attrs syntax. Also fixing bookmarklet loading wrong JS/CSS in development. 2022-11-24 12:53:40 -05:00
Samuel Clay
855a96d633 Fixing NYPost feed by ensuring enclosures with href attrbiutes are added to stories even without content. Thanks @kleinmatic! 2022-11-23 12:21:46 -05:00
Samuel Clay
f93eb2520c Adding file size and archive count to rss feed fetch status board. 2022-09-11 22:13:54 -04:00
Samuel Clay
57cf284f13 Adding slow log aggregator to uncover which users and paths are slowest. 2022-09-11 21:37:02 -04:00
Samuel Clay
e83a8e5a9b Revert "Bumping min to fetch by 1.5X to root out 12 minute broken feed. If this succeeds, narrow down the min_to_decay and then find the feed."
This reverts commit fd16bb0b73.
2022-09-11 08:45:29 -04:00
Samuel Clay
fd16bb0b73 Bumping min to fetch by 1.5X to root out 12 minute broken feed. If this succeeds, narrow down the min_to_decay and then find the feed. 2022-09-11 04:54:35 -04:00
Samuel Clay
cdf5a36b81 Don't delete redis keys because they take time to rebuild and subs can be counted incorrectly during that time. 2022-08-10 17:45:25 -04:00
Samuel Clay
fa77ff6a27 Also scrub twitter authors for xml invalid chars. 2022-07-02 13:11:25 -04:00
Samuel Clay
b26817ba77 Refactoring xml serialization. 2022-07-02 13:09:37 -04:00
Samuel Clay
80a1d57944 Remove unserializable control characters from tweets. 2022-07-02 12:58:30 -04:00
Samuel Clay
63635f42e1 Replacing @username's in tweets with links. 2022-07-01 17:00:22 -04:00
Samuel Clay
fa78ec335a Handling missing feed 2022-07-01 13:37:55 -04:00
Samuel Clay
3cf7f1e7c8 Need a feed to process. 2022-07-01 13:32:33 -04:00
Samuel Clay
1d1859db7d Handling missing feed on exception retry. 2022-07-01 13:29:49 -04:00
Samuel Clay
d4833a0d56 Updating redis query log to include response time as well. Huge improvement in redis monitoring. 2022-06-24 12:30:29 -04:00
Samuel Clay
4fe6110e96 Moving redis introspection to send_command, not just pack_command 2022-06-23 17:57:31 -04:00
Samuel Clay
81f51f1723 Handling soft time limits while fetching archives. 2022-06-22 17:40:16 -04:00
Samuel Clay
1665dc73a1 Don't skip pages on paged archives. 2022-06-22 14:49:31 -04:00
Samuel Clay
6c86bb479a Using correct url for rfc5005. 2022-06-22 12:11:52 -04:00
Samuel Clay
295bd65add Adding Twitter search. 2022-06-21 16:07:12 -04:00
Samuel Clay
6d5c1f29d5 Merge branch 'master' into pro
* master:
  Substack now uses a <text> element, so we need to add it to the sanitizer.
2022-06-07 09:36:18 -07:00
Samuel Clay
a269abc96d Substack now uses a <text> element, so we need to add it to the sanitizer. 2022-06-07 09:36:13 -07:00
Samuel Clay
9899816196 Fixing starting page for archives. 2022-05-24 11:40:49 -04:00
Samuel Clay
000c9b0954 Adding starting page for archive to help debug. 2022-05-24 11:37:54 -04:00
Samuel Clay
e6cf58a1d8 Boosting archive to 150 pages back. 2022-05-24 11:36:16 -04:00
Samuel Clay
b0a50f1834 Turning up the full archive. 2022-05-24 11:32:58 -04:00
Samuel Clay
8a5748fa37 Adding # of stories per dashboard module. Also fixing long-standing double load bug on initial page load. Fixed issues around phantom read stories popping in and out of existance on page load. 2022-05-16 17:21:46 -04:00
Samuel Clay
3a106b51bb Merge branch 'master' into pro
* master:
  Don't change file permissions, let's see what happens.
  Bumping postgres server size, fixing monitor disk usage cronjob.
2022-05-16 10:34:40 -04:00
Samuel Clay
b2cdff8a85 Bumping postgres server size, fixing monitor disk usage cronjob. 2022-05-13 16:59:57 -04:00
Samuel Clay
bfdfa10bfd Adding file system size (bytes) for feeds, decompressing stories to figure out their true size. Needs to be integrated into the UI. 2022-05-11 15:00:23 -04:00
Samuel Clay
9bb9700cc5 Merge branch 'master' into pro
* master:
  Handling getMore mongo log.
  Naming pg_basebackup
2022-05-11 13:12:06 -04:00
Samuel Clay
6b1d8cef78 Handling getMore mongo log. 2022-05-11 13:09:59 -04:00
Samuel Clay
5ff6a340f1 Merge branch 'master' into pro
* master:
  Upgrading to latest pymongo v3 to work with mongodb v4.0. Required rewrite of mongodb introspection, now using built-in pymongo monitoring. Should work on both web and celery.
  Removing unused mongo mms utils and configs.
  Updating pymongo to mongodb 4.0 compatible. Any further and we lose mongo wire protocol inspection.
  Full arm64 setup for imageproxy.
  Rearranging TF vars
  Upgrading to MongoDB 4.0, and upping mongo servers.
2022-05-10 20:59:03 -04:00
Samuel Clay
31de0f7a97 Upgrading to latest pymongo v3 to work with mongodb v4.0. Required rewrite of mongodb introspection, now using built-in pymongo monitoring. Should work on both web and celery. 2022-05-10 20:58:47 -04:00
Samuel Clay
7870eee728 Merge branch 'master' into pro
* master:
  Moving pg basebackup to a backup role
  Adding a secondary postgres that starts from a pg_basebackup. Still needs testing.
  Adding timestamps to backup logs
  No TTY/input needed on backups.
  Paths are in different environments.
  Renaming postgresql backups
  Adding postgres restore command and updating backup script.
2022-05-02 11:42:52 -04:00