Commit graph

574 commits

Author SHA1 Message Date
Samuel Clay
6c86bb479a Using correct url for rfc5005. 2022-06-22 12:11:52 -04:00
Samuel Clay
6d5c1f29d5 Merge branch 'master' into pro
* master:
  Substack now uses a <text> element, so we need to add it to the sanitizer.
2022-06-07 09:36:18 -07:00
Samuel Clay
a269abc96d Substack now uses a <text> element, so we need to add it to the sanitizer. 2022-06-07 09:36:13 -07:00
Samuel Clay
9899816196 Fixing starting page for archives. 2022-05-24 11:40:49 -04:00
Samuel Clay
000c9b0954 Adding starting page for archive to help debug. 2022-05-24 11:37:54 -04:00
Samuel Clay
e6cf58a1d8 Boosting archive to 150 pages back. 2022-05-24 11:36:16 -04:00
Samuel Clay
b0a50f1834 Turning up the full archive. 2022-05-24 11:32:58 -04:00
Samuel Clay
bfdfa10bfd Adding file system size (bytes) for feeds, decompressing stories to figure out their true size. Needs to be integrated into the UI. 2022-05-11 15:00:23 -04:00
Samuel Clay
de1fb57406 Finishing RFC 5005 archive fetcher (for #1109). Also fixing mongo commands and adding checks for archive subscribers. 2022-04-29 11:31:32 -04:00
Samuel Clay
4f69d15fcb RFC5005 fetching should be working, but the test site is changing urls to a weird http://x.com:443, so maybe we need to normalize the url? 2022-04-20 22:38:51 -04:00
Samuel Clay
4ee5525d71 Removing pdb 2022-04-19 15:27:59 -04:00
Samuel Clay
59b44088f6 Checking RFC5005 archive links in feeds. Needs testing, but first a fix for URL normalization. For #1109. 2022-04-19 13:26:39 -04:00
Samuel Clay
b4c9c4e896 Time to cleanup logging on archive. 2022-04-18 14:29:08 -04:00
Samuel Clay
5981b24588 Counting pre-archive pages. Need to format the email to better show what the user gained. And also to make it shareable. 2022-04-18 14:08:28 -04:00
Samuel Clay
ac593494db Sending all of a user's feeds through the archive fetcher to fill out their backfill, then email them the new stories. 2022-04-18 13:29:13 -04:00
Samuel Clay
43695b8cd4 Back-filling archive pages. This closes #1564. 2022-04-13 10:43:29 -04:00
Samuel Clay
f33cbcf312 Merge branch 'master' into pro
* master: (72 commits)
  Missing import
  Correctly decoding shared stories in RSS feed.
  Adding scroll to feed and story options popover (Style menu).
  Magazine view blog post.
  Refactoring story content preview with newlines preserved.
  Boosting db profiling from 1% to 5%.
  Ignore data uris for widget, they are never the big images
  Wait to strip data uri header until it's time.
  Moving encoding
  Don't double encode base64 images for widgets.
  Preserving paragraphs in magazine and grid views.
  Fixing bad instrumentation on mongo and redis logs.
  Wrong url for db profiler.
  Let's try showing only single feeds in the DB profiler to see if it's better at showing abnormalities with less noise.
  Fixing strict bug where  wasn't defined.
  Showing grid image previews. Not sure how this ever worked before.
  Grid has image if found already.
  Showing grid image on load if youtube
  Only monitor river_stories for db times
  Not hitting a db shouldnt be colored any differently than a small value.
  ...
2022-03-09 11:41:40 -05:00
Samuel Clay
c800dcaddf Black formatting 2022-03-02 10:55:46 -05:00
Samuel Clay
f0f6c0a828 Fixing issue with some sites spewing tons of headers back and Python failing. 2022-03-02 10:29:24 -05:00
Samuel Clay
39047ee938 Merge branch 'master' into pro
* master:
  Task deploy needs single sentry update.
  Turning off PuSH for standalone installs.
2022-01-25 15:46:25 -05:00
Samuel Clay
6a61620d32 Turning off PuSH for standalone installs. 2022-01-25 15:27:29 -05:00
Samuel Clay
2792e7305a Syncing redis on premium upgrade to sync new unread date possibilities for archive users. 2022-01-13 16:56:30 -05:00
Samuel Clay
1dc79686bc Skipping requests for RSS, letting feedparser do the fetching. Requests si still used for json feeds. 2021-12-08 20:44:16 -05:00
Samuel Clay
7081c95217
Merge pull request #1541 from louis-pre/fix_youtube_duration
fix YouTube video duration calculation
2021-12-08 14:37:00 -05:00
Samuel Clay
153675b43d Adding feeds as users to feed fetching in sentry. 2021-09-12 13:27:24 -04:00
Louis Pré
4bef983db3 fix YouTube video duration calculation 2021-08-26 21:38:33 -07:00
Samuel Clay
3804fbba5d Correcting mongodb on local installs since there is no auth. 2021-08-04 16:26:41 -04:00
Samuel Clay
3d63d95671 Forcing reconnect of analytics. 2021-08-03 21:17:37 -04:00
Samuel Clay
fe3a8bca3e Don't reconnect analytics server. 2021-08-03 21:16:16 -04:00
Samuel Clay
6286b798a7 Fixing JSON feeds. 2021-07-29 17:25:09 -04:00
Samuel Clay
047c86f8c8 Don't strip iframes from RSS. 2021-07-16 11:20:59 -04:00
Samuel Clay
520438aaec Newsletters should also send notifications. 2021-05-13 10:44:02 -04:00
Samuel Clay
fe1d0a28ba Not sanitizing youtube feeds. 2021-04-19 16:11:02 -04:00
Samuel Clay
c766e781ed JSON Feeds 1.1. 2021-04-02 13:27:33 -04:00
Samuel Clay
d52a8d362c Reducing number of workers on docker-compose. Also removing extraneous task servers and consolidating to task-celery for feeds and task-work for cron and work 2021-03-17 20:13:58 -04:00
Samuel Clay
28b09b5f3b Adding another exception handler to feed fetcher. 2021-03-03 09:30:20 -05:00
Samuel Clay
b266e3da4e badStatusLine exception handling. 2021-03-02 19:05:37 -05:00
Samuel Clay
5bf3cc251f IncompleteRead error 2021-03-02 19:03:57 -05:00
Samuel Clay
5e2e237871 Adding IncompleteRead error handling, fixing spacing. 2021-02-28 20:18:53 -05:00
Samuel Clay
37559d7237 Fixing InvalidURL 2021-02-26 11:39:36 -05:00
Samuel Clay
414c05bfbe Fixing ConnectionResetError. 2021-02-26 11:35:13 -05:00
Samuel Clay
d54de1cd23 Handling ConnectionResetError 2021-02-25 20:04:24 -05:00
Samuel Clay
ba6a8c81e0 Switching from raven to sentry sdk. 2021-02-25 19:52:36 -05:00
Samuel Clay
c5f4734de4 Finally getting sentry to work in feed fetching. 2021-02-25 19:32:48 -05:00
Samuel Clay
7f328b11c8 Typo 2021-02-25 19:16:28 -05:00
Samuel Clay
1d16ec88b1 Wrapping feed fetcher in sentry exception capturing. 2021-02-25 19:13:21 -05:00
Samuel Clay
4a608d2ce5 Finding all old celery invocations. 2021-01-19 17:30:11 -05:00
Samuel Clay
82fecb23a6 Merge branch 'docker_django3.1' into dashboard3
* docker_django3.1: (117 commits)
  Adding docker to node scripts.
  Removing node_modules from repo
  Whoops
  Better logging for broken paypal ipns.
  Adding RSS Tracker for Windows to Goodies.
  Refactoring socketio so real-time works around username restrictions.
  Socket.IO v3 on the server too.
  Switching to Socket.IO v3.
  Haproxy splitting for socketio 3
  Socket.io 3
  Handling missing feed.
  Only nb.local.com uses ports for websockets.
  Adding GrazeTEN
  Full node install on setup.
  Updating toprc
  correct newsblur to newsblur_web in tasks
  Changing mongodb server.
  Fixing toprc
  Attempting to collect queries for 1% of requests by using DEBUG.
  Fixing mongo install.
  ...
2021-01-05 19:36:49 -05:00
Samuel Clay
c790b7c5f7 Fixing v2 to v3 pickle issue by namespacing cache. 2020-12-17 11:55:06 -05:00
Samuel Clay
6f97446b3b Slightly refactoring feed fetcher to deal with weakref's when pickling functions. 2020-12-15 17:24:01 -05:00