Commit graph

106 commits

Author SHA1 Message Date
Samuel Clay
23abeb0b0a Handling feedparser backup failing. 2021-02-28 20:39:57 -05:00
Samuel Clay
ba6a8c81e0 Switching from raven to sentry sdk. 2021-02-25 19:52:36 -05:00
Samuel Clay
c7f4cbb13d Fixing a substantial number of encoding errors. Also fixing OPML uploads. 2021-01-24 21:05:22 -05:00
Jonathan Math
79f249878b remove use of ORIGINAL_PAGE_SERVER 2021-01-03 11:09:25 -05:00
Samuel Clay
aea4fcbe5b Merge branch 'django3.0' into django3.1
* django3.0: (27 commits)
  Removing log override
  Moving logging over to the newsblur log.
  Fixing search indexer background task for new celery.
  Attempting to add gunicorn errors to console/log.
  Better handling of missing subs.
  Handling missing user sub on feed delete.
  Correct encoding for strings on systems that don't have utf-8 as default encoding.
  Writing in the real urllib3 dependency for requests.
  Upgrading requests due to urllib3 incompatibility.
  Login required should use the next parameter.
  Upgrading django oauth toolkit for django 1.11.
  Handling newsletters with multiple recipients.
  Extracting image urls sometimes fails.
  Handling ajax errors in json views.
  Adding timeouts to most outbound requests.
  Sentry SDK 0.19.4.
  Removing imperfect proxy warning for every story.
  Found four more GET/POST crosses.
  Feed unread count may need a POST.
  Namespacing settings.
  ...
2020-12-08 09:10:51 -05:00
Samuel Clay
8371c635f7 Merge branch 'master' into django2.0
* master: (27 commits)
  Removing log override
  Moving logging over to the newsblur log.
  Fixing search indexer background task for new celery.
  Attempting to add gunicorn errors to console/log.
  Better handling of missing subs.
  Handling missing user sub on feed delete.
  Correct encoding for strings on systems that don't have utf-8 as default encoding.
  Writing in the real urllib3 dependency for requests.
  Upgrading requests due to urllib3 incompatibility.
  Login required should use the next parameter.
  Upgrading django oauth toolkit for django 1.11.
  Handling newsletters with multiple recipients.
  Extracting image urls sometimes fails.
  Handling ajax errors in json views.
  Adding timeouts to most outbound requests.
  Sentry SDK 0.19.4.
  Removing imperfect proxy warning for every story.
  Found four more GET/POST crosses.
  Feed unread count may need a POST.
  Namespacing settings.
  ...
2020-12-08 09:09:25 -05:00
Samuel Clay
1a5d440582 Adding timeouts to most outbound requests. 2020-12-06 11:37:01 -05:00
Samuel Clay
4bb9d47144 Upgrading all dependencies. This probably won't work (looking at you timezones). 2020-12-03 16:50:54 -05:00
Samuel Clay
b89e7dc429 Merge branch 'django1.11' into django2.0
* django1.11: (152 commits)
  request.raw_post_data -> request.body (django 1.6)
  Upgrading pgbouncer to 1.15.0.
  Finishing off Postgresql 13 upgrade.
  Upgrading to Postgresql 13.
  Ubuntu 20.04
  Fixing supervisor path issues
  Upgrading setuptools
  Fixing flask
  Handling over capacity for twitter.
  Max length for image_urls.
  Properly filtering newsletter feeds.
  Fixing issue with text importer on feed-less urls.
  Removing dependency, fixing encoding issue for pages.
  Fixing DB Monitor.
  Updating User Agent for all fetchers.
  Ignoring VSCode.
  Fixing DB Monitor.
  Updating User Agent for all fetchers.
  Ignoring VSCode.
  Fixing Statistics by fixing how timezones are handled.
  ...
2020-12-03 14:04:26 -05:00
Samuel Clay
e75da8771c Removing dependency, fixing encoding issue for pages. 2020-11-30 18:17:26 -05:00
Samuel Clay
21f40c253f Updating User Agent for all fetchers. 2020-11-30 18:03:15 -05:00
Samuel Clay
1351aee6a8 Vendorizing new feedparser. 2020-07-01 18:38:37 -04:00
Samuel Clay
0d6cb69548 Merge branch 'django1.11' into django2.0
* django1.11: (73 commits)
  Switching to new celery 4 standalone binary.
  Fixing various mongo data calls.
  Upgrading to latest celery 4 (holy moly), which required some big changes to project layout. Still needs supervisor scripts updated.
  Removing unused log on cookies.
  I believe this Context wrapping is still preserved. See this django ticket: https://code.djangoproject.com/ticket/28125. Reverting this fixes the error, so I'm assuming this is that type of render.
  Have to revert 3f122d5e03 because this broke existing sessions (logged me out) because the model has changed and the serialized model stored in redis no longer matches. Whew, this took a while to figure out.
  Upgrading redis cache.
  Adding cookies to path inspector.
  Removing dupe db log.
  Fixing missing DB logs (redis and mongo) due to this change in django 1.8: "connections.queries is now a read-only attribute."
  Removing migrations that set a default date of 2020-05-08. Not sure why this was committed. I thought we resolved the issue with default datetimes?
  Fixing CallableBool.
  Missing import
  Fixing runtime errors on django 1.10
  Fixing OAuth connect.
  Fixing various django1.9 issues, mainly around templates.
  BASE_DIR
  Not every story is from a feed.
  Styling background colors for newsletters.
  Styling more newsletter elements.
  ...
2020-06-30 12:34:59 -04:00
Samuel Clay
3fa55aff17 Upgrading to latest celery 4 (holy moly), which required some big changes to project layout. Still needs supervisor scripts updated. 2020-06-29 17:39:55 -04:00
jmath1
6021afaec3 2to3 apps/rss_feeds 2020-06-15 02:54:37 -04:00
Samuel Clay
0a608abb87 Handling broken page importer. 2020-05-04 09:50:01 -04:00
Samuel Clay
cef29f0165 Adding royalroad.com to excluded original page sites. 2020-02-27 10:23:23 -05:00
Samuel Clay
01f092e662 Handle no page 2019-12-21 10:21:27 -05:00
Samuel Clay
3780cddb47 Handle no page 2019-12-21 10:20:59 -05:00
Samuel Clay
ec35f2156a Don't bother decoding latin1 for pages. 2019-12-21 10:04:58 -05:00
Samuel Clay
4290b57370 Better handling of broken original text. 2018-07-16 10:55:17 -04:00
Samuel Clay
2b83ccabc1 Handling missig feed on page importer. 2018-07-16 10:50:29 -04:00
Samuel Clay
0d6d4de92e Removing ESPN from original pages. 2017-11-02 16:39:36 -07:00
Samuel Clay
461c1c4b65 Changing feed log format to include id at the beginning. Also normalizing all feed titles in logs for better searchability. 2017-03-31 19:52:24 -07:00
Samuel Clay
440d2c5841 Fixing broken S3 names. 2017-03-28 21:17:11 -07:00
Samuel Clay
9f15958c0e Only connect to S3 when needed. 2017-03-28 10:43:31 -07:00
Samuel Clay
c4830e3e95 Handling unicode encode errors in page/text handling. Also adding upgrade command for fabric when pip is non-trivial. 2016-12-05 22:09:05 -08:00
Samuel Clay
3ed96e338c Fixing page and text importer to correctly handling non-breaking spaces. 2016-12-05 17:40:39 -08:00
Samuel Clay
3a93ad7248 Fixing original page server. 2016-11-30 17:17:40 -08:00
Samuel Clay
2d6343b7ae Fixing feedfinder beautifulsoup issues, unicode issues in feed, and unicode comparisons issues with page_data (since it wasnt being decompressed when compared). 2016-02-05 14:43:31 -08:00
Samuel Clay
b6e1e03bd5 Upgrading feedparser, cleaing up logging on feed fetch. 2015-08-24 14:27:01 -07:00
Samuel Clay
22ae0e65e4 Removing gamespot.com feed from page and text fetchers. 2015-08-03 20:09:36 -07:00
Samuel Clay
b27590506c Fixing notuniqueerrors. 2015-07-07 13:34:06 -07:00
Samuel Clay
5001ad7602 Handling timeout on original page fetch. 2015-03-11 11:50:46 -07:00
Samuel Clay
02c91bb8b0 Fixing timeout error on PUSH. Also fixing requests error on page importing. 2015-03-11 11:44:47 -07:00
Samuel Clay
28f0860fff Fixing issue when resizing window causing story progress bar to resurface. 2014-10-30 13:24:24 -07:00
Samuel Clay
b91b65772a Adding proxy for Story view to handle https loading http sites. Needs testing, but done. 2014-10-29 16:16:50 -07:00
Samuel Clay
5e064fcc2e Handling PyAsn1Error 2014-05-27 13:08:21 -07:00
Samuel Clay
def51705c6 Handling the many OpenSSL errors. 2014-05-22 15:10:29 -07:00
Samuel Clay
0db2a1d110 Revert "Ripping out SNI/openssl/requests latest. Turns out pyOpenSSL barfs on a ton of pages."
This reverts commit 177b8aea09.
2014-05-22 15:09:08 -07:00
Samuel Clay
177b8aea09 Ripping out SNI/openssl/requests latest. Turns out pyOpenSSL barfs on a ton of pages. 2014-05-22 15:06:16 -07:00
Samuel Clay
c26086b2f6 Explicitly closing connection on page fetch. 2014-05-22 14:54:15 -07:00
Samuel Clay
889be221d5 Fixing broken original pages by auto-fixing the has_page field on successful page fetch. Thanks to The Verge for uncovering this one. 2013-07-23 10:43:28 -07:00
Samuel Clay
1936515aa0 Fixing Dilbert-specific encoding error. 2013-07-18 15:17:15 -07:00
Samuel Clay
a36955ee32 Fixing encoding on Original pages. Thanks @bruceyou for the wonderful help! 2013-07-18 15:03:12 -07:00
Samuel Clay
5f7ae25fc4 Normaling user agents between feed/page/icon fetchers. Adding full site permalink. 2013-07-10 14:14:55 -07:00
Samuel Clay
275b0ff9e4 Handling socket error on page fetch. 2013-05-29 15:59:54 -07:00
Samuel Clay
b3744e3ac6 Merge branch 'master' into circular
* master:
  Warning on missing maintenance.
  Further pushing out feed updates.
  Using @dcramer's excellent getsentry.com for all exceptions. Turning off exception emails.
  Further pushing down feed fetches to handle load until db is scaled out.
  Scrolling code blocks.
  New task servers.
  Turning down feed fetches.
  If requests.raw won't work, sut fudge it with a StringIO for now.
  Further turning down feed fetches until mongo is ready to replicate.
  Fixing Facebook image calculation.
  Fixing facebook posting.
  Moving shared story unread recalc to background task. Will greatly speed up sharing.
  Ramping down feed fetching until they can be handled.
  Only show time breakdown when > 1 sec.
  Fixed API docs for /reader/river_stories
  make fabfile.py use requirements.txt
  Make sure to send the proper content-type with social feeds
  Restore default whitespace to pre/code blocks
  Phase 2: switching to using story_hash index instead of compound unique story_feed_id/story_guid index.
  document keyboard shortcut to add site/folder

Conflicts:
	apps/social/views.py
2013-03-20 10:49:09 -07:00
Samuel Clay
a75554374b Using @dcramer's excellent getsentry.com for all exceptions. Turning off exception emails. 2013-03-20 08:24:11 -07:00
Samuel Clay
a3383e1bd6 New icons. 2013-03-02 11:18:12 -08:00