Commit graph

119 commits

Author SHA1 Message Date
Samuel Clay
909722b361 Showing reason why an original page won't load. Fixes #1519. 2022-05-02 16:57:15 -04:00
Samuel Clay
fa13a0f9d4 No using extraargs for s3. 2021-08-26 16:21:36 -04:00
Samuel Clay
1469992f2d S3 content-type to contenttype. 2021-08-26 15:50:11 -04:00
Samuel Clay
f03be46425 Removing boto2, upgrading to boto3. Needs testing. 2021-08-04 20:04:53 -04:00
Samuel Clay
c6ce8cc36a Everything is smart bytes when it comes to original pages. 2021-05-12 21:19:09 -04:00
Samuel Clay
16e5a0bd4f Reading icon url from node page instead of s3 page. Also preparing to turn off s3 page. 2021-05-05 11:45:44 -04:00
Samuel Clay
b571e74242 Dont bother saving bunk error 2021-04-20 18:18:52 -04:00
Samuel Clay
c9a3025204 Handling page importer url error. 2021-04-19 16:19:27 -04:00
Samuel Clay
3f6c9bbb74 String compression zlib vs gzip. 2021-04-02 14:29:43 -04:00
Samuel Clay
87e076a9ae One more str->bytes. 2021-04-02 14:23:03 -04:00
Samuel Clay
2f501fff6d Can only compress bytes, not str. 2021-04-02 12:55:07 -04:00
Samuel Clay
918ddb5200 Fixing issue with both requests and urllib fallback. 2021-04-02 12:51:48 -04:00
Samuel Clay
1bd5ac2219 Fixing #1434. Original pages weren't working until now due to encoding issues. 2021-04-01 18:22:50 -04:00
Samuel Clay
23abeb0b0a Handling feedparser backup failing. 2021-02-28 20:39:57 -05:00
Samuel Clay
ba6a8c81e0 Switching from raven to sentry sdk. 2021-02-25 19:52:36 -05:00
Samuel Clay
c7f4cbb13d Fixing a substantial number of encoding errors. Also fixing OPML uploads. 2021-01-24 21:05:22 -05:00
Jonathan Math
79f249878b remove use of ORIGINAL_PAGE_SERVER 2021-01-03 11:09:25 -05:00
Samuel Clay
aea4fcbe5b Merge branch 'django3.0' into django3.1
* django3.0: (27 commits)
  Removing log override
  Moving logging over to the newsblur log.
  Fixing search indexer background task for new celery.
  Attempting to add gunicorn errors to console/log.
  Better handling of missing subs.
  Handling missing user sub on feed delete.
  Correct encoding for strings on systems that don't have utf-8 as default encoding.
  Writing in the real urllib3 dependency for requests.
  Upgrading requests due to urllib3 incompatibility.
  Login required should use the next parameter.
  Upgrading django oauth toolkit for django 1.11.
  Handling newsletters with multiple recipients.
  Extracting image urls sometimes fails.
  Handling ajax errors in json views.
  Adding timeouts to most outbound requests.
  Sentry SDK 0.19.4.
  Removing imperfect proxy warning for every story.
  Found four more GET/POST crosses.
  Feed unread count may need a POST.
  Namespacing settings.
  ...
2020-12-08 09:10:51 -05:00
Samuel Clay
8371c635f7 Merge branch 'master' into django2.0
* master: (27 commits)
  Removing log override
  Moving logging over to the newsblur log.
  Fixing search indexer background task for new celery.
  Attempting to add gunicorn errors to console/log.
  Better handling of missing subs.
  Handling missing user sub on feed delete.
  Correct encoding for strings on systems that don't have utf-8 as default encoding.
  Writing in the real urllib3 dependency for requests.
  Upgrading requests due to urllib3 incompatibility.
  Login required should use the next parameter.
  Upgrading django oauth toolkit for django 1.11.
  Handling newsletters with multiple recipients.
  Extracting image urls sometimes fails.
  Handling ajax errors in json views.
  Adding timeouts to most outbound requests.
  Sentry SDK 0.19.4.
  Removing imperfect proxy warning for every story.
  Found four more GET/POST crosses.
  Feed unread count may need a POST.
  Namespacing settings.
  ...
2020-12-08 09:09:25 -05:00
Samuel Clay
1a5d440582 Adding timeouts to most outbound requests. 2020-12-06 11:37:01 -05:00
Samuel Clay
4bb9d47144 Upgrading all dependencies. This probably won't work (looking at you timezones). 2020-12-03 16:50:54 -05:00
Samuel Clay
b89e7dc429 Merge branch 'django1.11' into django2.0
* django1.11: (152 commits)
  request.raw_post_data -> request.body (django 1.6)
  Upgrading pgbouncer to 1.15.0.
  Finishing off Postgresql 13 upgrade.
  Upgrading to Postgresql 13.
  Ubuntu 20.04
  Fixing supervisor path issues
  Upgrading setuptools
  Fixing flask
  Handling over capacity for twitter.
  Max length for image_urls.
  Properly filtering newsletter feeds.
  Fixing issue with text importer on feed-less urls.
  Removing dependency, fixing encoding issue for pages.
  Fixing DB Monitor.
  Updating User Agent for all fetchers.
  Ignoring VSCode.
  Fixing DB Monitor.
  Updating User Agent for all fetchers.
  Ignoring VSCode.
  Fixing Statistics by fixing how timezones are handled.
  ...
2020-12-03 14:04:26 -05:00
Samuel Clay
e75da8771c Removing dependency, fixing encoding issue for pages. 2020-11-30 18:17:26 -05:00
Samuel Clay
21f40c253f Updating User Agent for all fetchers. 2020-11-30 18:03:15 -05:00
Samuel Clay
1351aee6a8 Vendorizing new feedparser. 2020-07-01 18:38:37 -04:00
Samuel Clay
0d6cb69548 Merge branch 'django1.11' into django2.0
* django1.11: (73 commits)
  Switching to new celery 4 standalone binary.
  Fixing various mongo data calls.
  Upgrading to latest celery 4 (holy moly), which required some big changes to project layout. Still needs supervisor scripts updated.
  Removing unused log on cookies.
  I believe this Context wrapping is still preserved. See this django ticket: https://code.djangoproject.com/ticket/28125. Reverting this fixes the error, so I'm assuming this is that type of render.
  Have to revert 3f122d5e03 because this broke existing sessions (logged me out) because the model has changed and the serialized model stored in redis no longer matches. Whew, this took a while to figure out.
  Upgrading redis cache.
  Adding cookies to path inspector.
  Removing dupe db log.
  Fixing missing DB logs (redis and mongo) due to this change in django 1.8: "connections.queries is now a read-only attribute."
  Removing migrations that set a default date of 2020-05-08. Not sure why this was committed. I thought we resolved the issue with default datetimes?
  Fixing CallableBool.
  Missing import
  Fixing runtime errors on django 1.10
  Fixing OAuth connect.
  Fixing various django1.9 issues, mainly around templates.
  BASE_DIR
  Not every story is from a feed.
  Styling background colors for newsletters.
  Styling more newsletter elements.
  ...
2020-06-30 12:34:59 -04:00
Samuel Clay
3fa55aff17 Upgrading to latest celery 4 (holy moly), which required some big changes to project layout. Still needs supervisor scripts updated. 2020-06-29 17:39:55 -04:00
jmath1
6021afaec3 2to3 apps/rss_feeds 2020-06-15 02:54:37 -04:00
Samuel Clay
0a608abb87 Handling broken page importer. 2020-05-04 09:50:01 -04:00
Samuel Clay
cef29f0165 Adding royalroad.com to excluded original page sites. 2020-02-27 10:23:23 -05:00
Samuel Clay
01f092e662 Handle no page 2019-12-21 10:21:27 -05:00
Samuel Clay
3780cddb47 Handle no page 2019-12-21 10:20:59 -05:00
Samuel Clay
ec35f2156a Don't bother decoding latin1 for pages. 2019-12-21 10:04:58 -05:00
Samuel Clay
4290b57370 Better handling of broken original text. 2018-07-16 10:55:17 -04:00
Samuel Clay
2b83ccabc1 Handling missig feed on page importer. 2018-07-16 10:50:29 -04:00
Samuel Clay
0d6d4de92e Removing ESPN from original pages. 2017-11-02 16:39:36 -07:00
Samuel Clay
461c1c4b65 Changing feed log format to include id at the beginning. Also normalizing all feed titles in logs for better searchability. 2017-03-31 19:52:24 -07:00
Samuel Clay
440d2c5841 Fixing broken S3 names. 2017-03-28 21:17:11 -07:00
Samuel Clay
9f15958c0e Only connect to S3 when needed. 2017-03-28 10:43:31 -07:00
Samuel Clay
c4830e3e95 Handling unicode encode errors in page/text handling. Also adding upgrade command for fabric when pip is non-trivial. 2016-12-05 22:09:05 -08:00
Samuel Clay
3ed96e338c Fixing page and text importer to correctly handling non-breaking spaces. 2016-12-05 17:40:39 -08:00
Samuel Clay
3a93ad7248 Fixing original page server. 2016-11-30 17:17:40 -08:00
Samuel Clay
2d6343b7ae Fixing feedfinder beautifulsoup issues, unicode issues in feed, and unicode comparisons issues with page_data (since it wasnt being decompressed when compared). 2016-02-05 14:43:31 -08:00
Samuel Clay
b6e1e03bd5 Upgrading feedparser, cleaing up logging on feed fetch. 2015-08-24 14:27:01 -07:00
Samuel Clay
22ae0e65e4 Removing gamespot.com feed from page and text fetchers. 2015-08-03 20:09:36 -07:00
Samuel Clay
b27590506c Fixing notuniqueerrors. 2015-07-07 13:34:06 -07:00
Samuel Clay
5001ad7602 Handling timeout on original page fetch. 2015-03-11 11:50:46 -07:00
Samuel Clay
02c91bb8b0 Fixing timeout error on PUSH. Also fixing requests error on page importing. 2015-03-11 11:44:47 -07:00
Samuel Clay
28f0860fff Fixing issue when resizing window causing story progress bar to resurface. 2014-10-30 13:24:24 -07:00
Samuel Clay
b91b65772a Adding proxy for Story view to handle https loading http sites. Needs testing, but done. 2014-10-29 16:16:50 -07:00