Commit graph

85 commits

Author SHA1 Message Date
Samuel Clay
8371c635f7 Merge branch 'master' into django2.0
* master: (27 commits)
  Removing log override
  Moving logging over to the newsblur log.
  Fixing search indexer background task for new celery.
  Attempting to add gunicorn errors to console/log.
  Better handling of missing subs.
  Handling missing user sub on feed delete.
  Correct encoding for strings on systems that don't have utf-8 as default encoding.
  Writing in the real urllib3 dependency for requests.
  Upgrading requests due to urllib3 incompatibility.
  Login required should use the next parameter.
  Upgrading django oauth toolkit for django 1.11.
  Handling newsletters with multiple recipients.
  Extracting image urls sometimes fails.
  Handling ajax errors in json views.
  Adding timeouts to most outbound requests.
  Sentry SDK 0.19.4.
  Removing imperfect proxy warning for every story.
  Found four more GET/POST crosses.
  Feed unread count may need a POST.
  Namespacing settings.
  ...
2020-12-08 09:09:25 -05:00
Samuel Clay
1a5d440582 Adding timeouts to most outbound requests. 2020-12-06 11:37:01 -05:00
Samuel Clay
b89e7dc429 Merge branch 'django1.11' into django2.0
* django1.11: (152 commits)
  request.raw_post_data -> request.body (django 1.6)
  Upgrading pgbouncer to 1.15.0.
  Finishing off Postgresql 13 upgrade.
  Upgrading to Postgresql 13.
  Ubuntu 20.04
  Fixing supervisor path issues
  Upgrading setuptools
  Fixing flask
  Handling over capacity for twitter.
  Max length for image_urls.
  Properly filtering newsletter feeds.
  Fixing issue with text importer on feed-less urls.
  Removing dependency, fixing encoding issue for pages.
  Fixing DB Monitor.
  Updating User Agent for all fetchers.
  Ignoring VSCode.
  Fixing DB Monitor.
  Updating User Agent for all fetchers.
  Ignoring VSCode.
  Fixing Statistics by fixing how timezones are handled.
  ...
2020-12-03 14:04:26 -05:00
Samuel Clay
21f40c253f Updating User Agent for all fetchers. 2020-11-30 18:03:15 -05:00
Samuel Clay
fa43eed1b8 Updating User Agent for all fetchers. 2020-11-30 15:48:59 -05:00
Samuel Clay
a8e0a7544b Handling various feed fetching exceptions. 2020-07-01 19:14:38 -04:00
Samuel Clay
1351aee6a8 Vendorizing new feedparser. 2020-07-01 18:38:37 -04:00
Samuel Clay
6f14455115 Proper base64 encode/decode 2020-07-01 16:59:21 -04:00
Samuel Clay
bc7055b971 StringIO -> BytesIO 2020-07-01 11:57:51 -04:00
jmath1
6021afaec3 2to3 apps/rss_feeds 2020-06-15 02:54:37 -04:00
Samuel Clay
09f5ae7a67 Handling MemoryError. 2019-10-13 09:16:17 -04:00
Samuel Clay
471d5434c6 Facebook pages feed gets the correct favicon. 2018-03-26 19:05:22 -07:00
Diego Elio Pettenò
4ff8ed7798 favicon fetcher: provide an Accept header on requests.
This makes the fetcher work with the default ModSecurity ruleset, and would also avoid providing unsupported favicon formats (e.g. webp), by explictly preferring PNG and
x-icon formats.
2017-08-16 18:21:14 +00:00
Samuel Clay
78568f0914 Handling broken URL for icons. 2017-03-31 20:30:43 -07:00
Samuel Clay
461c1c4b65 Changing feed log format to include id at the beginning. Also normalizing all feed titles in logs for better searchability. 2017-03-31 19:52:24 -07:00
Samuel Clay
440d2c5841 Fixing broken S3 names. 2017-03-28 21:17:11 -07:00
Samuel Clay
9f15958c0e Only connect to S3 when needed. 2017-03-28 10:43:31 -07:00
Samuel Clay
4dc17b751b Handling error when fetching icn for bad feed link. 2016-12-11 18:32:37 -08:00
Samuel Clay
44e9b131a0 Fixing icon saving to s3 bug. 2016-04-06 17:06:32 -07:00
Samuel Clay
e1cf12888a Handling mis-shapen icons. 2016-02-16 13:24:09 -08:00
Samuel Clay
61fb9f1719 Handling IncompleteRead from requests during icon fetch. 2016-02-16 13:22:11 -08:00
Samuel Clay
7c8f85c24a Newsletter icons now work. Time to try it out on prod. 2016-02-09 17:14:59 -08:00
Samuel Clay
6edbf67a50 Fix to scipy and the icon importer credited entirely to the work of @sbilly. 2016-02-04 21:31:49 -08:00
Samuel Clay
e3df276232 Trying to save S3 saving if only color is different. k-means clustering is non-deterministic, so this led to a ton of multi-uploads. 2016-02-02 18:54:34 -08:00
Samuel Clay
6be803a08f Icon importer needs to actually be different, not just forced. 2016-01-03 15:55:22 -08:00
Slavik Svyrydiuk
017170cbb4 PEP8 fixes 2015-12-02 12:15:56 +01:00
Samuel Clay
11ec92f9f0 Fixing yet another requests error on icon fetching. 2015-11-28 13:52:18 -08:00
Samuel Clay
d1cb93433c Handling another requests error when fetching icons. 2015-11-28 13:28:28 -08:00
Samuel Clay
da26ecdb46 1/2 Cleaning up feed fetcher's updates to feeds table. Still need to find 6 dupes. 2015-07-22 13:18:54 -07:00
Samuel Clay
82fbccd1c2 Handling invalid urls for icons. 2015-07-07 13:35:00 -07:00
Samuel Clay
94ce13183b Handling requests issue with too many redirects. 2015-07-07 13:30:46 -07:00
Samuel Clay
5b99ab01ec Adding error handling for new icon feed link fetcher. 2015-05-19 16:31:25 -07:00
Samuel Clay
e1f31b9277 Attempting to fetch Tumblr feed icons. 2015-05-18 17:51:23 -07:00
Samuel Clay
b746dda72d Fixing massive icon problems by removing support for Windows ICO files. 2014-05-16 12:33:06 -07:00
Samuel Clay
403a4ced90 Converting from PIL to Pillow. 2014-05-16 12:18:59 -07:00
Samuel Clay
07c5f95c36 Fixing a couple small bugs. 2013-10-07 16:16:38 -07:00
Samuel Clay
d3fa017cd9 Fixing broken icon bug. 2013-08-21 12:40:04 -07:00
Samuel Clay
5f7ae25fc4 Normaling user agents between feed/page/icon fetchers. Adding full site permalink. 2013-07-10 14:14:55 -07:00
Samuel Clay
8d9da758fd Better handling of busted favicons. 2013-07-05 17:43:14 -07:00
Samuel Clay
2408156024 Reverting most PIL cleanup. 2013-06-21 13:15:04 -07:00
Samuel Clay
bb95ea8984 Cleaning up PIL. 2013-06-21 13:11:54 -07:00
Samuel Clay
cc33a3e663 Fixing icon fetch for images > 500KB. 2013-06-04 16:30:21 -07:00
Samuel Clay
4de16b77f8 Adding expires header to icons. 2013-03-22 18:29:54 -07:00
Samuel Clay
279041249e Fixing missing scipy import missing form PILlow. 2013-03-22 15:33:26 -07:00
Samuel Clay
acb1efd3ea Last imports for pillow. 2013-03-22 15:27:10 -07:00
Samuel Clay
4a7516cb83 Fixing a bunch of feed fetch errors. 2012-10-01 19:31:33 -07:00
Samuel Clay
1252819c8c Fixing issue when opening feed in Story view but second story gets selected instead of first. 2012-09-21 11:36:27 -07:00
Samuel Clay
838cabbd42 Typo in s3 icon. 2012-09-20 12:44:41 -07:00
Samuel Clay
a8975a3bc4 Forcing icon save if the icon is not yet in s3. 2012-09-20 12:43:34 -07:00
Samuel Clay
1ace793648 Completing feed icon migration to S3. 2012-09-20 12:29:52 -07:00