Commit graph

521 commits

Author SHA1 Message Date
Samuel Clay
6f97446b3b Slightly refactoring feed fetcher to deal with weakref's when pickling functions. 2020-12-15 17:24:01 -05:00
Samuel Clay
aea4fcbe5b Merge branch 'django3.0' into django3.1
* django3.0: (27 commits)
  Removing log override
  Moving logging over to the newsblur log.
  Fixing search indexer background task for new celery.
  Attempting to add gunicorn errors to console/log.
  Better handling of missing subs.
  Handling missing user sub on feed delete.
  Correct encoding for strings on systems that don't have utf-8 as default encoding.
  Writing in the real urllib3 dependency for requests.
  Upgrading requests due to urllib3 incompatibility.
  Login required should use the next parameter.
  Upgrading django oauth toolkit for django 1.11.
  Handling newsletters with multiple recipients.
  Extracting image urls sometimes fails.
  Handling ajax errors in json views.
  Adding timeouts to most outbound requests.
  Sentry SDK 0.19.4.
  Removing imperfect proxy warning for every story.
  Found four more GET/POST crosses.
  Feed unread count may need a POST.
  Namespacing settings.
  ...
2020-12-08 09:10:51 -05:00
Samuel Clay
8371c635f7 Merge branch 'master' into django2.0
* master: (27 commits)
  Removing log override
  Moving logging over to the newsblur log.
  Fixing search indexer background task for new celery.
  Attempting to add gunicorn errors to console/log.
  Better handling of missing subs.
  Handling missing user sub on feed delete.
  Correct encoding for strings on systems that don't have utf-8 as default encoding.
  Writing in the real urllib3 dependency for requests.
  Upgrading requests due to urllib3 incompatibility.
  Login required should use the next parameter.
  Upgrading django oauth toolkit for django 1.11.
  Handling newsletters with multiple recipients.
  Extracting image urls sometimes fails.
  Handling ajax errors in json views.
  Adding timeouts to most outbound requests.
  Sentry SDK 0.19.4.
  Removing imperfect proxy warning for every story.
  Found four more GET/POST crosses.
  Feed unread count may need a POST.
  Namespacing settings.
  ...
2020-12-08 09:09:25 -05:00
Samuel Clay
18db90df9a Extracting image urls sometimes fails. 2020-12-06 12:30:24 -05:00
Samuel Clay
1a5d440582 Adding timeouts to most outbound requests. 2020-12-06 11:37:01 -05:00
Samuel Clay
4bb9d47144 Upgrading all dependencies. This probably won't work (looking at you timezones). 2020-12-03 16:50:54 -05:00
Samuel Clay
5eaa368315 Wrong fix for missing tagline. SHould instead create a FeedData. 2020-11-13 18:53:08 -05:00
Samuel Clay
364d85afec Fixing missing tagline error. 2020-11-13 18:48:18 -05:00
Samuel Clay
c508c58224 Merge branch 'django1.11' into django2.0
* django1.11: (102 commits)
  Beginning fix of rss_feeds unit tests.
  Fixing unit test for profile app and signup.
  Remove highlights count when it reaches zero.
  Android v10.1b1.
  Stubbing in profile tests.
  Adding nginx.local.conf
  Adding original text and original story to API docs.
  #1282 Adding feed to root folder
  #1319 In app and external browser options
  Adding a smarter wakeup for real-time to handle cases where a laptop is re-opened but real-time is not immediately reestablished.
  #1348 (scroll indicators theme)
  #1344 (search loses focus)
  #1335 Auto theme option for OS level dark mode
  Fixing signup flow.
  #1347 Show pager with stories after using the intel trainer and refreshing
  #1272 Load HTML in comments
  New icon for Infrequent Site Stories.
  Allowing selection in private notes.
  Autoresizing private notes field.
  For #1035: Adding private notes to saved stories.
  ...
2020-08-10 17:53:48 -04:00
Samuel Clay
ac1471017d Merge branch 'django1.10' into django1.11
* django1.10: (102 commits)
  Beginning fix of rss_feeds unit tests.
  Fixing unit test for profile app and signup.
  Remove highlights count when it reaches zero.
  Android v10.1b1.
  Stubbing in profile tests.
  Adding nginx.local.conf
  Adding original text and original story to API docs.
  #1282 Adding feed to root folder
  #1319 In app and external browser options
  Adding a smarter wakeup for real-time to handle cases where a laptop is re-opened but real-time is not immediately reestablished.
  #1348 (scroll indicators theme)
  #1344 (search loses focus)
  #1335 Auto theme option for OS level dark mode
  Fixing signup flow.
  #1347 Show pager with stories after using the intel trainer and refreshing
  #1272 Load HTML in comments
  New icon for Infrequent Site Stories.
  Allowing selection in private notes.
  Autoresizing private notes field.
  For #1035: Adding private notes to saved stories.
  ...
2020-08-10 17:52:24 -04:00
Samuel Clay
dec27e7adc Beginning fix of rss_feeds unit tests. 2020-08-04 13:45:16 -04:00
Samuel Clay
a8e0a7544b Handling various feed fetching exceptions. 2020-07-01 19:14:38 -04:00
Samuel Clay
1351aee6a8 Vendorizing new feedparser. 2020-07-01 18:38:37 -04:00
Samuel Clay
bc7055b971 StringIO -> BytesIO 2020-07-01 11:57:51 -04:00
Samuel Clay
eeb44e3400 Fixing a bunch of feed fetch py3 errors. Fetching now works! 2020-06-30 20:50:30 -04:00
Samuel Clay
42d4a4211f Fixing encoding issues. 2020-06-30 17:22:47 -04:00
Samuel Clay
0d6cb69548 Merge branch 'django1.11' into django2.0
* django1.11: (73 commits)
  Switching to new celery 4 standalone binary.
  Fixing various mongo data calls.
  Upgrading to latest celery 4 (holy moly), which required some big changes to project layout. Still needs supervisor scripts updated.
  Removing unused log on cookies.
  I believe this Context wrapping is still preserved. See this django ticket: https://code.djangoproject.com/ticket/28125. Reverting this fixes the error, so I'm assuming this is that type of render.
  Have to revert 3f122d5e03 because this broke existing sessions (logged me out) because the model has changed and the serialized model stored in redis no longer matches. Whew, this took a while to figure out.
  Upgrading redis cache.
  Adding cookies to path inspector.
  Removing dupe db log.
  Fixing missing DB logs (redis and mongo) due to this change in django 1.8: "connections.queries is now a read-only attribute."
  Removing migrations that set a default date of 2020-05-08. Not sure why this was committed. I thought we resolved the issue with default datetimes?
  Fixing CallableBool.
  Missing import
  Fixing runtime errors on django 1.10
  Fixing OAuth connect.
  Fixing various django1.9 issues, mainly around templates.
  BASE_DIR
  Not every story is from a feed.
  Styling background colors for newsletters.
  Styling more newsletter elements.
  ...
2020-06-30 12:34:59 -04:00
Samuel Clay
3fa55aff17 Upgrading to latest celery 4 (holy moly), which required some big changes to project layout. Still needs supervisor scripts updated. 2020-06-29 17:39:55 -04:00
jmath1
66b2011707 2to3 NewsBlur/utils/ 2020-06-13 13:13:20 -04:00
Samuel Clay
d37ca03d07 Adding twitter lists to feed fetcher. 2020-06-04 16:57:04 -04:00
Samuel Clay
15d393cc26 No story title and no content means no story. 2018-08-02 11:32:24 -04:00
Samuel Clay
f2c1585da0 Fixing facebook pages match. 2018-04-03 12:38:49 -07:00
Samuel Clay
7fe5b63bdf Merge branch 'master' into facebook
* master:
  Faking user agent shouldnt use a randomized URL.
  Adding authed api url add.
2018-03-27 12:01:38 -07:00
Samuel Clay
1321cd40aa Faking user agent shouldnt use a randomized URL. 2018-03-27 12:01:31 -07:00
Samuel Clay
471d5434c6 Facebook pages feed gets the correct favicon. 2018-03-26 19:05:22 -07:00
Samuel Clay
da4741a3fa Adding videos to facebook posts. Seems to work pretty well. 2018-03-26 17:31:12 -07:00
Samuel Clay
364ff9dc85 Fixing YouTube to not verify SSL certs. 2018-03-09 09:44:05 -08:00
Daniel Aleksandersen
19a24d8b7e Support HTTP 308 from RFC 7238 2017-08-16 09:17:23 +02:00
Samuel Clay
3bcfba0a9b Fixing YouTube Playlists. 2017-07-27 17:14:15 -07:00
Samuel Clay
c1834703d9 Adding support for JSON Feeds. 2017-05-22 16:46:56 -07:00
Samuel Clay
6bc9a55bfc Rewriting twitter fetching. Now fetching truncated text, embedding quoted tweets, and fixing URLs so that they show the display url (and not t.co) and link to the expanded url. 2017-05-06 19:38:36 -07:00
Samuel Clay
85355a5c9c Tweets now show full, untruncated text. They also expand all URLs. 2017-05-06 17:52:31 -07:00
Samuel Clay
44de405195 Only fetch as many guidas as there are stories in the feed. No need to get everything assuming the feed has the latest N stories. 2017-05-01 12:06:56 -07:00
Samuel Clay
08c897e5a1 This is it, the big kahuna of fixes. This corrects for messed up guids that are causing lots of read stories to become unread. 2017-05-01 11:39:24 -07:00
Samuel Clay
b753cf6819 This won't fix it but in the event of missing guids on feed fetch, check for stories within the mark read period. 2017-05-01 10:51:04 -07:00
Samuel Clay
c9326a6f02 Perhaps this is the way to find the missing story hashes. Shouldn't cause an issue, but logic for dates may be backwards. 2017-05-01 09:27:31 -07:00
Samuel Clay
ffeeb170e0 Finally have a test case for the Google Blog duping. 2017-04-30 18:47:10 -07:00
Samuel Clay
9d9a5accbd Better handling of tweepy errors. 2017-04-13 11:43:39 -07:00
Samuel Clay
d84e2af636 New experimental data collector for debugging feeds over time. 2017-04-12 19:13:33 -07:00
Samuel Clay
557438008b Fixing a couple Twitter and YouTube fetch issues. 2017-04-12 11:19:42 -07:00
Samuel Clay
461c1c4b65 Changing feed log format to include id at the beginning. Also normalizing all feed titles in logs for better searchability. 2017-03-31 19:52:24 -07:00
Samuel Clay
937a67f67c Only using active twitter accounts. 2017-03-24 13:04:38 -07:00
Samuel Clay
4f8a1c43b3 Fixing attribution in twitter RTs. 2017-03-15 19:40:43 -07:00
Samuel Clay
a6fdf45d33 Adding support for native RTs in Twitter. 2017-03-15 19:26:39 -07:00
Samuel Clay
937a1fa999 Faking user agent when rss bans NewsBlur. 2017-02-17 12:19:55 -08:00
Samuel Clay
255b30f3c1 Better recording of non-200/300 status codes on feed fetch. 2017-02-17 12:05:27 -08:00
Samuel Clay
2954ef7318 Adding new debug to feed fetcher. 2017-02-17 11:45:43 -08:00
Samuel Clay
c5d5ea5001 Attempting to fix dashboard stories by no longer relying on full river reloads when a new story comes in but to fetch that story if it is newer (or older) than last visible dashboard story. 2016-12-13 16:29:42 -08:00
Samuel Clay
046f97558b Checking for memory errors while feed fetching. 2016-12-11 18:59:02 -08:00
Samuel Clay
d1e2e3e1d9 Handling MemoryError while feed fetching. 2016-12-11 18:29:00 -08:00