Commit graph

1236 commits

Author SHA1 Message Date
Samuel Clay
da4741a3fa Adding videos to facebook posts. Seems to work pretty well. 2018-03-26 17:31:12 -07:00
Samuel Clay
3d31ed2507 Handling connection error when finding feed. 2018-01-18 16:15:17 -08:00
Samuel Clay
8421f667d7 Fixing broken image handling from Mercury Reader that was causing image urls with a srcset to be concat'd together. This one's for @yesthatjwz. 2018-01-17 16:51:06 -08:00
Samuel Clay
b99ccd7045 YouTube feeds now have an auto extracted title. Thank you Bruno! 2017-12-18 21:48:26 -08:00
Samuel Clay
ef96f59c2c Only count months that have stories for the average count. 2017-12-15 17:12:14 -08:00
Samuel Clay
f2ab8145c5 Adding options to control infrequency of infrequent site stories feed. 2017-11-05 14:01:25 -08:00
Samuel Clay
6a27023e12 Merge branch 'master' into infrequent
* master:
  Handling no original doc returning in text importer.
  Assets for icons.
  No longer finding the largest image in a story if the text view already successfully found one. Also using Mercury's builtin image finder.
  Fixing warnings.
  Removing ESPN from original pages.
2017-11-03 14:46:02 -07:00
Samuel Clay
27688c7593 Handling no original doc returning in text importer. 2017-11-03 13:48:44 -07:00
Samuel Clay
2d05dc9222 No longer finding the largest image in a story if the text view already successfully found one. Also using Mercury's builtin image finder. 2017-11-03 13:47:17 -07:00
Samuel Clay
b7574a1ff7 No longer finding the largest image in a story if the text view already successfully found one. Also using Mercury's builtin image finder. 2017-11-02 22:09:37 -07:00
Samuel Clay
f543e408e9 Attempting new Infrequent Site Stories river. 2017-10-16 14:22:37 -07:00
Samuel Clay
920e4be4bd Handling case where every story is the same time. 2017-06-28 17:19:54 -07:00
Samuel Clay
c1834703d9 Adding support for JSON Feeds. 2017-05-22 16:46:56 -07:00
Samuel Clay
6bc9a55bfc Rewriting twitter fetching. Now fetching truncated text, embedding quoted tweets, and fixing URLs so that they show the display url (and not t.co) and link to the expanded url. 2017-05-06 19:38:36 -07:00
Samuel Clay
262d67abf9 Whoops, zrevrange is high to low, which means newest to oldest. 2017-05-01 12:08:56 -07:00
Samuel Clay
44de405195 Only fetch as many guidas as there are stories in the feed. No need to get everything assuming the feed has the latest N stories. 2017-05-01 12:06:56 -07:00
Samuel Clay
08c897e5a1 This is it, the big kahuna of fixes. This corrects for messed up guids that are causing lots of read stories to become unread. 2017-05-01 11:39:24 -07:00
Samuel Clay
c9326a6f02 Perhaps this is the way to find the missing story hashes. Shouldn't cause an issue, but logic for dates may be backwards. 2017-05-01 09:27:31 -07:00
Samuel Clay
ffeeb170e0 Finally have a test case for the Google Blog duping. 2017-04-30 18:47:10 -07:00
Samuel Clay
d84e2af636 New experimental data collector for debugging feeds over time. 2017-04-12 19:13:33 -07:00
Samuel Clay
461c1c4b65 Changing feed log format to include id at the beginning. Also normalizing all feed titles in logs for better searchability. 2017-03-31 19:52:24 -07:00
Samuel Clay
930abb9adb Merge branch 'master' into saved_searches
* master: (22 commits)
  Turn original text caching back on.
  Extracting images from original text's noscript.
  Fetcing the original text now extracts the image url for others.
  Monkey patching SSL for new python, since hostnames don't match with S3.
  Converting videos in email notifications to images.
  Upping quota to 100 shared stories a day.
  Bumping premium shares to 50 per day.
  Android v5.1.0
  Only 20 stories may be shared per day for premiums, 3 for free users. Also hits IFTTT sharing.
  Improving messaging on emails that have OPML backups. Thanks to @frenetic for bringing this up. Closes #1003
  Fixing up postgresql backup.
  Hiding cookie lost message.
  Parallel pgbouncer kill.
  Fixing attribution in twitter RTs.
  Downgrading to elasticsearch 2.4.4, since pyes isn't ready for ES 5.
  Adding support for native RTs in Twitter.
  Automatically disbaling transparent huge pages (THP) on mongo and redis. Also upgrading to elasticsearch 5.2.2, although its untested.
  Add two buttons to get the app
  Fixing broken getsatisfaction community feedback.
  Goodbye Turn Touch campaign.
  ...
2017-03-23 16:54:37 -07:00
Samuel Clay
fdfcc8e798 Turn original text caching back on. 2017-03-23 16:29:15 -07:00
Samuel Clay
82cdae1e4d Extracting images from original text's noscript. 2017-03-23 16:28:47 -07:00
Samuel Clay
2c195cde2a Fetcing the original text now extracts the image url for others. 2017-03-23 16:06:06 -07:00
Samuel Clay
e1bd42612f Adding unique index. 2017-03-07 12:28:21 -08:00
Samuel Clay
2c0bf76e20 Saved search for feeds now works. Need to fix scroll to selected feed (and keyboard shortcuts). Also need to hook up other search types. 2017-03-06 19:55:18 -08:00
Samuel Clay
3435bac504 Showing saved searches. Titles need work. And they don't show the feed yet. 2017-03-03 18:12:27 -05:00
Samuel Clay
00695b04af Merge branch 'master' into saved_searches
* master: (771 commits)
  Fixing horrible session loss bug. Turns out the django-redis-sessions library changed timeout behavior, from silently retrying to silently failing. Adding necessary SESSION_* vars.
  Better handling of booleans on /reader/feeds.
  Fixing redis bootstrap
  Add Firefox to story browser list
  Dashboard should not show old stories.
  Faking user agent when rss bans NewsBlur.
  Better recording of non-200/300 status codes on feed fetch.
  Adding new debug to feed fetcher.
  Embedding kickstarter module for TT on dashboard.
  notifications alpha 2
  Updating TT launch email.
  Linking image on TT email.
  No sudo on virtualenv.
  Adding missing feeds for users as part of cleanup.
  Fixing spacing issues in Turn Touch email.
  Updating Turn Touch email.
  Upgrading Readability and forcing images to remain. THis should add a bunch of images back to the Text view.
  Andorid v5.0.
  Fixing bug that would clear river cache while reading the river, caused by the dashboard river reloading when it wasn't visible.
  Revert "Moving from Whitney to Whitney ScreenSmart."
  ...
2017-02-24 17:05:51 -08:00
Samuel Clay
7cb2213509 Ignoring feedsportal. 2017-02-24 16:46:58 -08:00
Samuel Clay
937a1fa999 Faking user agent when rss bans NewsBlur. 2017-02-17 12:19:55 -08:00
Samuel Clay
255b30f3c1 Better recording of non-200/300 status codes on feed fetch. 2017-02-17 12:05:27 -08:00
Samuel Clay
2954ef7318 Adding new debug to feed fetcher. 2017-02-17 11:45:43 -08:00
Samuel Clay
a7fcd82d6d Handling missing feed urls. 2017-01-08 18:14:48 -08:00
Samuel Clay
1f8f1db9e9 Simplifying the conditional formatting. 2017-01-08 18:13:00 -08:00
Samuel Clay
9e69db2590 Fixing broken feed urls on keyword popularity xls. 2017-01-08 17:54:51 -08:00
Samuel Clay
0f2c90ee2a Can't believe .write_column() is a thing. 2017-01-05 19:20:03 -08:00
Samuel Clay
72ffa295e8 Cleaning up popularity query. 2017-01-05 19:13:55 -08:00
Samuel Clay
ea713d011a Adding comments to popularity query xlsx. 2017-01-05 19:06:45 -08:00
Samuel Clay
1523f115fe Turning off feed stats that are unecessarily slowing down collection. 2017-01-05 18:39:01 -08:00
Samuel Clay
0235cb38b0 Adding popularity query email. 2017-01-05 18:26:50 -08:00
Samuel Clay
52a76e7811 Updating popularity xls to include share counts and classifiers. Refactored xls creation to better handle new columns. 2017-01-05 16:44:19 -08:00
Samuel Clay
c5d5ea5001 Attempting to fix dashboard stories by no longer relying on full river reloads when a new story comes in but to fetch that story if it is newer (or older) than last visible dashboard story. 2016-12-13 16:29:42 -08:00
Samuel Clay
3a93ad7248 Fixing original page server. 2016-11-30 17:17:40 -08:00
Samuel Clay
a40ebc3940 Adding linebreaks back into iOS story notifications. 2016-11-20 14:18:13 -08:00
Samuel Clay
56b3dfa659 More aggregation changes. 2016-11-20 13:11:56 -08:00
Samuel Clay
8f6bfa1c55 Notifications now support hidden stories and focus stories. Web also all done. Just need to build iOS push and iOS notification chooser. 2016-11-16 17:49:43 -08:00
Samuel Clay
0284793b3a Adding in push notifications for web. Needs filtering. 2016-11-15 20:45:59 -08:00
Samuel Clay
1be440c407 DynamicDocument allows starred stories to work with stories that have more fields. 2016-11-10 18:02:34 -08:00
Samuel Clay
afb93fd26d Turning off strict feed validation for mongoengine. 2016-11-10 17:56:08 -08:00