Commit graph

584 commits

Author SHA1 Message Date
Samuel Clay
19902ec7bf Attempting to fix SNI issues by using new feedfinder and using requests instead of feedparser. This needs virtualenv, stat. 2016-02-04 12:28:55 -08:00
Samuel Clay
d88ebec119 Handling missing page links. 2016-02-03 13:13:31 -08:00
Samuel Clay
92f36821a9 Typo in feed parser log entry. 2016-02-03 12:55:39 -08:00
Samuel Clay
7c4410226f Some feeds are really broken. 2016-02-03 12:54:34 -08:00
Samuel Clay
3134866414 Fixing address issue when updating page link. 2016-02-03 12:38:18 -08:00
Samuel Clay
0bf0bb02e2 Only remove cache busting query parameter when added. 2016-02-03 12:29:16 -08:00
Samuel Clay
4bda8ff0f0 Adding qurl library to better add and remove cache buster while maintain url query string order. 2016-02-03 12:18:35 -08:00
Samuel Clay
ee2573ee75 Cleaning cache buster from feed and page urls. 2016-02-03 12:11:22 -08:00
Samuel Clay
66aadc45a1 Adding soft time limit to celery tasks. 2016-02-01 23:06:04 -08:00
Samuel Clay
988bcd7aee Fixing redirect behavior by saving redirect history for 25 fetches when one 400+ error is found. 2016-01-13 12:42:39 -08:00
Samuel Clay
81bc426eeb Better handling youtube feed errors. 2016-01-07 20:04:29 -08:00
Slavik Svyrydiuk
753d5fc018 use django.utils.encoding.smart_unicode instead of utils.feed_functions.utf8encode 2015-12-02 06:46:18 +01:00
Samuel Clay
101e8477e0 Handling feeds that go missing. 2015-11-28 13:54:21 -08:00
Samuel Clay
f8775f8d54 Handling more errors when fetching feeds. 2015-11-28 13:48:20 -08:00
Samuel Clay
c72d0d4c06 Turning off annotations on Youtube videos by default. 2015-10-23 15:58:55 -07:00
Samuel Clay
3d92dfd0ad Handling YouTube client errors gracefully. 2015-08-24 14:51:01 -07:00
Samuel Clay
a774cddcb3 Fixing broken sub counts when reids sub count expires. 2015-08-05 13:59:26 -07:00
Samuel Clay
9cd8124419 Combing through all updates to feeds table when fetching feed. 2015-07-22 13:53:20 -07:00
Samuel Clay
da26ecdb46 1/2 Cleaning up feed fetcher's updates to feeds table. Still need to find 6 dupes. 2015-07-22 13:18:54 -07:00
Samuel Clay
45a6243bfd Handling youtube feed brokenness. 2015-07-07 13:40:22 -07:00
Samuel Clay
e446103392 Handling broken youtube feeds. 2015-07-07 13:37:07 -07:00
Samuel Clay
e6fef18976 Fixing redirect history saving. 2015-06-01 18:06:39 -07:00
Samuel Clay
2817181f33 Forcing all 301 redirects to occur 20 times before committing, saving feeds from disappearing. 2015-06-01 17:14:04 -07:00
Samuel Clay
29ca355ef3 Fixing list_id 2015-05-12 08:41:45 -07:00
Samuel Clay
fb4ac3757b Fixing YouTube playlist urls. 2015-05-11 17:49:40 -07:00
Samuel Clay
d22d81859a Adding duration and channel info to youtube videos. 2015-05-08 15:34:35 -07:00
Samuel Clay
23fd3cccc1 Adding channel_id youtube.com video feeds. 2015-05-08 14:50:44 -07:00
Samuel Clay
93cf7e5baf Allowing fullscreen videos for youtube. 2015-05-08 14:17:48 -07:00
Samuel Clay
5eb46a278e YouTube API doesn't always close the iframe tag. Better close it. 2015-04-29 18:07:00 -07:00
Samuel Clay
d7daf21694 Adding another youtube url style: https://www.youtube.com/feeds/videos.xml?user=SciShow 2015-04-29 17:58:44 -07:00
Samuel Clay
bb100fa0d8 Typo in youtube fetch error handler. 2015-04-29 17:50:05 -07:00
Samuel Clay
d7f6fa3928 Fixing two small bugs with youtube fetch. 2015-04-29 17:49:04 -07:00
Samuel Clay
dabd81608d Fixing three youtube issues: broken usernames/urls (non-uploads), 404 pages, and over-expanding content due to fixvid. 2015-04-29 17:16:15 -07:00
Samuel Clay
bdd12cf2ee Allowing both youtube/feeds/base and youtube/feeds/api. 2015-04-29 16:59:05 -07:00
Samuel Clay
367282c934 Fixing expanding youtube iframe. 2015-04-29 16:55:51 -07:00
Samuel Clay
f78db7e17e Using YouTube's dumb permalink. 2015-04-29 16:43:48 -07:00
Samuel Clay
83c7517b35 Changing permalink for youtube videos to original rss permalink. 2015-04-29 16:40:12 -07:00
Samuel Clay
1656762299 Trying new YouTube v3 API feed fetcher. 2015-04-29 16:30:44 -07:00
Samuel Clay
0da661c71b Fixing A&L Daily bug where stories all have the same GUID and permalink. Using permalink or timestamp, whichever is unique across the feed. 2015-02-19 10:39:10 -08:00
Samuel Clay
870f21f51f Boosting time limit on feed fetches. 2014-11-18 13:37:55 -08:00
Samuel Clay
0aaf661da1 Removing feedparser trunk. 2014-11-18 11:08:02 -08:00
Samuel Clay
c20ef4e5d0 Adding updates_off run-time setting for handling large spikes of updates. 2014-05-29 17:53:16 -07:00
Samuel Clay
0fee7af7c6 Moving to stricter story diff check. Using story hashes instead of grab bag by date when checking existing stories. 2014-04-17 12:10:04 -07:00
Samuel Clay
a4efcde236 Allowing feeds to fail fetch. 2014-03-29 17:14:10 -07:00
Samuel Clay
534e16fb84 Handling odd compressed stream error. 2014-03-13 16:53:20 -07:00
Samuel Clay
bb3e0a6567 Putting Google Alerts/News back into rotation. 2013-09-30 12:40:14 -07:00
Samuel Clay
70ea542481 User-dependent days of unread. 14 for free users, 30 for premium. 2013-09-16 16:42:49 -07:00
Samuel Clay
831a771309 Fixing logging message. 2013-09-16 12:11:26 -07:00
Samuel Clay
886f6c0d8b Fixing logging message. 2013-09-16 12:08:51 -07:00
Samuel Clay
5911acaa35 Moving the majority of story fetches to mongo secondaries. 2013-09-16 12:07:15 -07:00