Commit graph

444 commits

Author SHA1 Message Date
Samuel Clay
8795c46204 Wrong feed_id 2016-05-20 16:15:18 -07:00
Samuel Clay
679e2ddfd3 Using get_by_id when refreshing feed. 2016-05-20 16:13:51 -07:00
Samuel Clay
c1d1534d67 Sanitizing text/plain stories. 2016-05-11 12:01:27 -07:00
Samuel Clay
594e677357 Fix somewhat long-standing error logging bug that spits out a ton of data on feeds that don't work with requests lib. 2016-03-18 18:30:01 -07:00
Samuel Clay
f901f583a3 Sanity check for broken mtime. 2016-03-08 13:33:16 -08:00
Samuel Clay
586ff3f572 Fixing unicode issues in feed fetcher. Also fixing broken last-modified cache header. 2016-03-08 13:26:34 -08:00
Samuel Clay
3c7717aff5 Updating newsletter icon. 2016-02-09 16:59:58 -08:00
Samuel Clay
39b29bfd27 Splitting feed error codes. 2016-02-09 16:22:34 -08:00
Samuel Clay
3b3d76dcd2 Relative URI handling in feedparser changes when it doesn't have the URL of the feed. Thank goodness I caught this early, otherwise a bunch of sites' images would have been mysteriously broken. 2016-02-05 18:43:08 -08:00
Samuel Clay
2d6343b7ae Fixing feedfinder beautifulsoup issues, unicode issues in feed, and unicode comparisons issues with page_data (since it wasnt being decompressed when compared). 2016-02-05 14:43:31 -08:00
Samuel Clay
19902ec7bf Attempting to fix SNI issues by using new feedfinder and using requests instead of feedparser. This needs virtualenv, stat. 2016-02-04 12:28:55 -08:00
Samuel Clay
d88ebec119 Handling missing page links. 2016-02-03 13:13:31 -08:00
Samuel Clay
92f36821a9 Typo in feed parser log entry. 2016-02-03 12:55:39 -08:00
Samuel Clay
7c4410226f Some feeds are really broken. 2016-02-03 12:54:34 -08:00
Samuel Clay
3134866414 Fixing address issue when updating page link. 2016-02-03 12:38:18 -08:00
Samuel Clay
0bf0bb02e2 Only remove cache busting query parameter when added. 2016-02-03 12:29:16 -08:00
Samuel Clay
4bda8ff0f0 Adding qurl library to better add and remove cache buster while maintain url query string order. 2016-02-03 12:18:35 -08:00
Samuel Clay
ee2573ee75 Cleaning cache buster from feed and page urls. 2016-02-03 12:11:22 -08:00
Samuel Clay
66aadc45a1 Adding soft time limit to celery tasks. 2016-02-01 23:06:04 -08:00
Samuel Clay
988bcd7aee Fixing redirect behavior by saving redirect history for 25 fetches when one 400+ error is found. 2016-01-13 12:42:39 -08:00
Samuel Clay
81bc426eeb Better handling youtube feed errors. 2016-01-07 20:04:29 -08:00
Slavik Svyrydiuk
753d5fc018 use django.utils.encoding.smart_unicode instead of utils.feed_functions.utf8encode 2015-12-02 06:46:18 +01:00
Samuel Clay
101e8477e0 Handling feeds that go missing. 2015-11-28 13:54:21 -08:00
Samuel Clay
f8775f8d54 Handling more errors when fetching feeds. 2015-11-28 13:48:20 -08:00
Samuel Clay
c72d0d4c06 Turning off annotations on Youtube videos by default. 2015-10-23 15:58:55 -07:00
Samuel Clay
3d92dfd0ad Handling YouTube client errors gracefully. 2015-08-24 14:51:01 -07:00
Samuel Clay
a774cddcb3 Fixing broken sub counts when reids sub count expires. 2015-08-05 13:59:26 -07:00
Samuel Clay
9cd8124419 Combing through all updates to feeds table when fetching feed. 2015-07-22 13:53:20 -07:00
Samuel Clay
da26ecdb46 1/2 Cleaning up feed fetcher's updates to feeds table. Still need to find 6 dupes. 2015-07-22 13:18:54 -07:00
Samuel Clay
45a6243bfd Handling youtube feed brokenness. 2015-07-07 13:40:22 -07:00
Samuel Clay
e446103392 Handling broken youtube feeds. 2015-07-07 13:37:07 -07:00
Samuel Clay
e6fef18976 Fixing redirect history saving. 2015-06-01 18:06:39 -07:00
Samuel Clay
2817181f33 Forcing all 301 redirects to occur 20 times before committing, saving feeds from disappearing. 2015-06-01 17:14:04 -07:00
Samuel Clay
29ca355ef3 Fixing list_id 2015-05-12 08:41:45 -07:00
Samuel Clay
fb4ac3757b Fixing YouTube playlist urls. 2015-05-11 17:49:40 -07:00
Samuel Clay
d22d81859a Adding duration and channel info to youtube videos. 2015-05-08 15:34:35 -07:00
Samuel Clay
23fd3cccc1 Adding channel_id youtube.com video feeds. 2015-05-08 14:50:44 -07:00
Samuel Clay
93cf7e5baf Allowing fullscreen videos for youtube. 2015-05-08 14:17:48 -07:00
Samuel Clay
5eb46a278e YouTube API doesn't always close the iframe tag. Better close it. 2015-04-29 18:07:00 -07:00
Samuel Clay
d7daf21694 Adding another youtube url style: https://www.youtube.com/feeds/videos.xml?user=SciShow 2015-04-29 17:58:44 -07:00
Samuel Clay
bb100fa0d8 Typo in youtube fetch error handler. 2015-04-29 17:50:05 -07:00
Samuel Clay
d7f6fa3928 Fixing two small bugs with youtube fetch. 2015-04-29 17:49:04 -07:00
Samuel Clay
dabd81608d Fixing three youtube issues: broken usernames/urls (non-uploads), 404 pages, and over-expanding content due to fixvid. 2015-04-29 17:16:15 -07:00
Samuel Clay
bdd12cf2ee Allowing both youtube/feeds/base and youtube/feeds/api. 2015-04-29 16:59:05 -07:00
Samuel Clay
367282c934 Fixing expanding youtube iframe. 2015-04-29 16:55:51 -07:00
Samuel Clay
f78db7e17e Using YouTube's dumb permalink. 2015-04-29 16:43:48 -07:00
Samuel Clay
83c7517b35 Changing permalink for youtube videos to original rss permalink. 2015-04-29 16:40:12 -07:00
Samuel Clay
1656762299 Trying new YouTube v3 API feed fetcher. 2015-04-29 16:30:44 -07:00
Samuel Clay
0da661c71b Fixing A&L Daily bug where stories all have the same GUID and permalink. Using permalink or timestamp, whichever is unique across the feed. 2015-02-19 10:39:10 -08:00
Samuel Clay
870f21f51f Boosting time limit on feed fetches. 2014-11-18 13:37:55 -08:00