Commit graph

495 commits

Author SHA1 Message Date
Samuel Clay
e3445bb134 Native Twitter support for converting the Twitter API to RSS. If you're reading this, you can test it by subscribing to twitter usernames like so: https://twitter.com/newsblur 2016-05-26 14:30:26 -07:00
Samuel Clay
8795c46204 Wrong feed_id 2016-05-20 16:15:18 -07:00
Samuel Clay
679e2ddfd3 Using get_by_id when refreshing feed. 2016-05-20 16:13:51 -07:00
Samuel Clay
c1d1534d67 Sanitizing text/plain stories. 2016-05-11 12:01:27 -07:00
Samuel Clay
594e677357 Fix somewhat long-standing error logging bug that spits out a ton of data on feeds that don't work with requests lib. 2016-03-18 18:30:01 -07:00
Samuel Clay
f901f583a3 Sanity check for broken mtime. 2016-03-08 13:33:16 -08:00
Samuel Clay
586ff3f572 Fixing unicode issues in feed fetcher. Also fixing broken last-modified cache header. 2016-03-08 13:26:34 -08:00
Samuel Clay
3c7717aff5 Updating newsletter icon. 2016-02-09 16:59:58 -08:00
Samuel Clay
39b29bfd27 Splitting feed error codes. 2016-02-09 16:22:34 -08:00
Samuel Clay
3b3d76dcd2 Relative URI handling in feedparser changes when it doesn't have the URL of the feed. Thank goodness I caught this early, otherwise a bunch of sites' images would have been mysteriously broken. 2016-02-05 18:43:08 -08:00
Samuel Clay
2d6343b7ae Fixing feedfinder beautifulsoup issues, unicode issues in feed, and unicode comparisons issues with page_data (since it wasnt being decompressed when compared). 2016-02-05 14:43:31 -08:00
Samuel Clay
19902ec7bf Attempting to fix SNI issues by using new feedfinder and using requests instead of feedparser. This needs virtualenv, stat. 2016-02-04 12:28:55 -08:00
Samuel Clay
d88ebec119 Handling missing page links. 2016-02-03 13:13:31 -08:00
Samuel Clay
92f36821a9 Typo in feed parser log entry. 2016-02-03 12:55:39 -08:00
Samuel Clay
7c4410226f Some feeds are really broken. 2016-02-03 12:54:34 -08:00
Samuel Clay
3134866414 Fixing address issue when updating page link. 2016-02-03 12:38:18 -08:00
Samuel Clay
0bf0bb02e2 Only remove cache busting query parameter when added. 2016-02-03 12:29:16 -08:00
Samuel Clay
4bda8ff0f0 Adding qurl library to better add and remove cache buster while maintain url query string order. 2016-02-03 12:18:35 -08:00
Samuel Clay
ee2573ee75 Cleaning cache buster from feed and page urls. 2016-02-03 12:11:22 -08:00
Samuel Clay
66aadc45a1 Adding soft time limit to celery tasks. 2016-02-01 23:06:04 -08:00
Samuel Clay
988bcd7aee Fixing redirect behavior by saving redirect history for 25 fetches when one 400+ error is found. 2016-01-13 12:42:39 -08:00
Samuel Clay
81bc426eeb Better handling youtube feed errors. 2016-01-07 20:04:29 -08:00
Slavik Svyrydiuk
753d5fc018 use django.utils.encoding.smart_unicode instead of utils.feed_functions.utf8encode 2015-12-02 06:46:18 +01:00
Samuel Clay
101e8477e0 Handling feeds that go missing. 2015-11-28 13:54:21 -08:00
Samuel Clay
f8775f8d54 Handling more errors when fetching feeds. 2015-11-28 13:48:20 -08:00
Samuel Clay
c72d0d4c06 Turning off annotations on Youtube videos by default. 2015-10-23 15:58:55 -07:00
Samuel Clay
3d92dfd0ad Handling YouTube client errors gracefully. 2015-08-24 14:51:01 -07:00
Samuel Clay
a774cddcb3 Fixing broken sub counts when reids sub count expires. 2015-08-05 13:59:26 -07:00
Samuel Clay
9cd8124419 Combing through all updates to feeds table when fetching feed. 2015-07-22 13:53:20 -07:00
Samuel Clay
da26ecdb46 1/2 Cleaning up feed fetcher's updates to feeds table. Still need to find 6 dupes. 2015-07-22 13:18:54 -07:00
Samuel Clay
45a6243bfd Handling youtube feed brokenness. 2015-07-07 13:40:22 -07:00
Samuel Clay
e446103392 Handling broken youtube feeds. 2015-07-07 13:37:07 -07:00
Samuel Clay
e6fef18976 Fixing redirect history saving. 2015-06-01 18:06:39 -07:00
Samuel Clay
2817181f33 Forcing all 301 redirects to occur 20 times before committing, saving feeds from disappearing. 2015-06-01 17:14:04 -07:00
Samuel Clay
29ca355ef3 Fixing list_id 2015-05-12 08:41:45 -07:00
Samuel Clay
fb4ac3757b Fixing YouTube playlist urls. 2015-05-11 17:49:40 -07:00
Samuel Clay
d22d81859a Adding duration and channel info to youtube videos. 2015-05-08 15:34:35 -07:00
Samuel Clay
23fd3cccc1 Adding channel_id youtube.com video feeds. 2015-05-08 14:50:44 -07:00
Samuel Clay
93cf7e5baf Allowing fullscreen videos for youtube. 2015-05-08 14:17:48 -07:00
Samuel Clay
5eb46a278e YouTube API doesn't always close the iframe tag. Better close it. 2015-04-29 18:07:00 -07:00
Samuel Clay
d7daf21694 Adding another youtube url style: https://www.youtube.com/feeds/videos.xml?user=SciShow 2015-04-29 17:58:44 -07:00
Samuel Clay
bb100fa0d8 Typo in youtube fetch error handler. 2015-04-29 17:50:05 -07:00
Samuel Clay
d7f6fa3928 Fixing two small bugs with youtube fetch. 2015-04-29 17:49:04 -07:00
Samuel Clay
dabd81608d Fixing three youtube issues: broken usernames/urls (non-uploads), 404 pages, and over-expanding content due to fixvid. 2015-04-29 17:16:15 -07:00
Samuel Clay
bdd12cf2ee Allowing both youtube/feeds/base and youtube/feeds/api. 2015-04-29 16:59:05 -07:00
Samuel Clay
367282c934 Fixing expanding youtube iframe. 2015-04-29 16:55:51 -07:00
Samuel Clay
f78db7e17e Using YouTube's dumb permalink. 2015-04-29 16:43:48 -07:00
Samuel Clay
83c7517b35 Changing permalink for youtube videos to original rss permalink. 2015-04-29 16:40:12 -07:00
Samuel Clay
1656762299 Trying new YouTube v3 API feed fetcher. 2015-04-29 16:30:44 -07:00
Samuel Clay
0da661c71b Fixing A&L Daily bug where stories all have the same GUID and permalink. Using permalink or timestamp, whichever is unique across the feed. 2015-02-19 10:39:10 -08:00