Commit graph

1452 commits

Author SHA1 Message Date
Samuel Clay
2c6c3cc9cf Fixing missing years in stats. 2015-09-02 14:58:58 -07:00
Samuel Clay
b6e1e03bd5 Upgrading feedparser, cleaing up logging on feed fetch. 2015-08-24 14:27:01 -07:00
Samuel Clay
622d0814d6 Attempting a fix for feed url changes not sticking. 2015-08-12 14:43:17 -07:00
Samuel Clay
e82b1974a1 Logging sub counts before expired recount. 2015-08-05 14:00:58 -07:00
Samuel Clay
a774cddcb3 Fixing broken sub counts when reids sub count expires. 2015-08-05 13:59:26 -07:00
Samuel Clay
bebccafb3c Easing subscriber expirations for now. 2015-08-05 13:16:15 -07:00
Samuel Clay
793a560958 Adding story deletion logging. 2015-08-05 13:13:01 -07:00
Samuel Clay
c227a68be3 Adding story deletion logging. 2015-08-05 13:12:15 -07:00
Samuel Clay
f02128afdf Adding story deletion logging. 2015-08-05 13:11:53 -07:00
Samuel Clay
43d1339ef0 Fixing broken story url. 2015-08-03 20:12:51 -07:00
Samuel Clay
22ae0e65e4 Removing gamespot.com feed from page and text fetchers. 2015-08-03 20:09:36 -07:00
Samuel Clay
fdd76f6c9f Logging expired sub counts. 2015-07-30 13:55:45 -07:00
Samuel Clay
e285c95145 Only clearing recount scores if they exists. 2015-07-30 13:52:56 -07:00
Samuel Clay
40a1692a09 Further refining feed search. Upgrading to elasticsearch 1.7.1. 2015-07-30 11:51:21 -07:00
Samuel Clay
62b607ebfc Correcting discrepency of active subs when reading cache or having just recounted. 2015-07-29 17:01:16 -07:00
Samuel Clay
1e56d4f3a1 Adjust feed schedule duration. Looks like the bug has to do with exception feeds not being set correctly. Better fix that next. 2015-07-29 16:34:48 -07:00
Samuel Clay
e602b52a5d Typo. 2015-07-29 12:08:48 -07:00
Samuel Clay
89db8a5b9f Easing off of forcing updates when not counted in redis. 2015-07-29 12:07:22 -07:00
Samuel Clay
9b89a07fe1 Adding cleanup sub counts for users on cleanup. Also enabling redis sub store for all feeds. 2015-07-29 11:44:39 -07:00
Samuel Clay
98ac28d8f8 Adding random factor in to convert feeds over to redis cache. 2015-07-28 20:58:56 -07:00
Samuel Clay
f399604369 Handling only case where a recount isn't actually necessary. Still need to add in automatic feed conversion to redis subscriber cache. 2015-07-28 20:48:51 -07:00
Samuel Clay
a81864d8cd Fixing counts when taking into account muted feeds. 2015-07-28 20:46:30 -07:00
Samuel Clay
ce45ed8d6c Cleaning up pipeline. 2015-07-28 19:19:58 -07:00
Samuel Clay
cf43747b20 Adding user sub counts to redis, using new redis cache to count subscribers only when feed is entirely cached. Still need automatic conversion from postgres to redis sub counts. 2015-07-28 18:46:37 -07:00
Samuel Clay
6a20478a7f Renaming redis feed pool to redis feed update pool in prep for redis feed sub pool. 2015-07-27 18:35:25 -07:00
Samuel Clay
97b0910e67 Breaking up tasking of normal feeds and broken feeds so broken feeds are a less drain on postgresql. 2015-07-23 16:29:47 -07:00
Samuel Clay
487b79674c Simplifying common call to update next scheduled update. 2015-07-22 15:08:15 -07:00
Samuel Clay
9cd8124419 Combing through all updates to feeds table when fetching feed. 2015-07-22 13:53:20 -07:00
Samuel Clay
da26ecdb46 1/2 Cleaning up feed fetcher's updates to feeds table. Still need to find 6 dupes. 2015-07-22 13:18:54 -07:00
Samuel Clay
0df32112fb Adding ratelimit to stats. 2015-07-21 10:20:06 -07:00
Samuel Clay
48949f22a7 Deleting old shared stories by free users no longer using the service. 2015-07-09 18:03:30 -07:00
Samuel Clay
82fbccd1c2 Handling invalid urls for icons. 2015-07-07 13:35:00 -07:00
Samuel Clay
b27590506c Fixing notuniqueerrors. 2015-07-07 13:34:06 -07:00
Samuel Clay
94ce13183b Handling requests issue with too many redirects. 2015-07-07 13:30:46 -07:00
Samuel Clay
2817181f33 Forcing all 301 redirects to occur 20 times before committing, saving feeds from disappearing. 2015-06-01 17:14:04 -07:00
Samuel Clay
5b99ab01ec Adding error handling for new icon feed link fetcher. 2015-05-19 16:31:25 -07:00
Samuel Clay
e1f31b9277 Attempting to fetch Tumblr feed icons. 2015-05-18 17:51:23 -07:00
Samuel Clay
23fd3cccc1 Adding channel_id youtube.com video feeds. 2015-05-08 14:50:44 -07:00
Samuel Clay
d7f6fa3928 Fixing two small bugs with youtube fetch. 2015-04-29 17:49:04 -07:00
Samuel Clay
df21449e74 Adding support for entering html page for youtube, like: https://www.youtube.com/user/CookingChannel 2015-04-29 17:26:21 -07:00
Samuel Clay
84b6140faa Adding latitude to autocomplete blacklist. 2015-04-27 17:16:50 -07:00
Samuel Clay
d8a7a9c772 Correcting premium-preview next fetch dates in statistics. 2015-04-01 16:21:08 -07:00
Samuel Clay
5001ad7602 Handling timeout on original page fetch. 2015-03-11 11:50:46 -07:00
Samuel Clay
02c91bb8b0 Fixing timeout error on PUSH. Also fixing requests error on page importing. 2015-03-11 11:44:47 -07:00
Samuel Clay
46b78f70a9 Updating push logging message when skipping. 2015-02-23 13:31:42 -08:00
Samuel Clay
e351b6602b Chopping off milliseconds in push delta. 2015-02-19 14:21:08 -08:00
Samuel Clay
ed32cb5aed Chopping off milliseconds in push delta. 2015-02-19 14:20:08 -08:00
Samuel Clay
5fae9b6362 Logging last push date delta. 2015-02-19 13:41:09 -08:00
Samuel Clay
0da661c71b Fixing A&L Daily bug where stories all have the same GUID and permalink. Using permalink or timestamp, whichever is unique across the feed. 2015-02-19 10:39:10 -08:00
Samuel Clay
86a73d7c91 Deleting unstarred story as part of starred story cleanup. 2015-02-11 15:55:18 -08:00