708 commits

Author	SHA1	Message	Date
Samuel Clay	f7f9565756	Correcting for feedburner stupidity.	2013-05-13 15:41:58 -07:00
Samuel Clay	777d5a5e9c	Removing Redis autocomplete. Upgrading elasticsearch.	2013-05-08 12:23:10 -07:00
Samuel Clay	365e0c2813	Using haystack and elasticsearch for a third autocomplete feature.	2013-05-08 12:23:09 -07:00
Samuel Clay	d91cc66178	Preventing infinite loop on merged dupe feeds.	2013-05-07 16:23:30 -07:00
Samuel Clay	5ceb0c45bf	Migrating comment feeds to non-comment feeds.	2013-05-07 15:37:36 -07:00
Samuel Clay	477d928d70	Disallowing branched feeds from feed search.	2013-05-07 11:37:01 -07:00
Samuel Clay	e27b7f5688	Removing sparse constraint on MStory's story_hash index. Goodbye old stories that didn't make the cut.	2013-05-04 19:03:02 -07:00
Samuel Clay	009010f879	Removing unused redis story objectid db.	2013-05-02 12:27:37 -07:00
Samuel Clay	49245ce9e3	Another bug found by @kyleniemeyer's story. a - b was supposed to be b - a.	2013-05-01 14:33:12 -07:00
Samuel Clay	38d01df4ab	Fixing Global shared stories by checking for missing story hashes properly.	2013-05-01 14:33:12 -07:00
Samuel Clay	ed1e71e407	Using index in msharedstory search.	2013-05-01 14:33:12 -07:00
Samuel Clay	c01d8e7ba9	Using new story_hashes instead of story object ids.	2013-05-01 14:33:12 -07:00
Samuel Clay	8d516fc493	Adding story_hash to starred stories, user stories, and shared stories.	2013-04-29 16:07:08 -07:00
Samuel Clay	fbe5f4422e	If a feed has no stories, don't choke on latest story.	2013-04-23 21:22:49 -07:00
Samuel Clay	1584208677	Adding metric for last story date, used to calculate next scheduled update.	2013-04-23 17:04:21 -07:00
Samuel Clay	9f8009632a	Correcting for premiums being counted twice.	2013-04-23 16:03:45 -07:00
Samuel Clay	0b6a5bd7ce	Use average stories per month in calc.	2013-04-23 15:58:02 -07:00
Samuel Clay	630029cfcb	Complete rewrite of feed update schedule.	2013-04-23 15:44:31 -07:00
Samuel Clay	2b991fd031	Forcing earlier fetch if stats calc is less than scheduled fetch.	2013-04-23 13:46:07 -07:00
Samuel Clay	e24a30bbed	Fixing small bug for feeds that have no feed link.	2013-04-19 11:09:31 -07:00
Samuel Clay	7468a8e305	Upping feed fetching.	2013-04-18 17:19:27 -07:00
Samuel Clay	5db1cebecf	Removing random interval in feed fetches.	2013-04-18 16:56:54 -07:00
Samuel Clay	2d1a81694a	No longer random intervals for dead feeds.	2013-04-18 16:47:55 -07:00
Samuel Clay	409b829d24	Fixing autocomplete errors when feed address is invalid.	2013-04-17 09:04:45 -07:00
Samuel Clay	cafab8c86b	Boosting the hell out of feed fetches.	2013-04-16 11:49:37 -07:00
Samuel Clay	45e137b22c	Upping feed fetches.	2013-04-15 19:26:41 -07:00
Samuel Clay	2ccd7b6cfe	Fixing race condition between primary adn secondaries when creating feed fetch histories.	2013-04-15 15:09:04 -07:00
Samuel Clay	abbc05660f	Reversing order for fetch histories.	2013-04-15 14:38:43 -07:00
Samuel Clay	6011e4f2da	Moving from feed/page/push histories to capped fetch history.	2013-04-15 14:30:31 -07:00
Samuel Clay	1b482a78c6	Beginning big redis migration from story objectid to story guid hash.	2013-04-13 22:31:05 -07:00
Samuel Clay	13616665c1	Dumb, dumb bug causing division by zero errors because the autocomplete boost was 0.	2013-04-13 19:44:43 -07:00
Samuel Clay	3eaad16adf	Upping feed fetches.	2013-04-12 17:41:18 -07:00
Samuel Clay	bd2ad514ea	Further pushing up feed fetches. Let the good times roll.	2013-04-11 16:57:10 -07:00
Samuel Clay	4ee659a1a4	Upping feed fetching (4->3, still need 3->2 and 2->1) before I crank it.	2013-04-11 13:41:44 -07:00
Samuel Clay	ccb6d38fa3	Adding support for automatically converting google alerts.	2013-04-10 19:14:19 -07:00
Samuel Clay	20c089175b	Rejiggering autocomplete to only include > 10 subs.	2013-04-10 11:23:03 -07:00
Samuel Clay	47c2257ef7	Using regular expressions for comment stripping, unless the feed has an error count, in which case switch to lxml.	2013-04-08 16:14:48 -07:00
Samuel Clay	33e605875e	Switch to lxml for comment stripping.	2013-04-08 15:38:23 -07:00
Samuel Clay	9565c27784	Moving to 1 feed per task queue (from 12).	2013-04-08 14:39:00 -07:00
Samuel Clay	83c8443ed0	Prioritizing autocomplete matches on title before address.	2013-04-08 13:12:11 -07:00
Samuel Clay	c25c4478e6	Adding new autocomplete. Let's hope this doesn't destroy performance in Redis.	2013-04-08 12:54:02 -07:00
Samuel Clay	46f919f919	Using error_feeds to push broken feeds further out in the pipeline.	2013-04-08 10:50:50 -07:00
Samuel Clay	d63556bc06	Skipping tasking of feeds when queue is too high. Use the random queue.	2013-04-08 09:49:21 -07:00
Samuel Clay	00591ee7ad	Pushing out feed updates to cope with influx.	2013-04-08 08:34:12 -07:00
Samuel Clay	5c0b9de511	Increasing fetches for long tail feeds.	2013-04-07 17:31:43 -07:00
Samuel Clay	48c5d4d53e	Adding logging, feed cleanup to forced fetch feeds, and expirations on user read stories.	2013-04-07 17:19:59 -07:00
Samuel Clay	d238dafba7	Bumping up feed fetching for premiums.	2013-04-05 17:59:33 -07:00
Samuel Clay	9c79076aab	Better verbose logging for updating feeds.	2013-04-05 12:09:42 -07:00
Samuel Clay	1ba468d27d	Checking if feed id has changed, then removing from task queue.	2013-04-04 21:57:58 -07:00
Samuel Clay	d831088a46	Fixed the ridiculous bug with feed queuing: while checking to see if the feed id changed, self.pk changed along with the feed, resulting in the cached value changing!	2013-04-04 20:51:56 -07:00