692 commits

Author	SHA1	Message	Date
Samuel Clay	0b6a5bd7ce	Use average stories per month in calc.	2013-04-23 15:58:02 -07:00
Samuel Clay	630029cfcb	Complete rewrite of feed update schedule.	2013-04-23 15:44:31 -07:00
Samuel Clay	2b991fd031	Forcing earlier fetch if stats calc is less than scheduled fetch.	2013-04-23 13:46:07 -07:00
Samuel Clay	e24a30bbed	Fixing small bug for feeds that have no feed link.	2013-04-19 11:09:31 -07:00
Samuel Clay	7468a8e305	Upping feed fetching.	2013-04-18 17:19:27 -07:00
Samuel Clay	5db1cebecf	Removing random interval in feed fetches.	2013-04-18 16:56:54 -07:00
Samuel Clay	2d1a81694a	No longer random intervals for dead feeds.	2013-04-18 16:47:55 -07:00
Samuel Clay	409b829d24	Fixing autocomplete errors when feed address is invalid.	2013-04-17 09:04:45 -07:00
Samuel Clay	cafab8c86b	Boosting the hell out of feed fetches.	2013-04-16 11:49:37 -07:00
Samuel Clay	45e137b22c	Upping feed fetches.	2013-04-15 19:26:41 -07:00
Samuel Clay	2ccd7b6cfe	Fixing race condition between primary adn secondaries when creating feed fetch histories.	2013-04-15 15:09:04 -07:00
Samuel Clay	abbc05660f	Reversing order for fetch histories.	2013-04-15 14:38:43 -07:00
Samuel Clay	6011e4f2da	Moving from feed/page/push histories to capped fetch history.	2013-04-15 14:30:31 -07:00
Samuel Clay	1b482a78c6	Beginning big redis migration from story objectid to story guid hash.	2013-04-13 22:31:05 -07:00
Samuel Clay	13616665c1	Dumb, dumb bug causing division by zero errors because the autocomplete boost was 0.	2013-04-13 19:44:43 -07:00
Samuel Clay	3eaad16adf	Upping feed fetches.	2013-04-12 17:41:18 -07:00
Samuel Clay	bd2ad514ea	Further pushing up feed fetches. Let the good times roll.	2013-04-11 16:57:10 -07:00
Samuel Clay	4ee659a1a4	Upping feed fetching (4->3, still need 3->2 and 2->1) before I crank it.	2013-04-11 13:41:44 -07:00
Samuel Clay	ccb6d38fa3	Adding support for automatically converting google alerts.	2013-04-10 19:14:19 -07:00
Samuel Clay	20c089175b	Rejiggering autocomplete to only include > 10 subs.	2013-04-10 11:23:03 -07:00
Samuel Clay	47c2257ef7	Using regular expressions for comment stripping, unless the feed has an error count, in which case switch to lxml.	2013-04-08 16:14:48 -07:00
Samuel Clay	33e605875e	Switch to lxml for comment stripping.	2013-04-08 15:38:23 -07:00
Samuel Clay	9565c27784	Moving to 1 feed per task queue (from 12).	2013-04-08 14:39:00 -07:00
Samuel Clay	83c8443ed0	Prioritizing autocomplete matches on title before address.	2013-04-08 13:12:11 -07:00
Samuel Clay	c25c4478e6	Adding new autocomplete. Let's hope this doesn't destroy performance in Redis.	2013-04-08 12:54:02 -07:00
Samuel Clay	46f919f919	Using error_feeds to push broken feeds further out in the pipeline.	2013-04-08 10:50:50 -07:00
Samuel Clay	d63556bc06	Skipping tasking of feeds when queue is too high. Use the random queue.	2013-04-08 09:49:21 -07:00
Samuel Clay	00591ee7ad	Pushing out feed updates to cope with influx.	2013-04-08 08:34:12 -07:00
Samuel Clay	5c0b9de511	Increasing fetches for long tail feeds.	2013-04-07 17:31:43 -07:00
Samuel Clay	48c5d4d53e	Adding logging, feed cleanup to forced fetch feeds, and expirations on user read stories.	2013-04-07 17:19:59 -07:00
Samuel Clay	d238dafba7	Bumping up feed fetching for premiums.	2013-04-05 17:59:33 -07:00
Samuel Clay	9c79076aab	Better verbose logging for updating feeds.	2013-04-05 12:09:42 -07:00
Samuel Clay	1ba468d27d	Checking if feed id has changed, then removing from task queue.	2013-04-04 21:57:58 -07:00
Samuel Clay	d831088a46	Fixed the ridiculous bug with feed queuing: while checking to see if the feed id changed, self.pk changed along with the feed, resulting in the cached value changing!	2013-04-04 20:51:56 -07:00
Samuel Clay	341db243c3	Better logging for queues.	2013-04-04 20:47:41 -07:00
Samuel Clay	681968033b	Fixing broken JPG image sizer.	2013-04-04 17:49:38 -07:00
Samuel Clay	ee647be85b	Handling timeouts for feed link change.	2013-04-04 17:27:31 -07:00
Samuel Clay	3d190cc3ec	Fixing a few small feed collision errors.	2013-04-04 17:18:27 -07:00
Samuel Clay	e7ad04946d	Fixing feed address change by feed link when colliding with existing feed.	2013-04-04 17:13:06 -07:00
Samuel Clay	8a7ea248fd	Fixing feed address change by feed link when colliding with existing feed.	2013-04-04 17:09:16 -07:00
Samuel Clay	95f97a4718	Adding logging.	2013-04-03 22:51:03 -07:00
Samuel Clay	400cad89e6	Tossing deleted feeds from queue.	2013-04-03 21:11:40 -07:00
Samuel Clay	3627221968	Tossing deleted feeds from queue.	2013-04-03 21:10:37 -07:00
Samuel Clay	4651906f08	What the f? strip_comments now eats CPU on bad stories. But why now and not a few months ago when I introduced it?	2013-04-03 18:20:42 -07:00
Samuel Clay	7a41b6a949	Using proper hostname for task server. Adding verbosity to feed fetching.	2013-04-03 18:11:23 -07:00
Samuel Clay	8f8c88ca47	Adding tasked_feeds set to handle dropped feeds.	2013-04-03 17:22:45 -07:00
Samuel Clay	842afe7068	Handling decayed feeds with no more active subscribers. They don't get returned to the scheduled_updates queue until there are enough active users.	2013-04-01 15:45:22 -07:00
Samuel Clay	ba5919d760	Moving task queue from postgres db backed to redis backed.	2013-03-30 19:05:13 -07:00
Samuel Clay	2cc5ab6202	Boosting feed fetching.	2013-03-30 11:39:49 -07:00
Samuel Clay	03c27b42e8	Removing unused feed hash index (keeping the unique constraint).	2013-03-29 22:41:16 -07:00