682 commits

Author	SHA1	Message	Date
Samuel Clay	2ccd7b6cfe	Fixing race condition between primary adn secondaries when creating feed fetch histories.	2013-04-15 15:09:04 -07:00
Samuel Clay	abbc05660f	Reversing order for fetch histories.	2013-04-15 14:38:43 -07:00
Samuel Clay	6011e4f2da	Moving from feed/page/push histories to capped fetch history.	2013-04-15 14:30:31 -07:00
Samuel Clay	1b482a78c6	Beginning big redis migration from story objectid to story guid hash.	2013-04-13 22:31:05 -07:00
Samuel Clay	13616665c1	Dumb, dumb bug causing division by zero errors because the autocomplete boost was 0.	2013-04-13 19:44:43 -07:00
Samuel Clay	3eaad16adf	Upping feed fetches.	2013-04-12 17:41:18 -07:00
Samuel Clay	bd2ad514ea	Further pushing up feed fetches. Let the good times roll.	2013-04-11 16:57:10 -07:00
Samuel Clay	4ee659a1a4	Upping feed fetching (4->3, still need 3->2 and 2->1) before I crank it.	2013-04-11 13:41:44 -07:00
Samuel Clay	ccb6d38fa3	Adding support for automatically converting google alerts.	2013-04-10 19:14:19 -07:00
Samuel Clay	20c089175b	Rejiggering autocomplete to only include > 10 subs.	2013-04-10 11:23:03 -07:00
Samuel Clay	47c2257ef7	Using regular expressions for comment stripping, unless the feed has an error count, in which case switch to lxml.	2013-04-08 16:14:48 -07:00
Samuel Clay	33e605875e	Switch to lxml for comment stripping.	2013-04-08 15:38:23 -07:00
Samuel Clay	9565c27784	Moving to 1 feed per task queue (from 12).	2013-04-08 14:39:00 -07:00
Samuel Clay	83c8443ed0	Prioritizing autocomplete matches on title before address.	2013-04-08 13:12:11 -07:00
Samuel Clay	c25c4478e6	Adding new autocomplete. Let's hope this doesn't destroy performance in Redis.	2013-04-08 12:54:02 -07:00
Samuel Clay	46f919f919	Using error_feeds to push broken feeds further out in the pipeline.	2013-04-08 10:50:50 -07:00
Samuel Clay	d63556bc06	Skipping tasking of feeds when queue is too high. Use the random queue.	2013-04-08 09:49:21 -07:00
Samuel Clay	00591ee7ad	Pushing out feed updates to cope with influx.	2013-04-08 08:34:12 -07:00
Samuel Clay	5c0b9de511	Increasing fetches for long tail feeds.	2013-04-07 17:31:43 -07:00
Samuel Clay	48c5d4d53e	Adding logging, feed cleanup to forced fetch feeds, and expirations on user read stories.	2013-04-07 17:19:59 -07:00
Samuel Clay	d238dafba7	Bumping up feed fetching for premiums.	2013-04-05 17:59:33 -07:00
Samuel Clay	9c79076aab	Better verbose logging for updating feeds.	2013-04-05 12:09:42 -07:00
Samuel Clay	1ba468d27d	Checking if feed id has changed, then removing from task queue.	2013-04-04 21:57:58 -07:00
Samuel Clay	d831088a46	Fixed the ridiculous bug with feed queuing: while checking to see if the feed id changed, self.pk changed along with the feed, resulting in the cached value changing!	2013-04-04 20:51:56 -07:00
Samuel Clay	341db243c3	Better logging for queues.	2013-04-04 20:47:41 -07:00
Samuel Clay	681968033b	Fixing broken JPG image sizer.	2013-04-04 17:49:38 -07:00
Samuel Clay	ee647be85b	Handling timeouts for feed link change.	2013-04-04 17:27:31 -07:00
Samuel Clay	3d190cc3ec	Fixing a few small feed collision errors.	2013-04-04 17:18:27 -07:00
Samuel Clay	e7ad04946d	Fixing feed address change by feed link when colliding with existing feed.	2013-04-04 17:13:06 -07:00
Samuel Clay	8a7ea248fd	Fixing feed address change by feed link when colliding with existing feed.	2013-04-04 17:09:16 -07:00
Samuel Clay	95f97a4718	Adding logging.	2013-04-03 22:51:03 -07:00
Samuel Clay	400cad89e6	Tossing deleted feeds from queue.	2013-04-03 21:11:40 -07:00
Samuel Clay	3627221968	Tossing deleted feeds from queue.	2013-04-03 21:10:37 -07:00
Samuel Clay	4651906f08	What the f? strip_comments now eats CPU on bad stories. But why now and not a few months ago when I introduced it?	2013-04-03 18:20:42 -07:00
Samuel Clay	7a41b6a949	Using proper hostname for task server. Adding verbosity to feed fetching.	2013-04-03 18:11:23 -07:00
Samuel Clay	8f8c88ca47	Adding tasked_feeds set to handle dropped feeds.	2013-04-03 17:22:45 -07:00
Samuel Clay	842afe7068	Handling decayed feeds with no more active subscribers. They don't get returned to the scheduled_updates queue until there are enough active users.	2013-04-01 15:45:22 -07:00
Samuel Clay	ba5919d760	Moving task queue from postgres db backed to redis backed.	2013-03-30 19:05:13 -07:00
Samuel Clay	2cc5ab6202	Boosting feed fetching.	2013-03-30 11:39:49 -07:00
Samuel Clay	03c27b42e8	Removing unused feed hash index (keeping the unique constraint).	2013-03-29 22:41:16 -07:00
Samuel Clay	33bd64f2a0	Adding two indexes and killing one unused one from feeds.	2013-03-29 22:11:36 -07:00
Samuel Clay	93794aef36	Returning index on active subs.	2013-03-29 21:48:38 -07:00
Samuel Clay	22b2bd0604	Changing logging on feed queueing.	2013-03-29 21:46:04 -07:00
Samuel Clay	a47fa4db0f	Removing unused indexes.	2013-03-29 20:24:06 -07:00
Samuel Clay	07ebbd63b5	Boosting feed fetching way up for the weekend.	2013-03-29 15:56:43 -07:00
Samuel Clay	aa4bf71b34	Upping feed fetching by quite a bit.	2013-03-29 14:14:29 -07:00
Samuel Clay	fc53550392	Making a clearer description of slow feeds.	2013-03-28 12:17:30 -07:00
Samuel Clay	e39dee98be	Upping feed fetches, slowing down fetch interval for less active feeds, setting premium expire time to one year from most recent payment date (as opposed to borked payment gap logic), and adding logginf or tasking feeds.	2013-03-28 11:16:43 -07:00
Samuel Clay	79200d7bc1	Adding RStats, necessary to remove mongodb analytics.	2013-03-25 11:10:41 -07:00
Samuel Clay	e91d11e068	Adding cached template loader. Adding new mongodb.	2013-03-22 13:36:32 -07:00