59 commits

Author	SHA1	Message	Date
Samuel Clay	03c27b42e8	Removing unused feed hash index (keeping the unique constraint).	2013-03-29 22:41:16 -07:00
Samuel Clay	22b2bd0604	Changing logging on feed queueing.	2013-03-29 21:46:04 -07:00
Samuel Clay	ae953ee208	Cleaning up week old feeds.	2013-03-29 16:02:32 -07:00
Samuel Clay	07ebbd63b5	Boosting feed fetching way up for the weekend.	2013-03-29 15:56:43 -07:00
Samuel Clay	aa4bf71b34	Upping feed fetching by quite a bit.	2013-03-29 14:14:29 -07:00
Samuel Clay	e39dee98be	Upping feed fetches, slowing down fetch interval for less active feeds, setting premium expire time to one year from most recent payment date (as opposed to borked payment gap logic), and adding logginf or tasking feeds.	2013-03-28 11:16:43 -07:00
Samuel Clay	c9f6012a1e	Upping feed fetches.	2013-03-27 21:12:55 -07:00
Samuel Clay	2acb9e1940	Boosting fetches.	2013-03-27 12:58:24 -07:00
Samuel Clay	2a5bd136da	Boosting feed fetches.	2013-03-25 13:15:14 -07:00
Samuel Clay	86a64fa74e	Boosting feed fetches.	2013-03-22 15:10:21 -07:00
Samuel Clay	c4b6bb6fd8	Cutting number of feeds for fetching until mongo is sharded.	2013-03-21 21:15:18 -07:00
Samuel Clay	5e6b1e5fee	Grabbing forced refresh feeds.	2013-03-21 10:57:57 -07:00
Samuel Clay	ca43ea8555	Further pushing down feed fetches to handle load until db is scaled out.	2013-03-20 08:13:07 -07:00
Samuel Clay	7ad6c594a3	Turning down feed fetches.	2013-03-19 17:57:24 -07:00
Samuel Clay	0db41bc853	Further turning down feed fetches until mongo is ready to replicate.	2013-03-19 17:38:36 -07:00
Samuel Clay	db92d3ca10	Ramping down feed fetching until they can be handled.	2013-03-19 15:32:25 -07:00
Samuel Clay	80fe5666f2	Fetching more feeds.	2013-03-14 21:33:06 -07:00
Samuel Clay	91c07a89e7	Bumping up premium refresh rates.	2013-03-14 16:15:21 -07:00
Samuel Clay	762b8aceb5	Tasking only popular feeds for now.	2013-03-14 16:01:28 -07:00
Samuel Clay	ffe07db5c8	Only 5k feeds for minute.	2013-03-14 12:59:28 -07:00
Samuel Clay	9052636418	Moving task feeds to its own beat queue.	2013-03-14 12:55:15 -07:00
Samuel Clay	2dd4a84c1a	Only fetching premium feeds.	2013-03-14 11:48:09 -07:00
Samuel Clay	3bdb4af70a	Fixing the timeout on premium setups by doing the scheduling of premium setups in a task, only immediately activating all of a user's feeds.	2013-02-15 09:52:11 -08:00
Samuel Clay	5ed2109bcf	Using queue for scheduling feed fetches on load.	2013-01-02 12:27:08 -08:00
Samuel Clay	e80d5fdcae	Preventing the double tasking fo feeds.	2012-12-25 15:11:15 -08:00
Samuel Clay	66184506e9	Preventing the double tasking fo feeds.	2012-12-25 15:09:46 -08:00
Samuel Clay	62d210ffd6	Migration to increase max length of feed address.	2012-12-25 14:58:59 -08:00
Samuel Clay	f94939b83e	Cleaning up task feeds logging.	2012-12-25 12:08:17 -08:00
Samuel Clay	8d158676b3	Moving statistics tasks.	2012-12-10 13:33:57 -08:00
Samuel Clay	8f2dc95b18	S3 utils.	2012-12-10 11:40:35 -08:00
Samuel Clay	edc3a80667	Adding backup mongo task to be run by cron instead.	2012-12-10 11:37:35 -08:00
Samuel Clay	e572b3d53f	Moving to better feed finder (in the case of dupe feeds).	2012-10-25 16:14:25 -07:00
Samuel Clay	83ebd2391a	Turning down replica lag unread count give up time to 10 sec from 60s.	2012-10-23 09:30:14 -07:00
Samuel Clay	47434c2823	Trimming mongodb lag cutoff to force feeds to correct themselves when slave lag becomes an issue.	2012-09-13 13:54:40 -07:00
Samuel Clay	2fa00fd737	Forcing the feed fetcher to not count scores when there's replica lag. God this had better work.	2012-08-16 23:56:21 -07:00
Samuel Clay	f0dc76b71e	Merging separate task feeds tasks.	2012-08-09 21:18:45 -07:00
Samuel Clay	d722e9c4eb	Moving cronjob tasks over to celerybeat. Starting with task_feeds, collect_feedback, collect_stats, and mark_read for homepage freshening.	2012-07-16 23:40:17 -07:00
Samuel Clay	a74be1b604	Adding push feeds queue. Ignoring fat ping, since it might not match URLs.	2012-03-28 15:49:21 -07:00
Samuel Clay	5e981635d1	Small cleanup to parameters of feed update method.	2012-03-26 17:04:35 -07:00
Samuel Clay	1321ade17a	Adding hints, revising indexes on user stories, and trying to quicken feed fetching meta work.	2012-03-26 12:40:13 -07:00
Samuel Clay	ac639380c0	Using a probability from redis to determine whether or not to skip a fetch.	2012-02-27 16:41:34 -08:00
Samuel Clay	d901ccdae1	Switching to redis to feed queuing. Also adding a weight for skipping feed fetches under high load.	2012-02-26 22:33:06 -08:00
Samuel Clay	ce536200f6	Adding ability to wind down feed queue by faking requests.	2012-02-24 16:43:08 -08:00
Samuel Clay	f3d2f8e5b5	Moving from slave_db to ReplicaSetConnection in pymongo 2.1.	2011-12-11 10:54:07 -08:00
Samuel Clay	63e6b86d8e	Finishing up realtime for testing on prod.	2011-11-06 16:05:42 -08:00
Samuel Clay	040b755694	Reading stories from slave db during feed update.	2011-11-04 09:45:10 -07:00
Samuel Clay	8e2936fc1b	Utils -> vendor	2011-04-11 21:57:45 -04:00
Samuel Clay	b98c63dd14	Abstracting feed queueing so I can queue a queryset of feeds from the command line. Also adding in queueing of feeds that have not been updated in a week (for those that fall through the cracks.	2010-12-23 12:32:24 -05:00
Samuel Clay	3865cb77fa	Small visual tweaks as well as tightening up some logging.	2010-11-07 00:40:26 -04:00
Samuel Clay	9f4f27f4aa	Trimming 50% more stories. They're just unneeded.	2010-10-11 22:31:43 -04:00