49 commits

Author	SHA1	Message	Date
Samuel Clay	c4b6bb6fd8	Cutting number of feeds for fetching until mongo is sharded.	2013-03-21 21:15:18 -07:00
Samuel Clay	5e6b1e5fee	Grabbing forced refresh feeds.	2013-03-21 10:57:57 -07:00
Samuel Clay	ca43ea8555	Further pushing down feed fetches to handle load until db is scaled out.	2013-03-20 08:13:07 -07:00
Samuel Clay	7ad6c594a3	Turning down feed fetches.	2013-03-19 17:57:24 -07:00
Samuel Clay	0db41bc853	Further turning down feed fetches until mongo is ready to replicate.	2013-03-19 17:38:36 -07:00
Samuel Clay	db92d3ca10	Ramping down feed fetching until they can be handled.	2013-03-19 15:32:25 -07:00
Samuel Clay	80fe5666f2	Fetching more feeds.	2013-03-14 21:33:06 -07:00
Samuel Clay	91c07a89e7	Bumping up premium refresh rates.	2013-03-14 16:15:21 -07:00
Samuel Clay	762b8aceb5	Tasking only popular feeds for now.	2013-03-14 16:01:28 -07:00
Samuel Clay	ffe07db5c8	Only 5k feeds for minute.	2013-03-14 12:59:28 -07:00
Samuel Clay	9052636418	Moving task feeds to its own beat queue.	2013-03-14 12:55:15 -07:00
Samuel Clay	2dd4a84c1a	Only fetching premium feeds.	2013-03-14 11:48:09 -07:00
Samuel Clay	3bdb4af70a	Fixing the timeout on premium setups by doing the scheduling of premium setups in a task, only immediately activating all of a user's feeds.	2013-02-15 09:52:11 -08:00
Samuel Clay	5ed2109bcf	Using queue for scheduling feed fetches on load.	2013-01-02 12:27:08 -08:00
Samuel Clay	e80d5fdcae	Preventing the double tasking fo feeds.	2012-12-25 15:11:15 -08:00
Samuel Clay	66184506e9	Preventing the double tasking fo feeds.	2012-12-25 15:09:46 -08:00
Samuel Clay	62d210ffd6	Migration to increase max length of feed address.	2012-12-25 14:58:59 -08:00
Samuel Clay	f94939b83e	Cleaning up task feeds logging.	2012-12-25 12:08:17 -08:00
Samuel Clay	8d158676b3	Moving statistics tasks.	2012-12-10 13:33:57 -08:00
Samuel Clay	8f2dc95b18	S3 utils.	2012-12-10 11:40:35 -08:00
Samuel Clay	edc3a80667	Adding backup mongo task to be run by cron instead.	2012-12-10 11:37:35 -08:00
Samuel Clay	e572b3d53f	Moving to better feed finder (in the case of dupe feeds).	2012-10-25 16:14:25 -07:00
Samuel Clay	83ebd2391a	Turning down replica lag unread count give up time to 10 sec from 60s.	2012-10-23 09:30:14 -07:00
Samuel Clay	47434c2823	Trimming mongodb lag cutoff to force feeds to correct themselves when slave lag becomes an issue.	2012-09-13 13:54:40 -07:00
Samuel Clay	2fa00fd737	Forcing the feed fetcher to not count scores when there's replica lag. God this had better work.	2012-08-16 23:56:21 -07:00
Samuel Clay	f0dc76b71e	Merging separate task feeds tasks.	2012-08-09 21:18:45 -07:00
Samuel Clay	d722e9c4eb	Moving cronjob tasks over to celerybeat. Starting with task_feeds, collect_feedback, collect_stats, and mark_read for homepage freshening.	2012-07-16 23:40:17 -07:00
Samuel Clay	a74be1b604	Adding push feeds queue. Ignoring fat ping, since it might not match URLs.	2012-03-28 15:49:21 -07:00
Samuel Clay	5e981635d1	Small cleanup to parameters of feed update method.	2012-03-26 17:04:35 -07:00
Samuel Clay	1321ade17a	Adding hints, revising indexes on user stories, and trying to quicken feed fetching meta work.	2012-03-26 12:40:13 -07:00
Samuel Clay	ac639380c0	Using a probability from redis to determine whether or not to skip a fetch.	2012-02-27 16:41:34 -08:00
Samuel Clay	d901ccdae1	Switching to redis to feed queuing. Also adding a weight for skipping feed fetches under high load.	2012-02-26 22:33:06 -08:00
Samuel Clay	ce536200f6	Adding ability to wind down feed queue by faking requests.	2012-02-24 16:43:08 -08:00
Samuel Clay	f3d2f8e5b5	Moving from slave_db to ReplicaSetConnection in pymongo 2.1.	2011-12-11 10:54:07 -08:00
Samuel Clay	63e6b86d8e	Finishing up realtime for testing on prod.	2011-11-06 16:05:42 -08:00
Samuel Clay	040b755694	Reading stories from slave db during feed update.	2011-11-04 09:45:10 -07:00
Samuel Clay	8e2936fc1b	Utils -> vendor	2011-04-11 21:57:45 -04:00
Samuel Clay	b98c63dd14	Abstracting feed queueing so I can queue a queryset of feeds from the command line. Also adding in queueing of feeds that have not been updated in a week (for those that fall through the cracks.	2010-12-23 12:32:24 -05:00
Samuel Clay	3865cb77fa	Small visual tweaks as well as tightening up some logging.	2010-11-07 00:40:26 -04:00
Samuel Clay	9f4f27f4aa	Trimming 50% more stories. They're just unneeded.	2010-10-11 22:31:43 -04:00
Samuel Clay	50488db7ca	Close the connection for every set of tasks.	2010-10-08 16:08:41 -04:00
Samuel Clay	111459d644	Logging feed updates in broken task scheduler.	2010-10-08 14:34:41 -04:00
Samuel Clay	032505591d	Explicit queue management.	2010-09-28 05:16:00 -04:00
Samuel Clay	9d265e9db0	Adding a new queue that handles new feeds, so they can be prioritized over updating old feeds.	2010-09-20 19:22:19 -04:00
Samuel Clay	976e6c2c86	Merge branch 'master' of github.com:samuelclay/NewsBlur	2010-09-01 10:38:00 -04:00
Samuel Clay	ab3d052065	Using a list of feed_pks instead of one. Less task overhead.	2010-09-01 10:37:58 -04:00
Samuel Clay	c42cabcae2	Celery options for performance.	2010-09-01 08:19:58 -04:00
Samuel Clay	27209c3164	Defining the run task for celery's refresh feeds.	2010-08-31 18:18:40 -04:00
Samuel Clay	f906513657	Adding celery tasks, config, and djcelery app.	2010-08-31 18:07:47 -04:00