80 commits

Author	SHA1	Message	Date
Samuel Clay	8f2dc95b18	S3 utils.	2012-12-10 11:40:35 -08:00
Samuel Clay	edc3a80667	Adding backup mongo task to be run by cron instead.	2012-12-10 11:37:35 -08:00
Samuel Clay	e572b3d53f	Moving to better feed finder (in the case of dupe feeds).	2012-10-25 16:14:25 -07:00
Samuel Clay	83ebd2391a	Turning down replica lag unread count give up time to 10 sec from 60s.	2012-10-23 09:30:14 -07:00
Samuel Clay	47434c2823	Trimming mongodb lag cutoff to force feeds to correct themselves when slave lag becomes an issue.	2012-09-13 13:54:40 -07:00
Samuel Clay	2fa00fd737	Forcing the feed fetcher to not count scores when there's replica lag. God this had better work.	2012-08-16 23:56:21 -07:00
Samuel Clay	f0dc76b71e	Merging separate task feeds tasks.	2012-08-09 21:18:45 -07:00
Samuel Clay	d722e9c4eb	Moving cronjob tasks over to celerybeat. Starting with task_feeds, collect_feedback, collect_stats, and mark_read for homepage freshening.	2012-07-16 23:40:17 -07:00
Samuel Clay	a74be1b604	Adding push feeds queue. Ignoring fat ping, since it might not match URLs.	2012-03-28 15:49:21 -07:00
Samuel Clay	5e981635d1	Small cleanup to parameters of feed update method.	2012-03-26 17:04:35 -07:00
Samuel Clay	1321ade17a	Adding hints, revising indexes on user stories, and trying to quicken feed fetching meta work.	2012-03-26 12:40:13 -07:00
Samuel Clay	ac639380c0	Using a probability from redis to determine whether or not to skip a fetch.	2012-02-27 16:41:34 -08:00
Samuel Clay	d901ccdae1	Switching to redis to feed queuing. Also adding a weight for skipping feed fetches under high load.	2012-02-26 22:33:06 -08:00
Samuel Clay	ce536200f6	Adding ability to wind down feed queue by faking requests.	2012-02-24 16:43:08 -08:00
Samuel Clay	f3d2f8e5b5	Moving from slave_db to ReplicaSetConnection in pymongo 2.1.	2011-12-11 10:54:07 -08:00
Samuel Clay	63e6b86d8e	Finishing up realtime for testing on prod.	2011-11-06 16:05:42 -08:00
Samuel Clay	040b755694	Reading stories from slave db during feed update.	2011-11-04 09:45:10 -07:00
Samuel Clay	8e2936fc1b	Utils -> vendor	2011-04-11 21:57:45 -04:00
Samuel Clay	b98c63dd14	Abstracting feed queueing so I can queue a queryset of feeds from the command line. Also adding in queueing of feeds that have not been updated in a week (for those that fall through the cracks.	2010-12-23 12:32:24 -05:00
Samuel Clay	3865cb77fa	Small visual tweaks as well as tightening up some logging.	2010-11-07 00:40:26 -04:00
Samuel Clay	9f4f27f4aa	Trimming 50% more stories. They're just unneeded.	2010-10-11 22:31:43 -04:00
Samuel Clay	50488db7ca	Close the connection for every set of tasks.	2010-10-08 16:08:41 -04:00
Samuel Clay	111459d644	Logging feed updates in broken task scheduler.	2010-10-08 14:34:41 -04:00
Samuel Clay	032505591d	Explicit queue management.	2010-09-28 05:16:00 -04:00
Samuel Clay	9d265e9db0	Adding a new queue that handles new feeds, so they can be prioritized over updating old feeds.	2010-09-20 19:22:19 -04:00
Samuel Clay	976e6c2c86	Merge branch 'master' of github.com:samuelclay/NewsBlur	2010-09-01 10:38:00 -04:00
Samuel Clay	ab3d052065	Using a list of feed_pks instead of one. Less task overhead.	2010-09-01 10:37:58 -04:00
Samuel Clay	c42cabcae2	Celery options for performance.	2010-09-01 08:19:58 -04:00
Samuel Clay	27209c3164	Defining the run task for celery's refresh feeds.	2010-08-31 18:18:40 -04:00
Samuel Clay	f906513657	Adding celery tasks, config, and djcelery app.	2010-08-31 18:07:47 -04:00