80 commits

Author	SHA1	Message	Date
Samuel Clay	18d750091d	Shortening task error restart to 10m from 1h.	2013-05-24 13:18:05 -07:00
Samuel Clay	f6f053b285	Adding duplicate feed address+link truncation.	2013-05-15 15:40:38 -07:00
Samuel Clay	6a3d043ced	Upping to 5000 feeds per minute.	2013-04-23 21:17:03 -07:00
Samuel Clay	45e137b22c	Upping feed fetches.	2013-04-15 19:26:41 -07:00
Samuel Clay	528c86a1eb	Fixing small bug around missing feeds during update pipeline.	2013-04-15 15:16:53 -07:00
Samuel Clay	a6837724e3	Increasing task queue size.	2013-04-08 17:11:40 -07:00
Samuel Clay	46f919f919	Using error_feeds to push broken feeds further out in the pipeline.	2013-04-08 10:50:50 -07:00
Samuel Clay	7fc9b494af	Adding error_feeds queue to keep track of broken feeds.	2013-04-08 10:04:40 -07:00
Samuel Clay	d63556bc06	Skipping tasking of feeds when queue is too high. Use the random queue.	2013-04-08 09:49:21 -07:00
Samuel Clay	c41bf80b4e	Removing force from new feeds.	2013-04-07 20:22:46 -07:00
Samuel Clay	48c5d4d53e	Adding logging, feed cleanup to forced fetch feeds, and expirations on user read stories.	2013-04-07 17:19:59 -07:00
Samuel Clay	1ba468d27d	Checking if feed id has changed, then removing from task queue.	2013-04-04 21:57:58 -07:00
Samuel Clay	341db243c3	Better logging for queues.	2013-04-04 20:47:41 -07:00
Samuel Clay	a36a960566	Bumping numbers on task fetching.	2013-04-03 22:59:25 -07:00
Samuel Clay	6b262f8557	Cleaning up counts in munin and work log.	2013-04-03 19:04:30 -07:00
Samuel Clay	4461bb9c99	Removing old inactive feeds refresher.	2013-04-03 17:31:37 -07:00
Samuel Clay	6a2a144747	Requeueing old tasked feeds.	2013-04-03 17:28:51 -07:00
Samuel Clay	8f8c88ca47	Adding tasked_feeds set to handle dropped feeds.	2013-04-03 17:22:45 -07:00
Samuel Clay	e0457bf0d0	Fixing missing feed bug.	2013-04-03 16:48:42 -07:00
Samuel Clay	ba5919d760	Moving task queue from postgres db backed to redis backed.	2013-03-30 19:05:13 -07:00
Samuel Clay	2cc5ab6202	Boosting feed fetching.	2013-03-30 11:39:49 -07:00
Samuel Clay	03c27b42e8	Removing unused feed hash index (keeping the unique constraint).	2013-03-29 22:41:16 -07:00
Samuel Clay	22b2bd0604	Changing logging on feed queueing.	2013-03-29 21:46:04 -07:00
Samuel Clay	ae953ee208	Cleaning up week old feeds.	2013-03-29 16:02:32 -07:00
Samuel Clay	07ebbd63b5	Boosting feed fetching way up for the weekend.	2013-03-29 15:56:43 -07:00
Samuel Clay	aa4bf71b34	Upping feed fetching by quite a bit.	2013-03-29 14:14:29 -07:00
Samuel Clay	e39dee98be	Upping feed fetches, slowing down fetch interval for less active feeds, setting premium expire time to one year from most recent payment date (as opposed to borked payment gap logic), and adding logginf or tasking feeds.	2013-03-28 11:16:43 -07:00
Samuel Clay	c9f6012a1e	Upping feed fetches.	2013-03-27 21:12:55 -07:00
Samuel Clay	2acb9e1940	Boosting fetches.	2013-03-27 12:58:24 -07:00
Samuel Clay	2a5bd136da	Boosting feed fetches.	2013-03-25 13:15:14 -07:00
Samuel Clay	86a64fa74e	Boosting feed fetches.	2013-03-22 15:10:21 -07:00
Samuel Clay	c4b6bb6fd8	Cutting number of feeds for fetching until mongo is sharded.	2013-03-21 21:15:18 -07:00
Samuel Clay	5e6b1e5fee	Grabbing forced refresh feeds.	2013-03-21 10:57:57 -07:00
Samuel Clay	ca43ea8555	Further pushing down feed fetches to handle load until db is scaled out.	2013-03-20 08:13:07 -07:00
Samuel Clay	7ad6c594a3	Turning down feed fetches.	2013-03-19 17:57:24 -07:00
Samuel Clay	0db41bc853	Further turning down feed fetches until mongo is ready to replicate.	2013-03-19 17:38:36 -07:00
Samuel Clay	db92d3ca10	Ramping down feed fetching until they can be handled.	2013-03-19 15:32:25 -07:00
Samuel Clay	80fe5666f2	Fetching more feeds.	2013-03-14 21:33:06 -07:00
Samuel Clay	91c07a89e7	Bumping up premium refresh rates.	2013-03-14 16:15:21 -07:00
Samuel Clay	762b8aceb5	Tasking only popular feeds for now.	2013-03-14 16:01:28 -07:00
Samuel Clay	ffe07db5c8	Only 5k feeds for minute.	2013-03-14 12:59:28 -07:00
Samuel Clay	9052636418	Moving task feeds to its own beat queue.	2013-03-14 12:55:15 -07:00
Samuel Clay	2dd4a84c1a	Only fetching premium feeds.	2013-03-14 11:48:09 -07:00
Samuel Clay	3bdb4af70a	Fixing the timeout on premium setups by doing the scheduling of premium setups in a task, only immediately activating all of a user's feeds.	2013-02-15 09:52:11 -08:00
Samuel Clay	5ed2109bcf	Using queue for scheduling feed fetches on load.	2013-01-02 12:27:08 -08:00
Samuel Clay	e80d5fdcae	Preventing the double tasking fo feeds.	2012-12-25 15:11:15 -08:00
Samuel Clay	66184506e9	Preventing the double tasking fo feeds.	2012-12-25 15:09:46 -08:00
Samuel Clay	62d210ffd6	Migration to increase max length of feed address.	2012-12-25 14:58:59 -08:00
Samuel Clay	f94939b83e	Cleaning up task feeds logging.	2012-12-25 12:08:17 -08:00
Samuel Clay	8d158676b3	Moving statistics tasks.	2012-12-10 13:33:57 -08:00