21 commits

Author	SHA1	Message	Date
Samuel Clay	e26ee9dbcc	Moving feed_pages from PostgreSQL to Mongo. Adding compression.	2010-08-29 12:35:09 -04:00
Samuel Clay	44969c7eee	Upgrading feedparser.	2010-08-24 15:35:05 -04:00
Samuel Clay	f3522cf0ec	Adding logging in many other places.	2010-08-16 15:45:35 -04:00
Samuel Clay	fcca02717f	Silencing logs from the backgorund importer.	2010-08-15 12:09:40 -04:00
Samuel Clay	3df0722dcc	Moving from print-based logging to python-based logging. Quite the improvement. Console/Stream output in development, logfile in production.	2010-08-15 12:04:26 -04:00
Samuel Clay	2f86bbbb24	Fixing bad page URLs to use the feed fetcher to find the correct address.	2010-07-21 12:29:09 -04:00
Samuel Clay	2c0b78a319	Adding page fetching history. Also denormaling story authors.	2010-07-08 11:37:54 -04:00
Samuel Clay	671121b29c	Fixing broken Last Modified Date bug which causes all stories to be discarded.	2010-07-08 01:07:37 -04:00
Samuel Clay	2e15f84a0d	Additional instrumentation on feed fetching, parsing, and importing.	2010-07-06 18:16:41 -04:00
Samuel Clay	5b1bb231ef	Removing unnecessary cruft above the page importer.	2010-07-01 17:33:58 -04:00
Samuel Clay	53ac3fa0c4	Migrating page_data to its own table so as to not slow down the feeds table.	2010-06-27 21:03:29 -04:00
Samuel Clay	9c4cbc9031	Prevent blank original sites from being saved.	2010-06-24 16:31:38 -04:00
Samuel Clay	3d4cf207d3	Adding a feed punishment for slow feeds. Slow feeds get updated less often.	2010-06-24 15:27:25 -04:00
Samuel Clay	00ed629443	Ignoring fix urls path for now.	2010-04-23 10:44:55 -04:00
Samuel Clay	0edbb7ed74	I just don't feel comfortable with Django's thread-unsafe ORM. Putting locks back in place around all database .save() calls. It's The Right Thing To Do.	2009-09-16 04:00:37 +00:00
Samuel Clay	7b08770b3a	Getting rid of Queues entirely. Replaced with values being distributed directly to Processes. One day this will change. Until then, all feeds fit in memory.	2009-09-16 03:54:33 +00:00
Samuel Clay	4aa0271047	Adding semaphore locks on all mysql db requests in threads/processes.	2009-09-16 02:34:04 +00:00
Samuel Clay	3bd43a8cb4	Updating algorithm to insert the necessary base information into the feed's page.	2009-08-26 03:12:55 +00:00
Samuel Clay	492d1fb229	Better imports through error handling.	2009-08-15 15:10:21 +00:00
Samuel Clay	d97c5ff0c3	Catching the proper exception.	2009-08-13 04:05:49 +00:00
Samuel Clay	0f4cfdffb7	Importing feed's original page along with RSS stories. Wowzers.	2009-08-13 03:26:12 +00:00