Samuel Clay
|
4f35c2fade
|
Making feed.min_to_decay reflect the actual decay time. Using it in statistics, and refreshing it in feed fetching. Also splitting up feed munin.
|
2010-12-23 13:29:31 -05:00 |
|
Samuel Clay
|
97ad4838f1
|
Fixing stories in tabs, and adding option to specify username for refresh_feeds, to assist in development.
|
2010-12-14 23:10:13 -05:00 |
|
Samuel Clay
|
fa663e910a
|
Faster feed refreshing (insta-fetching) by only computing the score for the current user and leaving the other users with dirty subscriptions.
|
2010-11-10 18:04:17 -05:00 |
|
Samuel Clay
|
4cf9b05f0c
|
Changing all datetime.datetime.now() to datetime.datetime.utcnow().
|
2010-10-10 23:55:00 -04:00 |
|
Samuel Clay
|
50488db7ca
|
Close the connection for every set of tasks.
|
2010-10-08 16:08:41 -04:00 |
|
Samuel Clay
|
39d9e13449
|
Forcing refresh of feed information at every stage of feed fetch pipeline. This really needs a rewrite. Someday.
|
2010-10-03 18:04:40 -04:00 |
|
Samuel Clay
|
15a27e840f
|
Bootstrapping mongo to use story_guids as unqiue to feeds.
|
2010-09-08 18:30:46 -07:00 |
|
Samuel Clay
|
41be0abde3
|
Moving feeds out of the queue when they get updated.
|
2010-08-31 22:04:26 -04:00 |
|
Samuel Clay
|
25c0450020
|
Commenting out map/reduce statistics because mongo keeps crashing.
|
2010-08-30 13:33:29 -04:00 |
|
Samuel Clay
|
abe73a1d0e
|
Barring the ability to use multiprocessing for feed fetching, in the meantime we'll have to use separate processes, which means potential for fetchin gthe same feeds.
|
2010-08-23 08:09:58 -04:00 |
|
Samuel Clay
|
ac53f33788
|
Further Mongo work. Stories are now saved correctly, with tags + authors. Still need to do UserStories.
|
2010-08-21 20:42:38 -04:00 |
|
Samuel Clay
|
8b9c28e8f8
|
Cleaning up exception handling of feeds to show to users during import.
|
2010-08-18 21:56:30 -04:00 |
|
Samuel Clay
|
68b0ae7254
|
Adding exception handling UI. Also disabled feeds that are just chock full of exceptions. No need to continually fetch buggy feeds. Still need to figure out a way to let users fix problematic feeds, but letting them know is an improvement.
|
2010-08-18 20:35:45 -04:00 |
|
Samuel Clay
|
55683fc1d5
|
Skipping feeds with fewer than X stories_per_month. Used to catch the queue up and toss inactive feeds.
|
2010-08-16 13:21:32 -04:00 |
|
Samuel Clay
|
588207384c
|
Logging tweaks to handle console and file handlers.
|
2010-08-16 12:52:39 -04:00 |
|
Samuel Clay
|
3c6fdfdea7
|
Statistics on all feeds, stories, averages. Shown in the new statistics modal (not yet finished).
|
2010-07-25 23:13:27 -04:00 |
|
Samuel Clay
|
2c6433b0de
|
Parallelizing feed fetcher now recognizes other feed fetchers so they do not replicate work. Now multi-threaded and multi-processed aware
|
2010-04-27 13:44:53 -04:00 |
|
Samuel Clay
|
0a664b44a8
|
Adding feed update history. Gotta get some metrics on the speed of the biggest cpu/network suck.
|
2010-04-23 21:19:19 -04:00 |
|
Samuel Clay
|
71b746a48d
|
Feed update cascading. Feeds update relative to how often they have new content. This should save a LOT of processing power. Whew.
|
2010-04-19 12:09:04 -04:00 |
|
Samuel Clay
|
f169569710
|
Adding single threaded feed updating (used for tests and debugging)
|
2010-04-09 16:37:19 -04:00 |
|
Samuel Clay
|
391aa76dd6
|
Lowering worker threads from 8 to 4 to conserve memory.
|
2010-03-02 17:04:54 +00:00 |
|
samuelclay
|
d9c7ae7771
|
Auto-compressing all stories and feeds at the db level for better database lookups.
|
2010-01-27 16:29:11 -05:00 |
|
Samuel Clay
|
7b08770b3a
|
Getting rid of Queues entirely. Replaced with values being distributed directly to Processes. One day this will change. Until then, all feeds fit in memory.
|
2009-09-16 03:54:33 +00:00 |
|
Samuel Clay
|
a7888c2aae
|
Switching from threads to processes. Sure, no big deal. Django is not thread-safe, so I can't use threads. I can, however, use multiprocessing to pass a Queue around between Processes.
|
2009-09-16 02:22:27 +00:00 |
|
Samuel Clay
|
0446ff2ee3
|
Adding locks to feed processing.
|
2009-09-12 20:42:38 +00:00 |
|
Samuel Clay
|
bec2f00979
|
Exiting after it's over.
|
2009-09-10 04:24:51 +00:00 |
|
Samuel Clay
|
0b9b778382
|
Queues not working. Reverting back to 1 feed at a time. Ugh.
|
2009-09-10 04:16:50 +00:00 |
|
Samuel Clay
|
8437ef29f1
|
Handling empty queues and increasing thread count
|
2009-09-10 03:52:43 +00:00 |
|
Samuel Clay
|
67ab0d6c66
|
Let's try changing the load.
|
2009-09-10 03:48:22 +00:00 |
|
Samuel Clay
|
02a4d292e8
|
Moving from threadpool to a Queue. Much simpler.
|
2009-09-10 03:33:05 +00:00 |
|
Samuel Clay
|
0caf49e9a4
|
OK. Turning on threading for feed fetching. Enough logging to catch the bugs. Fixed the segmentation fault on exit.
|
2009-09-10 02:31:55 +00:00 |
|
Samuel Clay
|
9e522ca528
|
Woo - Checking for content and title differences to resolve duplicate entries. Integration tests prove a number of differences are successfully resolved.
|
2009-08-30 00:43:13 +00:00 |
|
Samuel Clay
|
bdd91d714d
|
Refactored the feed fetcher. Added integration tests for dupe matching. Also stubbed in AI score calculations so I can do something with the output soon.
|
2009-08-29 19:34:42 +00:00 |
|
Samuel Clay
|
d24b56fbf4
|
Adding new "Return to Page" button. When the user visits other pages in a feed, security prevents NB from reading the iframe, so we provide a convenient hook to bring the iframe back. Call it SnapBack, except, don't.
|
2009-08-26 02:31:35 +00:00 |
|
Samuel Clay
|
ce9ec58d00
|
Some feeds don't have pages. Who knew.
|
2009-08-23 00:07:37 +00:00 |
|
Samuel Clay
|
9c522416a1
|
Gotta get rid of threading until I can figure out how to thread-safe the Django ORM. It's corrupting my tables!
|
2009-08-23 00:00:55 +00:00 |
|
Samuel Clay
|
fa123e7f3d
|
Updating feeds to crash with noise.
|
2009-08-21 13:14:44 +00:00 |
|
Samuel Clay
|
6cf04497ee
|
Checking for threadpool
|
2009-08-20 02:53:41 +00:00 |
|
Samuel Clay
|
8545e09d9a
|
Using threadpool module and feedjack to import feeds. Largely the same functionality, but now more debugging info, as well as better error handling and threads.
|
2009-08-20 02:43:01 +00:00 |
|
Samuel Clay
|
88ccfa7e77
|
Created admin commands to run refresh_feed/s without invoking Apache. Finally!
|
2009-08-14 01:48:21 +00:00 |
|