Samuel Clay
dbec2708da
Only show active feeds in dashboard river. Also link dashboard river title and icon.
2021-03-04 10:24:17 -05:00
jmath1
6021afaec3
2to3 apps/rss_feeds
2020-06-15 02:54:37 -04:00
jmath1
ebd0da2d63
change management commands to use argparse instead of optparse
2020-06-08 07:55:17 -04:00
Samuel Clay
da26ecdb46
1/2 Cleaning up feed fetcher's updates to feeds table. Still need to find 6 dupes.
2015-07-22 13:18:54 -07:00
Samuel Clay
c20ef4e5d0
Adding updates_off run-time setting for handling large spikes of updates.
2014-05-29 17:53:16 -07:00
Samuel Clay
1584208677
Adding metric for last story date, used to calculate next scheduled update.
2013-04-23 17:04:21 -07:00
Samuel Clay
c41bf80b4e
Removing force from new feeds.
2013-04-07 20:22:46 -07:00
Samuel Clay
2200aa282a
Subscribe to blog during intro.
2012-03-19 15:46:59 -07:00
Samuel Clay
d901ccdae1
Switching to redis to feed queuing. Also adding a weight for skipping feed fetches under high load.
2012-02-26 22:33:06 -08:00
Samuel Clay
ce536200f6
Adding ability to wind down feed queue by faking requests.
2012-02-24 16:43:08 -08:00
Samuel Clay
b5efad9142
Merge branch 'master' into requests
...
* master: (66 commits)
Too many stories? Don't animate transitions when switching intelligence levels.
Fixing typo in iphone app that preventing mark folder as read when the visible stories option was showing.
Updating the mongo db copy util to also look for updated stories.
Globals.is_staff on user, not profile. Make the stats referesh every minute instead of every 10 minutes for staff.
Fixing offsets in river of news.
Adding ciphering to usernames in log, so I can finally take screenshots of the most colorful logs of all time.
Adding my very special mongo serialization backup utility to get around the damn unrepairable mongo database. This is taking 14 hours to run.
Auto-refreshing feedback (1 min for staff, 10 min for everybody).
Fixing exception around multiple feeds during Google Reader import process. Also switching rate limit to status code 429.
Adding Nokia MeeGo client to user agents.
Only show raw feeds in feed autocomplete.
Handling iphone version for new users on ios app.
FIXING THE WORST BUG OF MY LIFE -- finally figured out what was causing the story-shows-as-unread bug. Also fixed enclosures on certain types of feeds.
Fixing menu manage open position to compensate for additional menu items.
Reducing the amount of work done by feed fetching when there are no new stories.
Fixing emergency bug around trimming feeds where the cursor is changing. Dammit mongo.
Simplifying ufw ports in fabfile. Adding env.user.
Launching the iPhone app on the front-page. Big deal.
Minor cleanup of river stories view.
Cleaning up mongoengine imports and settings for default MongoDB.
...
Conflicts:
local_settings.py.template
2011-12-20 21:47:55 -08:00
Samuel Clay
f3d2f8e5b5
Moving from slave_db to ReplicaSetConnection in pymongo 2.1.
2011-12-11 10:54:07 -08:00
Samuel Clay
b112717d95
Revert "Revert "Switching to requests from urllib2/httplib. Bring on the page errors.""
...
This reverts commit a827b2c864
2011-11-27 02:41:12 -05:00
Samuel Clay
a827b2c864
Revert "Switching to requests from urllib2/httplib. Bring on the page errors."
...
This reverts commit 617dc3bb56
2011-11-27 02:40:44 -05:00
Samuel Clay
d07b03a7a4
Revert "Revert "Switching to requests from urllib2/httplib. Bring on the page errors.""
...
This reverts commit 7972d0bb5c
2011-11-26 11:15:01 -05:00
Samuel Clay
7972d0bb5c
Revert "Switching to requests from urllib2/httplib. Bring on the page errors."
...
This reverts commit 617dc3bb56
2011-11-26 02:16:14 -05:00
Samuel Clay
617dc3bb56
Switching to requests from urllib2/httplib. Bring on the page errors.
2011-11-26 02:07:31 -05:00
Samuel Clay
ac3609bf7d
Merge branch 'master' into realtime
...
* master:
Adding 'E' as a keyboard shortcut for the Everything view.
Doubling the amount Space pages in a story.
Auto-linkify urls in stories.
Reading stories from slave db during feed update.
Conflicts:
settings.py
2011-11-06 15:28:37 -08:00
Samuel Clay
07fa01ed9a
Adding server half of redis real-time unread count updates. That was the easy part.
2011-11-05 17:08:31 -07:00
Samuel Clay
040b755694
Reading stories from slave db during feed update.
2011-11-04 09:45:10 -07:00
Samuel Clay
4f35c2fade
Making feed.min_to_decay reflect the actual decay time. Using it in statistics, and refreshing it in feed fetching. Also splitting up feed munin.
2010-12-23 13:29:31 -05:00
Samuel Clay
97ad4838f1
Fixing stories in tabs, and adding option to specify username for refresh_feeds, to assist in development.
2010-12-14 23:10:13 -05:00
Samuel Clay
fa663e910a
Faster feed refreshing (insta-fetching) by only computing the score for the current user and leaving the other users with dirty subscriptions.
2010-11-10 18:04:17 -05:00
Samuel Clay
4cf9b05f0c
Changing all datetime.datetime.now() to datetime.datetime.utcnow().
2010-10-10 23:55:00 -04:00
Samuel Clay
50488db7ca
Close the connection for every set of tasks.
2010-10-08 16:08:41 -04:00
Samuel Clay
39d9e13449
Forcing refresh of feed information at every stage of feed fetch pipeline. This really needs a rewrite. Someday.
2010-10-03 18:04:40 -04:00
Samuel Clay
15a27e840f
Bootstrapping mongo to use story_guids as unqiue to feeds.
2010-09-08 18:30:46 -07:00
Samuel Clay
41be0abde3
Moving feeds out of the queue when they get updated.
2010-08-31 22:04:26 -04:00
Samuel Clay
25c0450020
Commenting out map/reduce statistics because mongo keeps crashing.
2010-08-30 13:33:29 -04:00
Samuel Clay
abe73a1d0e
Barring the ability to use multiprocessing for feed fetching, in the meantime we'll have to use separate processes, which means potential for fetchin gthe same feeds.
2010-08-23 08:09:58 -04:00
Samuel Clay
ac53f33788
Further Mongo work. Stories are now saved correctly, with tags + authors. Still need to do UserStories.
2010-08-21 20:42:38 -04:00
Samuel Clay
8b9c28e8f8
Cleaning up exception handling of feeds to show to users during import.
2010-08-18 21:56:30 -04:00
Samuel Clay
68b0ae7254
Adding exception handling UI. Also disabled feeds that are just chock full of exceptions. No need to continually fetch buggy feeds. Still need to figure out a way to let users fix problematic feeds, but letting them know is an improvement.
2010-08-18 20:35:45 -04:00
Samuel Clay
55683fc1d5
Skipping feeds with fewer than X stories_per_month. Used to catch the queue up and toss inactive feeds.
2010-08-16 13:21:32 -04:00
Samuel Clay
588207384c
Logging tweaks to handle console and file handlers.
2010-08-16 12:52:39 -04:00
Samuel Clay
3c6fdfdea7
Statistics on all feeds, stories, averages. Shown in the new statistics modal (not yet finished).
2010-07-25 23:13:27 -04:00
Samuel Clay
2c6433b0de
Parallelizing feed fetcher now recognizes other feed fetchers so they do not replicate work. Now multi-threaded and multi-processed aware
2010-04-27 13:44:53 -04:00
Samuel Clay
0a664b44a8
Adding feed update history. Gotta get some metrics on the speed of the biggest cpu/network suck.
2010-04-23 21:19:19 -04:00
Samuel Clay
71b746a48d
Feed update cascading. Feeds update relative to how often they have new content. This should save a LOT of processing power. Whew.
2010-04-19 12:09:04 -04:00
Samuel Clay
f169569710
Adding single threaded feed updating (used for tests and debugging)
2010-04-09 16:37:19 -04:00
Samuel Clay
391aa76dd6
Lowering worker threads from 8 to 4 to conserve memory.
2010-03-02 17:04:54 +00:00
samuelclay
d9c7ae7771
Auto-compressing all stories and feeds at the db level for better database lookups.
2010-01-27 16:29:11 -05:00
Samuel Clay
7b08770b3a
Getting rid of Queues entirely. Replaced with values being distributed directly to Processes. One day this will change. Until then, all feeds fit in memory.
2009-09-16 03:54:33 +00:00
Samuel Clay
a7888c2aae
Switching from threads to processes. Sure, no big deal. Django is not thread-safe, so I can't use threads. I can, however, use multiprocessing to pass a Queue around between Processes.
2009-09-16 02:22:27 +00:00
Samuel Clay
0446ff2ee3
Adding locks to feed processing.
2009-09-12 20:42:38 +00:00
Samuel Clay
bec2f00979
Exiting after it's over.
2009-09-10 04:24:51 +00:00
Samuel Clay
0b9b778382
Queues not working. Reverting back to 1 feed at a time. Ugh.
2009-09-10 04:16:50 +00:00
Samuel Clay
8437ef29f1
Handling empty queues and increasing thread count
2009-09-10 03:52:43 +00:00
Samuel Clay
67ab0d6c66
Let's try changing the load.
2009-09-10 03:48:22 +00:00
Samuel Clay
02a4d292e8
Moving from threadpool to a Queue. Much simpler.
2009-09-10 03:33:05 +00:00
