Samuel Clay
c9cb7dcbc0
Adding new feeds db column 'known_good', for forcing updates of feeds that were once good but are now throwing 500s.
2012-02-01 17:59:46 -08:00
Samuel Clay
b5efad9142
Merge branch 'master' into requests
...
* master: (66 commits)
Too many stories? Don't animate transitions when switching intelligence levels.
Fixing typo in iphone app that preventing mark folder as read when the visible stories option was showing.
Updating the mongo db copy util to also look for updated stories.
Globals.is_staff on user, not profile. Make the stats referesh every minute instead of every 10 minutes for staff.
Fixing offsets in river of news.
Adding ciphering to usernames in log, so I can finally take screenshots of the most colorful logs of all time.
Adding my very special mongo serialization backup utility to get around the damn unrepairable mongo database. This is taking 14 hours to run.
Auto-refreshing feedback (1 min for staff, 10 min for everybody).
Fixing exception around multiple feeds during Google Reader import process. Also switching rate limit to status code 429.
Adding Nokia MeeGo client to user agents.
Only show raw feeds in feed autocomplete.
Handling iphone version for new users on ios app.
FIXING THE WORST BUG OF MY LIFE -- finally figured out what was causing the story-shows-as-unread bug. Also fixed enclosures on certain types of feeds.
Fixing menu manage open position to compensate for additional menu items.
Reducing the amount of work done by feed fetching when there are no new stories.
Fixing emergency bug around trimming feeds where the cursor is changing. Dammit mongo.
Simplifying ufw ports in fabfile. Adding env.user.
Launching the iPhone app on the front-page. Big deal.
Minor cleanup of river stories view.
Cleaning up mongoengine imports and settings for default MongoDB.
...
Conflicts:
local_settings.py.template
2011-12-20 21:47:55 -08:00
Samuel Clay
f3d2f8e5b5
Moving from slave_db to ReplicaSetConnection in pymongo 2.1.
2011-12-11 10:54:07 -08:00
Samuel Clay
b112717d95
Revert "Revert "Switching to requests from urllib2/httplib. Bring on the page errors.""
...
This reverts commit a827b2c864
2011-11-27 02:41:12 -05:00
Samuel Clay
a827b2c864
Revert "Switching to requests from urllib2/httplib. Bring on the page errors."
...
This reverts commit 617dc3bb56
2011-11-27 02:40:44 -05:00
Samuel Clay
d07b03a7a4
Revert "Revert "Switching to requests from urllib2/httplib. Bring on the page errors.""
...
This reverts commit 7972d0bb5c
2011-11-26 11:15:01 -05:00
Samuel Clay
7972d0bb5c
Revert "Switching to requests from urllib2/httplib. Bring on the page errors."
...
This reverts commit 617dc3bb56
2011-11-26 02:16:14 -05:00
Samuel Clay
617dc3bb56
Switching to requests from urllib2/httplib. Bring on the page errors.
2011-11-26 02:07:31 -05:00
Samuel Clay
ac3609bf7d
Merge branch 'master' into realtime
...
* master:
Adding 'E' as a keyboard shortcut for the Everything view.
Doubling the amount Space pages in a story.
Auto-linkify urls in stories.
Reading stories from slave db during feed update.
Conflicts:
settings.py
2011-11-06 15:28:37 -08:00
Samuel Clay
07fa01ed9a
Adding server half of redis real-time unread count updates. That was the easy part.
2011-11-05 17:08:31 -07:00
Samuel Clay
040b755694
Reading stories from slave db during feed update.
2011-11-04 09:45:10 -07:00
Samuel Clay
c2b1d7eb35
Fixing the infernal FeedNotFound error whenever a dupe feed is created in the middle of another transaction. Yarr!
2011-05-19 19:04:10 -04:00
Samuel Clay
8e2936fc1b
Utils -> vendor
2011-04-11 21:57:45 -04:00
Samuel Clay
1254d350d6
Only tasking feeds that are active. Clears the exception pile.
2011-04-07 12:36:38 -04:00
Samuel Clay
bdc8e7bdea
Separating nginx configuration into a setup and a configure, since one half is much more common than the other.
2011-03-31 18:51:23 -04:00
Samuel Clay
09021ea08c
Trying to fix stale feeds by checking for a last update of a day.
2011-02-22 09:37:09 -05:00
Samuel Clay
3723160155
A huge number of misc fixes: fixing potential errors on paypal return screen, mailing all feed/page parse errors, and swithcing to mapreduce item_frequencies in mongoengine.
2011-02-05 15:34:43 -05:00
Samuel Clay
e14ba054ad
Fixing up trim feeds so we can trim old unused feeds.
2011-01-22 11:32:49 -05:00
Samuel Clay
443fe69156
Moving text fields from Feed to FeedData. Migration includes deletion of old fields on Feed, now unused.
2011-01-17 22:48:38 -05:00
Samuel Clay
4f35c2fade
Making feed.min_to_decay reflect the actual decay time. Using it in statistics, and refreshing it in feed fetching. Also splitting up feed munin.
2010-12-23 13:29:31 -05:00
Samuel Clay
e039d9a1bd
Queueing fallen-through-the-cracks feeds appropriately, giving a day of leniency.
2010-12-23 12:39:28 -05:00
Samuel Clay
b98c63dd14
Abstracting feed queueing so I can queue a queryset of feeds from the command line. Also adding in queueing of feeds that have not been updated in a week (for those that fall through the cracks.
2010-12-23 12:32:24 -05:00
Samuel Clay
97ad4838f1
Fixing stories in tabs, and adding option to specify username for refresh_feeds, to assist in development.
2010-12-14 23:10:13 -05:00
Samuel Clay
fa663e910a
Faster feed refreshing (insta-fetching) by only computing the score for the current user and leaving the other users with dirty subscriptions.
2010-11-10 18:04:17 -05:00
Samuel Clay
cbb04ba6ef
Adding a few keybaord shortcuts, and adding duplicate feed counts to munin.
2010-10-12 13:17:43 -04:00
Samuel Clay
4cf9b05f0c
Changing all datetime.datetime.now() to datetime.datetime.utcnow().
2010-10-10 23:55:00 -04:00
Samuel Clay
50255cf928
Using UTCnow() for dates in marking read stories. Off by 4 hours right now.
2010-10-10 23:51:31 -04:00
Samuel Clay
50488db7ca
Close the connection for every set of tasks.
2010-10-08 16:08:41 -04:00
Samuel Clay
59b32e6248
Further fixing feed fetching errors by retrying feeds [nearly] immediately when they can be corrected by visiting the feed's site and finding the correct RSS feed.
2010-10-03 22:38:53 -04:00
Samuel Clay
39d9e13449
Forcing refresh of feed information at every stage of feed fetch pipeline. This really needs a rewrite. Someday.
2010-10-03 18:04:40 -04:00
Samuel Clay
2e3f7d415b
Fixing celery queues and exchanges by adding explicit bindings for the new and update queues.
2010-09-28 19:50:29 -04:00
Samuel Clay
032505591d
Explicit queue management.
2010-09-28 05:16:00 -04:00
Samuel Clay
e647edf025
Trimming more than 500 stories from feeds. One day these can remain, but for now, it's too much to handle.
2010-09-22 14:35:26 -04:00
Samuel Clay
2bd9f72acb
Fetching feedids with folder delete.
2010-09-21 11:16:22 -04:00
Samuel Clay
7099107899
Merge branch 'master' into folder_actions
2010-09-20 19:32:17 -04:00
Samuel Clay
9d265e9db0
Adding a new queue that handles new feeds, so they can be prioritized over updating old feeds.
2010-09-20 19:22:19 -04:00
Samuel Clay
d3e2c6ae1f
Optimizing distribution of tasked feeds.
2010-09-17 12:53:05 -04:00
Samuel Clay
fa73da5bfc
Turning off feed fetching for feeds with no active subscribers. No way to turn them back on, but who cares? They're inactive
2010-09-17 12:51:25 -04:00
Samuel Clay
b7dc554801
Turning off feed fetching for feeds with no active subscribers. No way to turn them back on, but who cares? They're inactive
2010-09-17 12:50:46 -04:00
Samuel Clay
154ac4d3a9
Fixing delete feeds to only delete the one feed you want to delete, not all of the others. Also fixed add URL to check more aggressively for duplicate feeds.
2010-09-14 23:47:21 -04:00
Samuel Clay
15a27e840f
Bootstrapping mongo to use story_guids as unqiue to feeds.
2010-09-08 18:30:46 -07:00
Samuel Clay
44b6606b32
Finalizing merge_feeds.
2010-09-08 16:43:00 -07:00
Samuel Clay
7b28861090
MongoDB requires integers for integers.
2010-09-08 16:08:01 -07:00
Samuel Clay
a021c71286
MongoDB requires integers for integers.
2010-09-08 16:06:27 -07:00
Samuel Clay
70fcab8a40
Off by 1 in checking for similar stories.
2010-09-08 16:04:12 -07:00
Samuel Clay
3dd61cbbc6
Merge feeds, now for MongoDB. Checks feed titles, taglines, and links, and then guarantees at least 3 stories are the same.
2010-09-08 16:00:20 -07:00
Samuel Clay
132c561be5
Queue date for feeds for better metrics.
2010-09-08 13:58:05 -07:00
Samuel Clay
30e1e032c1
Task feeds in a grouped list rather than individually. Saves task creation overhead.
2010-09-01 10:44:29 -04:00
Samuel Clay
b10c223f18
Only active feeds get updated.
2010-08-31 22:17:31 -04:00
Samuel Clay
41be0abde3
Moving feeds out of the queue when they get updated.
2010-08-31 22:04:26 -04:00
