Commit graph

251 commits

Author SHA1 Message Date
Samuel Clay
d74139e2ed Fixing major bug around stale data when updating feeds. 2012-03-21 16:05:52 -07:00
Samuel Clay
3a87a7c418 Fixing issue around feed titles that disappear. This just confirms that the new feed title is substantive. 2012-03-19 20:47:06 -07:00
Samuel Clay
d5926fe577 Fixing rss feed tests. 2012-03-06 16:11:27 -08:00
Samuel Clay
33fa13cf69 Fixing issue around feeds that are mistakenly declared as not having a page. 2012-03-05 13:12:50 -08:00
Samuel Clay
936088f09c Allowing any password to be used on accounts with no password set. 2012-02-27 16:36:39 -08:00
Samuel Clay
d901ccdae1 Switching to redis to feed queuing. Also adding a weight for skipping feed fetches under high load. 2012-02-26 22:33:06 -08:00
Samuel Clay
ff00e16682 Fixing last update problem that has been choking feed fetchers. 2012-02-24 17:00:55 -08:00
Samuel Clay
bdf84cb64c Typo in feed pool drain. 2012-02-24 16:48:58 -08:00
Samuel Clay
ce536200f6 Adding ability to wind down feed queue by faking requests. 2012-02-24 16:43:08 -08:00
Samuel Clay
1ffefaa419 Geometrically delaying the fetching of bad feeds, while they're still somewhat fresh. 2012-02-24 11:47:38 -08:00
Samuel Clay
6c7034343b Geometrically decaying feed fetches on known good feeds that have gone bad. 2012-02-23 13:20:10 -08:00
Samuel Clay
dc986a0a3e Preserving feed fetch error histories for 2 weeks. 2012-02-02 07:46:04 -08:00
Samuel Clay
c9cb7dcbc0 Adding new feeds db column 'known_good', for forcing updates of feeds that were once good but are now throwing 500s. 2012-02-01 17:59:46 -08:00
Samuel Clay
619f03f610 Fixing typo in aggressive feed fetching. 2012-01-23 09:12:38 -08:00
Samuel Clay
c37bcdad63 No longer being aggressive about finding a feed's correct address if it returns a 400 or 500 error code, unless it is still the first run or an exception refresh. 2012-01-23 09:08:55 -08:00
Samuel Clay
ccc8878e88 Colorizing feed fetching. 2012-01-04 18:47:40 -08:00
Samuel Clay
b0aa1764b4 Colorizing feed fetching. 2012-01-04 18:30:43 -08:00
Samuel Clay
10ef120e34 Merge branch 'master' into requests
* master:
  Abort old requests and don't cache responses. Trying to fix IE read story bug.
  Force jquery to never cache.
  Checking for exceptions earlier in order to show fixed site.
  Changing user agent to hopefully override overly-sniffy websites.
  Fixing inaccurate substitution on feed urls. Ugh, can't believe this was here this long.
  Adding the setting of a feed's view settings to Site Settings dialog. This should've been in there a loooong time ago.
2012-01-02 16:00:09 -08:00
Samuel Clay
4fd9965949 Changing user agent to hopefully override overly-sniffy websites. 2012-01-02 14:31:50 -08:00
Samuel Clay
6637c05adf Adding a little bit more leeway in page importer by falling back to urllib2 when requests fail. 2011-12-20 22:30:55 -08:00
Samuel Clay
0418538381 Adding slight bit of color to feed fetcher. 2011-12-20 22:02:52 -08:00
Samuel Clay
b5efad9142 Merge branch 'master' into requests
* master: (66 commits)
  Too many stories? Don't animate transitions when switching intelligence levels.
  Fixing typo in iphone app that preventing mark folder as read when the visible stories option was showing.
  Updating the mongo db copy util to also look for updated stories.
  Globals.is_staff on user, not profile. Make the stats referesh every minute instead of every 10 minutes for staff.
  Fixing offsets in river of news.
  Adding ciphering to usernames in log, so I can finally take screenshots of the most colorful logs of all time.
  Adding my very special mongo serialization backup utility to get around the damn unrepairable mongo database. This is taking 14 hours to run.
  Auto-refreshing feedback (1 min for staff, 10 min for everybody).
  Fixing exception around multiple feeds during Google Reader import process. Also switching rate limit to status code 429.
  Adding Nokia MeeGo client to user agents.
  Only show raw feeds in feed autocomplete.
  Handling iphone version for new users on ios app.
  FIXING THE WORST BUG OF MY LIFE -- finally figured out what was causing the story-shows-as-unread bug. Also fixed enclosures on certain types of feeds.
  Fixing menu manage open position to compensate for additional menu items.
  Reducing the amount of work done by feed fetching when there are no new stories.
  Fixing emergency bug around trimming feeds where the cursor is changing. Dammit mongo.
  Simplifying ufw ports in fabfile. Adding env.user.
  Launching the iPhone app on the front-page. Big deal.
  Minor cleanup of river stories view.
  Cleaning up mongoengine imports and settings for default MongoDB.
  ...

Conflicts:
	local_settings.py.template
2011-12-20 21:47:55 -08:00
Samuel Clay
d1b0552942 Reducing the amount of work done by feed fetching when there are no new stories. 2011-12-14 14:18:01 -08:00
Samuel Clay
1828d949d8 Killing off the old slave_db pattern. 2011-12-11 11:06:42 -08:00
Samuel Clay
85feadb3f1 Fixing the crap out of the move feed/folder dialog in iOS app. Completely done with this dialog. Yay! 2011-12-04 13:55:57 -08:00
Samuel Clay
f8f2e6ba3a Merge branch 'master' into requests
* master:
  Fixing bogus type error, caused by duplicate feeds.
  Correcting styling on site settings loader icon.
  Removing dupe migration from feed_settings.
  Adding migration for unique hashes on feed addresses + links.
  Counting subscribers on branched feeds correctly, so as to maintain update frequency, even though it's branched.
  Hooking up feed address change nd branching. Also fixing those infernal feed address change bugs for good.
  Finishing up feed_link corrections. Just need feed_address (which is essentially a refactor and abstraction of this feed_link code), and then duplicate UI, which is pretty easy. Yay.
  Refactoring feed rewriting to take advantage of new feed link changer. Seems to work. Still needs some better duplicate locking during fetch.
  Uniqifying address+link hash. Allowing feed_link branches, but it does not yet work. Needs proper dupe detection.
  Beginning new branch feed logic. Needs a huge db migration.
  Swapping titles on feed settings when necessary.
  Swapping rss and website url.
  Stubbing out entire feed settings modal. Needs duplicate feed list. Also needs backend splitting of subscriptions.
  Adding feed setting menu item. Cleaning up feed exception dialog.
2011-11-27 03:19:00 -05:00
Samuel Clay
36105bf851 Merge branch 'master' into feed_settings
* master:
  Revert "Switching to requests from urllib2/httplib. Bring on the page errors."
  Fixing #41: Adding REDIS to local_settings.py.template.
  Revert "Revert "Switching to requests from urllib2/httplib. Bring on the page errors.""
  Revert "Switching to requests from urllib2/httplib. Bring on the page errors."
  Switching to requests from urllib2/httplib. Bring on the page errors.
  Adding a bunch fo keyboard shortcuts.
  Adding elapsed time field to river and feed view.
  Fixing autolinking of text links to no longer indiscriminately replace text, causing double encoding of html entities.
  Fixing the space bar key to not queue, so it's actually useful now!
  Refining media enclosures by adding all types and linking to their source.
  Refining media enclosure detection.
2011-11-27 02:41:40 -05:00
Samuel Clay
b112717d95 Revert "Revert "Switching to requests from urllib2/httplib. Bring on the page errors.""
This reverts commit a827b2c864.
2011-11-27 02:41:12 -05:00
Samuel Clay
a827b2c864 Revert "Switching to requests from urllib2/httplib. Bring on the page errors."
This reverts commit 617dc3bb56.

Conflicts:

	local_settings.py.template
2011-11-27 02:40:44 -05:00
Samuel Clay
d07b03a7a4 Revert "Revert "Switching to requests from urllib2/httplib. Bring on the page errors.""
This reverts commit 7972d0bb5c.
2011-11-26 11:15:01 -05:00
Samuel Clay
7972d0bb5c Revert "Switching to requests from urllib2/httplib. Bring on the page errors."
This reverts commit 617dc3bb56.
2011-11-26 02:16:14 -05:00
Samuel Clay
617dc3bb56 Switching to requests from urllib2/httplib. Bring on the page errors. 2011-11-26 02:07:31 -05:00
Samuel Clay
221b356494 Refining media enclosures by adding all types and linking to their source. 2011-11-25 10:38:39 -05:00
Samuel Clay
3e68aff667 Merge branch 'master' into feed_settings
* master:
  Correcting for feeds that have incorrect media content.
  Adding media enclosures in an audio tag.
  Using story date when there is no guid.
  Using story date when there is no guid.
  Adding verbose flag to feed updates to diagnose errors.
  Fixing issue with stories that have no permalink.

Conflicts:
	apps/rss_feeds/models.py
2011-11-25 00:18:32 -05:00
Samuel Clay
7ea2e01496 Adding verbose flag to feed updates to diagnose errors. 2011-11-24 15:29:26 -05:00
Samuel Clay
fb237e1049 Finishing up feed_link corrections. Just need feed_address (which is essentially a refactor and abstraction of this feed_link code), and then duplicate UI, which is pretty easy. Yay. 2011-11-16 18:41:36 -08:00
Samuel Clay
14ed618c6c File under "fucking finally": Adding client-side error handling and callbacks for feeds and river. 2011-11-15 18:19:09 -08:00
Samuel Clay
3e1d9eb626 Brief cleanup. 2011-11-12 15:36:38 -08:00
Samuel Clay
54e42f6012 Adding bullshit user agent string because some sites are sniffing for browsers in order to serve the correct site. 2011-11-10 10:29:25 -08:00
Samuel Clay
ac3609bf7d Merge branch 'master' into realtime
* master:
  Adding 'E' as a keyboard shortcut for the Everything view.
  Doubling the amount Space pages in a story.
  Auto-linkify urls in stories.
  Reading stories from slave db during feed update.

Conflicts:
	settings.py
2011-11-06 15:28:37 -08:00
Samuel Clay
4691345d91 Preparing realtime for deployment. 2011-11-06 12:28:06 -08:00
Samuel Clay
07fa01ed9a Adding server half of redis real-time unread count updates. That was the easy part. 2011-11-05 17:08:31 -07:00
Samuel Clay
040b755694 Reading stories from slave db during feed update. 2011-11-04 09:45:10 -07:00
Samuel Clay
acfd9e7614 Adding feed.has_page to weed out feeds with no original page. This should've been in 2 years ago. 2011-09-04 10:59:29 -07:00
Samuel Clay
47556ec303 Locking feed links for testing. This also helps future feed links that need locking. Probsbly should include feed_address_lock's as well. 2011-09-01 09:34:57 -07:00
Samuel Clay
8f5a4c612d Updating rss_feeds tests for completeness. Updating README to include how to run test suite. 2011-09-01 09:11:29 -07:00
Samuel Clay
4f46b358db Major optimization on mongo queries. No longer using .rewind(), which is actually re-running the damn query! Wrapping repetitive querysets in list() for performance. 2011-06-09 15:36:24 -04:00
Samuel Clay
06a46993d2 Backup mongo + psql. Removing ridiculous FeedUpdateHistory table. 2011-05-18 14:36:31 -04:00
Samuel Clay
a757b9f88f Fixing a few feed fetching bugs. 2011-04-04 20:53:12 -04:00
Samuel Clay
bb4e81b278 Fixing a whole lot of feed fetch bugs. 2011-02-15 21:08:40 -05:00