What a difference a few months make. NewsBlur was a side-project of mine for two years. In March of this year, I committed myself full-time and went from developing NewsBlur almost entirely on the NYC subway to writing code every waking minute of the day. And now there are three big announcements to make. # 1. NewsBlur is now a *social* news reader The big news of the day is that you can now share stories on NewsBlur. When you share a story, your comments and the original story are posted to your blurblog. Your blurblog is a simple and customizable website. People can comment and reply directly on your blurblog, and you can follow your friends to read the news stories and blog posts that they care about.Since you’re good at picking your friends, and your friends are good at picking their friends, you will see friends of friends show up, expanding your network with shared stories that you will enjoy. It’s a new way of sharing the news. And because NewsBlur is already an easy to use news reader, it’s simple to find and share stories that your friends will care about. Every NewsBlur user has their own blurblog. All you have to do is signup for an account on www.newsblur.com and share interesting stories. # 2. Y Combinator For those of you who work with computer science, you may know that a Y-combinator generalizes recursion, abstracting its implementation, and thereby separating it from the actual work of the function in question.[^1] I’m pleased as punch to announce an investment in NewsBlur by Y Combinator, the investment firm. Over the past two months, we’ve been humbled by the roster of experienced partners giving us candid advice. It’s their tough love that is the catalyst for the next few months of transitioning NewsBlur from side project to world-class news reader. Expect NewsBlur to become simpler and more refined. # 3. Introducing Roy Yang When Y Combinator accepted me as a solo founder, their first piece of advice was to find a co-founder. Looking at every successful startup, a common pattern emerges. Every great startup has multiple people carrying the load when the company takes off. There is one person on this planet that I would trust as a co-founder. His name is Roy Yang and we have been friends since we met in New York four years ago. We worked together for nearly two years at Daylife, another news startup. I attended his wedding last year in Mexico, and he was the only person I called when I knew I needed somebody talented, focused, and able to complement me on a project that demands enormous time and effort.Roy is now responsible for both iOS apps and is instrumental in challenging me when I think I’m right and am clearly not. He’s got the patience of a monk and the determination of a true New Yorker. Follow Roy’s blurblog to keep up with him. # A glimpse into the future of NewsBlur This summer marks the beginning of NewsBlur as a full-time startup. Look forward to new mobile apps, new designs, and new features. Here’s a quick idea of what we’re working on for the next few weeks:
Until then, follow @newsblur on Twitter and start sharing news and blogs on NewsBlur.
Today, NewsBlur is going real-time. Blogs using the PubSubHubbub protocol (PuSH), which includes all Blogger, Tumblr, and many Wordpress blogs, will instantaneously show new updates to subscribers on NewsBlur. Making this happen, while not for the faint of heart, was straight-forward enough that I’m sharing the recipe I used to get everything hooked up and running smoothly.
Every user, both premium and standard, will now receive instantaneous updates. I’ve been beta-testing this feature for the past few weeks, and I’ve been quite pleased in knowing that I’m now reading on the bleeding-edge.
If you are a developer, you may be interested in how this was done. There are two components in a real-time feed: detecting updates and then informing users of those updates.
Get blog updates in real-time
If you are building a system that consumes an RSS feed and you want it to push to you, you’ll have to subscribe to a special PubSubHubbub hub url that the RSS feed gives you in the original RSS feed.
Take a look at the <feed> section in the NewsBlur Blog’s RSS feed:
If there’s a rel="hub" node under links, then the RSS feed is advertising its PubSubHubbub abilities. If you make a subscription request to that address, then the feed will push out updates to your callback URL.
The publisher will then ping your server back to confirm the subscription. Once the publisher is configured to send blog updates to your server, you just have to let users know when there’s a new story, and that’s takes some COMET/push technology with the help of WebSockets.
Serving updates to visitors in real-time
When a publisher pushes a new story to your server, apart from dupe detection and storing it in your database, you need to alert users who are currently on the site.
Redis is your new best friend. One of its primary data structures, apart from hashes, sets, sorted sets, and key-value, is a pubsub type that is perfect for this kind of update. Users subscribe to the updates of all of the feeds to which they subscribe. When these sites have a new story, they publish a simple notification to each of the feed’s subscribers.
Here the feed fetcher is publishing to any listening subscribers.
# Python
def publish_to_subscribers(self, feed):
try:
r = redis.Redis(connection_pool=settings.REDIS_POOL)
listeners_count = r.publish(str(feed.pk), 'story:new')
if listeners_count:
logging.debug(" ---> [%-30s] Published to %s subscribers" % (
feed.title[:30], listeners_count))
except redis.ConnectionError:
logging.debug(" ***> [%-30s] Redis is unavailable for real-time." % (
feed.title[:30],))
These subscribers have subscribed via Redis. To know that a user is currently connected and wants to be notified of updates, Socket.io is used to connect the browser to a Node.js server that will subscribe to updates via Redis.
The browser opens up a WebSocket and listens for updates for the feeds that they care about:
// JavaScript
setup_socket_realtime_unread_counts: function() {
if (!this.socket) {
var server = window.location.protocol + '//' +
window.location.hostname + ':8888';
this.socket = this.socket || io.connect(server);
this.socket.on('connect', _.bind(function() {
var active_feeds = this.send_socket_active_feeds();
console.log(["Connected to real-time pubsub with " +
active_feeds.length + " feeds."]);
this.socket.on('feed:update', _.bind(function(feed_id, message) {
console.log(['Real-time feed update', feed_id, message]);
this.force_feeds_refresh(false, false, parseInt(feed_id, 10));
}, this));
this.flags.feed_refreshing_in_realtime = true;
this.setup_feed_refresh();
}, this));
this.socket.on('disconnect', _.bind(function() {
console.log(["Lost connection to real-time pubsub. Falling back to polling."]);
this.setup_feed_refresh();
}, this));
}
},
The app server is ready to handle thousands of concurrent subscription requests, being Node.js and asynchronous:
That’s all there is to it. There a lot going on, but it’s effectively a small circle composed of subscribers and publishers, using Redis to maintain pubsub connections between the many clients and their many feeds.
Hey NewsBlurians, I’m applying for a grant from the Knight Foundation. In a previous life I worked on DocumentCloud, a successful Knight grantee, building open-source libraries. I’m looking to continue the fine tradition of building for both users and for other developers.
I’m asking for enough to fund a year of development with the help of another engineer. Please vote for the NewsBlur grant application on Knight’s website, reblogged below.
To build an intelligent social news reader for web and mobile called NewsBlur.
2. Is anyone doing something like this now and how is your project different? [30 words]
RSS feed readers exist, none have the sharing model, original site view, and intelligence classifiers of NewsBlur. Since Google Reader phased out sharing, there’s a sizable community looking to share and discuss news.
3. Describe the network with which you intend to build or work. [50 words]
NewsBlur will surface stories shared by friends and friends of friends by combining the imported networks of Twitter/Facebook with communities on NewsBlur. These communities make it easy to expand your network by showing popular comments from outside your network. NewsBlur also has intelligence classifiers which allow the user to filter and highlight comments across all networks.
4. Why will it work? [100 words]
Because it’s worked before, just under a different model in Google Reader’s now defunct all-or-nothing community. NewsBlur’s network will be oriented more towards showing relationships and distance between you and the other active commenters on a story. NewsBlur will capitalize on the value of pre-existing networks with an intuitive and clean interface that highlights the distance between users. Surfacing relative connections between people will result in a more active community and increased engagement between like-minded readers. NewsBlur further benefits newspapers, publishers, and individual writers by showing the original site (including ads and design), as well as encouraging reading through NewsBlur’s intelligence filters.
5. Who is working on it? [100 words]
I started working on NewsBlur as a side project in June 2009. Over the past 2.5 years, NewsBlur has become self-sustainable through organic growth (word-of-mouth, blog posts, github activity). Because NewsBlur is open-source, a number of contributors from the NewsBlur community have developed their own pet features which have been integrated back into the website. This also works well for finding typos in documentation and allowing users to submit a simple pull request to get it fixed.
6. What part of the project have you already built? [100 words]
On the back-end: distributed feed fetchers and parsers. On the front-end: the feed reader itself, intelligence training, and an iPhone app. There is an actively used API, on top of which NewsBlur’s users have built a mobile website, an Android app, a Windows Phone app, and a Nokia MeeGo app. What’s not built is the entire social layer. A prototype has already been developed to surface any network relationships on comments and shared stories.
7. How would you sustain the project after the funding expires? [50 words]
NewsBlur is free, but there is also a premium subscription that costs between $1 - $3 per month. Users can choose how much they’d like to pay, but that means that NewsBlur is able to pay for its 8 servers. The gap between costs and revenue (also known as profit) is increasing every day.
Requested amount from Knight News Challenge: $150,000
Expected amount of time required to complete project: 1 year
And what a gorgeous mobile app it is. App developer Róbert Márki just released Web Feeds, the first NewsBlur app for Nokia MeeGo. Take a look at these screenshots:
In that time, every measure of traffic, paid subscriptions, and community feedback has cleared new heights on my revered munin charts. But I kept my full-time job and only worked on NewsBlur as a side-project, as I have been doing for the past two years.
I’m at a slow but steady trickle of users, enough of whom convert into paying premium subscribers to pay for the increasing number of servers (now costing almost $800/month!). But a vocal support forum and an ever-increasing trickle of payments aren’t what convinced me to dedicate myself full-time to my humble project.
What convinced me is that life’s too short.
The only way to do great work is to love what you do
Yesterday was my last day at Tasty Labs. Today is my first day as a full-time indie developer.
I’m following the better-half of my inner-Jobs and choosing to follow the path of doing what I love to do.1 It’s going to be expensive on my part. There will be no paycheck, health insurance runs out in a few months, and even on an exceptionally good day, premium subscriptions bring in only about half of my former salary.
Paying customers are keeping things afloat
It’s not that big a risk. Users are choosing to pay and choosing how much to pay. Some users opt to pay a $12/year at a buck a month, while some choose to pay $36/year at $3/month. The servers cost roughly .75x what premium subscriptions bring in. That gap between cost and revenue is growing steadily, as the servers I have are running at 25% capacity.
Maciej Ceglowski, of Pinboard fame, has written extensively about the scourge of free services not taking payments from users. From a recent February 2012 presentation, he writes of the three types of services [pdf]:
Business Model #1: Charge Money
This is a groundbreaking new approach where a site accepts fungible tokens of value in return for a product or service.
Business Model #2: Burn Money
Find a sponsor with deep pockets and run at a loss indefinitely.
Business Model #3: Offer a Free Service and Fail
The most popular business model has been to offer a free service and then shut down (or transition to model #1).
You can look forward to new features, bug fixes, and an unending stream of progress, all because you’re paying me for the work I do. It’s a simple setup that helps me sleep at night. I’m building an honest product. You know what you’re paying for. And NewsBlur will continue to be ad-free, because ads are not part of the product that I want to use myself.
So much time, so little to do. Strike that! Reverse it.
In what I can only consider a bizarrely wonderful coincidence, I see this the morning after I declared freedom:
The wind’s at my back, and I’m feeling pretty good about this.2
The worse-half of my inner-Jobs insists on perfection and never gets anything shipped. The balancing act between better and done is about the only constant in this crazy dream to be a self-sustaining indie dev. ↩︎
Stripe.js integration means you can use a credit card to pay for a premium account.
In order to integrate Stripe.js into NewsBlur, I had to add SSL support, which was not the easiest task. Generating signing requests, signing certifications, and providing additional security is no basic task.
Since you’re reading this, it would be a safe assumption that you have already gone premium. To which, firstly, I thank you. And secondly, if you’re invested in the success of a product, you want to watch it grow. But that means you also will not have a chance to see the new payment form, so here’s the new premium upsell:
And the new payment form:
I’m extraordinarily pleased with how much easier this was relative to setting up PayPal about a year ago for recurring payments. Thanks to Stripe.js for being such a hassle-free payment provider.
Twelve months can be a quick flyby if you don’t stop to write everything down. Luckily, a habit I’ve kept since July 2009, when I started recording monthly goals for my project, is still going strong.
This gives me the opportunity to review everything that we’ve accomplished in the past year. There were numerous new features, about half of which were created as a result of feedback from the community. You asked for it, I found the time.
It’s been a pretty good cycle that looks like it’s going to keep going for years.
Premium accounts are keeping NewsBlur alive
At only a $1/month, nearly 800 people have subscribed to NewsBlur. Not much of it goes back to me, since the servers, between the eight of them, cost $650 every month. But that gap is increasing and my prediction is that these servers should be good for more than a doubling of traffic.
Social is coming
In October Google Reader dropped its social features and there was an enormous influx of new users. Unfortunately, NewsBlur doesn’t have any social features that could be considered a replacement for the community lost in the Reader sunsetting.
But that’s changing soon. There’s only a couple more weeks of development needed before the Social branch is ready for beta testers. And I’ll be taking feedback on the changes, iterating until sharebro communities are able to form. It’ll be a great time, watching a favorite service evolve into a social community.
If you’re a developer, you might enjoy watching the development of NewsBlur on GitHub.
2011 Calendar
The biggest features built this year were the River of News, the iPhone app, and the API. There were tons of other big features, but these three top the list for their importance.
January 2011
River of News - read all stories in a folder
“Show N hidden stories” button
Bookmarklet - browser bookmark that allows you to subscribe to a site from the site itself
Favicon fetching and color decomposing - the color gradient that matches the site’s favicon.
February 2011
New dashboard
Replicated PostgreSQL - a slave relational DB for redundancy
Hide and show story changes - see the story being modified after being published
Graphs on the dashboard
Upgraded to Django 1.3
Feed fetch histories in Site Statistics
May 2011
Send story by email - the original social network
Community feedback on the dashboard
The tutorial - built because blogger Louis Gray didn’t easily grasp how to get started with NewsBlur, and he’s an early adopter, so good luck to any regular users without the help of a tutorial
June 2011
Prototype mobile site - using jQuery UI, was a disaster in terms of performance
MongoDB is now replicated - for higher availability
Lots of backend changes to account for increased load - increased speed with more load using better code
You can follow updates in real-time on Twitter by following both @samuelclay and @newsblur. Lots more to come.
Keep the feedback, ideas, praise, and bugs coming. I couldn’t do this without the external motivation coming from dozens of your voices talking about NewsBlur.
NewsBlur was released exactly one year ago. You can read the initial reaction on Hacker News: http://news.ycombinator.com/item?id=1834305. Since then, so much has changed and all for the better. Usage is up—way, way up. Premium users are helping the site run. Load times are approaching the goal of less than 100 ms (0.10 sec) per page. In short, things couldn’t be better.
Recently, my main competitor, Google Reader, announced plans to decommission the social features of the site and integrate them into Google+. While Google is busy refocusing on what’s important to them, many users feel left behind. My goal with NewsBlur is to make a better, complete experience for reading sites. This includes social features that make reading a social experience.
Social is a major planned feature. It’s highly prioritized right after I build two other big ticket items: mobile and search. The mobile iPhone app is wrapping up and is already at version 1.1 on the App Store, although I have not publicly launched it because it still needs a few more features (specifically, training and the river of news) to be considered feature-complete.
Once that’s out the door, I have to build search to be able to support social. Search won’t be impossible, since the UI design decisions are fairly straight-foward, and the backend is a no-brainer in terms of design. But it’ll take some time to get right, make fast, and get integrated into the massive database that is quickly accumulating.
After that, I have so many social ideas swimming in my head that I’m ready to drown. I’ve had these social features planned since day one. But I never built them because I needed to make a proof-of-concept first, to be sure that social features would be even feasible.
Also, you can’t discount the immense network effort that building social features would bring. I’m a solo indie developer working on this part-time. Having social features would crush my servers if used as much as I think they would. But know that I am planning some wonderful features to be released as soon as the pieces are in place. Look for the first inkling in the new year.
Let me know on Twitter what your ideas of a good social experience are: @samuelclay.
The iPhone app is now only a few days away from launching. But it took 3 weeks of sitting around in the App Store approval queue before getting here. During that time, I started working on the new customizations that folks have been asking for.
The first of these customizations is the story title pane. Most other readers allow you to position it wherever you like, and NewsBlur is now joining them in giving you that freedom.
You can change the positioning under Manage > Preferences.
Left positioning
Top positioning
Bottom positioning
The old tried and true, this is the default and what you’re already used to.
I hope you like these new positions. There are a few other niceties in this release, namely that resizing window panes is now a live operation, showing you the content stretching in realtime.
There’s a lot more customizations coming out soon. Follow me on GitHub to watch them happen: http://github.com/samuelclay.
This Summer is shaping up to be the season for mobile apps. Blar, a new Android client for NewsBlur, has just been released. It’s available on the Android Market here: https://market.android.com/details?id=bitwrit.Blar. It is created by Harris Munir, who you can contact through his site.
In other news, the official NewsBlur iPhone app is nearly finished and is ready to be beta tested. If you are a premium user, email me your UUID and username and I will send you your own free copy of the NewsBlur iPhone app. Just know that it is under heavy development and might change for the better in the near-term future.
Blar was created using the just released NewsBlur API. If you create your own mobile app, bookmarklet, browser extension, or even desktop client, I’ll feature it either on the dashboard of every user or the Goodies section. I’m also happy to assist you in making the app, and even making new API endpoints for you if you wish. Just let me know and keep the NewsBlur apps rolling.
</td> <td>
</td> </tr><tr><td>
</td> <td>
</td> </tr><tr><td>
</td> <td>
</td> </tr></table>
