Commit graph

650 commits

Author SHA1 Message Date
Samuel Clay
0f53acdf6d Merge branch 'master' into circular
* master:
  Fixing stripe reimport
  Upping feed fetching by quite a bit.
  Fixing double counting stripe payment bug by clearing out striep purchase history when inserting.
2013-03-29 14:27:04 -07:00
Samuel Clay
aa4bf71b34 Upping feed fetching by quite a bit. 2013-03-29 14:14:29 -07:00
Samuel Clay
a67df54101 Merge branch 'master' into circular
* master:
  Adding task servers to db firewall.
  Adding redis cache. Returning favicon server.
  Better munin plugins for redis and mongo.
  Removing unused redis keys for shared/comments. Dry-run.
  Adding read preference to migration.
  Migration to create story_hash for missing feeds.
  Making a clearer description of slow feeds.
  Upping feed fetches, slowing down fetch interval for less active feeds, setting premium expire time to one year from most recent payment date (as opposed to borked payment gap logic), and adding logginf or tasking feeds.
  Switching check for HAProxy'ied socket.io to dev instead of www. Thanks @anaconda!
  Sync delay in mongodb and updating story hash migration.
  Upping feed fetches.
  Fix force close when attempting to fetch a user with a null user.id value
  Fix force close updating ProfileActivityFragment if no profile response was received
  Move creating ImageLoader in NewsBlurApplication to onCreate to ensure that base context has been set and prevent force close
  Boosting fetches.
  Fix duplicate requirement

Conflicts:
	fabfile.py
2013-03-28 19:09:14 -07:00
Samuel Clay
fc53550392 Making a clearer description of slow feeds. 2013-03-28 12:17:30 -07:00
Samuel Clay
e39dee98be Upping feed fetches, slowing down fetch interval for less active feeds, setting premium expire time to one year from most recent payment date (as opposed to borked payment gap logic), and adding logginf or tasking feeds. 2013-03-28 11:16:43 -07:00
Samuel Clay
1b80b37dc9 Merge branch 'master' into circular
* master:
  Adding dev and debug to firewall
  Adding hyphens to allowed hosts for blurblogs.
  Adding X-nginx-server and X-gunicorn-server.
  Adding RStats, necessary to remove mongodb analytics.
  Fixing firewall setup.
  Using secondary mongos for feed fetch histories.
  Moving to a single hosts and hosts.yml file, used to control logs, deployment, firewalls, and communication. Awesome!
  Moving hosts to secrets.
  Cache busting feeds on force and 1% of the time.

Conflicts:
	config/nginx.newsblur.conf
2013-03-25 12:47:16 -07:00
Samuel Clay
79200d7bc1 Adding RStats, necessary to remove mongodb analytics. 2013-03-25 11:10:41 -07:00
Samuel Clay
568081eb0b Merge branch 'master' into circular
* master:
  Fixing Google+ sharing.
  64 sites, not 12 sites. Doh!
  Adding expires header to icons.
  Downgrading from Pillow, which is incompatible with PIL.
  Pushing feed history deletion to end of stats collection.
  Logging feed fetch deletions.
  Fixing missing scipy import missing form PILlow.
  Last imports for pillow.
  Boosting feed fetches.
  Upgrading from PIL to Pillow.
  Upgrading from PIL to Pillow.
  Upgrading from PIL to Pillow.
  Adding cached template loader. Adding new mongodb.
  Cutting number of feeds for fetching until mongo is sharded.

Conflicts:
	fabfile.py
2013-03-24 21:13:32 -07:00
Samuel Clay
e91d11e068 Adding cached template loader. Adding new mongodb. 2013-03-22 13:36:32 -07:00
Samuel Clay
1338c69edb Merge branch 'master' into circular
* master:
  Using user's username on signup if no password.
  Using user's username on signup if no password.
  Fixing passwords if left blank.
  Patching Django 1.5 to handle underscores in subdomains.
  Allowed hosts
  Allowed hosts
  If not active subs, turn off real-time.
  If not active subs, turn off real-time.
  Skipping real-time fetch for non-premium active feeds.
  Upgrading django 1.5 manage.py.
  Upgrade to django 1.5 script. Also handling story lookups with objectids.
  Updated Django to always use the latest 1.5 version. Updated django-redis-session to latest version.
  Moving HAProxy's config to secrets so I can set a password.
  Allowing_hosts for django 1.5.
  Upgrading django from 1.3 to 1.5.
  Keep Raven from making settings puke.
  Added django-redis-sessions to requirements.
  Added a check for testing the email address provided during registration and throwing an error if the email address is already associated with an existing account.
  Adding revsys fab command.
  Froze requirements to make installation a bit easier.

Conflicts:
	templates/reader/dashboard.xhtml
	templates/social/social_page.xhtml
2013-03-20 18:57:06 -07:00
Samuel Clay
d8a43d394f Upgrade to django 1.5 script. Also handling story lookups with objectids. 2013-03-20 15:43:35 -07:00
Samuel Clay
b3744e3ac6 Merge branch 'master' into circular
* master:
  Warning on missing maintenance.
  Further pushing out feed updates.
  Using @dcramer's excellent getsentry.com for all exceptions. Turning off exception emails.
  Further pushing down feed fetches to handle load until db is scaled out.
  Scrolling code blocks.
  New task servers.
  Turning down feed fetches.
  If requests.raw won't work, sut fudge it with a StringIO for now.
  Further turning down feed fetches until mongo is ready to replicate.
  Fixing Facebook image calculation.
  Fixing facebook posting.
  Moving shared story unread recalc to background task. Will greatly speed up sharing.
  Ramping down feed fetching until they can be handled.
  Only show time breakdown when > 1 sec.
  Fixed API docs for /reader/river_stories
  make fabfile.py use requirements.txt
  Make sure to send the proper content-type with social feeds
  Restore default whitespace to pre/code blocks
  Phase 2: switching to using story_hash index instead of compound unique story_feed_id/story_guid index.
  document keyboard shortcut to add site/folder

Conflicts:
	apps/social/views.py
2013-03-20 10:49:09 -07:00
Samuel Clay
5eebcfc546 Merge branch 'master' into story_hash_2
# By Samuel Clay (10) and others
* master:
  Further pushing out feed updates.
  Using @dcramer's excellent getsentry.com for all exceptions. Turning off exception emails.
  Further pushing down feed fetches to handle load until db is scaled out.
  Scrolling code blocks.
  New task servers.
  Turning down feed fetches.
  If requests.raw won't work, sut fudge it with a StringIO for now.
  Further turning down feed fetches until mongo is ready to replicate.
  Fixing Facebook image calculation.
  Fixing facebook posting.
  Fixed API docs for /reader/river_stories
  make fabfile.py use requirements.txt
  Make sure to send the proper content-type with social feeds
  Restore default whitespace to pre/code blocks
  document keyboard shortcut to add site/folder
2013-03-20 10:22:30 -07:00
Samuel Clay
d167788f29 Further pushing out feed updates. 2013-03-20 10:04:25 -07:00
Samuel Clay
27a2dfbfa0 Merge branch 'master' into story_hash_2
# By Samuel Clay (70) and adriano petrich (1)
# Via Samuel Clay (3) and adriano petrich (1)
* master: (71 commits)
  Adding new task servers.
  Activating premiums immediately for stripe users.
  Adding X-server to headers to figure out request pinning.
  removing dependency to dop.client for common fab tasks
  Increase file descriptor limits with fabric.
  Opening up gunicorn for use by HAProxy directly.
  Naively connecting to real-time socket.io.
  Preventing socket.io buildups.
  Rewriting HAProxy config to include gunicorn and maintenance. Much better monitoring.
  Fixing HAProxy to use http headers so a 502 error code doesn't get served with a 200 status code. I mean seriously, WTF HAProxy?!
  Trying out a gunicorn haproxy backend.
  Re-upping 12 to 64 free feeds.
  Removing Redis state for socket.io. Too many open mongo connections.
  Referencing new secret hosts.
  Removing hosts.
  Adding dev hosts.
  Using new secrets repo instead of ignored files.
  Fixing homepage copy.
  Using redisstore for socketio.
  Forcing reconnection for socket.io.
  ...
2013-03-18 13:35:17 -07:00
Samuel Clay
7f9893f1f3 Merge branch 'master' into circular
* master:
  Refactoring fabfile for DO
  Moving redis servers for real-time.
  Increasing feed fetches quite a bit.
  More gunicorn servers.
  More gunicorn servers.
  Upping gunicorn processes.
  Hiding 12 option to cut back on number of new premiums.
  Bumping up premium refresh rates.
  Tasking only popular feeds for now.
  Pushing out feed updates until able to handle the giant influx.
2013-03-14 20:38:30 -07:00
Samuel Clay
e73f3aa66d Increasing feed fetches quite a bit. 2013-03-14 19:26:22 -07:00
Samuel Clay
85c85ab565 Pushing out feed updates until able to handle the giant influx. 2013-03-14 15:52:34 -07:00
Samuel Clay
66709d6908 Merge branch 'master' into circular
* master:
  db03 is no longer a db.
  Redis to db01.
  Logging stripe use.
  Paywall on signup.
  Extending random holds for feeds while being crushed.
  Minimum of MEMORY/256 workers.
  Minimum of MEMORY/256 workers.
  Bumping gunicorn workers.
  Only recount feeds if necessary.
  Only recount feeds if necessary.

Conflicts:
	media/css/reader.css
2013-03-14 10:54:12 -07:00
Samuel Clay
9c5c210568 Extending random holds for feeds while being crushed. 2013-03-14 00:47:18 -07:00
Samuel Clay
13e6246860 Merge branch 'master' into circular
* master:
  Counting average stories if stories were detected in the last month. This should fix feeds that have long delays but shouldn't.
  Fixing maintenance mode message.
  Maintenace message.
  First half of story hash migration. Setting up unique sparse index, then migration to calculate hashes for every story.
  Fixing the timeout on premium setups by doing the scheduling of premium setups in a task, only immediately activating all of a user's feeds.
2013-02-25 12:05:50 -08:00
Samuel Clay
4dba6ce380 Counting average stories if stories were detected in the last month. This should fix feeds that have long delays but shouldn't. 2013-02-25 11:11:47 -08:00
Samuel Clay
52205ab89d New dashboard widget: search sites and people 2013-02-22 16:02:51 -08:00
Samuel Clay
a2c1eb8ae3 Phase 2: switching to using story_hash index instead of compound unique story_feed_id/story_guid index. 2013-02-20 16:08:14 -08:00
Samuel Clay
fc8b2bd4ec First half of story hash migration. Setting up unique sparse index, then migration to calculate hashes for every story. 2013-02-20 15:42:40 -08:00
Samuel Clay
4ffb4424a9 Fixing the timeout on premium setups by doing the scheduling of premium setups in a task, only immediately activating all of a user's feeds. 2013-02-15 17:51:37 -08:00
Samuel Clay
3bdb4af70a Fixing the timeout on premium setups by doing the scheduling of premium setups in a task, only immediately activating all of a user's feeds. 2013-02-15 09:52:11 -08:00
Samuel Clay
0627e6fdb2 Merge branch 'master' into circular
* master:
  If there are no unread stories in a folder, don't request river_stories with no feeds, since that will return all unread stories. Fetch all feeds in folder. Thanks to @afita for pushing this.
  Fixing sharing story by email if the story has no feed.
  Fixing sharing story by email if the story has no feed.
  Adding Tafiti, Windows 8 app, to logs.
  1 month max on next fetch.

Conflicts:
	media/js/newsblur/reader/reader.js
2013-02-14 17:21:15 -08:00
Samuel Clay
0d75d9ecd8 1 month max on next fetch. 2013-02-13 12:54:14 -08:00
Samuel Clay
99b1efbb47 Adding stats to filter popover. 2013-02-11 16:07:08 -08:00
Samuel Clay
6c6ad7ed3d If a feed has no stories in the past month, fetch it 1/12 as often. 2013-02-07 15:30:35 -08:00
Samuel Clay
e2d6b8d6f3 Fixing broken branch id in unicode/str for feed. 2013-01-29 15:58:15 -08:00
Samuel Clay
c18c8b54bc Correcting feeds that get merged but are originally branches. 2013-01-29 14:37:01 -08:00
Samuel Clay
79c6ea09cd Don't collide stories that are too short to be collided. 2013-01-28 16:56:01 -08:00
Samuel Clay
7e52fc37ef Fixing assortment of small bugs. 2013-01-28 15:43:00 -08:00
Samuel Clay
36028cc719 Text view. Needs a few tweaks and a premium only view. 2013-01-08 18:33:30 -08:00
Samuel Clay
0b1b3ee9b7 Undoing some broken changes on reading. 2013-01-08 14:22:53 -08:00
Samuel Clay
4dd1d10d5a Adding avatar photo uploading. 2013-01-08 14:11:59 -08:00
Samuel Clay
cb30fe2ae8 Bumping up duplicate address length for feeds. 2013-01-07 16:35:29 -08:00
Samuel Clay
bc0192c3df Adding broken search for feeds. 2013-01-04 16:34:27 -08:00
Samuel Clay
aefd8de78c Backporting collections.Counter to python2.6 2013-01-03 18:15:00 -08:00
Samuel Clay
40fa172edc Prototype of a dupe checker using real-time update times to figure out which feeds are the same. Last time I walked down this road I got run over. 2013-01-03 13:47:38 -08:00
Samuel Clay
5ed2109bcf Using queue for scheduling feed fetches on load. 2013-01-02 12:27:08 -08:00
Samuel Clay
cba0733a28 Attempting to correctly save duplicate feeds. 2012-12-28 22:03:48 -08:00
Samuel Clay
db63f48bf5 Adding tons of logging (especially user-facing) to feed exception handling. Much much better feed address and link fetch history. 2012-12-28 21:42:52 -08:00
Samuel Clay
70f2da6218 Reducing queue size of push feeds when overloaded. 2012-12-26 02:41:13 -08:00
Samuel Clay
1ca58ed515 Showing maintenance mode message. One day ... put in a live chat. 2012-12-25 15:28:13 -08:00
Samuel Clay
62d210ffd6 Migration to increase max length of feed address. 2012-12-25 14:58:59 -08:00
Samuel Clay
f94939b83e Cleaning up task feeds logging. 2012-12-25 12:08:17 -08:00
Samuel Clay
6311bf41d7 Updating sessions handling on page importer in anticipation of ugprading to latest python requests. 2012-12-24 00:39:52 -08:00