public-mirrors/NewsBlur
1
0
Fork 0
mirror of https://github.com/samuelclay/NewsBlur.git synced 2025-08-05 16:58:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
5245 commits 16 branches 12 tags 925 MiB
Commit graph

668 commits

Author SHA1 Message Date
Samuel Clay
c25c4478e6 Adding new autocomplete. Let's hope this doesn't destroy performance in Redis. 2013-04-08 12:54:02 -07:00
Samuel Clay
46f919f919 Using error_feeds to push broken feeds further out in the pipeline. 2013-04-08 10:50:50 -07:00
Samuel Clay
d63556bc06 Skipping tasking of feeds when queue is too high. Use the random queue. 2013-04-08 09:49:21 -07:00
Samuel Clay
00591ee7ad Pushing out feed updates to cope with influx. 2013-04-08 08:34:12 -07:00
Samuel Clay
5c0b9de511 Increasing fetches for long tail feeds. 2013-04-07 17:31:43 -07:00
Samuel Clay
48c5d4d53e Adding logging, feed cleanup to forced fetch feeds, and expirations on user read stories. 2013-04-07 17:19:59 -07:00
Samuel Clay
d238dafba7 Bumping up feed fetching for premiums. 2013-04-05 17:59:33 -07:00
Samuel Clay
9c79076aab Better verbose logging for updating feeds. 2013-04-05 12:09:42 -07:00
Samuel Clay
1ba468d27d Checking if feed id has changed, then removing from task queue. 2013-04-04 21:57:58 -07:00
Samuel Clay
d831088a46 Fixed the ridiculous bug with feed queuing: while checking to see if the feed id changed, self.pk changed along with the feed, resulting in the cached value changing! 2013-04-04 20:51:56 -07:00
Samuel Clay
341db243c3 Better logging for queues. 2013-04-04 20:47:41 -07:00
Samuel Clay
681968033b Fixing broken JPG image sizer. 2013-04-04 17:49:38 -07:00
Samuel Clay
ee647be85b Handling timeouts for feed link change. 2013-04-04 17:27:31 -07:00
Samuel Clay
3d190cc3ec Fixing a few small feed collision errors. 2013-04-04 17:18:27 -07:00
Samuel Clay
e7ad04946d Fixing feed address change by feed link when colliding with existing feed. 2013-04-04 17:13:06 -07:00
Samuel Clay
8a7ea248fd Fixing feed address change by feed link when colliding with existing feed. 2013-04-04 17:09:16 -07:00
Samuel Clay
95f97a4718 Adding logging. 2013-04-03 22:51:03 -07:00
Samuel Clay
400cad89e6 Tossing deleted feeds from queue. 2013-04-03 21:11:40 -07:00
Samuel Clay
3627221968 Tossing deleted feeds from queue. 2013-04-03 21:10:37 -07:00
Samuel Clay
4651906f08 What the f? strip_comments now eats CPU on bad stories. But why now and not a few months ago when I introduced it? 2013-04-03 18:20:42 -07:00
Samuel Clay
7a41b6a949 Using proper hostname for task server. Adding verbosity to feed fetching. 2013-04-03 18:11:23 -07:00
Samuel Clay
8f8c88ca47 Adding tasked_feeds set to handle dropped feeds. 2013-04-03 17:22:45 -07:00
Samuel Clay
842afe7068 Handling decayed feeds with no more active subscribers. They don't get returned to the scheduled_updates queue until there are enough active users. 2013-04-01 15:45:22 -07:00
Samuel Clay
ba5919d760 Moving task queue from postgres db backed to redis backed. 2013-03-30 19:05:13 -07:00
Samuel Clay
2cc5ab6202 Boosting feed fetching. 2013-03-30 11:39:49 -07:00
Samuel Clay
03c27b42e8 Removing unused feed hash index (keeping the unique constraint). 2013-03-29 22:41:16 -07:00
Samuel Clay
33bd64f2a0 Adding two indexes and killing one unused one from feeds. 2013-03-29 22:11:36 -07:00
Samuel Clay
93794aef36 Returning index on active subs. 2013-03-29 21:48:38 -07:00
Samuel Clay
22b2bd0604 Changing logging on feed queueing. 2013-03-29 21:46:04 -07:00
Samuel Clay
a47fa4db0f Removing unused indexes. 2013-03-29 20:24:06 -07:00
Samuel Clay
07ebbd63b5 Boosting feed fetching way up for the weekend. 2013-03-29 15:56:43 -07:00
Samuel Clay
aa4bf71b34 Upping feed fetching by quite a bit. 2013-03-29 14:14:29 -07:00
Samuel Clay
fc53550392 Making a clearer description of slow feeds. 2013-03-28 12:17:30 -07:00
Samuel Clay
e39dee98be Upping feed fetches, slowing down fetch interval for less active feeds, setting premium expire time to one year from most recent payment date (as opposed to borked payment gap logic), and adding logginf or tasking feeds. 2013-03-28 11:16:43 -07:00
Samuel Clay
79200d7bc1 Adding RStats, necessary to remove mongodb analytics. 2013-03-25 11:10:41 -07:00
Samuel Clay
e91d11e068 Adding cached template loader. Adding new mongodb. 2013-03-22 13:36:32 -07:00
Samuel Clay
d8a43d394f Upgrade to django 1.5 script. Also handling story lookups with objectids. 2013-03-20 15:43:35 -07:00
Samuel Clay
5eebcfc546 Merge branch 'master' into story_hash_2 
# By Samuel Clay (10) and others
* master:
  Further pushing out feed updates.
  Using @dcramer's excellent getsentry.com for all exceptions. Turning off exception emails.
  Further pushing down feed fetches to handle load until db is scaled out.
  Scrolling code blocks.
  New task servers.
  Turning down feed fetches.
  If requests.raw won't work, sut fudge it with a StringIO for now.
  Further turning down feed fetches until mongo is ready to replicate.
  Fixing Facebook image calculation.
  Fixing facebook posting.
  Fixed API docs for /reader/river_stories
  make fabfile.py use requirements.txt
  Make sure to send the proper content-type with social feeds
  Restore default whitespace to pre/code blocks
  document keyboard shortcut to add site/folder
 2013-03-20 10:22:30 -07:00
Samuel Clay
d167788f29 Further pushing out feed updates. 2013-03-20 10:04:25 -07:00
Samuel Clay
27a2dfbfa0 Merge branch 'master' into story_hash_2 
# By Samuel Clay (70) and adriano petrich (1)
# Via Samuel Clay (3) and adriano petrich (1)
* master: (71 commits)
  Adding new task servers.
  Activating premiums immediately for stripe users.
  Adding X-server to headers to figure out request pinning.
  removing dependency to dop.client for common fab tasks
  Increase file descriptor limits with fabric.
  Opening up gunicorn for use by HAProxy directly.
  Naively connecting to real-time socket.io.
  Preventing socket.io buildups.
  Rewriting HAProxy config to include gunicorn and maintenance. Much better monitoring.
  Fixing HAProxy to use http headers so a 502 error code doesn't get served with a 200 status code. I mean seriously, WTF HAProxy?!
  Trying out a gunicorn haproxy backend.
  Re-upping 12 to 64 free feeds.
  Removing Redis state for socket.io. Too many open mongo connections.
  Referencing new secret hosts.
  Removing hosts.
  Adding dev hosts.
  Using new secrets repo instead of ignored files.
  Fixing homepage copy.
  Using redisstore for socketio.
  Forcing reconnection for socket.io.
  ...
 2013-03-18 13:35:17 -07:00
Samuel Clay
e73f3aa66d Increasing feed fetches quite a bit. 2013-03-14 19:26:22 -07:00
Samuel Clay
85c85ab565 Pushing out feed updates until able to handle the giant influx. 2013-03-14 15:52:34 -07:00
Samuel Clay
9c5c210568 Extending random holds for feeds while being crushed. 2013-03-14 00:47:18 -07:00
Samuel Clay
4dba6ce380 Counting average stories if stories were detected in the last month. This should fix feeds that have long delays but shouldn't. 2013-02-25 11:11:47 -08:00
Samuel Clay
a2c1eb8ae3 Phase 2: switching to using story_hash index instead of compound unique story_feed_id/story_guid index. 2013-02-20 16:08:14 -08:00
Samuel Clay
fc8b2bd4ec First half of story hash migration. Setting up unique sparse index, then migration to calculate hashes for every story. 2013-02-20 15:42:40 -08:00
Samuel Clay
3bdb4af70a Fixing the timeout on premium setups by doing the scheduling of premium setups in a task, only immediately activating all of a user's feeds. 2013-02-15 09:52:11 -08:00
Samuel Clay
0d75d9ecd8 1 month max on next fetch. 2013-02-13 12:54:14 -08:00
Samuel Clay
6c6ad7ed3d If a feed has no stories in the past month, fetch it 1/12 as often. 2013-02-07 15:30:35 -08:00
Samuel Clay
e2d6b8d6f3 Fixing broken branch id in unicode/str for feed. 2013-01-29 15:58:15 -08:00