Commit graph

392 commits

Author SHA1 Message Date
Samuel Clay
0fee7af7c6 Moving to stricter story diff check. Using story hashes instead of grab bag by date when checking existing stories. 2014-04-17 12:10:04 -07:00
Samuel Clay
a4efcde236 Allowing feeds to fail fetch. 2014-03-29 17:14:10 -07:00
Samuel Clay
534e16fb84 Handling odd compressed stream error. 2014-03-13 16:53:20 -07:00
Samuel Clay
bb3e0a6567 Putting Google Alerts/News back into rotation. 2013-09-30 12:40:14 -07:00
Samuel Clay
70ea542481 User-dependent days of unread. 14 for free users, 30 for premium. 2013-09-16 16:42:49 -07:00
Samuel Clay
831a771309 Fixing logging message. 2013-09-16 12:11:26 -07:00
Samuel Clay
886f6c0d8b Fixing logging message. 2013-09-16 12:08:51 -07:00
Samuel Clay
5911acaa35 Moving the majority of story fetches to mongo secondaries. 2013-09-16 12:07:15 -07:00
Samuel Clay
143ee770ce Skipping google news feeds again. Need to tone down these fetches. 2013-09-15 23:41:17 -07:00
Samuel Clay
3fc40acd47 Removing double. 2013-09-13 19:10:41 -07:00
Samuel Clay
2fe65adc4d Re-enabling Google News fetches. 2013-09-11 10:50:20 -07:00
Samuel Clay
00a412960a Skipping google alerts. 2013-09-09 13:50:05 -07:00
Samuel Clay
2b2c731bd5 Fixing a couple exceptions and turning off google news rss feeds temporarily. 2013-09-09 13:46:14 -07:00
Samuel Clay
dec22ac776 Moving to simpler read story endpoint. 2013-09-06 12:42:39 -07:00
Samuel Clay
2394e31f89 Moving feed parsing hotspot to specific seq ratio check. 2013-08-06 13:54:06 -07:00
Samuel Clay
ecf11eed98 Upping process feed time limit. 2013-08-06 13:42:53 -07:00
Samuel Clay
683140af2e Moving feed processing to timelimit due to some feeds sucking up all the time. 2013-08-06 13:38:35 -07:00
Samuel Clay
88f2a69a93 Fixing a dozen text and feed fetching bugs. 2013-08-06 13:18:55 -07:00
Samuel Clay
d49e9569a5 Fixing #377. Killing XSS attacks by sanitizing feed titles. 2013-07-29 14:25:43 -07:00
Samuel Clay
ee5c00bce5 Using latest feedparser. 2013-07-10 16:10:19 -07:00
Samuel Clay
cf36df47fa Bumping feed blast limit from 50 to 100 stories at a time. 2013-07-10 15:13:19 -07:00
Samuel Clay
5f7ae25fc4 Normaling user agents between feed/page/icon fetchers. Adding full site permalink. 2013-07-10 14:14:55 -07:00
Samuel Clay
6fda0b6a35 Timing out on PuSH hubs when down. 2013-07-01 18:46:21 -07:00
Samuel Clay
9826a6499b Moving all pubsub to a dedicated redis box (called REDIS2). 2013-06-18 12:21:27 -07:00
Samuel Clay
e3f2677ee7 Upgrading mongoengine prep. 2013-05-29 18:00:09 -07:00
Samuel Clay
b98d129d1b Removing MUserStory entirely. 2013-05-10 16:11:30 -07:00
Samuel Clay
550a33d9ab Using story cache when calculating unreads, for those who need unread counts updating. 2013-05-03 09:43:21 -07:00
Samuel Clay
6011e4f2da Moving from feed/page/push histories to capped fetch history. 2013-04-15 14:30:31 -07:00
Samuel Clay
532d1a67a7 No longer doing overly zealous read story cleanup. 2013-04-08 11:47:39 -07:00
Samuel Clay
36e76759e8 Adding in timing on feed cleanup. 2013-04-07 17:49:34 -07:00
Samuel Clay
48c5d4d53e Adding logging, feed cleanup to forced fetch feeds, and expirations on user read stories. 2013-04-07 17:19:59 -07:00
Samuel Clay
f2281ec7bc Refunding stripe payments. 2013-04-05 17:54:22 -07:00
Samuel Clay
3d190cc3ec Fixing a few small feed collision errors. 2013-04-04 17:18:27 -07:00
Samuel Clay
8a7ea248fd Fixing feed address change by feed link when colliding with existing feed. 2013-04-04 17:09:16 -07:00
Samuel Clay
5898ea9df0 Feed may not longer exist. Toss it. 2013-04-03 21:20:19 -07:00
Samuel Clay
9e2fa3115d Cache busting feeds on force and 1% of the time. 2013-03-24 15:50:57 -07:00
Samuel Clay
d793946e3a If not active subs, turn off real-time. 2013-03-20 16:17:20 -07:00
Samuel Clay
a75554374b Using @dcramer's excellent getsentry.com for all exceptions. Turning off exception emails. 2013-03-20 08:24:11 -07:00
Samuel Clay
e4fc8b6c49 Turning off PuSH if no hub url found. 2013-03-07 10:58:05 -05:00
Samuel Clay
bdd60bf3f7 Fixing push re-sub logging. 2013-03-06 14:47:15 -08:00
Samuel Clay
00f59821be Resubscribing to PuSH feeds that have expired. 2013-03-06 14:45:01 -08:00
Samuel Clay
5170b56689 If last_modified fails, clear it. 2013-01-31 12:21:10 -08:00
Samuel Clay
79c6ea09cd Don't collide stories that are too short to be collided. 2013-01-28 16:56:01 -08:00
Samuel Clay
4e09e4db81 Fixing user agent. 2013-01-22 18:18:07 -08:00
Samuel Clay
bcc24c1253 Fixing PuSH by using the correct hub url. Thanks to @donmelton for finding the bug. 2013-01-22 18:15:28 -08:00
Samuel Clay
cb30fe2ae8 Bumping up duplicate address length for feeds. 2013-01-07 16:35:29 -08:00
Samuel Clay
1cc128b5a2 Doubling resync likelihood. 2012-12-17 18:20:26 -08:00
Samuel Clay
53b5c4e2db Clearing out old read stories for 1% of all feed fetches with new stories. 2012-12-17 17:28:45 -08:00
Samuel Clay
5a5aa0d17b Turning off deletion of old stories until the db can be prepped. 2012-12-17 16:19:28 -08:00
Samuel Clay
28d5f44182 Deleting old stories. 2012-12-17 15:48:43 -08:00