Commit graph

91 commits

Author SHA1 Message Date
Samuel Clay
0a608abb87 Handling broken page importer. 2020-05-04 09:50:01 -04:00
Samuel Clay
cef29f0165 Adding royalroad.com to excluded original page sites. 2020-02-27 10:23:23 -05:00
Samuel Clay
01f092e662 Handle no page 2019-12-21 10:21:27 -05:00
Samuel Clay
3780cddb47 Handle no page 2019-12-21 10:20:59 -05:00
Samuel Clay
ec35f2156a Don't bother decoding latin1 for pages. 2019-12-21 10:04:58 -05:00
Samuel Clay
4290b57370 Better handling of broken original text. 2018-07-16 10:55:17 -04:00
Samuel Clay
2b83ccabc1 Handling missig feed on page importer. 2018-07-16 10:50:29 -04:00
Samuel Clay
0d6d4de92e Removing ESPN from original pages. 2017-11-02 16:39:36 -07:00
Samuel Clay
461c1c4b65 Changing feed log format to include id at the beginning. Also normalizing all feed titles in logs for better searchability. 2017-03-31 19:52:24 -07:00
Samuel Clay
440d2c5841 Fixing broken S3 names. 2017-03-28 21:17:11 -07:00
Samuel Clay
9f15958c0e Only connect to S3 when needed. 2017-03-28 10:43:31 -07:00
Samuel Clay
c4830e3e95 Handling unicode encode errors in page/text handling. Also adding upgrade command for fabric when pip is non-trivial. 2016-12-05 22:09:05 -08:00
Samuel Clay
3ed96e338c Fixing page and text importer to correctly handling non-breaking spaces. 2016-12-05 17:40:39 -08:00
Samuel Clay
3a93ad7248 Fixing original page server. 2016-11-30 17:17:40 -08:00
Samuel Clay
2d6343b7ae Fixing feedfinder beautifulsoup issues, unicode issues in feed, and unicode comparisons issues with page_data (since it wasnt being decompressed when compared). 2016-02-05 14:43:31 -08:00
Samuel Clay
b6e1e03bd5 Upgrading feedparser, cleaing up logging on feed fetch. 2015-08-24 14:27:01 -07:00
Samuel Clay
22ae0e65e4 Removing gamespot.com feed from page and text fetchers. 2015-08-03 20:09:36 -07:00
Samuel Clay
b27590506c Fixing notuniqueerrors. 2015-07-07 13:34:06 -07:00
Samuel Clay
5001ad7602 Handling timeout on original page fetch. 2015-03-11 11:50:46 -07:00
Samuel Clay
02c91bb8b0 Fixing timeout error on PUSH. Also fixing requests error on page importing. 2015-03-11 11:44:47 -07:00
Samuel Clay
28f0860fff Fixing issue when resizing window causing story progress bar to resurface. 2014-10-30 13:24:24 -07:00
Samuel Clay
b91b65772a Adding proxy for Story view to handle https loading http sites. Needs testing, but done. 2014-10-29 16:16:50 -07:00
Samuel Clay
5e064fcc2e Handling PyAsn1Error 2014-05-27 13:08:21 -07:00
Samuel Clay
def51705c6 Handling the many OpenSSL errors. 2014-05-22 15:10:29 -07:00
Samuel Clay
0db2a1d110 Revert "Ripping out SNI/openssl/requests latest. Turns out pyOpenSSL barfs on a ton of pages."
This reverts commit 177b8aea09.
2014-05-22 15:09:08 -07:00
Samuel Clay
177b8aea09 Ripping out SNI/openssl/requests latest. Turns out pyOpenSSL barfs on a ton of pages. 2014-05-22 15:06:16 -07:00
Samuel Clay
c26086b2f6 Explicitly closing connection on page fetch. 2014-05-22 14:54:15 -07:00
Samuel Clay
889be221d5 Fixing broken original pages by auto-fixing the has_page field on successful page fetch. Thanks to The Verge for uncovering this one. 2013-07-23 10:43:28 -07:00
Samuel Clay
1936515aa0 Fixing Dilbert-specific encoding error. 2013-07-18 15:17:15 -07:00
Samuel Clay
a36955ee32 Fixing encoding on Original pages. Thanks @bruceyou for the wonderful help! 2013-07-18 15:03:12 -07:00
Samuel Clay
5f7ae25fc4 Normaling user agents between feed/page/icon fetchers. Adding full site permalink. 2013-07-10 14:14:55 -07:00
Samuel Clay
275b0ff9e4 Handling socket error on page fetch. 2013-05-29 15:59:54 -07:00
Samuel Clay
b3744e3ac6 Merge branch 'master' into circular
* master:
  Warning on missing maintenance.
  Further pushing out feed updates.
  Using @dcramer's excellent getsentry.com for all exceptions. Turning off exception emails.
  Further pushing down feed fetches to handle load until db is scaled out.
  Scrolling code blocks.
  New task servers.
  Turning down feed fetches.
  If requests.raw won't work, sut fudge it with a StringIO for now.
  Further turning down feed fetches until mongo is ready to replicate.
  Fixing Facebook image calculation.
  Fixing facebook posting.
  Moving shared story unread recalc to background task. Will greatly speed up sharing.
  Ramping down feed fetching until they can be handled.
  Only show time breakdown when > 1 sec.
  Fixed API docs for /reader/river_stories
  make fabfile.py use requirements.txt
  Make sure to send the proper content-type with social feeds
  Restore default whitespace to pre/code blocks
  Phase 2: switching to using story_hash index instead of compound unique story_feed_id/story_guid index.
  document keyboard shortcut to add site/folder

Conflicts:
	apps/social/views.py
2013-03-20 10:49:09 -07:00
Samuel Clay
a75554374b Using @dcramer's excellent getsentry.com for all exceptions. Turning off exception emails. 2013-03-20 08:24:11 -07:00
Samuel Clay
a3383e1bd6 New icons. 2013-03-02 11:18:12 -08:00
Samuel Clay
ce5663b20b Preparing original pages node server for launch. 2013-01-07 11:16:52 -08:00
Samuel Clay
a90518fd31 Merge branch 'master' into original_pages_node
* master: (81 commits)
  Adding font size choices to web preferences.
  Closing #66 with both integration of #68 and using the same fonts. Users can add an optional class if they want these other convenient fonts. Also using a few fonts as backups for non-Mac users.
  Fix default theming hook
  Add multiple targeted font stacks
  Fixing #75: shared stories should use story permalink, not story guid. Doh. Thanks @denubis!
  Categorizing preferences. Adding window title count back in.
  Adding email lookup to forgot password flow.
  Adding email lookup to forgot password flow.
  Fixing typo in logging.
  Fixing autocomplete behavior on add dialog.
  Adding broken search for feeds.
  Logging
  Switching to guid-based read story counts from range based counts. Let's see what this screws up.
  Everybody gets unread count in title.
  Backporting collections.Counter to python2.6
  Fixing broken reply interaction with links.
  Prototype of a dupe checker using real-time update times to figure out which feeds are the same. Last time I walked down this road I got run over.
  Adding delete user flow. About time, since I'm sick of doing this myself.
  Deractivating premiums.
  Adding a lock for the intelligence slider for focus stories. If the user selects focus and there are actually focus stories, lock it for the future when there are no unread focus stories.
  ...

Conflicts:
	fabfile.py
2013-01-07 10:48:42 -08:00
Samuel Clay
17899d0e2a Adding a lock for the intelligence slider for focus stories. If the user selects focus and there are actually focus stories, lock it for the future when there are no unread focus stories. 2013-01-02 18:40:12 -08:00
Samuel Clay
6311bf41d7 Updating sessions handling on page importer in anticipation of ugprading to latest python requests. 2012-12-24 00:39:52 -08:00
Samuel Clay
9f896d0493 Removing unused original pages from S3 when stored on node server. 2012-12-17 18:42:55 -08:00
Samuel Clay
e4fa40b428 Refactoring original page saving toa ccount for node server being down. 2012-12-17 18:30:14 -08:00
Samuel Clay
a00c462457 Moving original pages to a node server from s3. Still needs conversion from s3. 2012-12-14 16:28:06 -08:00
Samuel Clay
2b5a13ce8c Allowing S3 to be turned off and on again. 2012-12-10 11:13:45 -08:00
Samuel Clay
4a7516cb83 Fixing a bunch of feed fetch errors. 2012-10-01 19:31:33 -07:00
Samuel Clay
79d07cac05 Fixing JS issues around stories with no feeds clobbering the river view. 2012-09-27 10:44:55 -07:00
Samuel Clay
db3b2ae4d6 Removing old feed pages from mongo. 2012-09-19 12:36:50 -07:00
Samuel Clay
00ba259c66 Migrating to S3 for feed pages. 2012-09-18 17:09:07 -07:00
Samuel Clay
f4add87dc6 Fixing a couple of update bugs -- story cascading potentially resulting in invalid ids (probably not), and not unique errors on new page fetches. 2012-09-17 17:01:56 -07:00
Samuel Clay
6e9c604241 Fixing bug where publisher-disabled original pages were not properly disabled. 2012-09-05 12:18:06 -07:00
Samuel Clay
15df6b6b47 Making the icon importer a bit faster and more slave lag resilient by using page fetch data. Also fixing title on blurblogs. 2012-07-30 11:12:58 -07:00