Samuel Clay
97b0910e67
Breaking up tasking of normal feeds and broken feeds so broken feeds are a less drain on postgresql.
2015-07-23 16:29:47 -07:00
Samuel Clay
da26ecdb46
1/2 Cleaning up feed fetcher's updates to feeds table. Still need to find 6 dupes.
2015-07-22 13:18:54 -07:00
Samuel Clay
c20ef4e5d0
Adding updates_off run-time setting for handling large spikes of updates.
2014-05-29 17:53:16 -07:00
Samuel Clay
3f50a1b311
Adding stories to search index.
2014-04-11 17:25:13 -07:00
Samuel Clay
073098750c
Swapping out haystack for pyes.
2014-04-11 15:40:58 -07:00
Samuel Clay
1584208677
Adding metric for last story date, used to calculate next scheduled update.
2013-04-23 17:04:21 -07:00
Samuel Clay
bd9f992262
Fixing credit card images. Doh\!
2013-04-19 09:51:30 -07:00
Samuel Clay
c41bf80b4e
Removing force from new feeds.
2013-04-07 20:22:46 -07:00
Samuel Clay
edc3a80667
Adding backup mongo task to be run by cron instead.
2012-12-10 11:37:35 -08:00
Samuel Clay
c19713eabf
Adding offsets to unread count fixing.
2012-10-30 11:41:01 -07:00
Samuel Clay
769a1e084e
Manually counting can now use offsets in case of broken feeds.
2012-10-30 09:11:39 -07:00
Samuel Clay
e572b3d53f
Moving to better feed finder (in the case of dupe feeds).
2012-10-25 16:14:25 -07:00
Samuel Clay
f0dc76b71e
Merging separate task feeds tasks.
2012-08-09 21:18:45 -07:00
Samuel Clay
34ad54e1f5
Fixing feed update times for inactive feeds to never go inadctive but to instead go into geometrically longer feed fetches.
2012-08-09 13:26:52 -07:00
Samuel Clay
e40b93882a
Fixing issue around feeds thinking they are unfetched.
2012-06-18 15:59:31 -07:00
Samuel Clay
b5c5cdeda3
Merge branch 'master' into social
...
* master:
Fixing icon fetching error when the page is corrupted.
Writing a glorious hack that forces a count of unread scores.
Conflicts:
apps/reader/models.py
apps/rss_feeds/management/commands/refresh_feeds.py
fabfile.py
utils/feed_fetcher.py
2012-04-24 17:55:20 -07:00
Samuel Clay
c1b5aef983
Fixing icon fetching error when the page is corrupted.
2012-04-24 17:40:34 -07:00
Samuel Clay
d209aa5e25
Writing a glorious hack that forces a count of unread scores.
2012-04-24 17:02:55 -07:00
Samuel Clay
2200aa282a
Subscribe to blog during intro.
2012-03-19 15:46:59 -07:00
Samuel Clay
d901ccdae1
Switching to redis to feed queuing. Also adding a weight for skipping feed fetches under high load.
2012-02-26 22:33:06 -08:00
Samuel Clay
cbc8df2c1c
Skipping feed resurrection.
2012-02-24 17:07:42 -08:00
Samuel Clay
ce536200f6
Adding ability to wind down feed queue by faking requests.
2012-02-24 16:43:08 -08:00
Samuel Clay
c9cb7dcbc0
Adding new feeds db column 'known_good', for forcing updates of feeds that were once good but are now throwing 500s.
2012-02-01 17:59:46 -08:00
Samuel Clay
b5efad9142
Merge branch 'master' into requests
...
* master: (66 commits)
Too many stories? Don't animate transitions when switching intelligence levels.
Fixing typo in iphone app that preventing mark folder as read when the visible stories option was showing.
Updating the mongo db copy util to also look for updated stories.
Globals.is_staff on user, not profile. Make the stats referesh every minute instead of every 10 minutes for staff.
Fixing offsets in river of news.
Adding ciphering to usernames in log, so I can finally take screenshots of the most colorful logs of all time.
Adding my very special mongo serialization backup utility to get around the damn unrepairable mongo database. This is taking 14 hours to run.
Auto-refreshing feedback (1 min for staff, 10 min for everybody).
Fixing exception around multiple feeds during Google Reader import process. Also switching rate limit to status code 429.
Adding Nokia MeeGo client to user agents.
Only show raw feeds in feed autocomplete.
Handling iphone version for new users on ios app.
FIXING THE WORST BUG OF MY LIFE -- finally figured out what was causing the story-shows-as-unread bug. Also fixed enclosures on certain types of feeds.
Fixing menu manage open position to compensate for additional menu items.
Reducing the amount of work done by feed fetching when there are no new stories.
Fixing emergency bug around trimming feeds where the cursor is changing. Dammit mongo.
Simplifying ufw ports in fabfile. Adding env.user.
Launching the iPhone app on the front-page. Big deal.
Minor cleanup of river stories view.
Cleaning up mongoengine imports and settings for default MongoDB.
...
Conflicts:
local_settings.py.template
2011-12-20 21:47:55 -08:00
Samuel Clay
f3d2f8e5b5
Moving from slave_db to ReplicaSetConnection in pymongo 2.1.
2011-12-11 10:54:07 -08:00
Samuel Clay
b112717d95
Revert "Revert "Switching to requests from urllib2/httplib. Bring on the page errors.""
...
This reverts commit a827b2c864
2011-11-27 02:41:12 -05:00
Samuel Clay
a827b2c864
Revert "Switching to requests from urllib2/httplib. Bring on the page errors."
...
This reverts commit 617dc3bb56
2011-11-27 02:40:44 -05:00
Samuel Clay
d07b03a7a4
Revert "Revert "Switching to requests from urllib2/httplib. Bring on the page errors.""
...
This reverts commit 7972d0bb5c
2011-11-26 11:15:01 -05:00
Samuel Clay
7972d0bb5c
Revert "Switching to requests from urllib2/httplib. Bring on the page errors."
...
This reverts commit 617dc3bb56
2011-11-26 02:16:14 -05:00
Samuel Clay
617dc3bb56
Switching to requests from urllib2/httplib. Bring on the page errors.
2011-11-26 02:07:31 -05:00
Samuel Clay
ac3609bf7d
Merge branch 'master' into realtime
...
* master:
Adding 'E' as a keyboard shortcut for the Everything view.
Doubling the amount Space pages in a story.
Auto-linkify urls in stories.
Reading stories from slave db during feed update.
Conflicts:
settings.py
2011-11-06 15:28:37 -08:00
Samuel Clay
07fa01ed9a
Adding server half of redis real-time unread count updates. That was the easy part.
2011-11-05 17:08:31 -07:00
Samuel Clay
040b755694
Reading stories from slave db during feed update.
2011-11-04 09:45:10 -07:00
Samuel Clay
c2b1d7eb35
Fixing the infernal FeedNotFound error whenever a dupe feed is created in the middle of another transaction. Yarr!
2011-05-19 19:04:10 -04:00
Samuel Clay
8e2936fc1b
Utils -> vendor
2011-04-11 21:57:45 -04:00
Samuel Clay
1254d350d6
Only tasking feeds that are active. Clears the exception pile.
2011-04-07 12:36:38 -04:00
Samuel Clay
bdc8e7bdea
Separating nginx configuration into a setup and a configure, since one half is much more common than the other.
2011-03-31 18:51:23 -04:00
Samuel Clay
09021ea08c
Trying to fix stale feeds by checking for a last update of a day.
2011-02-22 09:37:09 -05:00
Samuel Clay
3723160155
A huge number of misc fixes: fixing potential errors on paypal return screen, mailing all feed/page parse errors, and swithcing to mapreduce item_frequencies in mongoengine.
2011-02-05 15:34:43 -05:00
Samuel Clay
e14ba054ad
Fixing up trim feeds so we can trim old unused feeds.
2011-01-22 11:32:49 -05:00
Samuel Clay
443fe69156
Moving text fields from Feed to FeedData. Migration includes deletion of old fields on Feed, now unused.
2011-01-17 22:48:38 -05:00
Samuel Clay
4f35c2fade
Making feed.min_to_decay reflect the actual decay time. Using it in statistics, and refreshing it in feed fetching. Also splitting up feed munin.
2010-12-23 13:29:31 -05:00
Samuel Clay
e039d9a1bd
Queueing fallen-through-the-cracks feeds appropriately, giving a day of leniency.
2010-12-23 12:39:28 -05:00
Samuel Clay
b98c63dd14
Abstracting feed queueing so I can queue a queryset of feeds from the command line. Also adding in queueing of feeds that have not been updated in a week (for those that fall through the cracks.
2010-12-23 12:32:24 -05:00
Samuel Clay
97ad4838f1
Fixing stories in tabs, and adding option to specify username for refresh_feeds, to assist in development.
2010-12-14 23:10:13 -05:00
Samuel Clay
fa663e910a
Faster feed refreshing (insta-fetching) by only computing the score for the current user and leaving the other users with dirty subscriptions.
2010-11-10 18:04:17 -05:00
Samuel Clay
cbb04ba6ef
Adding a few keybaord shortcuts, and adding duplicate feed counts to munin.
2010-10-12 13:17:43 -04:00
Samuel Clay
4cf9b05f0c
Changing all datetime.datetime.now() to datetime.datetime.utcnow().
2010-10-10 23:55:00 -04:00
Samuel Clay
50255cf928
Using UTCnow() for dates in marking read stories. Off by 4 hours right now.
2010-10-10 23:51:31 -04:00
Samuel Clay
50488db7ca
Close the connection for every set of tasks.
2010-10-08 16:08:41 -04:00
