public-mirrors/NewsBlur-viq
1
0
Fork 0
mirror of https://github.com/viq/NewsBlur.git synced 2025-08-05 16:49:45 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
13131 commits 2 branches 0 tags 863 MiB
Commit graph

93 commits

Author SHA1 Message Date
Samuel Clay
2d6343b7ae Fixing feedfinder beautifulsoup issues, unicode issues in feed, and unicode comparisons issues with page_data (since it wasnt being decompressed when compared). 2016-02-05 14:43:31 -08:00
Samuel Clay
66aadc45a1 Adding soft time limit to celery tasks. 2016-02-01 23:06:04 -08:00
Samuel Clay
4a12c220fe Risky business: now capturing db times (sql/mongo/redis) for 1% of all tasks. Also updating munin to recognize new db profiling times. 2015-12-16 17:31:41 -08:00
Samuel Clay
6a20478a7f Renaming redis feed pool to redis feed update pool in prep for redis feed sub pool. 2015-07-27 18:35:25 -07:00
Samuel Clay
97b0910e67 Breaking up tasking of normal feeds and broken feeds so broken feeds are a less drain on postgresql. 2015-07-23 16:29:47 -07:00
Samuel Clay
374e546175 Merge branch 'master' into saved 
* master:
  Fixing updates_off in tasks.
  Adding updates_off run-time setting for handling large spikes of updates.
  Draining feeds also clears out error feeds.
 2014-05-29 18:38:56 -07:00
Samuel Clay
c20ef4e5d0 Adding updates_off run-time setting for handling large spikes of updates. 2014-05-29 17:53:16 -07:00
Samuel Clay
425775eb0d Adding feed counts for saved stories. Need to show counts next to feeds. 2014-05-28 15:30:30 -07:00
Samuel Clay
ab12b913c3 Correcting user tag order. 2014-05-20 16:00:57 -07:00
Samuel Clay
9173bee736 Moving to a naive story tag counter. Can schedule re-counts if necessary. 2014-04-29 12:36:42 -07:00
Samuel Clay
97bf655f1d Handling up to 10000 feed updates per 5 minutes. 2014-04-21 17:19:50 -07:00
Samuel Clay
4787f70ebb Fixing issue with scheduling feeds. 2014-04-21 16:38:27 -07:00
Samuel Clay
9599cf65f2 Only fetching feeds immediately if necessary. 2014-04-21 16:26:10 -07:00
Samuel Clay
18d750091d Shortening task error restart to 10m from 1h. 2013-05-24 13:18:05 -07:00
Samuel Clay
f6f053b285 Adding duplicate feed address+link truncation. 2013-05-15 15:40:38 -07:00
Samuel Clay
6a3d043ced Upping to 5000 feeds per minute. 2013-04-23 21:17:03 -07:00
Samuel Clay
45e137b22c Upping feed fetches. 2013-04-15 19:26:41 -07:00
Samuel Clay
528c86a1eb Fixing small bug around missing feeds during update pipeline. 2013-04-15 15:16:53 -07:00
Samuel Clay
a6837724e3 Increasing task queue size. 2013-04-08 17:11:40 -07:00
Samuel Clay
46f919f919 Using error_feeds to push broken feeds further out in the pipeline. 2013-04-08 10:50:50 -07:00
Samuel Clay
7fc9b494af Adding error_feeds queue to keep track of broken feeds. 2013-04-08 10:04:40 -07:00
Samuel Clay
d63556bc06 Skipping tasking of feeds when queue is too high. Use the random queue. 2013-04-08 09:49:21 -07:00
Samuel Clay
c41bf80b4e Removing force from new feeds. 2013-04-07 20:22:46 -07:00
Samuel Clay
48c5d4d53e Adding logging, feed cleanup to forced fetch feeds, and expirations on user read stories. 2013-04-07 17:19:59 -07:00
Samuel Clay
1ba468d27d Checking if feed id has changed, then removing from task queue. 2013-04-04 21:57:58 -07:00
Samuel Clay
341db243c3 Better logging for queues. 2013-04-04 20:47:41 -07:00
Samuel Clay
a36a960566 Bumping numbers on task fetching. 2013-04-03 22:59:25 -07:00
Samuel Clay
6b262f8557 Cleaning up counts in munin and work log. 2013-04-03 19:04:30 -07:00
Samuel Clay
4461bb9c99 Removing old inactive feeds refresher. 2013-04-03 17:31:37 -07:00
Samuel Clay
6a2a144747 Requeueing old tasked feeds. 2013-04-03 17:28:51 -07:00
Samuel Clay
8f8c88ca47 Adding tasked_feeds set to handle dropped feeds. 2013-04-03 17:22:45 -07:00
Samuel Clay
e0457bf0d0 Fixing missing feed bug. 2013-04-03 16:48:42 -07:00
Samuel Clay
ba5919d760 Moving task queue from postgres db backed to redis backed. 2013-03-30 19:05:13 -07:00
Samuel Clay
2cc5ab6202 Boosting feed fetching. 2013-03-30 11:39:49 -07:00
Samuel Clay
03c27b42e8 Removing unused feed hash index (keeping the unique constraint). 2013-03-29 22:41:16 -07:00
Samuel Clay
22b2bd0604 Changing logging on feed queueing. 2013-03-29 21:46:04 -07:00
Samuel Clay
ae953ee208 Cleaning up week old feeds. 2013-03-29 16:02:32 -07:00
Samuel Clay
07ebbd63b5 Boosting feed fetching way up for the weekend. 2013-03-29 15:56:43 -07:00
Samuel Clay
aa4bf71b34 Upping feed fetching by quite a bit. 2013-03-29 14:14:29 -07:00
Samuel Clay
e39dee98be Upping feed fetches, slowing down fetch interval for less active feeds, setting premium expire time to one year from most recent payment date (as opposed to borked payment gap logic), and adding logginf or tasking feeds. 2013-03-28 11:16:43 -07:00
Samuel Clay
c9f6012a1e Upping feed fetches. 2013-03-27 21:12:55 -07:00
Samuel Clay
2acb9e1940 Boosting fetches. 2013-03-27 12:58:24 -07:00
Samuel Clay
2a5bd136da Boosting feed fetches. 2013-03-25 13:15:14 -07:00
Samuel Clay
86a64fa74e Boosting feed fetches. 2013-03-22 15:10:21 -07:00
Samuel Clay
c4b6bb6fd8 Cutting number of feeds for fetching until mongo is sharded. 2013-03-21 21:15:18 -07:00
Samuel Clay
5e6b1e5fee Grabbing forced refresh feeds. 2013-03-21 10:57:57 -07:00
Samuel Clay
ca43ea8555 Further pushing down feed fetches to handle load until db is scaled out. 2013-03-20 08:13:07 -07:00
Samuel Clay
7ad6c594a3 Turning down feed fetches. 2013-03-19 17:57:24 -07:00
Samuel Clay
0db41bc853 Further turning down feed fetches until mongo is ready to replicate. 2013-03-19 17:38:36 -07:00
Samuel Clay
db92d3ca10 Ramping down feed fetching until they can be handled. 2013-03-19 15:32:25 -07:00