public-mirrors/NewsBlur-viq
1
0
Fork 0
mirror of https://github.com/viq/NewsBlur.git synced 2025-08-21 05:44:54 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
17114 commits 2 branches 0 tags 863 MiB
Commit graph

104 commits

Author SHA1 Message Date
Samuel Clay
d1dafe7606 Black formatting. 2024-04-24 09:43:56 -04:00
Samuel Clay
1f5e9053dd Handling too many feeds tasked. 2024-02-28 09:33:10 -05:00
Samuel Clay
92c117bb9f Spit out feeds being fetched to spot an errant one. 2022-09-10 10:11:22 -04:00
Samuel Clay
31de0f7a97 Upgrading to latest pymongo v3 to work with mongodb v4.0. Required rewrite of mongodb introspection, now using built-in pymongo monitoring. Should work on both web and celery. 2022-05-10 20:58:47 -04:00
Samuel Clay
f03be46425 Removing boto2, upgrading to boto3. Needs testing. 2021-08-04 20:04:53 -04:00
Samuel Clay
4f37953df4 Mongo backups should use a dual-docker solution: 1) Use docker to backup mongo from the mongo container, 2) use newsblur_web container to upload to S3. 2021-07-16 12:25:25 -04:00
Jonathan Math
a2c7acf3ff correct newsblur to newsblur_web in tasks 2021-01-04 13:33:56 -05:00
Samuel Clay
f041bbceae 2to3 2020-12-03 14:16:47 -05:00
Samuel Clay
3147fee8af Switching to shared celery app task. 2020-11-13 13:26:25 -05:00
Samuel Clay
0f77d1c441 Migrating to celery 4.0 2020-11-13 12:14:37 -05:00
Samuel Clay
3fa55aff17 Upgrading to latest celery 4 (holy moly), which required some big changes to project layout. Still needs supervisor scripts updated. 2020-06-29 17:39:55 -04:00
Samuel Clay
2d6343b7ae Fixing feedfinder beautifulsoup issues, unicode issues in feed, and unicode comparisons issues with page_data (since it wasnt being decompressed when compared). 2016-02-05 14:43:31 -08:00
Samuel Clay
66aadc45a1 Adding soft time limit to celery tasks. 2016-02-01 23:06:04 -08:00
Samuel Clay
4a12c220fe Risky business: now capturing db times (sql/mongo/redis) for 1% of all tasks. Also updating munin to recognize new db profiling times. 2015-12-16 17:31:41 -08:00
Samuel Clay
6a20478a7f Renaming redis feed pool to redis feed update pool in prep for redis feed sub pool. 2015-07-27 18:35:25 -07:00
Samuel Clay
97b0910e67 Breaking up tasking of normal feeds and broken feeds so broken feeds are a less drain on postgresql. 2015-07-23 16:29:47 -07:00
Samuel Clay
374e546175 Merge branch 'master' into saved 
* master:
  Fixing updates_off in tasks.
  Adding updates_off run-time setting for handling large spikes of updates.
  Draining feeds also clears out error feeds.
 2014-05-29 18:38:56 -07:00
Samuel Clay
c20ef4e5d0 Adding updates_off run-time setting for handling large spikes of updates. 2014-05-29 17:53:16 -07:00
Samuel Clay
425775eb0d Adding feed counts for saved stories. Need to show counts next to feeds. 2014-05-28 15:30:30 -07:00
Samuel Clay
ab12b913c3 Correcting user tag order. 2014-05-20 16:00:57 -07:00
Samuel Clay
9173bee736 Moving to a naive story tag counter. Can schedule re-counts if necessary. 2014-04-29 12:36:42 -07:00
Samuel Clay
97bf655f1d Handling up to 10000 feed updates per 5 minutes. 2014-04-21 17:19:50 -07:00
Samuel Clay
4787f70ebb Fixing issue with scheduling feeds. 2014-04-21 16:38:27 -07:00
Samuel Clay
9599cf65f2 Only fetching feeds immediately if necessary. 2014-04-21 16:26:10 -07:00
Samuel Clay
18d750091d Shortening task error restart to 10m from 1h. 2013-05-24 13:18:05 -07:00
Samuel Clay
f6f053b285 Adding duplicate feed address+link truncation. 2013-05-15 15:40:38 -07:00
Samuel Clay
6a3d043ced Upping to 5000 feeds per minute. 2013-04-23 21:17:03 -07:00
Samuel Clay
45e137b22c Upping feed fetches. 2013-04-15 19:26:41 -07:00
Samuel Clay
528c86a1eb Fixing small bug around missing feeds during update pipeline. 2013-04-15 15:16:53 -07:00
Samuel Clay
a6837724e3 Increasing task queue size. 2013-04-08 17:11:40 -07:00
Samuel Clay
46f919f919 Using error_feeds to push broken feeds further out in the pipeline. 2013-04-08 10:50:50 -07:00
Samuel Clay
7fc9b494af Adding error_feeds queue to keep track of broken feeds. 2013-04-08 10:04:40 -07:00
Samuel Clay
d63556bc06 Skipping tasking of feeds when queue is too high. Use the random queue. 2013-04-08 09:49:21 -07:00
Samuel Clay
c41bf80b4e Removing force from new feeds. 2013-04-07 20:22:46 -07:00
Samuel Clay
48c5d4d53e Adding logging, feed cleanup to forced fetch feeds, and expirations on user read stories. 2013-04-07 17:19:59 -07:00
Samuel Clay
1ba468d27d Checking if feed id has changed, then removing from task queue. 2013-04-04 21:57:58 -07:00
Samuel Clay
341db243c3 Better logging for queues. 2013-04-04 20:47:41 -07:00
Samuel Clay
a36a960566 Bumping numbers on task fetching. 2013-04-03 22:59:25 -07:00
Samuel Clay
6b262f8557 Cleaning up counts in munin and work log. 2013-04-03 19:04:30 -07:00
Samuel Clay
4461bb9c99 Removing old inactive feeds refresher. 2013-04-03 17:31:37 -07:00
Samuel Clay
6a2a144747 Requeueing old tasked feeds. 2013-04-03 17:28:51 -07:00
Samuel Clay
8f8c88ca47 Adding tasked_feeds set to handle dropped feeds. 2013-04-03 17:22:45 -07:00
Samuel Clay
e0457bf0d0 Fixing missing feed bug. 2013-04-03 16:48:42 -07:00
Samuel Clay
ba5919d760 Moving task queue from postgres db backed to redis backed. 2013-03-30 19:05:13 -07:00
Samuel Clay
2cc5ab6202 Boosting feed fetching. 2013-03-30 11:39:49 -07:00
Samuel Clay
03c27b42e8 Removing unused feed hash index (keeping the unique constraint). 2013-03-29 22:41:16 -07:00
Samuel Clay
22b2bd0604 Changing logging on feed queueing. 2013-03-29 21:46:04 -07:00
Samuel Clay
ae953ee208 Cleaning up week old feeds. 2013-03-29 16:02:32 -07:00
Samuel Clay
07ebbd63b5 Boosting feed fetching way up for the weekend. 2013-03-29 15:56:43 -07:00
Samuel Clay
aa4bf71b34 Upping feed fetching by quite a bit. 2013-03-29 14:14:29 -07:00