Commit graph

1668 commits

Author SHA1 Message Date
Samuel Clay
31de0f7a97 Upgrading to latest pymongo v3 to work with mongodb v4.0. Required rewrite of mongodb introspection, now using built-in pymongo monitoring. Should work on both web and celery. 2022-05-10 20:58:47 -04:00
Samuel Clay
1dc0e98448 Renaming postgresql backups 2022-04-29 16:53:47 -04:00
Samuel Clay
464fecd18e Adding postgres restore command and updating backup script. 2022-04-29 16:45:17 -04:00
Samuel Clay
d0571e2ab0 Wait to strip data uri header until it's time. 2022-03-08 13:11:08 -05:00
Samuel Clay
ca728af375 Don't double encode base64 images for widgets. 2022-03-08 12:48:57 -05:00
Samuel Clay
0e63da2d74 Fixing bad instrumentation on mongo and redis logs. 2022-03-07 15:57:51 -05:00
Samuel Clay
b1de95dd20 Not hitting a db shouldnt be colored any differently than a small value. 2022-03-04 12:51:43 -05:00
Samuel Clay
0bae64e6fd Recoloring db times logs. 2022-03-04 12:49:01 -05:00
Samuel Clay
c800dcaddf Black formatting 2022-03-02 10:55:46 -05:00
Samuel Clay
f0f6c0a828 Fixing issue with some sites spewing tons of headers back and Python failing. 2022-03-02 10:29:24 -05:00
Samuel Clay
06c25df199 No longer sending tons of mail to admins. Logging is where that goes. 2022-02-22 10:19:51 -05:00
Samuel Clay
b6763d872f XoneB 4 LIFE 2022-02-08 15:32:43 -05:00
Samuel Clay
ed08c035cc Explicitly uploading mongo backups 2022-02-03 16:37:15 -05:00
Samuel Clay
43e61cc5aa Only need date once in filename 2022-02-03 16:31:33 -05:00
Samuel Clay
0f09de6218 Rotating mongo backups 2022-02-03 16:17:32 -05:00
Samuel Clay
922269a945 Using full path for progress. 2022-02-03 16:11:13 -05:00
Samuel Clay
d8f1081935 Showing full path of postgres backup. 2022-02-03 16:08:45 -05:00
Samuel Clay
386861a479 Separating out restoring backups and storing backups. 2022-02-03 15:53:42 -05:00
Samuel Clay
5bde153147 Compressing Postgres db sql backup. 2022-02-03 15:39:36 -05:00
Samuel Clay
1397693d54 Adding date to tarball to allow for versioning. 2022-02-03 15:24:49 -05:00
Samuel Clay
5fdeedbbb2 Correcting permissions on mongo backup 2022-02-03 15:20:08 -05:00
Samuel Clay
a3b35933c3 Fixing mongo upload to s3 by using host networking mode. 2022-02-03 14:34:34 -05:00
Samuel Clay
d05c3b1599 Removing unused backup code. 2022-02-03 14:24:12 -05:00
Samuel Clay
cd23d068c3 Forgot import 2022-02-03 14:23:13 -05:00
Samuel Clay
2b2d947dca Adding progress bar to mongo backup. 2022-02-03 14:22:43 -05:00
Samuel Clay
e8a88740bb Refactoring aws keys for mongo backups. 2022-02-03 14:03:58 -05:00
Samuel Clay
eadcfc4426 Adjusting backup for mongo to live on mounted volume. 2022-02-03 13:38:36 -05:00
Samuel Clay
7889fe4cf0 Fixing bookmarklet and saved story tags. 2022-01-28 17:05:23 -05:00
Samuel Clay
55f0bd9324 Adding a RawJSNode to literally read the <script> tags spit out by django-pipeline, and stick those files into embedded JS. But it works and should stay current as it points to the non-cached version. If this gets millions of hits a day, we may want to optimize this, but for now it's fine. 2022-01-28 13:59:51 -05:00
Samuel Clay
6a61620d32 Turning off PuSH for standalone installs. 2022-01-25 15:27:29 -05:00
Samuel Clay
71eae2c640 Better handling of Twitter being down (and returning a 503). 2022-01-21 06:52:10 -05:00
Samuel Clay
0011249cad Cleaning up django collection pipeline. 2022-01-20 16:32:37 -05:00
Samuel Clay
ced2e7d6a6 Upgrading jQuery and jQuery UI, fixing closure compiler language output to work in older browser. 2022-01-20 13:19:35 -05:00
Samuel Clay
88d000a2f9 Turning off logging for debug_assets=True. 2022-01-11 11:02:40 -05:00
Samuel Clay
a8d73f3685 Fixing static files so that at least newsblur is packaged correctly. Admin static is still not able to be processed. 2022-01-06 12:42:13 -05:00
Samuel Clay
ddeaf55be0 Cleaning up pipeline and adding admin static assets, but now certain admin static assets are being post-processed multiple times but with different cache manifests, resulting in a file not found error. This needs work. 2022-01-06 11:01:35 -05:00
Samuel Clay
76ffe7bb8a Fixing bookmarklet, moving endlne fleuron, adding load-line, archviing Jammit. 2022-01-05 11:18:31 -05:00
Samuel Clay
b8eccdbed4 Merge branch 'master' into pipeline
* master:
  Fixing names for state timeline, also fixing node server bootstrap.
  Moving up haproxy state timeline
  Bumping haproxy state timeline scrapes.
  Only need haproxy for haproxy metrics
  Refactored state timeline to work as a flask metrics server on haproxy. Needs ansible configs to deploy on www server.
  Forgot to save merge conflict.
  fix service state timeline key error and change scrape interval
  Revert "Revert "Merge branch 'state_timeline'""
  Revert "Merge branch 'state_timeline'"
  Fixing syntax error
  Solving docker networking container issues by turning iptables back on but using ufw after.rules that allow ufw to retain blocking ability. Works on Redis, now to move to mongo.
  Mongo analytics monitor not reayd yet
  Turning consul monitor back on.
  Turning on analytics
  Need to strip csv values.
  Binding redis to public interface.
  add grafana dashboard backup functionality
2022-01-04 14:44:57 -05:00
Samuel Clay
0db15560da Merge branch 'master' into state_timeline
* master: (29 commits)
  Pruning docker, updating sentry release webhook only on local.
  Delete .github directory
  Adding Sentry release webhook to deployment of web, task, monitor, and node.
  Add Sentry action workflow for actions
  Moving from hosted Sentry to self-hosted sentry.
  Flask transactions are way too often.
  Cleanup
  Seems redundant with newsblur_web/docker_local_settings.py
  Using new repo password for postgres.
  Moving around postgres config
  Moving directories.
  Adding postgresql backup restoration. Working on config.
  Using user log for db times.
  Last 5 minutes is actually last minute.
  Adding 5min loadtime to prometheus graphs.
  Switching back to nbanalytics with user auth.
  Mongo auth
  Deploying new analytics server.
  Create mongodb database user. Only runs on analytics, but should work fine on main mongo.
  Adding second mongo analytics db, cleaning up mongo analytics auth.
  ...
2022-01-04 11:18:48 -05:00
Samuel Clay
9432e9a5b8 Merge branch 'master' into pipeline
* master:
  Pruning docker, updating sentry release webhook only on local.
  Delete .github directory
  Adding Sentry release webhook to deployment of web, task, monitor, and node.
  Add Sentry action workflow for actions
  Moving from hosted Sentry to self-hosted sentry.
  Flask transactions are way too often.
  Cleanup
  Seems redundant with newsblur_web/docker_local_settings.py
  Using new repo password for postgres.
  Moving around postgres config
  Moving directories.
  Adding postgresql backup restoration. Working on config.
  Using user log for db times.
  Last 5 minutes is actually last minute.
  Adding 5min loadtime to prometheus graphs.
  Switching back to nbanalytics with user auth.
  Mongo auth
2022-01-03 18:04:40 -05:00
Samuel Clay
647b1100fd Using user log for db times. 2021-12-17 13:55:35 -05:00
Samuel Clay
8be0192702 Cleaning up pipeline 2021-12-16 20:06:15 -05:00
Samuel Clay
cf948c725a Renaming css files and fixing debug assets pipeline collector by forcing it to munge the static url with the media, non-cached, name. nginx should do the caching, not the url. 2021-12-16 17:30:05 -05:00
Samuel Clay
9ced4ccadd Merge branch 'master' into pipeline
* master: (84 commits)
  Deploying new analytics server.
  Create mongodb database user. Only runs on analytics, but should work fine on main mongo.
  Adding second mongo analytics db, cleaning up mongo analytics auth.
  Adding RSS autodiscovery to blog. Thanks https://forum.newsblur.com/t/irony-newsblur-cant-auto-extract-rss-feed-from-blog-newsblur-com/8849.
  Revert "Merge branch 'state_timeline'"
  Fixing syntax error
  Solving docker networking container issues by turning iptables back on but using ufw after.rules that allow ufw to retain blocking ability. Works on Redis, now to move to mongo.
  fix flask_metrics ansible role
  add state timeline dashboard to grafana
  move state timeline to one single flask that gets astate for all services
  Mongo analytics monitor not reayd yet
  Turning consul monitor back on.
  Turning on analytics
  Need to strip csv values.
  Binding redis to public interface.
  Restore postgres backup. Needs work.
  Prepping postgres backup
  Prepping postgres transition, writing redis db maintenance.
  Prepping redis for moving back to primary.
  Updating redis backup to include percentage progress and access to the correct data dump.
  ...
2021-12-16 15:41:57 -05:00
Jonathan Math
6189dd4c01 add grafana dashboard backup functionality 2021-12-15 13:40:23 -05:00
Samuel Clay
777e69ae82 Prepping postgres backup 2021-12-13 17:09:37 -05:00
Samuel Clay
8bceaada80 Updating redis backup to include percentage progress and access to the correct data dump. 2021-12-13 14:26:20 -05:00
Samuel Clay
be3237cdc3 Updating to boto3 syntax. 2021-12-13 12:45:47 -05:00
Samuel Clay
d677373e3b No more settings.get. 2021-12-13 12:42:30 -05:00
Samuel Clay
1dc79686bc Skipping requests for RSS, letting feedparser do the fetching. Requests si still used for json feeds. 2021-12-08 20:44:16 -05:00