Commit graph

484 commits

Author SHA1 Message Date
Samuel Clay
ddeaf55be0 Cleaning up pipeline and adding admin static assets, but now certain admin static assets are being post-processed multiple times but with different cache manifests, resulting in a file not found error. This needs work. 2022-01-06 11:01:35 -05:00
Samuel Clay
688b3524a3 Fixing grafana sentry integration. 2022-01-05 15:18:26 -05:00
Samuel Clay
6108f2a067 Adding sentry grafana pluging. 2022-01-05 13:22:24 -05:00
Samuel Clay
76ffe7bb8a Fixing bookmarklet, moving endlne fleuron, adding load-line, archviing Jammit. 2022-01-05 11:18:31 -05:00
Samuel Clay
04c62b11c7 Cleaning up static assets. Needs to run on first make nb or DEBUG_ASSETS needs to always be set to True. 2022-01-04 15:54:37 -05:00
Samuel Clay
d06e871ee2 Boosting timeout for docker prune 2022-01-04 15:14:22 -05:00
Samuel Clay
3e348329b3 Don't prune docker unless needed. 2022-01-04 15:09:01 -05:00
Samuel Clay
b8eccdbed4 Merge branch 'master' into pipeline
* master:
  Fixing names for state timeline, also fixing node server bootstrap.
  Moving up haproxy state timeline
  Bumping haproxy state timeline scrapes.
  Only need haproxy for haproxy metrics
  Refactored state timeline to work as a flask metrics server on haproxy. Needs ansible configs to deploy on www server.
  Forgot to save merge conflict.
  fix service state timeline key error and change scrape interval
  Revert "Revert "Merge branch 'state_timeline'""
  Revert "Merge branch 'state_timeline'"
  Fixing syntax error
  Solving docker networking container issues by turning iptables back on but using ufw after.rules that allow ufw to retain blocking ability. Works on Redis, now to move to mongo.
  Mongo analytics monitor not reayd yet
  Turning consul monitor back on.
  Turning on analytics
  Need to strip csv values.
  Binding redis to public interface.
  add grafana dashboard backup functionality
2022-01-04 14:44:57 -05:00
Samuel Clay
8de2bfcbae Fixing names for state timeline, also fixing node server bootstrap. 2022-01-04 14:44:47 -05:00
Samuel Clay
af7524a976 Bumping haproxy state timeline scrapes. 2022-01-04 12:51:45 -05:00
Samuel Clay
9432e9a5b8 Merge branch 'master' into pipeline
* master:
  Pruning docker, updating sentry release webhook only on local.
  Delete .github directory
  Adding Sentry release webhook to deployment of web, task, monitor, and node.
  Add Sentry action workflow for actions
  Moving from hosted Sentry to self-hosted sentry.
  Flask transactions are way too often.
  Cleanup
  Seems redundant with newsblur_web/docker_local_settings.py
  Using new repo password for postgres.
  Moving around postgres config
  Moving directories.
  Adding postgresql backup restoration. Working on config.
  Using user log for db times.
  Last 5 minutes is actually last minute.
  Adding 5min loadtime to prometheus graphs.
  Switching back to nbanalytics with user auth.
  Mongo auth
2022-01-03 18:04:40 -05:00
Samuel Clay
e227e651ed Pruning docker, updating sentry release webhook only on local. 2022-01-03 17:24:57 -05:00
Samuel Clay
259dff477e Adding Sentry release webhook to deployment of web, task, monitor, and node. 2022-01-03 15:27:53 -05:00
Samuel Clay
e5e3162f89 Moving from hosted Sentry to self-hosted sentry. 2022-01-03 14:48:39 -05:00
Samuel Clay
f33c1b3600 Using new repo password for postgres. 2021-12-17 17:35:01 -05:00
Samuel Clay
6b8e747ebc Moving around postgres config 2021-12-17 17:11:54 -05:00
Samuel Clay
aee543d913 Moving directories. 2021-12-17 16:56:30 -05:00
Samuel Clay
62529e96c2 Adding postgresql backup restoration. Working on config. 2021-12-17 16:53:28 -05:00
Samuel Clay
f36a4f8fd5 Mongo auth 2021-12-16 20:30:27 -05:00
Samuel Clay
9ced4ccadd Merge branch 'master' into pipeline
* master: (84 commits)
  Deploying new analytics server.
  Create mongodb database user. Only runs on analytics, but should work fine on main mongo.
  Adding second mongo analytics db, cleaning up mongo analytics auth.
  Adding RSS autodiscovery to blog. Thanks https://forum.newsblur.com/t/irony-newsblur-cant-auto-extract-rss-feed-from-blog-newsblur-com/8849.
  Revert "Merge branch 'state_timeline'"
  Fixing syntax error
  Solving docker networking container issues by turning iptables back on but using ufw after.rules that allow ufw to retain blocking ability. Works on Redis, now to move to mongo.
  fix flask_metrics ansible role
  add state timeline dashboard to grafana
  move state timeline to one single flask that gets astate for all services
  Mongo analytics monitor not reayd yet
  Turning consul monitor back on.
  Turning on analytics
  Need to strip csv values.
  Binding redis to public interface.
  Restore postgres backup. Needs work.
  Prepping postgres backup
  Prepping postgres transition, writing redis db maintenance.
  Prepping redis for moving back to primary.
  Updating redis backup to include percentage progress and access to the correct data dump.
  ...
2021-12-16 15:41:57 -05:00
Samuel Clay
ef62d212ab Deploying new analytics server. 2021-12-16 15:31:12 -05:00
Samuel Clay
8b6452b094 Create mongodb database user. Only runs on analytics, but should work fine on main mongo. 2021-12-16 15:02:27 -05:00
Samuel Clay
ea6103bc8f Adding second mongo analytics db, cleaning up mongo analytics auth. 2021-12-16 13:48:20 -05:00
Samuel Clay
e70ba4a8c0 Revert "Merge branch 'state_timeline'"
This reverts commit 83e4609af1, reversing
changes made to 089161b20f.
2021-12-15 14:17:48 -05:00
Samuel Clay
5008d42a8e Merge branch 'master' into state_timeline
* master:
  Solving docker networking container issues by turning iptables back on but using ufw after.rules that allow ufw to retain blocking ability. Works on Redis, now to move to mongo.
  Mongo analytics monitor not reayd yet
  Turning consul monitor back on.
  Turning on analytics
  Need to strip csv values.
  Binding redis to public interface.
2021-12-15 13:34:11 -05:00
Samuel Clay
089161b20f Solving docker networking container issues by turning iptables back on but using ufw after.rules that allow ufw to retain blocking ability. Works on Redis, now to move to mongo. 2021-12-15 13:34:01 -05:00
Jonathan Math
cf917b43e5 fix flask_metrics ansible role 2021-12-15 11:37:43 -05:00
Jonathan Math
99dd44a435 move state timeline to one single flask that gets astate for all services 2021-12-15 11:35:16 -05:00
Samuel Clay
8da118f653 Restore postgres backup. Needs work. 2021-12-14 08:22:19 -05:00
Samuel Clay
777e69ae82 Prepping postgres backup 2021-12-13 17:09:37 -05:00
Samuel Clay
5b0ef4dbd6 Prepping redis for moving back to primary. 2021-12-13 15:31:21 -05:00
Samuel Clay
8bceaada80 Updating redis backup to include percentage progress and access to the correct data dump. 2021-12-13 14:26:20 -05:00
Samuel Clay
be3237cdc3 Updating to boto3 syntax. 2021-12-13 12:45:47 -05:00
Samuel Clay
58d6c00bfa Bumping mongo size 2021-12-09 13:20:02 -05:00
Samuel Clay
35bc9ad07f Rewriting readme from #1545. 2021-12-08 14:55:26 -05:00
Samuel Clay
d385495ef5 Explicit prometheus command, modeled after the dockerhub image. We're not using --web.console.libraries=/usr/share/prometheus/console_libraries --web.console.templates=/usr/share/prometheus/consoles 2021-12-07 16:23:05 -05:00
Samuel Clay
e63ad6193b Explicit config file 2021-12-07 11:16:00 -05:00
Samuel Clay
b7995547e4 Merge branch 'master' into prometheus_retention
* master:
  Turning refresh stale feeds back on.
  Terraform Digital Ocean v2.0
  Adjusting sizes
  Better topline metrics
  Cleaning up redis dashboard.
  Flask metrics may not have the port, use default.
2021-12-07 11:11:18 -05:00
Samuel Clay
0f81b9dfa1 Flask metrics may not have the port, use default. 2021-12-07 07:16:31 -05:00
Jonathan Math
cbe6c58b0c add 10 year retention for data in prometheus 2021-12-06 17:25:15 -05:00
Samuel Clay
ec33dfab48 Separating consul from haproxy health checks. Consul doesn't care what the status of the service is so long as it's reachable. Don't take out bad nodes, let the software do that. 2021-12-06 15:12:12 -05:00
Samuel Clay
966f0998fc Removing unused traces of older redis servers. 2021-12-06 13:56:09 -05:00
Samuel Clay
c4bbd3a7a8 New file system finder, finds static assets. 2021-11-16 11:27:09 -05:00
Samuel Clay
e6367605cd Merge branch 'master' into pipeline
* master:
  Removing dependencies from ssh helper.
  Specifying docker build platform (--platform linux/amd64) so M1 Macs don't build arm64/v8.
  Haproxy shoudl reload on config update, adding celery task group, and allowing task servers to update inventory.
2021-11-14 20:11:10 -05:00
Samuel Clay
38c2004479 Haproxy shoudl reload on config update, adding celery task group, and allowing task servers to update inventory. 2021-11-14 17:21:04 -05:00
Samuel Clay
b6db14dacd Switching from Jammit to django-pipeline. Doesn't work yet, needs compressor hooked up and assets checked. 2021-11-14 13:42:06 -05:00
Samuel Clay
bf70f2aaf7 Redis replica user 2021-10-25 21:24:00 -04:00
Samuel Clay
db5cb4bd94 Moving redis conf to template to distinguish servers. 2021-10-20 17:27:20 -04:00
Samuel Clay
24d6868108 Redis port 2021-10-20 17:18:28 -04:00
Samuel Clay
e0af49022e Adjusting redis conf dirs. 2021-10-20 17:12:24 -04:00