Commit graph

54 commits

Author SHA1 Message Date
Samuel Clay
d0fdbde96b Ensuring dependencies met for deployment on local. 2022-01-19 10:48:11 -05:00
Samuel Clay
0f27ba81e0 Updating to docker buildx for multi-arch builds (amd64 + arm64) 2022-01-18 17:25:07 -05:00
Samuel Clay
76ffe7bb8a Fixing bookmarklet, moving endlne fleuron, adding load-line, archviing Jammit. 2022-01-05 11:18:31 -05:00
Samuel Clay
04c62b11c7 Cleaning up static assets. Needs to run on first make nb or DEBUG_ASSETS needs to always be set to True. 2022-01-04 15:54:37 -05:00
Samuel Clay
b8eccdbed4 Merge branch 'master' into pipeline
* master:
  Fixing names for state timeline, also fixing node server bootstrap.
  Moving up haproxy state timeline
  Bumping haproxy state timeline scrapes.
  Only need haproxy for haproxy metrics
  Refactored state timeline to work as a flask metrics server on haproxy. Needs ansible configs to deploy on www server.
  Forgot to save merge conflict.
  fix service state timeline key error and change scrape interval
  Revert "Revert "Merge branch 'state_timeline'""
  Revert "Merge branch 'state_timeline'"
  Fixing syntax error
  Solving docker networking container issues by turning iptables back on but using ufw after.rules that allow ufw to retain blocking ability. Works on Redis, now to move to mongo.
  Mongo analytics monitor not reayd yet
  Turning consul monitor back on.
  Turning on analytics
  Need to strip csv values.
  Binding redis to public interface.
  add grafana dashboard backup functionality
2022-01-04 14:44:57 -05:00
Samuel Clay
af7524a976 Bumping haproxy state timeline scrapes. 2022-01-04 12:51:45 -05:00
Samuel Clay
9432e9a5b8 Merge branch 'master' into pipeline
* master:
  Pruning docker, updating sentry release webhook only on local.
  Delete .github directory
  Adding Sentry release webhook to deployment of web, task, monitor, and node.
  Add Sentry action workflow for actions
  Moving from hosted Sentry to self-hosted sentry.
  Flask transactions are way too often.
  Cleanup
  Seems redundant with newsblur_web/docker_local_settings.py
  Using new repo password for postgres.
  Moving around postgres config
  Moving directories.
  Adding postgresql backup restoration. Working on config.
  Using user log for db times.
  Last 5 minutes is actually last minute.
  Adding 5min loadtime to prometheus graphs.
  Switching back to nbanalytics with user auth.
  Mongo auth
2022-01-03 18:04:40 -05:00
Samuel Clay
e227e651ed Pruning docker, updating sentry release webhook only on local. 2022-01-03 17:24:57 -05:00
Samuel Clay
259dff477e Adding Sentry release webhook to deployment of web, task, monitor, and node. 2022-01-03 15:27:53 -05:00
Samuel Clay
e5e3162f89 Moving from hosted Sentry to self-hosted sentry. 2022-01-03 14:48:39 -05:00
Samuel Clay
62529e96c2 Adding postgresql backup restoration. Working on config. 2021-12-17 16:53:28 -05:00
Samuel Clay
9ced4ccadd Merge branch 'master' into pipeline
* master: (84 commits)
  Deploying new analytics server.
  Create mongodb database user. Only runs on analytics, but should work fine on main mongo.
  Adding second mongo analytics db, cleaning up mongo analytics auth.
  Adding RSS autodiscovery to blog. Thanks https://forum.newsblur.com/t/irony-newsblur-cant-auto-extract-rss-feed-from-blog-newsblur-com/8849.
  Revert "Merge branch 'state_timeline'"
  Fixing syntax error
  Solving docker networking container issues by turning iptables back on but using ufw after.rules that allow ufw to retain blocking ability. Works on Redis, now to move to mongo.
  fix flask_metrics ansible role
  add state timeline dashboard to grafana
  move state timeline to one single flask that gets astate for all services
  Mongo analytics monitor not reayd yet
  Turning consul monitor back on.
  Turning on analytics
  Need to strip csv values.
  Binding redis to public interface.
  Restore postgres backup. Needs work.
  Prepping postgres backup
  Prepping postgres transition, writing redis db maintenance.
  Prepping redis for moving back to primary.
  Updating redis backup to include percentage progress and access to the correct data dump.
  ...
2021-12-16 15:41:57 -05:00
Samuel Clay
0f81b9dfa1 Flask metrics may not have the port, use default. 2021-12-07 07:16:31 -05:00
Samuel Clay
966f0998fc Removing unused traces of older redis servers. 2021-12-06 13:56:09 -05:00
Samuel Clay
c4bbd3a7a8 New file system finder, finds static assets. 2021-11-16 11:27:09 -05:00
Samuel Clay
38c2004479 Haproxy shoudl reload on config update, adding celery task group, and allowing task servers to update inventory. 2021-11-14 17:21:04 -05:00
Samuel Clay
ce34d598ca Directory needs changing from elasticsearch_exporter to elasticsearch-exporter. 2021-10-19 11:26:22 -04:00
Jonathan Math
cadfc15e75 add elasticsearch-exporter role in ansible 2021-10-11 11:08:51 -05:00
Samuel Clay
532790610d Removing auth from mongo. 2021-09-30 18:02:00 -04:00
Samuel Clay
4e18ceb369 Removing unused code. 2021-09-21 17:00:17 -04:00
Jonathan Math
f96bdb2744 Merge branch 'add_grafana_monitors' into metrics 2021-09-16 11:58:36 -05:00
Jonathan Math
2ee9971cbe add flask_metrics role to setup_mongo.yml 2021-09-15 11:14:52 -05:00
Samuel Clay
683b55994c Mongo exporter is a metrics. 2021-09-07 11:38:18 -04:00
Samuel Clay
af390d5fd1 Adding elasticsearch to py3, upgrading monitor, need to fix boto -> boto3. 2021-08-04 18:42:58 -04:00
Samuel Clay
040c165164 Adding db mongo analytics back. 2021-08-04 18:04:06 -04:00
Samuel Clay
dd62491fd7 Mongo analytics needs correct user. 2021-08-03 19:59:45 -04:00
Samuel Clay
02b2546825 Adding hostnames to docker containers, changing mongo to just use host network mode dammit, and adding possibility for logging to dnsmasq. 2021-07-23 18:13:19 -04:00
Samuel Clay
4ddb345e32 Retrying mongodb keyfile auth. 2021-07-14 19:59:00 -04:00
Samuel Clay
dce7ece0aa Newsletters bouncing looks good. 2021-07-14 11:11:44 -04:00
Samuel Clay
5f58d9ccee Better rescue of broken task servers. 2021-07-09 17:39:34 -04:00
Samuel Clay
4c5a5907c5 Adding redirects for dashboard refreshes. Also attempting to cache global river for dashboard. 2021-07-02 09:15:22 -04:00
Samuel Clay
dd9de64b29 Updating dashboard3 -> master. 2021-07-01 17:11:11 -04:00
Samuel Clay
7d8ba31ee1 Adding blog role and server. 2021-06-22 12:50:08 -04:00
Samuel Clay
cc7251610a Disable consul services needs +ie staging. 2021-06-15 11:12:09 -04:00
Jonathan Math
3c5f26a26a add exporters for dbs 2021-05-26 12:16:33 -04:00
Samuel Clay
ee7bd47e8b Cleaning up deploy work and monitors. The task monitors need to be integrate with the flask db monitors. 2021-05-26 11:41:28 -04:00
Samuel Clay
deadb36c92 Adding metrics tag to all nodeexporter/prometheus/grafana roles. 2021-05-21 12:02:07 -04:00
Samuel Clay
82fddd415f Switching from network_mode host to using a docker network. This gets us off host networking and to defined ports. The docker network connects the container's dns resolver to the machine's dnsmasq, which is used for consul. 2021-05-19 17:26:47 -04:00
Samuel Clay
b1680f9e49 Adding db-metrics server. 2021-05-17 11:30:09 -04:00
Jonathan Math
6da0450fc3 add node-exporter to setup_ .yml files 2021-05-17 10:09:50 -04:00
Samuel Clay
d0f03779b3 Fixing sharing story issue with scrubbing encoding. 2021-05-14 20:52:28 -04:00
Samuel Clay
6a559e0f3f HAProxy now does a graceful reload. Cleaning up ansible. Splitting out firewall into ufw role. 2021-05-14 18:40:43 -04:00
Samuel Clay
62a6f664e1 Maintenance mode was easier to write than I thought. 2021-05-13 19:53:34 -04:00
Samuel Clay
54e4914804 Adding 3 consul servers. 2021-05-12 15:41:17 -04:00
Samuel Clay
1b3b42648f Adding benchmarking and volumes for spinning up new mongo clusters. 2021-05-10 13:58:43 -04:00
Samuel Clay
a22ce16a0e Attempting to handle mistaken week where stories were inserted with no doc type. 2021-04-27 15:25:46 -04:00
Jonathan Math
6276bc8448 begin development with ansible by adding to docker-compose and starting ansible roles 2021-04-27 08:00:06 -05:00
Jonathan Math
e1748b8e90 remove netdata 2021-04-27 07:58:19 -05:00
Samuel Clay
1773eb4dd5 Ignoring task env settings. 2021-04-26 16:36:04 -04:00
Samuel Clay
5d7ddb1d8a Adding netdata in to task servers. Also setting up logrotate. 2021-04-22 11:36:44 -04:00