325 commits

Author	SHA1	Message	Date
Samuel Clay	e11d4db4ea	Fixing prometheus to use consul internal dns.	2021-07-10 16:11:04 -04:00
Samuel Clay	5f58d9ccee	Better rescue of broken task servers.	2021-07-09 17:39:34 -04:00
Samuel Clay	f2e64ca36e	Fixing sanity checkers. None of them were running. Turns out they needed to be shell scripts if in the /etc/cron.hourly dir (and crontab format if in the /etc/cron.d dir).	2021-07-08 15:39:29 -04:00
Samuel Clay	4c5a5907c5	Adding redirects for dashboard refreshes. Also attempting to cache global river for dashboard.	2021-07-02 09:15:22 -04:00
Samuel Clay	dd9de64b29	Updating dashboard3 -> master.	2021-07-01 17:11:11 -04:00
Samuel Clay	f652941687	Adding autoheal to celery label.	2021-06-30 20:41:17 -04:00
Samuel Clay	e160a86d2a	Switching celery to local log driver for docker due to logs filling up disk space.	2021-06-30 00:22:09 -04:00
Samuel Clay	17e6f7f424	Adding logrotate for mongo.	2021-06-29 17:51:14 -04:00
Jonathan Math	0e81d2f346	no need to pass filename into backup_psql.py in ansible postgres backup. removing it	2021-06-27 14:40:00 -06:00
Jonathan Math	a78dec3ed9	fix syntax for backup_redis and duse socket for getting hostname instead of passing in as an argument	2021-06-27 14:36:24 -06:00
Jonathan Math	4846457bc4	fix syntax for mongo backups	2021-06-27 14:32:37 -06:00
Jonathan Math	8ede5a3987	change get_postgres_credentials.py to get_credentials.py so it can also be used to get the s3 bucket name. Fix postgres backups	2021-06-27 14:11:14 -06:00
Jonathan Math	baee7a798b	add multiline command syntax >- to cronjobs for sanity checkers	2021-06-27 13:24:07 -06:00
Jonathan Math	5cbe285888	run command to get redis log path in cronjob for redis bgsave. This way if the container id changes for any reason, the change will be picked up when the cronjob starts instead of being evaluated only when the cronjob is deployed with ansible	2021-06-25 11:49:38 -06:00
Jonathan Math	dfe72b3e04	add -it to disk usage commands for cronjobs. capitalize OUTPUT in disk usage commands	2021-06-25 11:46:09 -06:00
Jonathan Math	972f13b394	edit ansible files so that all disk usage sanity checkers run hourly	2021-06-25 11:42:21 -06:00
Samuel Clay	cea069d842	Fixing redis sanity checker.	2021-06-24 13:21:25 -04:00
Samuel Clay	57c2713029	Fixing redis bgsave and disk checker monitors.	2021-06-24 11:54:56 -04:00
Samuel Clay	ae95b25910	Cleaning up monitors, switching to cron file, adding hostname.	2021-06-24 11:50:38 -04:00
Samuel Clay	1d5173e7bc	Fixing monitor task fetches	2021-06-24 11:19:45 -04:00
Samuel Clay	b533d824f1	Turning off Docker iptables firewall exclusion. See https://stackoverflow.com/questions/30383845/what-is-the-best-practice-of-docker-ufw-under-ubuntu	2021-06-23 23:16:10 -04:00
Samuel Clay	7d8ba31ee1	Adding blog role and server.	2021-06-22 12:50:08 -04:00
Jonathan Math	15f40588f3	fix celery_task sanity checkers	2021-06-22 09:06:10 -06:00
Jonathan Math	309155faeb	fix docker commands for backups	2021-06-19 13:01:38 -06:00
Jonathan Math	4cea718d40	add redis backup and remove redundant redis backup scripts. Use one script and pass in hostname argument instead	2021-06-19 12:52:04 -06:00
Jonathan Math	c2bef5565b	add backup to ansible for postgres	2021-06-19 12:35:28 -06:00
Jonathan Math	a71069994c	add mongo backup cronjob for ansible and integrate backup_mongo.py with docker setup	2021-06-19 12:02:29 -06:00
Jonathan Math	8d6a9732b3	use argv instead of env vars for redis log path	2021-06-19 10:03:02 -06:00
Jonathan Math	bf140f1aea	fix disk usage sanity checker to check host machine and pass the data into the container to evaluate the disk usage and send an email	2021-06-18 14:57:29 -06:00
Jonathan Math	7394517a40	try to make redis bgsave monitor work in docker	2021-06-16 11:58:57 -06:00
Jonathan Math	7b9555560f	fix docker commands for redis role	2021-06-16 10:56:08 -06:00
Jonathan Math	b89fb9de3a	fix sanity checkers in redis role	2021-06-16 10:44:47 -06:00
Jonathan Math	56b44213b9	fix redis sanity checker 'when' directive	2021-06-16 10:26:13 -06:00
Jonathan Math	debafdc9ad	fix ansible sanity checkers	2021-06-16 10:12:41 -06:00
Jonathan Math	60e08d6786	make container cron jobs for sanity checkers	2021-06-16 10:00:08 -06:00
Jonathan Math	0b159b19cd	add consul manager service	2021-06-15 09:43:44 -06:00
Jonathan Math	6222db9828	add health checks to prometheus and grafana	2021-06-15 09:32:46 -06:00
Jonathan Math	67b0707eb7	dont register consul services for staging. change datasource for ansible grafana	2021-06-15 09:23:14 -06:00
Samuel Clay	cc7251610a	Disable consul services needs +ie staging.	2021-06-15 11:12:09 -04:00
Jonathan Math	a6d3fff2ba	pin grafana version in ansible and add to consul	2021-06-14 20:32:35 -06:00
Samuel Clay	576f932dea	Make generate to generate inventory.	2021-06-08 11:38:57 -04:00
Jonathan Math	23f1af298a	fix celery_task sanity checker docker container directive	2021-06-08 10:35:57 -04:00
Samuel Clay	8f5e0ba690	Ignore missing task_env.py.	2021-06-08 10:23:51 -04:00
Jonathan Math	2b49288aa7	pull latest grafana image	2021-06-08 08:50:40 -04:00
Jonathan Math	5bea910d46	keep sanity checkers alive, fix ln directive in ansible	2021-06-07 16:30:51 -04:00
Jonathan Math	78120ad11c	keep sanity checker container alive and only use on task_work servers	2021-06-07 16:27:46 -04:00
Jonathan Math	adf7955df5	add the rest of the sanity checkers to ansible	2021-06-07 15:57:01 -04:00
Jonathan Math	17c64e4acc	add disk usage sanity checker	2021-06-07 15:44:59 -04:00
Samuel Clay	d1b73bf86d	Reload haproxy on cert update	2021-06-07 11:20:18 -04:00
Jonathan Math	36f61bd4a8	disable ssl for postgres connection in postgres-exporter	2021-06-03 10:52:27 -04:00