Commit graph

358 commits

Author SHA1 Message Date
Samuel Clay
94fda89ee5 Paths are in different environments. 2022-04-29 17:13:17 -04:00
Samuel Clay
1dc0e98448 Renaming postgresql backups 2022-04-29 16:53:47 -04:00
Samuel Clay
464fecd18e Adding postgres restore command and updating backup script. 2022-04-29 16:45:17 -04:00
Samuel Clay
2d2ed0a30d Adding postgres restore command and updating backup script. 2022-04-29 16:44:46 -04:00
Samuel Clay
65fac88590 Merge branch 'master' into pro
* master:
  Renaming www in terraform until next www server
  Removing audio handler
  Removing unused audio handler shim that turned audio into flash for older browsers.
  Adding global shared stories feeds so they appear on the dashboard even when not subscribed.
  Adding metrics to a single server, let's see how these shake out.
  Don't show prometheus metrics for migrations.
  Adding django prometheus metrics.
  Beginning upgrade to Mongo 4.0 (eventually to 5.0 to get time-series dbs)
2022-04-01 16:47:03 -04:00
Samuel Clay
8e177cdce0 Adding metrics to a single server, let's see how these shake out. 2022-03-31 17:37:03 -04:00
Samuel Clay
893619596b Adding django prometheus metrics. 2022-03-31 15:34:33 -04:00
Samuel Clay
f33cbcf312 Merge branch 'master' into pro
* master: (72 commits)
  Missing import
  Correctly decoding shared stories in RSS feed.
  Adding scroll to feed and story options popover (Style menu).
  Magazine view blog post.
  Refactoring story content preview with newlines preserved.
  Boosting db profiling from 1% to 5%.
  Ignore data uris for widget, they are never the big images
  Wait to strip data uri header until it's time.
  Moving encoding
  Don't double encode base64 images for widgets.
  Preserving paragraphs in magazine and grid views.
  Fixing bad instrumentation on mongo and redis logs.
  Wrong url for db profiler.
  Let's try showing only single feeds in the DB profiler to see if it's better at showing abnormalities with less noise.
  Fixing strict bug where  wasn't defined.
  Showing grid image previews. Not sure how this ever worked before.
  Grid has image if found already.
  Showing grid image on load if youtube
  Only monitor river_stories for db times
  Not hitting a db shouldnt be colored any differently than a small value.
  ...
2022-03-09 11:41:40 -05:00
Samuel Clay
af4b6b2b02 Adding blog to docker and in progress magazine post. 2022-03-04 11:03:57 -05:00
Samuel Clay
2369556817 Adding gzip compression to static assets. 2022-03-02 11:42:44 -05:00
Samuel Clay
c64b614d8c Merge branch 'master' into pro
* master:
  All django boxes the same size.
2022-02-14 12:42:01 -05:00
Samuel Clay
56c4e80c18 All django boxes the same size. 2022-02-14 10:24:24 -05:00
Samuel Clay
5451d4a8eb Merge branch 'master' into pro
* master:
  XoneB 4 LIFE
  Handling recovering mongo db in metrics. (Typo)
  Handling recovering mongo db in metrics.
2022-02-08 15:33:26 -05:00
Samuel Clay
b6763d872f XoneB 4 LIFE 2022-02-08 15:32:43 -05:00
Samuel Clay
88d07b143c Merge branch 'master' into pro
* master: (31 commits)
  Adding postgres backup log.
  Handling operation failures in flask metrics.
  Locking consul to a working version. Also changing flask status code response to handle custom status codes.
  Explicitly uploading mongo backups
  Only need date once in filename
  Rotating mongo backups
  Using full path for progress.
  Showing full path of postgres backup.
  Separating out restoring backups and storing backups.
  Moving to postgres backup directory.
  Executable postgres backup script.
  Compressing Postgres db sql backup.
  Adding date to tarball to allow for versioning.
  Correcting permissions on mongo backup
  Fixing mongo upload to s3 by using host networking mode.
  Removing unused backup code.
  Forgot import
  Adding progress bar to mongo backup.
  Refactoring aws keys for mongo backups.
  Moving mongo backups
  ...
2022-02-04 13:55:38 -05:00
Samuel Clay
0f09de6218 Rotating mongo backups 2022-02-03 16:17:32 -05:00
Samuel Clay
386861a479 Separating out restoring backups and storing backups. 2022-02-03 15:53:42 -05:00
Samuel Clay
5e2c3f7a10 Moving to postgres backup directory. 2022-02-03 15:49:55 -05:00
Samuel Clay
4fb48aa754 Executable postgres backup script. 2022-02-03 15:41:35 -05:00
Samuel Clay
5bde153147 Compressing Postgres db sql backup. 2022-02-03 15:39:36 -05:00
Samuel Clay
1397693d54 Adding date to tarball to allow for versioning. 2022-02-03 15:24:49 -05:00
Samuel Clay
5fdeedbbb2 Correcting permissions on mongo backup 2022-02-03 15:20:08 -05:00
Samuel Clay
a3b35933c3 Fixing mongo upload to s3 by using host networking mode. 2022-02-03 14:34:34 -05:00
Samuel Clay
e8a88740bb Refactoring aws keys for mongo backups. 2022-02-03 14:03:58 -05:00
Samuel Clay
58abe3792e Moving mongo backups 2022-02-03 13:49:23 -05:00
Samuel Clay
eadcfc4426 Adjusting backup for mongo to live on mounted volume. 2022-02-03 13:38:36 -05:00
Samuel Clay
be6e5523d3 Moving starred/shared stories to separate backup. 2022-02-03 13:25:20 -05:00
Samuel Clay
699bbceeba Fixing postgres backup site. Also fixing Mongo backup collections. 2022-02-03 13:21:06 -05:00
Samuel Clay
6615745aac Showing next invoice. 2022-01-26 15:51:26 -05:00
Samuel Clay
02cb4cf2b4 Cleaning up docker compose 2022-01-26 13:35:06 -05:00
Samuel Clay
18aeafffa2 Cleaning up docker compose 2022-01-26 13:33:59 -05:00
Samuel Clay
4e8ffa2094 Deploying new asset pipeline 2022-01-19 15:22:05 -05:00
Samuel Clay
7f4209fd24 Removing unused google-closure-compiler executable. Now using java. 2022-01-19 14:56:26 -05:00
Samuel Clay
fc8f3cb03e Installing Java in deploy image to get closure compiler working. Also updating closure binary path. 2022-01-19 13:01:31 -05:00
Samuel Clay
d0fdbde96b Ensuring dependencies met for deployment on local. 2022-01-19 10:48:11 -05:00
Samuel Clay
0f27ba81e0 Updating to docker buildx for multi-arch builds (amd64 + arm64) 2022-01-18 17:25:07 -05:00
Samuel Clay
41469acd73 Looks like redis wasn't being saved properly in docker-compose. 2022-01-11 17:23:58 -05:00
Samuel Clay
61450de5f8 Merge branch 'master' into pipeline
* master:
  Celery also has a set user.
  Setting the user for celery and newsblur_web in both prod and docker-compose so that logs are always under the right user. Fixes #1593.
  Making add_site callback optional to support iOS. #1584
  Adding check for renaming a feed that a user isn't subscribed to.
  Finishing up PostgreSQL migration. Needs to test backups.
  Handling unreadable POST for push feeds and their fat pings.
  Android v11.2
  Schedule and cancel subscription sync services based on user auth state.
  Check for authenticated user before a subscription sync.
  Subscription sync service running every 24 hours to sync if necessary the user's premium subscription state between NewsBlur and Play Store.
  Upgrade subscriptions to billing 4.0
  Subscription manager to handle billing and subscription syncs between Play Store and NewsBlur.
  #1573 Feed search favicon loading.
  Replace cursor loaders (#9)
  Fixing grafana sentry integration.
  Adding sentry grafana pluging.
  Boosting timeout for docker prune
  Don't prune docker unless needed.
  Update docker-compose.yml
2022-01-11 11:00:06 -05:00
Samuel Clay
09551d5d8f Finishing up PostgreSQL migration. Needs to test backups. 2022-01-07 15:47:36 -05:00
Samuel Clay
5414b4c414 Integrating admin without gziping, as that's causing issues. Ready to go. 2022-01-06 13:32:40 -05:00
Samuel Clay
ddeaf55be0 Cleaning up pipeline and adding admin static assets, but now certain admin static assets are being post-processed multiple times but with different cache manifests, resulting in a file not found error. This needs work. 2022-01-06 11:01:35 -05:00
Samuel Clay
6108f2a067 Adding sentry grafana pluging. 2022-01-05 13:22:24 -05:00
Samuel Clay
b8eccdbed4 Merge branch 'master' into pipeline
* master:
  Fixing names for state timeline, also fixing node server bootstrap.
  Moving up haproxy state timeline
  Bumping haproxy state timeline scrapes.
  Only need haproxy for haproxy metrics
  Refactored state timeline to work as a flask metrics server on haproxy. Needs ansible configs to deploy on www server.
  Forgot to save merge conflict.
  fix service state timeline key error and change scrape interval
  Revert "Revert "Merge branch 'state_timeline'""
  Revert "Merge branch 'state_timeline'"
  Fixing syntax error
  Solving docker networking container issues by turning iptables back on but using ufw after.rules that allow ufw to retain blocking ability. Works on Redis, now to move to mongo.
  Mongo analytics monitor not reayd yet
  Turning consul monitor back on.
  Turning on analytics
  Need to strip csv values.
  Binding redis to public interface.
  add grafana dashboard backup functionality
2022-01-04 14:44:57 -05:00
Samuel Clay
8de2bfcbae Fixing names for state timeline, also fixing node server bootstrap. 2022-01-04 14:44:47 -05:00
Samuel Clay
888d226350 Moving up haproxy state timeline 2022-01-04 12:52:50 -05:00
Samuel Clay
af7524a976 Bumping haproxy state timeline scrapes. 2022-01-04 12:51:45 -05:00
Samuel Clay
213b60bcd6 Refactored state timeline to work as a flask metrics server on haproxy. Needs ansible configs to deploy on www server. 2022-01-04 12:26:47 -05:00
Samuel Clay
93077c79fa Forgot to save merge conflict. 2022-01-04 11:24:24 -05:00
Samuel Clay
0db15560da Merge branch 'master' into state_timeline
* master: (29 commits)
  Pruning docker, updating sentry release webhook only on local.
  Delete .github directory
  Adding Sentry release webhook to deployment of web, task, monitor, and node.
  Add Sentry action workflow for actions
  Moving from hosted Sentry to self-hosted sentry.
  Flask transactions are way too often.
  Cleanup
  Seems redundant with newsblur_web/docker_local_settings.py
  Using new repo password for postgres.
  Moving around postgres config
  Moving directories.
  Adding postgresql backup restoration. Working on config.
  Using user log for db times.
  Last 5 minutes is actually last minute.
  Adding 5min loadtime to prometheus graphs.
  Switching back to nbanalytics with user auth.
  Mongo auth
  Deploying new analytics server.
  Create mongodb database user. Only runs on analytics, but should work fine on main mongo.
  Adding second mongo analytics db, cleaning up mongo analytics auth.
  ...
2022-01-04 11:18:48 -05:00
Samuel Clay
97c489c034 Attempting symlinks for pipeline. 2022-01-04 11:17:04 -05:00