Commit graph

132 commits

Author SHA1 Message Date
Samuel Clay
5f02400567 Re-normalizing dense vectors from 1584 dimensions to 256 dimensions, applying bbq_hnsw index to save 96% memory usage on vector embeddings. 2024-12-24 19:40:48 -05:00
Samuel Clay
be7380a0a4 Small fix to requirements (wrong pyflakes) 2024-12-24 18:05:28 -05:00
Samuel Clay
153cb876b5 Moving discover indexing to its own queue. 2024-12-14 13:39:21 -08:00
Samuel Clay
a49a2f6ca3 Fixing folders disappearing on search. 2024-11-24 13:58:22 -08:00
Samuel Clay
92f881e54b If hostname has staging in it, only 2 workers 2024-07-10 12:28:02 -04:00
Samuel Clay
8d30cec2ec Setting TRANSFORMERS_CACHE 2024-07-10 11:51:20 -04:00
Samuel Clay
0f579cbc03 Using sentence transformers and the MiniLM model to create embeddings for feeds. 2024-06-30 09:13:25 -04:00
Samuel Clay
c63067754d Merge branch 'black-formatting' into discover
* black-formatting:
  Black formatting.
2024-04-24 09:49:17 -04:00
Samuel Clay
d1dafe7606 Black formatting. 2024-04-24 09:43:56 -04:00
Samuel Clay
573476e003 Merge branch 'master' into discover
* master: (84 commits)
  Cleanup postgres
  Adding support for hyphens in youtube usernames.
  New APNS updating cert instructions.
  Fiddling with metrics server.
  Handling broken youtube channel
  Youtube username/title
  Handling youtube usernames that are actually handles.
  Handling @handle youtube.com feeds when adding a feed.
  Users who are too far into paging now get a 404
  Updating youtube fetcher to use channels/playlists/users for everything, no longer relying on RSS/xml url.
  Updating certs.
  Handling too many feeds tasked.
  Bumping concurrency for feed fetchers.
  Icon importer also uses new consul domain.
  User not user agent
  Adding individual IP addresses from hetzner.
  Moving hetzner migration doc to archive
  Formatting
  Adding dwww
  Moving node consul
  ...
2024-04-24 09:25:45 -04:00
Samuel Clay
ec44039cc9 Updating certs. 2024-03-02 08:37:20 -05:00
Samuel Clay
25d6d84971 Bumping concurrency for feed fetchers. 2024-02-26 21:08:40 -05:00
Samuel Clay
dc59b019a3 Moving to new redis ports. 2024-02-25 13:54:14 -05:00
Samuel Clay
56f5ff501a Old task servers should use smaller concurrency. 2024-02-25 12:36:08 -05:00
Samuel Clay
27d5803ca9 SERVER_NAME needs the env 2024-02-18 08:09:08 -05:00
Samuel Clay
f17d12a10c Work servers only need 4 celery. 2024-02-18 08:08:25 -05:00
Samuel Clay
caa7e05ee9 Upping task concurrency 2024-02-18 07:03:21 -05:00
Samuel Clay
f709bd7e18 Updating consul and docker for arm64. Using old redis ports. 2024-02-17 17:14:35 -05:00
Samuel Clay
7a049c1e55 Switching to new redis ports. 2024-02-11 13:48:33 -05:00
Samuel Clay
95a0acbc30 Adding similiar feeds many-to-many field. Using KNN classifier for similar feeds using user-feed-rating csv. Now to productionalize. 2023-12-27 08:24:49 -05:00
Samuel Clay
29c99616e5 Set SURPRISE_DATA_FOLDER envvar. 2023-11-23 13:53:28 -05:00
Samuel Clay
f9b8e0850c Create the surprise data folder if it doesn't exist. 2023-11-23 13:48:15 -05:00
Samuel Clay
3acd6d1f82 It's a wash if this does anything. 2023-11-10 17:13:46 -05:00
Samuel Clay
2feaf087e7 Running scikit-surprise 2023-10-11 09:06:44 -04:00
Samuel Clay
58c01922da Max emails sent per user per day is now 20 emails for story notifications. This affects only two users. 2023-05-02 09:31:43 -04:00
Samuel Clay
9e21d03255 Merging SDIFFSTORE and ZINTERSTORE into a single ZDIFFSTORE, thanks to redis 6.2.0. Requires new docker image. 2022-07-19 21:02:42 -04:00
Samuel Clay
8f1655d7d1 Revert "Merging SDIFFSTORE and ZINTERSTORE into a single ZDIFFSTORE, thanks to redis 6.2.0. Requires new docker image."
This reverts commit b3ec9fd748.
2022-07-18 22:45:22 -04:00
Samuel Clay
b3ec9fd748 Merging SDIFFSTORE and ZINTERSTORE into a single ZDIFFSTORE, thanks to redis 6.2.0. Requires new docker image. 2022-07-18 22:34:07 -04:00
Samuel Clay
108d88b785 Attempting to reduce redis db size by deleting unread stories list when no longer needed. 2022-06-27 12:24:49 -04:00
Samuel Clay
9848bb5051 Bumping archive fetch to 15 minutes. 2022-06-23 10:58:52 -04:00
Samuel Clay
81f51f1723 Handling soft time limits while fetching archives. 2022-06-22 17:40:16 -04:00
Samuel Clay
c20eed028c Attempting to impose longer time limits for archive fetch. 2022-06-22 15:44:39 -04:00
Samuel Clay
5ff6a340f1 Merge branch 'master' into pro
* master:
  Upgrading to latest pymongo v3 to work with mongodb v4.0. Required rewrite of mongodb introspection, now using built-in pymongo monitoring. Should work on both web and celery.
  Removing unused mongo mms utils and configs.
  Updating pymongo to mongodb 4.0 compatible. Any further and we lose mongo wire protocol inspection.
  Full arm64 setup for imageproxy.
  Rearranging TF vars
  Upgrading to MongoDB 4.0, and upping mongo servers.
2022-05-10 20:59:03 -04:00
Samuel Clay
31de0f7a97 Upgrading to latest pymongo v3 to work with mongodb v4.0. Required rewrite of mongodb introspection, now using built-in pymongo monitoring. Should work on both web and celery. 2022-05-10 20:58:47 -04:00
Samuel Clay
0883c20378 Merge branch 'master' into pro
* master:
  Handling missing histogram prometheus counter.
2022-04-18 16:24:43 -04:00
Samuel Clay
dd646d1c4d Handling missing histogram prometheus counter. 2022-04-18 16:24:33 -04:00
Samuel Clay
11f4f57eba Merge branch 'master' into pro
* master:
  Clear prometheus cache on reload.
  Rearranging urls for paypal. Also updating to latest gunicorn.
  Pulling latest docker image in a separate task.
  Update elasticsearch:7.16.3 (Log4j 2.17.1)
2022-04-11 11:47:17 -04:00
Samuel Clay
f7f5425d8f Clear prometheus cache on reload. 2022-04-11 11:03:14 -04:00
Samuel Clay
65fac88590 Merge branch 'master' into pro
* master:
  Renaming www in terraform until next www server
  Removing audio handler
  Removing unused audio handler shim that turned audio into flash for older browsers.
  Adding global shared stories feeds so they appear on the dashboard even when not subscribed.
  Adding metrics to a single server, let's see how these shake out.
  Don't show prometheus metrics for migrations.
  Adding django prometheus metrics.
  Beginning upgrade to Mongo 4.0 (eventually to 5.0 to get time-series dbs)
2022-04-01 16:47:03 -04:00
Samuel Clay
90725d14c1 Don't show prometheus metrics for migrations. 2022-03-31 16:52:29 -04:00
Samuel Clay
893619596b Adding django prometheus metrics. 2022-03-31 15:34:33 -04:00
Samuel Clay
f33cbcf312 Merge branch 'master' into pro
* master: (72 commits)
  Missing import
  Correctly decoding shared stories in RSS feed.
  Adding scroll to feed and story options popover (Style menu).
  Magazine view blog post.
  Refactoring story content preview with newlines preserved.
  Boosting db profiling from 1% to 5%.
  Ignore data uris for widget, they are never the big images
  Wait to strip data uri header until it's time.
  Moving encoding
  Don't double encode base64 images for widgets.
  Preserving paragraphs in magazine and grid views.
  Fixing bad instrumentation on mongo and redis logs.
  Wrong url for db profiler.
  Let's try showing only single feeds in the DB profiler to see if it's better at showing abnormalities with less noise.
  Fixing strict bug where  wasn't defined.
  Showing grid image previews. Not sure how this ever worked before.
  Grid has image if found already.
  Showing grid image on load if youtube
  Only monitor river_stories for db times
  Not hitting a db shouldnt be colored any differently than a small value.
  ...
2022-03-09 11:41:40 -05:00
Samuel Clay
c227a335f3 Use strict 2022-02-28 15:10:25 -05:00
Samuel Clay
d7b81d7b68 Turning off use strict in prep for turning it right back on in the magazine branch. 2022-02-28 14:28:15 -05:00
Samuel Clay
0c8dbda555 Fixing path for closure compiler jar file. Thanks to @louis-pre. This fixes #1631. 2022-02-18 12:47:37 -05:00
Samuel Clay
a1517b6266 Adding paypal sub id and paypalrestsdk library. Paypal has what it needs, now to hook up the API which is still unauthorized. 2022-02-08 12:49:40 -05:00
Samuel Clay
88d07b143c Merge branch 'master' into pro
* master: (31 commits)
  Adding postgres backup log.
  Handling operation failures in flask metrics.
  Locking consul to a working version. Also changing flask status code response to handle custom status codes.
  Explicitly uploading mongo backups
  Only need date once in filename
  Rotating mongo backups
  Using full path for progress.
  Showing full path of postgres backup.
  Separating out restoring backups and storing backups.
  Moving to postgres backup directory.
  Executable postgres backup script.
  Compressing Postgres db sql backup.
  Adding date to tarball to allow for versioning.
  Correcting permissions on mongo backup
  Fixing mongo upload to s3 by using host networking mode.
  Removing unused backup code.
  Forgot import
  Adding progress bar to mongo backup.
  Refactoring aws keys for mongo backups.
  Moving mongo backups
  ...
2022-02-04 13:55:38 -05:00
Samuel Clay
22feba4443 Moving to S3 compatible backup bucket name. 2022-02-02 16:35:48 -05:00
Samuel Clay
d7cdc47016 Adjusting to new Paypal API, although the subscription is not yet relayed through, but it's captured client-side. 2022-01-28 12:08:05 -05:00
Samuel Clay
39047ee938 Merge branch 'master' into pro
* master:
  Task deploy needs single sentry update.
  Turning off PuSH for standalone installs.
2022-01-25 15:46:25 -05:00