Commit graph

1343 commits

Author SHA1 Message Date
Samuel Clay
5b9f6d402f No need to reindex stories once indexed. 2024-12-01 12:32:18 -05:00
Samuel Clay
4cfb64b6bd Adding index debug to all search models. 2024-11-24 13:57:48 -08:00
Samuel Clay
a45170f0dc Adding debug_index 2024-11-24 13:20:37 -08:00
Samuel Clay
74f0f25711 Cleaning up discover stories index, adding discover_indexed field, beginning to split discover/search indexing. Needs further splitting at the task/queue level. 2024-11-21 08:38:14 -08:00
Samuel Clay
a0cf47abaf Separating index discover from index stories. Need to separate out index discover and index story when initially indexing so as to return search results to the user faster. Also need to index stories when adding a feed for premium users. 2024-11-21 07:17:57 -08:00
Samuel Clay
d3280ff95b Splitting DiscoverStory from SearchStory so the indexes can be separate. 2024-11-20 08:37:31 -08:00
Samuel Clay
d04b4b038a Adding API endpoints for discover stories. Need to finish the UI for stories. Then prob add premium limits. 2024-11-06 21:29:07 -08:00
Samuel Clay
a2d3785c7c Merge branch 'master' into discover
* master:
  Backfilling Youtube videos for premium archive subscribers.
2024-11-04 21:58:20 -08:00
Samuel Clay
a5229ce5cb Backfilling Youtube videos for premium archive subscribers. 2024-11-04 21:32:15 -08:00
Samuel Clay
9e7b08b536 Indexing stories with content vectors. 2024-11-03 21:40:27 -08:00
Samuel Clay
1bd3de276f Showing 5 distinct related feeds in title bar. 2024-10-13 13:48:08 -07:00
Samuel Clay
914d07d51e Removing dupe feeds based on feed title. 2024-10-13 13:37:04 -07:00
Samuel Clay
f17f5f25ae Adding paging and infinite scroll to discover popover. Needs de-dupe and a refactor. 2024-10-13 13:32:27 -07:00
Samuel Clay
2931e5d578 Moving to OpenAI Embeddings API. 2024-10-10 21:56:49 -07:00
Samuel Clay
7e68b00d13 Adding discover to folders of feeds. Holy moly this is nearly done. 2024-07-12 18:16:11 -04:00
Samuel Clay
711a6fbbfc Removing unused scikit-learn code. 2024-07-06 08:01:28 -04:00
Samuel Clay
c31893f669 Merge branch 'master' into discover
* master:
  iOS 15 screenshots
  Don't use the underscore cache buster anymore.
  Correct archive sub count
  Don't resync stories on new sub if it already has archive subscribers.
2024-07-06 07:25:03 -04:00
Samuel Clay
aa84a2c1b1 Don't resync stories on new sub if it already has archive subscribers. 2024-06-30 20:52:02 -04:00
Samuel Clay
bb6400f59a Hooking up new elasticsearch discovery. 2024-06-30 20:27:30 -04:00
Samuel Clay
e862793e01 Merge branch 'master' into discover
* master:
  More numpy changes.
  Strip underscores from feed addresses, special case jwz
2024-06-30 12:34:14 -04:00
Samuel Clay
546f338b8c Strip underscores from feed addresses, special case jwz 2024-06-30 12:13:29 -04:00
Samuel Clay
0f579cbc03 Using sentence transformers and the MiniLM model to create embeddings for feeds. 2024-06-30 09:13:25 -04:00
Samuel Clay
55dda973da Merge branch 'master' into discover
* master:
  Allowing traffic on docker0 interface.
  Clean TXT records so letsencrypt doesn't fail
  Thresholding bounced newsletters email.
  Another type of invalid url on the widget.
  Handling broken urls on widget
  Handling broken urls on image extraction.
  Handling invalid forgot password form.
  Fixing global shared stories blurblog.
  https for stripe urls. Fixes #1865.
2024-06-29 08:19:51 -04:00
Samuel Clay
4a962dc7b7 Handling broken urls on image extraction. 2024-05-30 11:49:21 -04:00
Samuel Clay
da0275d7e6 Merge branch 'black-formatting' into discover
* black-formatting:
  Black formatting and isort
2024-04-24 09:52:22 -04:00
Samuel Clay
7165aa1bf6 Black formatting and isort 2024-04-24 09:50:42 -04:00
Samuel Clay
c63067754d Merge branch 'black-formatting' into discover
* black-formatting:
  Black formatting.
2024-04-24 09:49:17 -04:00
Samuel Clay
d1dafe7606 Black formatting. 2024-04-24 09:43:56 -04:00
Samuel Clay
573476e003 Merge branch 'master' into discover
* master: (84 commits)
  Cleanup postgres
  Adding support for hyphens in youtube usernames.
  New APNS updating cert instructions.
  Fiddling with metrics server.
  Handling broken youtube channel
  Youtube username/title
  Handling youtube usernames that are actually handles.
  Handling @handle youtube.com feeds when adding a feed.
  Users who are too far into paging now get a 404
  Updating youtube fetcher to use channels/playlists/users for everything, no longer relying on RSS/xml url.
  Updating certs.
  Handling too many feeds tasked.
  Bumping concurrency for feed fetchers.
  Icon importer also uses new consul domain.
  User not user agent
  Adding individual IP addresses from hetzner.
  Moving hetzner migration doc to archive
  Formatting
  Adding dwww
  Moving node consul
  ...
2024-04-24 09:25:45 -04:00
Samuel Clay
2f4d823303 Handling @handle youtube.com feeds when adding a feed. 2024-03-03 13:06:08 -05:00
Samuel Clay
c8162d142b Renaming discover to similar. 2023-12-27 11:41:40 -05:00
Samuel Clay
95a0acbc30 Adding similiar feeds many-to-many field. Using KNN classifier for similar feeds using user-feed-rating csv. Now to productionalize. 2023-12-27 08:24:49 -05:00
Samuel Clay
a22b25b1f9 Rounding reach score 2023-11-23 16:18:00 -05:00
Samuel Clay
bf49f47460 Faster discover feed count 2023-11-23 14:31:07 -05:00
Samuel Clay
849e3f1a76 Random feed ids. 2023-11-23 14:27:51 -05:00
Samuel Clay
6085d97a05 Only storing active feeds 2023-11-16 07:50:51 -05:00
Samuel Clay
aea20beb29 Merge branch 'master' into discover
* master: (41 commits)
  Handling missing story content.
  Android v13.1.2.
  Dashboard with saved searches for missing feeds shouldn't crash.
  Handle null is_archived flag on user details json.
  #1815 Story detail reading horizontal scrolling fix
  Android v13.1.1
  Gson proguard rules
  Android v13.1.0
  Use coroutines for thawing stories in the reading adapter
  Kt file extension
  Provide the image ok http client
  #1811 Verify error handling and show loading indicator to the user when adding a feed.
  Fix broken swipe to refresh dependency
  Use plugin information from buildSrc
  Use dependencies and constants from buildSrc
  Add buildSrc for dependency management
  Convert Groovy to Kotlin. Update dependencies
  Convert Groovy to Kotlin
  Update sub service to cancel job when requested. Execute on the IO dispatcher.
  Await for sub service termination
  ...
2023-10-09 13:34:48 -04:00
Samuel Clay
c25f3da5f1 Handling missing story content. 2023-10-09 13:34:29 -04:00
Samuel Clay
091b079b63 Asynchronously loading discover feed data. Need to show story previews and then hook up to read data. Also would like to add a folder chooser directly to add. 2023-08-13 12:28:00 -04:00
Samuel Clay
28b66f9ce6 Stubbing in discover feed icons in popover on feeds. 2023-06-28 07:45:28 -04:00
Samuel Clay
5aa06c393f Adding support for subscribing to Mastodon usernames (newsblur@mastodon.social for instance). 2022-11-25 10:08:07 -05:00
Samuel Clay
855a96d633 Fixing NYPost feed by ensuring enclosures with href attrbiutes are added to stories even without content. Thanks @kleinmatic! 2022-11-23 12:21:46 -05:00
Samuel Clay
e83a8e5a9b Revert "Bumping min to fetch by 1.5X to root out 12 minute broken feed. If this succeeds, narrow down the min_to_decay and then find the feed."
This reverts commit fd16bb0b73.
2022-09-11 08:45:29 -04:00
Samuel Clay
fd16bb0b73 Bumping min to fetch by 1.5X to root out 12 minute broken feed. If this succeeds, narrow down the min_to_decay and then find the feed. 2022-09-11 04:54:35 -04:00
Samuel Clay
cdf5a36b81 Don't delete redis keys because they take time to rebuild and subs can be counted incorrectly during that time. 2022-08-10 17:45:25 -04:00
Samuel Clay
78547af24b Adding cutoff from trimming stories for memory profiling. 2022-07-27 08:19:46 -04:00
Samuel Clay
381953ee78 Bumping highlights max from 1024 characters to 16384 characters. 2022-07-19 10:00:16 -04:00
Samuel Clay
c94bb6640d Improving redis performance by reading the config. 2022-07-18 21:10:42 -04:00
Samuel Clay
3c0cab8d8e Preserve feed even if no subs. 2022-07-01 13:48:10 -04:00
Samuel Clay
0e42edeb89 Including size in feed title 2022-06-27 13:16:08 -04:00