diff --git a/.gitignore b/.gitignore index 9b8cd37..56669e5 100644 --- a/.gitignore +++ b/.gitignore @@ -67,4 +67,5 @@ examples/*.flac examples/*.acc examples/*.ogg examples/speech.mp3 -examples/phoneme_examples/output/*.wav \ No newline at end of file +examples/phoneme_examples/output/*.wav +examples/assorted_checks/benchmarks/output_audio/* diff --git a/CHANGELOG.md b/CHANGELOG.md index cf31282..ac6d393 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,34 @@ Notable changes to this project will be documented in this file. +## [v0.1.4] - 2025-01-30 +### Added +- Smart Chunking System: + - New text_processor with smart_split for improved sentence boundary detection + - Dynamically adjusts chunk sizes based on sentence structure, using phoneme/token information in an intial pass + - Should avoid ever going over the 510 limit per chunk, while preserving natural cadence +- Web UI Added (To Be Replacing Gradio): + - Integrated streaming with tempfile generation + - Download links available in X-Download-Path header + - Configurable cleanup triggers for temp files +- Debug Endpoints: + - /debug/threads for thread information and stack traces + - /debug/storage for temp file and output directory monitoring + - /debug/system for system resource information + - /debug/session_pools for ONNX/CUDA session status +- Automated Model Management: + - Auto-download from releases page + - Included download scripts for manual installation + - Pre-packaged voice models in repository + +### Changed +- Significant architectural improvements: + - Multi-model architecture support + - Enhanced concurrency handling + - Improved streaming header management + - Better resource/session pool management + + ## [v0.1.2] - 2025-01-23 ### Structural Improvements - Models can be manually download and placed in api/src/models, or use included script diff --git a/README.md b/README.md index 44d9600..71f9d74 100644 --- a/README.md +++ b/README.md @@ -3,51 +3,58 @@

# _`FastKoko`_ -[![Tests](https://img.shields.io/badge/tests-104%20passed-darkgreen)]() +[![Tests](https://img.shields.io/badge/tests-100%20passed-darkgreen)]() [![Coverage](https://img.shields.io/badge/coverage-49%25-grey)]() [![Tested at Model Commit](https://img.shields.io/badge/last--tested--model--commit-a67f113-blue)](https://huggingface.co/hexgrad/Kokoro-82M/tree/c3b0d86e2a980e027ef71c28819ea02e351c2667) [![Try on Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Try%20on-Spaces-blue)](https://huggingface.co/spaces/Remsky/Kokoro-TTS-Zero) -> Pre-release. Not fully tested +> Support for Kokoro-82M v1.0 coming very soon! Dockerized FastAPI wrapper for [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) text-to-speech model - OpenAI-compatible Speech endpoint, with inline voice combination, and mapped naming/models for strict systems - NVIDIA GPU accelerated or CPU inference (ONNX, Pytorch) - very fast generation time - - 35x-100x+ real time speed via 4060Ti+ - - 5x+ real time speed via M3 Pro CPU -- streaming support w/ variable chunking to control latency, (new) improved concurrency + - ~35x-100x+ real time speed via 4060Ti+ + - ~5x+ real time speed via M3 Pro CPU +- streaming support & tempfile generation - phoneme based dev endpoints - (new) Integrated web UI on localhost:8880/web +- (new) Debug endpoints for monitoring threads, storage, and session pools -> [!Tip] -> You can try the new beta version from the `v0.1.2-pre` branch now: - - - - - -
- Voice Analysis Comparison - -
    -
  • Integrated web UI (on localhost:8880/web)
  • -
  • Better concurrency handling, baked in models and voices
  • -
  • Voice name/model mappings to OAI standard
  • -
     # with:
    -docker run -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-cpu:latest # CPU
    -docker run --gpus all -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-gpu:latest # Nvidia GPU
    -        
    -
-
-
-Quick Start +## Get Started -The service can be accessed through either the API endpoints or the Gradio web interface. +
+Quickest Start (docker run) + + +Pre built images are available to run, with arm/multi-arch support, and baked in models +Refer to the core/config.py file for a full list of variables which can be managed via the environment + +```bash + +docker run -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-cpu:v0.1.4 # CPU, or: +docker run --gpus all -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-gpu:v0.1.4 #NVIDIA GPU +``` + +Once running, access: +- API Documentation: http://localhost:8880/docs +- Web Interface: http://localhost:8880/web + +
+ API Documentation + Web UI Screenshot +
+ +
+ + +
+ +Quick Start (docker compose) 1. Install prerequisites, and start the service using Docker Compose (Full setup including UI): - Install [Docker](https://www.docker.com/products/docker-desktop/) - + - - Clone the repository: ```bash git clone https://github.com/remsky/Kokoro-FastAPI.git @@ -60,20 +67,18 @@ The service can be accessed through either the API endpoints or the Gradio web i # python ../scripts/download_model.py --type pth # for GPU # python ../scripts/download_model.py --type onnx # for CPU ``` - + + ```bash + Or directly via UV + ./start-cpu.sh + ./start-gpu.sh + ``` + Once started: - The API will be available at http://localhost:8880 - The *Web UI* can be tested at http://localhost:8880/web - The Gradio UI (deprecating) can be accessed at http://localhost:7860 - - __Or__ running the API alone using Docker (model + voice packs baked in) (Most Recent): - - ```bash - docker run -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-cpu:v0.1.2 # CPU - docker run --gpus all -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-gpu:v0.1.2 # Nvidia GPU - ``` - - + 2. Run locally as an OpenAI-Compatible Speech Endpoint ```python from openai import OpenAI @@ -91,13 +96,50 @@ The service can be accessed through either the API endpoints or the Gradio web i response.stream_to_file("output.mp3") ``` +
+Direct Run (via uv) -
-
- Beta Web UI - Voice Analysis Comparison -
-
+1. Install prerequisites (): + - Install [astral-uv](https://docs.astral.sh/uv/) + - Clone the repository: + ```bash + git clone https://github.com/remsky/Kokoro-FastAPI.git + cd Kokoro-FastAPI + + # if you are missing any models, run: + # python ../scripts/download_model.py --type pth # for GPU + # python ../scripts/download_model.py --type onnx # for CPU + ``` + + Start directly via UV (with hot-reload) + ```bash + ./start-cpu.sh OR + ./start-gpu.sh + ``` + + Once started: + - The API will be available at http://localhost:8880 + - The *Web UI* can be tested at http://localhost:8880/web + - The Gradio UI (deprecating) can be accessed at http://localhost:7860 + +2. Run locally as an OpenAI-Compatible Speech Endpoint + ```python + from openai import OpenAI + client = OpenAI( + base_url="http://localhost:8880/v1", + api_key="not-needed" + ) + + with client.audio.speech.with_streaming_response.create( + model="kokoro", + voice="af_sky+af_bella", #single or multiple voicepack combo + input="Hello world!", + response_format="mp3" + ) as response: + response.stream_to_file("output.mp3") + + ``` +
## Features
@@ -211,13 +253,12 @@ If you only want the API, just comment out everything in the docker-compose.yml Currently, voices created via the API are accessible here, but voice combination/creation has not yet been added -Running the UI Docker Service +Running the UI Docker Service [deprecating] - If you only want to run the Gradio web interface separately and connect it to an existing API service: ```bash docker run -p 7860:7860 \ -e API_HOST= \ -e API_PORT=8880 \ - ghcr.io/remsky/kokoro-fastapi-ui:v0.1.0 ``` - Replace `` with: @@ -236,7 +277,7 @@ environment: When running the Docker image directly: ```bash -docker run -p 7860:7860 -e DISABLE_LOCAL_SAVING=true ghcr.io/remsky/kokoro-fastapi-ui:latest +docker run -p 7860:7860 -e DISABLE_LOCAL_SAVING=true ghcr.io/remsky/kokoro-fastapi-ui:v0.1.4 ```
@@ -247,7 +288,7 @@ docker run -p 7860:7860 -e DISABLE_LOCAL_SAVING=true ghcr.io/remsky/kokoro-fasta # OpenAI-compatible streaming from openai import OpenAI client = OpenAI( - base_url="http://localhost:8880", api_key="not-needed") + base_url="http://localhost:8880/v1", api_key="not-needed") # Stream to file with client.audio.speech.with_streaming_response.create( @@ -329,17 +370,17 @@ Benchmarking was performed on generation via the local API using text lengths up

Key Performance Metrics: -- Realtime Speed: Ranges between 25-50x (generation time to output audio length) +- Realtime Speed: Ranges between 35x-100x (generation time to output audio length) - Average Processing Rate: 137.67 tokens/second (cl100k_base)
GPU Vs. CPU ```bash -# GPU: Requires NVIDIA GPU with CUDA 12.1 support (~35x realtime speed) +# GPU: Requires NVIDIA GPU with CUDA 12.1 support (~35x-100x realtime speed) docker compose up --build -# CPU: ONNX optimized inference (~2.4x realtime speed) +# CPU: ONNX optimized inference (~5x+ realtime speed on M3 Pro) docker compose -f docker-compose.cpu.yml up --build ``` *Note: Overall speed may have reduced somewhat with the structural changes to accomodate streaming. Looking into it* @@ -359,36 +400,61 @@ Convert text to phonemes and/or generate audio directly from phonemes: ```python import requests -# Convert text to phonemes -response = requests.post( - "http://localhost:8880/dev/phonemize", - json={ - "text": "Hello world!", - "language": "a" # "a" for American English - } -) -result = response.json() -phonemes = result["phonemes"] # Phoneme string e.g ðɪs ɪz ˈoʊnli ɐ tˈɛst -tokens = result["tokens"] # Token IDs including start/end tokens +def get_phonemes(text: str, language: str = "a"): + """Get phonemes and tokens for input text""" + response = requests.post( + "http://localhost:8880/dev/phonemize", + json={"text": text, "language": language} # "a" for American English + ) + response.raise_for_status() + result = response.json() + return result["phonemes"], result["tokens"] -# Generate audio from phonemes -response = requests.post( - "http://localhost:8880/dev/generate_from_phonemes", - json={ - "phonemes": phonemes, - "voice": "af_bella", - "speed": 1.0 - } -) +def generate_audio_from_phonemes(phonemes: str, voice: str = "af_bella"): + """Generate audio from phonemes""" + response = requests.post( + "http://localhost:8880/dev/generate_from_phonemes", + json={"phonemes": phonemes, "voice": voice}, + headers={"Accept": "audio/wav"} + ) + if response.status_code != 200: + print(f"Error: {response.text}") + return None + return response.content -# Save WAV audio -with open("speech.wav", "wb") as f: - f.write(response.content) +# Example usage +text = "Hello world!" +try: + # Convert text to phonemes + phonemes, tokens = get_phonemes(text) + print(f"Phonemes: {phonemes}") # e.g. ðɪs ɪz ˈoʊnli ɐ tˈɛst + print(f"Tokens: {tokens}") # Token IDs including start/end tokens + + # Generate and save audio + if audio_bytes := generate_audio_from_phonemes(phonemes): + with open("speech.wav", "wb") as f: + f.write(audio_bytes) + print(f"Generated {len(audio_bytes)} bytes of audio") +except Exception as e: + print(f"Error: {e}") ``` See `examples/phoneme_examples/generate_phonemes.py` for a sample script.
+
+Debug Endpoints + +Monitor system state and resource usage with these endpoints: + +- `/debug/threads` - Get thread information and stack traces +- `/debug/storage` - Monitor temp file and output directory usage +- `/debug/system` - Get system information (CPU, memory, GPU) +- `/debug/session_pools` - View ONNX session and CUDA stream status + +Useful for debugging resource exhaustion or performance issues. +
+ ## Known Issues
diff --git a/api/src/inference/session_pool.py b/api/src/inference/session_pool.py index d3d108e..2dafa61 100644 --- a/api/src/inference/session_pool.py +++ b/api/src/inference/session_pool.py @@ -119,13 +119,17 @@ class BaseSessionPool: # Clean expired sessions self._cleanup_expired() + # TODO: Change session tracking to use unique IDs instead of model paths + # This would allow multiple instances of the same model + # Check if session exists and is valid if model_path in self._sessions: session_info = self._sessions[model_path] session_info.last_used = time.time() return session_info.session - # Check if we can create new session + # TODO: Modify session limit check to count instances per model path + # Rather than total sessions across all models if len(self._sessions) >= self._max_size: raise RuntimeError( f"Maximum number of sessions reached ({self._max_size}). " diff --git a/api/src/services/tts_service.py b/api/src/services/tts_service.py index 916e236..f152b62 100644 --- a/api/src/services/tts_service.py +++ b/api/src/services/tts_service.py @@ -50,6 +50,10 @@ class TTSService: try: # Handle stream finalization if is_last: + # Skip format conversion for raw audio mode + if not output_format: + return np.array([], dtype=np.float32) + return await AudioService.convert_audio( np.array([0], dtype=np.float32), # Dummy data for type checking 24000, @@ -229,7 +233,7 @@ class TTSService: try: # Use streaming generator but collect all chunks async for chunk in self.generate_audio_stream( - text, voice, speed, output_format=None + text, voice, speed, # Default to WAV for raw audio ): if chunk is not None: chunks.append(chunk) diff --git a/assets/docs-screenshot.png b/assets/docs-screenshot.png new file mode 100644 index 0000000..d90bd5f Binary files /dev/null and b/assets/docs-screenshot.png differ diff --git a/assets/webui-screenshot.png b/assets/webui-screenshot.png new file mode 100644 index 0000000..9037e7e Binary files /dev/null and b/assets/webui-screenshot.png differ diff --git a/examples/assorted_checks/benchmarks/benchmark_tts_rtf.py b/examples/assorted_checks/benchmarks/benchmark_tts_rtf.py index 7e4940d..165da42 100644 --- a/examples/assorted_checks/benchmarks/benchmark_tts_rtf.py +++ b/examples/assorted_checks/benchmarks/benchmark_tts_rtf.py @@ -68,7 +68,7 @@ def main(): # Initialize system monitor monitor = SystemMonitor(interval=1.0) # 1 second interval # Set prefix for output files (e.g. "gpu", "cpu", "onnx", etc.) - prefix = "cpu" + prefix = "gpu" # Generate token sizes if "gpu" in prefix: token_sizes = generate_token_sizes( diff --git a/examples/assorted_checks/benchmarks/output_data/cpu_benchmark_results_rtf.json b/examples/assorted_checks/benchmarks/output_data/cpu_benchmark_results_rtf.json index 447b2d0..f9d41c9 100644 --- a/examples/assorted_checks/benchmarks/output_data/cpu_benchmark_results_rtf.json +++ b/examples/assorted_checks/benchmarks/output_data/cpu_benchmark_results_rtf.json @@ -1,2220 +1,138 @@ { "results": [ + { + "tokens": 100, + "processing_time": 0.98, + "output_length": 28.975, + "rtf": 0.03, + "elapsed_time": 1.02255 + }, + { + "tokens": 200, + "processing_time": 1.79, + "output_length": 58.45, + "rtf": 0.03, + "elapsed_time": 2.84766 + }, { "tokens": 300, - "processing_time": 56.61, - "output_length": 96.425, - "rtf": 0.59, - "elapsed_time": 56.70357 + "processing_time": 2.1, + "output_length": 86.75, + "rtf": 0.02, + "elapsed_time": 4.98201 }, { - "tokens": 600, - "processing_time": 105.39, - "output_length": 188.675, - "rtf": 0.56, - "elapsed_time": 162.20408 + "tokens": 400, + "processing_time": 2.66, + "output_length": 113.5, + "rtf": 0.02, + "elapsed_time": 7.67743 }, { - "tokens": 900, - "processing_time": 143.67, - "output_length": 283.425, - "rtf": 0.51, - "elapsed_time": 306.01951 + "tokens": 500, + "processing_time": 3.13, + "output_length": 140.225, + "rtf": 0.02, + "elapsed_time": 10.84279 } ], "system_metrics": [ { - "timestamp": "2025-01-06T01:59:21.480266", - "cpu_percent": 18.78, - "ram_percent": 47.7, - "ram_used_gb": 30.34305191040039, - "gpu_memory_used": 2069.0, - "relative_time": 0.09699058532714844 - }, - { - "timestamp": "2025-01-06T01:59:22.587112", - "cpu_percent": 45.13, - "ram_percent": 47.7, - "ram_used_gb": 30.35516357421875, - "gpu_memory_used": 2068.0, - "relative_time": 1.2145774364471436 - }, - { - "timestamp": "2025-01-06T01:59:23.704498", - "cpu_percent": 72.19, - "ram_percent": 47.7, - "ram_used_gb": 30.345977783203125, - "gpu_memory_used": 2064.0, - "relative_time": 2.308037042617798 - }, - { - "timestamp": "2025-01-06T01:59:24.792248", - "cpu_percent": 74.04, - "ram_percent": 47.7, - "ram_used_gb": 30.324016571044922, - "gpu_memory_used": 2063.0, - "relative_time": 3.423358201980591 - }, - { - "timestamp": "2025-01-06T01:59:25.912823", - "cpu_percent": 72.79, - "ram_percent": 47.7, - "ram_used_gb": 30.340232849121094, - "gpu_memory_used": 2060.0, - "relative_time": 4.514510154724121 - }, - { - "timestamp": "2025-01-06T01:59:27.003820", - "cpu_percent": 65.58, - "ram_percent": 47.7, - "ram_used_gb": 30.333145141601562, - "gpu_memory_used": 2060.0, - "relative_time": 5.624518871307373 - }, - { - "timestamp": "2025-01-06T01:59:28.108050", - "cpu_percent": 73.56, - "ram_percent": 47.7, - "ram_used_gb": 30.33941650390625, - "gpu_memory_used": 2060.0, - "relative_time": 6.728093385696411 - }, - { - "timestamp": "2025-01-06T01:59:29.209197", - "cpu_percent": 74.51, - "ram_percent": 47.7, - "ram_used_gb": 30.35086441040039, - "gpu_memory_used": 2060.0, - "relative_time": 7.804106950759888 - }, - { - "timestamp": "2025-01-06T01:59:30.285102", - "cpu_percent": 62.56, - "ram_percent": 47.7, - "ram_used_gb": 30.335765838623047, - "gpu_memory_used": 2060.0, - "relative_time": 8.90535569190979 - }, - { - "timestamp": "2025-01-06T01:59:31.388382", - "cpu_percent": 70.13, - "ram_percent": 47.7, - "ram_used_gb": 30.331851959228516, - "gpu_memory_used": 2060.0, - "relative_time": 10.00930905342102 - }, - { - "timestamp": "2025-01-06T01:59:32.493094", - "cpu_percent": 73.96, - "ram_percent": 47.7, - "ram_used_gb": 30.34304428100586, - "gpu_memory_used": 2060.0, - "relative_time": 11.135397672653198 - }, - { - "timestamp": "2025-01-06T01:59:33.626989", - "cpu_percent": 76.87, - "ram_percent": 47.7, - "ram_used_gb": 30.352691650390625, - "gpu_memory_used": 2060.0, - "relative_time": 12.232722282409668 - }, - { - "timestamp": "2025-01-06T01:59:34.720747", - "cpu_percent": 60.67, - "ram_percent": 47.7, - "ram_used_gb": 30.355037689208984, - "gpu_memory_used": 2060.0, - "relative_time": 13.359250783920288 - }, - { - "timestamp": "2025-01-06T01:59:35.840613", - "cpu_percent": 72.88, - "ram_percent": 47.7, - "ram_used_gb": 30.32884979248047, - "gpu_memory_used": 2060.0, - "relative_time": 14.481508016586304 - }, - { - "timestamp": "2025-01-06T01:59:36.967478", - "cpu_percent": 66.28, - "ram_percent": 47.7, - "ram_used_gb": 30.325313568115234, - "gpu_memory_used": 2060.0, - "relative_time": 15.61866569519043 - }, - { - "timestamp": "2025-01-06T01:59:38.100358", - "cpu_percent": 81.44, - "ram_percent": 47.7, - "ram_used_gb": 30.32506561279297, - "gpu_memory_used": 2055.0, - "relative_time": 16.75141143798828 - }, - { - "timestamp": "2025-01-06T01:59:39.234215", - "cpu_percent": 78.94, - "ram_percent": 47.7, - "ram_used_gb": 30.32697296142578, - "gpu_memory_used": 2064.0, - "relative_time": 17.868074655532837 - }, - { - "timestamp": "2025-01-06T01:59:40.353820", - "cpu_percent": 90.03, - "ram_percent": 47.6, - "ram_used_gb": 30.304851531982422, - "gpu_memory_used": 2064.0, - "relative_time": 18.969531536102295 - }, - { - "timestamp": "2025-01-06T01:59:41.459510", - "cpu_percent": 69.47, - "ram_percent": 47.6, - "ram_used_gb": 30.290199279785156, - "gpu_memory_used": 2064.0, - "relative_time": 20.0858416557312 - }, - { - "timestamp": "2025-01-06T01:59:42.576804", - "cpu_percent": 74.6, - "ram_percent": 47.7, - "ram_used_gb": 30.310714721679688, - "gpu_memory_used": 2064.0, - "relative_time": 21.232748985290527 - }, - { - "timestamp": "2025-01-06T01:59:43.714609", - "cpu_percent": 71.29, - "ram_percent": 47.6, - "ram_used_gb": 30.309490203857422, - "gpu_memory_used": 2064.0, - "relative_time": 22.3442165851593 - }, - { - "timestamp": "2025-01-06T01:59:44.828524", - "cpu_percent": 72.72, - "ram_percent": 47.6, - "ram_used_gb": 30.286724090576172, - "gpu_memory_used": 2064.0, - "relative_time": 23.47381353378296 - }, - { - "timestamp": "2025-01-06T01:59:45.965351", - "cpu_percent": 75.83, - "ram_percent": 47.6, - "ram_used_gb": 30.27065658569336, - "gpu_memory_used": 2064.0, - "relative_time": 24.611122608184814 - }, - { - "timestamp": "2025-01-06T01:59:47.101999", - "cpu_percent": 69.91, - "ram_percent": 47.6, - "ram_used_gb": 30.303489685058594, - "gpu_memory_used": 2064.0, - "relative_time": 25.740842819213867 - }, - { - "timestamp": "2025-01-06T01:59:48.224089", - "cpu_percent": 69.72, - "ram_percent": 47.6, - "ram_used_gb": 30.308269500732422, - "gpu_memory_used": 2064.0, - "relative_time": 26.859923839569092 - }, - { - "timestamp": "2025-01-06T01:59:49.344323", - "cpu_percent": 72.56, - "ram_percent": 47.7, - "ram_used_gb": 30.340702056884766, - "gpu_memory_used": 2077.0, - "relative_time": 27.983789205551147 - }, - { - "timestamp": "2025-01-06T01:59:50.476329", - "cpu_percent": 70.04, - "ram_percent": 47.6, - "ram_used_gb": 30.294044494628906, - "gpu_memory_used": 2077.0, - "relative_time": 29.103161811828613 - }, - { - "timestamp": "2025-01-06T01:59:51.584187", - "cpu_percent": 60.73, - "ram_percent": 47.6, - "ram_used_gb": 30.3031005859375, - "gpu_memory_used": 2077.0, - "relative_time": 30.226102352142334 - }, - { - "timestamp": "2025-01-06T01:59:52.714254", - "cpu_percent": 69.69, - "ram_percent": 47.6, - "ram_used_gb": 30.30960464477539, - "gpu_memory_used": 2076.0, - "relative_time": 31.348071575164795 - }, - { - "timestamp": "2025-01-06T01:59:53.833567", - "cpu_percent": 79.26, - "ram_percent": 47.7, - "ram_used_gb": 30.327259063720703, - "gpu_memory_used": 2076.0, - "relative_time": 32.47895312309265 - }, - { - "timestamp": "2025-01-06T01:59:54.969955", - "cpu_percent": 76.92, - "ram_percent": 47.6, - "ram_used_gb": 30.295856475830078, - "gpu_memory_used": 2075.0, - "relative_time": 33.593998670578 - }, - { - "timestamp": "2025-01-06T01:59:56.074960", - "cpu_percent": 81.26, - "ram_percent": 47.6, - "ram_used_gb": 30.309158325195312, - "gpu_memory_used": 2066.0, - "relative_time": 34.720394134521484 - }, - { - "timestamp": "2025-01-06T01:59:57.210513", - "cpu_percent": 82.42, - "ram_percent": 47.7, - "ram_used_gb": 30.345314025878906, - "gpu_memory_used": 2068.0, - "relative_time": 35.84532642364502 - }, - { - "timestamp": "2025-01-06T01:59:58.330832", - "cpu_percent": 95.36, - "ram_percent": 47.9, - "ram_used_gb": 30.43775177001953, - "gpu_memory_used": 2069.0, - "relative_time": 36.99730587005615 - }, - { - "timestamp": "2025-01-06T01:59:59.487468", - "cpu_percent": 100.0, - "ram_percent": 47.8, - "ram_used_gb": 30.407115936279297, - "gpu_memory_used": 2069.0, - "relative_time": 38.232134103775024 - }, - { - "timestamp": "2025-01-06T02:00:00.714760", - "cpu_percent": 100.0, - "ram_percent": 48.3, - "ram_used_gb": 30.697307586669922, - "gpu_memory_used": 2069.0, - "relative_time": 39.4050030708313 - }, - { - "timestamp": "2025-01-06T02:00:01.888028", - "cpu_percent": 95.29, - "ram_percent": 48.0, - "ram_used_gb": 30.559246063232422, - "gpu_memory_used": 2069.0, - "relative_time": 40.55751347541809 - }, - { - "timestamp": "2025-01-06T02:00:03.044225", - "cpu_percent": 96.41, - "ram_percent": 48.1, - "ram_used_gb": 30.57846450805664, - "gpu_memory_used": 2070.0, - "relative_time": 41.67921280860901 - }, - { - "timestamp": "2025-01-06T02:00:04.173396", - "cpu_percent": 98.56, - "ram_percent": 48.1, - "ram_used_gb": 30.579666137695312, - "gpu_memory_used": 2070.0, - "relative_time": 42.861982583999634 - }, - { - "timestamp": "2025-01-06T02:00:05.356206", - "cpu_percent": 98.14, - "ram_percent": 48.3, - "ram_used_gb": 30.72429656982422, - "gpu_memory_used": 2070.0, - "relative_time": 44.000171184539795 - }, - { - "timestamp": "2025-01-06T02:00:06.485051", - "cpu_percent": 95.54, - "ram_percent": 48.0, - "ram_used_gb": 30.531700134277344, - "gpu_memory_used": 2069.0, - "relative_time": 45.14318823814392 - }, - { - "timestamp": "2025-01-06T02:00:07.624181", - "cpu_percent": 96.77, - "ram_percent": 48.1, - "ram_used_gb": 30.568119049072266, - "gpu_memory_used": 2070.0, - "relative_time": 46.264710903167725 - }, - { - "timestamp": "2025-01-06T02:00:08.758279", - "cpu_percent": 99.74, - "ram_percent": 48.0, - "ram_used_gb": 30.530475616455078, - "gpu_memory_used": 2071.0, - "relative_time": 47.39181351661682 - }, - { - "timestamp": "2025-01-06T02:00:09.876762", - "cpu_percent": 98.61, - "ram_percent": 48.1, - "ram_used_gb": 30.617321014404297, - "gpu_memory_used": 2072.0, - "relative_time": 48.66594886779785 - }, - { - "timestamp": "2025-01-06T02:00:11.145450", - "cpu_percent": 100.0, - "ram_percent": 48.2, - "ram_used_gb": 30.632938385009766, - "gpu_memory_used": 2071.0, - "relative_time": 49.81949210166931 - }, - { - "timestamp": "2025-01-06T02:00:12.300318", - "cpu_percent": 100.0, - "ram_percent": 48.3, - "ram_used_gb": 30.732288360595703, - "gpu_memory_used": 2068.0, - "relative_time": 50.97749900817871 - }, - { - "timestamp": "2025-01-06T02:00:13.467022", - "cpu_percent": 98.76, - "ram_percent": 48.1, - "ram_used_gb": 30.61214828491211, - "gpu_memory_used": 2069.0, - "relative_time": 52.09201526641846 - }, - { - "timestamp": "2025-01-06T02:00:14.576042", - "cpu_percent": 100.0, - "ram_percent": 48.3, - "ram_used_gb": 30.70488739013672, - "gpu_memory_used": 2083.0, - "relative_time": 53.26009011268616 - }, - { - "timestamp": "2025-01-06T02:00:15.749641", - "cpu_percent": 99.92, - "ram_percent": 48.3, - "ram_used_gb": 30.736248016357422, - "gpu_memory_used": 2082.0, - "relative_time": 54.40339493751526 - }, - { - "timestamp": "2025-01-06T02:00:16.896314", - "cpu_percent": 85.65, - "ram_percent": 48.1, - "ram_used_gb": 30.594017028808594, - "gpu_memory_used": 2082.0, - "relative_time": 55.51317882537842 - }, - { - "timestamp": "2025-01-06T02:00:17.999548", - "cpu_percent": 89.3, - "ram_percent": 48.1, - "ram_used_gb": 30.596832275390625, - "gpu_memory_used": 2082.0, - "relative_time": 56.65741968154907 - }, - { - "timestamp": "2025-01-06T02:00:19.138009", - "cpu_percent": 29.01, - "ram_percent": 48.1, - "ram_used_gb": 30.623390197753906, - "gpu_memory_used": 2073.0, - "relative_time": 57.802518367767334 - }, - { - "timestamp": "2025-01-06T02:00:20.291026", - "cpu_percent": 67.97, - "ram_percent": 48.2, - "ram_used_gb": 30.647132873535156, - "gpu_memory_used": 2073.0, - "relative_time": 58.922008991241455 - }, - { - "timestamp": "2025-01-06T02:00:21.412675", - "cpu_percent": 82.7, - "ram_percent": 48.2, - "ram_used_gb": 30.6431884765625, - "gpu_memory_used": 2071.0, - "relative_time": 60.05585193634033 - }, - { - "timestamp": "2025-01-06T02:00:22.550916", - "cpu_percent": 77.03, - "ram_percent": 48.2, - "ram_used_gb": 30.654644012451172, - "gpu_memory_used": 2069.0, - "relative_time": 61.18237376213074 - }, - { - "timestamp": "2025-01-06T02:00:23.670003", - "cpu_percent": 85.95, - "ram_percent": 48.2, - "ram_used_gb": 30.648406982421875, - "gpu_memory_used": 2069.0, - "relative_time": 62.266594648361206 - }, - { - "timestamp": "2025-01-06T02:00:24.747817", - "cpu_percent": 78.11, - "ram_percent": 48.2, - "ram_used_gb": 30.651287078857422, - "gpu_memory_used": 2062.0, - "relative_time": 63.40120267868042 - }, - { - "timestamp": "2025-01-06T02:00:25.895696", - "cpu_percent": 83.28, - "ram_percent": 48.2, - "ram_used_gb": 30.665794372558594, - "gpu_memory_used": 2062.0, - "relative_time": 64.55349016189575 - }, - { - "timestamp": "2025-01-06T02:00:27.046675", - "cpu_percent": 84.96, - "ram_percent": 48.2, - "ram_used_gb": 30.64820098876953, - "gpu_memory_used": 2062.0, - "relative_time": 65.66735076904297 - }, - { - "timestamp": "2025-01-06T02:00:28.152257", - "cpu_percent": 84.77, - "ram_percent": 48.2, - "ram_used_gb": 30.66122817993164, - "gpu_memory_used": 2074.0, - "relative_time": 66.77637505531311 - }, - { - "timestamp": "2025-01-06T02:00:29.261199", - "cpu_percent": 99.39, - "ram_percent": 48.2, - "ram_used_gb": 30.667972564697266, - "gpu_memory_used": 2082.0, - "relative_time": 67.92725157737732 - }, - { - "timestamp": "2025-01-06T02:00:30.411510", - "cpu_percent": 85.6, - "ram_percent": 48.2, - "ram_used_gb": 30.687171936035156, - "gpu_memory_used": 2082.0, - "relative_time": 69.0316333770752 - }, - { - "timestamp": "2025-01-06T02:00:31.521991", - "cpu_percent": 94.26, - "ram_percent": 48.2, - "ram_used_gb": 30.66115951538086, - "gpu_memory_used": 2104.0, - "relative_time": 70.16249108314514 - }, - { - "timestamp": "2025-01-06T02:00:32.654440", - "cpu_percent": 91.09, - "ram_percent": 48.2, - "ram_used_gb": 30.67467498779297, - "gpu_memory_used": 2104.0, - "relative_time": 71.28759574890137 - }, - { - "timestamp": "2025-01-06T02:00:33.775019", - "cpu_percent": 78.39, - "ram_percent": 48.2, - "ram_used_gb": 30.673912048339844, - "gpu_memory_used": 2101.0, - "relative_time": 72.43123960494995 - }, - { - "timestamp": "2025-01-06T02:00:34.911532", - "cpu_percent": 65.79, - "ram_percent": 48.2, - "ram_used_gb": 30.675281524658203, - "gpu_memory_used": 2097.0, - "relative_time": 73.5528736114502 - }, - { - "timestamp": "2025-01-06T02:00:36.044747", - "cpu_percent": 84.46, - "ram_percent": 48.2, - "ram_used_gb": 30.673004150390625, - "gpu_memory_used": 2097.0, - "relative_time": 74.67303442955017 - }, - { - "timestamp": "2025-01-06T02:00:37.166882", - "cpu_percent": 80.36, - "ram_percent": 48.2, - "ram_used_gb": 30.672035217285156, - "gpu_memory_used": 2085.0, - "relative_time": 75.80277967453003 - }, - { - "timestamp": "2025-01-06T02:00:38.287897", - "cpu_percent": 82.75, - "ram_percent": 48.2, - "ram_used_gb": 30.677658081054688, - "gpu_memory_used": 2071.0, - "relative_time": 76.90284442901611 - }, - { - "timestamp": "2025-01-06T02:00:39.391528", - "cpu_percent": 81.51, - "ram_percent": 48.2, - "ram_used_gb": 30.668991088867188, - "gpu_memory_used": 2071.0, - "relative_time": 78.05355834960938 - }, - { - "timestamp": "2025-01-06T02:00:40.541574", - "cpu_percent": 89.08, - "ram_percent": 48.2, - "ram_used_gb": 30.680435180664062, - "gpu_memory_used": 2070.0, - "relative_time": 79.18938493728638 - }, - { - "timestamp": "2025-01-06T02:00:41.681131", - "cpu_percent": 77.08, - "ram_percent": 48.2, - "ram_used_gb": 30.680683135986328, - "gpu_memory_used": 2075.0, - "relative_time": 80.32201266288757 - }, - { - "timestamp": "2025-01-06T02:00:42.815968", - "cpu_percent": 92.74, - "ram_percent": 48.0, - "ram_used_gb": 30.51242446899414, - "gpu_memory_used": 2075.0, - "relative_time": 81.43525695800781 - }, - { - "timestamp": "2025-01-06T02:00:43.925903", - "cpu_percent": 69.67, - "ram_percent": 48.0, - "ram_used_gb": 30.508132934570312, - "gpu_memory_used": 2073.0, - "relative_time": 82.55602645874023 - }, - { - "timestamp": "2025-01-06T02:00:45.048108", - "cpu_percent": 76.51, - "ram_percent": 47.9, - "ram_used_gb": 30.498050689697266, - "gpu_memory_used": 2073.0, - "relative_time": 83.67183542251587 - }, - { - "timestamp": "2025-01-06T02:00:46.157342", - "cpu_percent": 71.96, - "ram_percent": 47.9, - "ram_used_gb": 30.499706268310547, - "gpu_memory_used": 2073.0, - "relative_time": 84.79261541366577 - }, - { - "timestamp": "2025-01-06T02:00:47.282643", - "cpu_percent": 70.89, - "ram_percent": 47.9, - "ram_used_gb": 30.492206573486328, - "gpu_memory_used": 2073.0, - "relative_time": 85.89637041091919 - }, - { - "timestamp": "2025-01-06T02:00:48.385779", - "cpu_percent": 72.06, - "ram_percent": 47.9, - "ram_used_gb": 30.494159698486328, - "gpu_memory_used": 2072.0, - "relative_time": 86.99722957611084 - }, - { - "timestamp": "2025-01-06T02:00:49.489042", - "cpu_percent": 69.02, - "ram_percent": 48.0, - "ram_used_gb": 30.5125732421875, - "gpu_memory_used": 2072.0, - "relative_time": 88.13648676872253 - }, - { - "timestamp": "2025-01-06T02:00:50.619717", - "cpu_percent": 89.34, - "ram_percent": 48.0, - "ram_used_gb": 30.516311645507812, - "gpu_memory_used": 2072.0, - "relative_time": 89.25186491012573 - }, - { - "timestamp": "2025-01-06T02:00:51.730818", - "cpu_percent": 88.79, - "ram_percent": 48.0, - "ram_used_gb": 30.56262969970703, - "gpu_memory_used": 2067.0, - "relative_time": 90.39395761489868 - }, - { - "timestamp": "2025-01-06T02:00:52.879559", - "cpu_percent": 81.64, - "ram_percent": 48.2, - "ram_used_gb": 30.665996551513672, - "gpu_memory_used": 2078.0, - "relative_time": 91.57850098609924 - }, - { - "timestamp": "2025-01-06T02:00:54.072796", - "cpu_percent": 97.22, - "ram_percent": 48.3, - "ram_used_gb": 30.707321166992188, - "gpu_memory_used": 2079.0, - "relative_time": 92.74180054664612 - }, - { - "timestamp": "2025-01-06T02:00:55.221833", - "cpu_percent": 79.49, - "ram_percent": 48.1, - "ram_used_gb": 30.58167266845703, - "gpu_memory_used": 2079.0, - "relative_time": 93.87428188323975 - }, - { - "timestamp": "2025-01-06T02:00:56.368456", - "cpu_percent": 89.45, - "ram_percent": 48.2, - "ram_used_gb": 30.635387420654297, - "gpu_memory_used": 2079.0, - "relative_time": 95.03554892539978 - }, - { - "timestamp": "2025-01-06T02:00:57.518791", - "cpu_percent": 82.44, - "ram_percent": 48.2, - "ram_used_gb": 30.68875503540039, - "gpu_memory_used": 2075.0, - "relative_time": 96.16753339767456 - }, - { - "timestamp": "2025-01-06T02:00:58.651967", - "cpu_percent": 87.59, - "ram_percent": 48.3, - "ram_used_gb": 30.73841094970703, - "gpu_memory_used": 2075.0, - "relative_time": 97.31249833106995 - }, - { - "timestamp": "2025-01-06T02:00:59.802502", - "cpu_percent": 72.6, - "ram_percent": 48.5, - "ram_used_gb": 30.828655242919922, - "gpu_memory_used": 2074.0, - "relative_time": 98.44322228431702 - }, - { - "timestamp": "2025-01-06T02:01:00.930471", - "cpu_percent": 92.01, - "ram_percent": 48.6, - "ram_used_gb": 30.916194915771484, - "gpu_memory_used": 2057.0, - "relative_time": 99.55358910560608 - }, - { - "timestamp": "2025-01-06T02:01:02.046338", - "cpu_percent": 77.78, - "ram_percent": 48.7, - "ram_used_gb": 30.970348358154297, - "gpu_memory_used": 2059.0, - "relative_time": 100.7326455116272 - }, - { - "timestamp": "2025-01-06T02:01:03.221413", - "cpu_percent": 87.99, - "ram_percent": 48.8, - "ram_used_gb": 31.030376434326172, - "gpu_memory_used": 2065.0, - "relative_time": 101.87276792526245 - }, - { - "timestamp": "2025-01-06T02:01:04.365536", - "cpu_percent": 83.47, - "ram_percent": 48.8, - "ram_used_gb": 31.04818344116211, - "gpu_memory_used": 2066.0, - "relative_time": 103.00344848632812 - }, - { - "timestamp": "2025-01-06T02:01:05.495547", - "cpu_percent": 73.88, - "ram_percent": 48.3, - "ram_used_gb": 30.707366943359375, - "gpu_memory_used": 2066.0, - "relative_time": 104.14199256896973 - }, - { - "timestamp": "2025-01-06T02:01:06.627593", - "cpu_percent": 81.92, - "ram_percent": 48.3, - "ram_used_gb": 30.735427856445312, - "gpu_memory_used": 2065.0, - "relative_time": 105.27145409584045 - }, - { - "timestamp": "2025-01-06T02:01:07.762940", - "cpu_percent": 93.88, - "ram_percent": 48.3, - "ram_used_gb": 30.71178436279297, - "gpu_memory_used": 2073.0, - "relative_time": 106.42445015907288 - }, - { - "timestamp": "2025-01-06T02:01:08.908747", - "cpu_percent": 78.18, - "ram_percent": 48.2, - "ram_used_gb": 30.685108184814453, - "gpu_memory_used": 2073.0, - "relative_time": 107.53882956504822 - }, - { - "timestamp": "2025-01-06T02:01:10.023117", - "cpu_percent": 80.25, - "ram_percent": 48.3, - "ram_used_gb": 30.693721771240234, - "gpu_memory_used": 2073.0, - "relative_time": 108.62077617645264 - }, - { - "timestamp": "2025-01-06T02:01:11.125442", - "cpu_percent": 74.96, - "ram_percent": 45.8, - "ram_used_gb": 29.10287094116211, - "gpu_memory_used": 1121.0, - "relative_time": 109.79581832885742 - }, - { - "timestamp": "2025-01-06T02:01:12.277407", - "cpu_percent": 82.89, - "ram_percent": 45.6, - "ram_used_gb": 28.987510681152344, - "gpu_memory_used": 1115.0, - "relative_time": 110.89319515228271 - }, - { - "timestamp": "2025-01-06T02:01:13.382871", - "cpu_percent": 79.66, - "ram_percent": 45.6, - "ram_used_gb": 28.993927001953125, - "gpu_memory_used": 1119.0, - "relative_time": 112.00745177268982 - }, - { - "timestamp": "2025-01-06T02:01:14.498991", - "cpu_percent": 86.41, - "ram_percent": 45.6, - "ram_used_gb": 29.035633087158203, - "gpu_memory_used": 1119.0, - "relative_time": 113.1147940158844 - }, - { - "timestamp": "2025-01-06T02:01:15.600091", - "cpu_percent": 94.06, - "ram_percent": 45.6, - "ram_used_gb": 28.978424072265625, - "gpu_memory_used": 1115.0, - "relative_time": 114.23285436630249 - }, - { - "timestamp": "2025-01-06T02:01:16.722096", - "cpu_percent": 72.28, - "ram_percent": 45.5, - "ram_used_gb": 28.96765899658203, - "gpu_memory_used": 1113.0, - "relative_time": 115.35184216499329 - }, - { - "timestamp": "2025-01-06T02:01:17.840265", - "cpu_percent": 74.65, - "ram_percent": 45.5, - "ram_used_gb": 28.946372985839844, - "gpu_memory_used": 1115.0, - "relative_time": 116.5044412612915 - }, - { - "timestamp": "2025-01-06T02:01:18.991455", - "cpu_percent": 87.44, - "ram_percent": 45.5, - "ram_used_gb": 28.957916259765625, - "gpu_memory_used": 1123.0, - "relative_time": 117.65444111824036 - }, - { - "timestamp": "2025-01-06T02:01:20.139485", - "cpu_percent": 86.77, - "ram_percent": 45.5, - "ram_used_gb": 28.956340789794922, - "gpu_memory_used": 1126.0, - "relative_time": 118.77146029472351 - }, - { - "timestamp": "2025-01-06T02:01:21.266216", - "cpu_percent": 75.49, - "ram_percent": 45.5, - "ram_used_gb": 28.958038330078125, - "gpu_memory_used": 1126.0, - "relative_time": 119.87909436225891 - }, - { - "timestamp": "2025-01-06T02:01:22.363169", - "cpu_percent": 79.51, - "ram_percent": 45.5, - "ram_used_gb": 28.949172973632812, - "gpu_memory_used": 1124.0, - "relative_time": 121.01380467414856 - }, - { - "timestamp": "2025-01-06T02:01:23.498096", - "cpu_percent": 85.16, - "ram_percent": 45.5, - "ram_used_gb": 28.958885192871094, - "gpu_memory_used": 1124.0, - "relative_time": 122.11237931251526 - }, - { - "timestamp": "2025-01-06T02:01:24.603852", - "cpu_percent": 67.16, - "ram_percent": 45.5, - "ram_used_gb": 28.936931610107422, - "gpu_memory_used": 1124.0, - "relative_time": 123.20061945915222 - }, - { - "timestamp": "2025-01-06T02:01:25.691913", - "cpu_percent": 68.53, - "ram_percent": 45.5, - "ram_used_gb": 28.933303833007812, - "gpu_memory_used": 1118.0, - "relative_time": 124.33519053459167 - }, - { - "timestamp": "2025-01-06T02:01:26.828745", - "cpu_percent": 80.47, - "ram_percent": 45.5, - "ram_used_gb": 28.923110961914062, - "gpu_memory_used": 1118.0, - "relative_time": 125.48865795135498 - }, - { - "timestamp": "2025-01-06T02:01:27.979825", - "cpu_percent": 79.44, - "ram_percent": 45.4, - "ram_used_gb": 28.900054931640625, - "gpu_memory_used": 1125.0, - "relative_time": 126.59582543373108 - }, - { - "timestamp": "2025-01-06T02:01:29.080446", - "cpu_percent": 72.07, - "ram_percent": 45.4, - "ram_used_gb": 28.896743774414062, - "gpu_memory_used": 1125.0, - "relative_time": 127.70408391952515 - }, - { - "timestamp": "2025-01-06T02:01:30.187503", - "cpu_percent": 79.71, - "ram_percent": 45.4, - "ram_used_gb": 28.90612030029297, - "gpu_memory_used": 1126.0, - "relative_time": 128.84098839759827 - }, - { - "timestamp": "2025-01-06T02:01:31.327260", - "cpu_percent": 82.71, - "ram_percent": 45.4, - "ram_used_gb": 28.899269104003906, - "gpu_memory_used": 1126.0, - "relative_time": 129.94981002807617 - }, - { - "timestamp": "2025-01-06T02:01:32.435487", - "cpu_percent": 69.43, - "ram_percent": 45.5, - "ram_used_gb": 28.913204193115234, - "gpu_memory_used": 1124.0, - "relative_time": 131.05302381515503 - }, - { - "timestamp": "2025-01-06T02:01:33.538589", - "cpu_percent": 69.85, - "ram_percent": 45.4, - "ram_used_gb": 28.902801513671875, - "gpu_memory_used": 1117.0, - "relative_time": 132.18232107162476 - }, - { - "timestamp": "2025-01-06T02:01:34.676977", - "cpu_percent": 76.86, - "ram_percent": 45.4, - "ram_used_gb": 28.90363311767578, - "gpu_memory_used": 1113.0, - "relative_time": 133.3194751739502 - }, - { - "timestamp": "2025-01-06T02:01:35.813043", - "cpu_percent": 92.46, - "ram_percent": 45.5, - "ram_used_gb": 28.924484252929688, - "gpu_memory_used": 1111.0, - "relative_time": 134.4480812549591 - }, - { - "timestamp": "2025-01-06T02:01:36.934534", - "cpu_percent": 91.6, - "ram_percent": 45.5, - "ram_used_gb": 28.93694305419922, - "gpu_memory_used": 1112.0, - "relative_time": 135.552565574646 - }, - { - "timestamp": "2025-01-06T02:01:38.046439", - "cpu_percent": 87.57, - "ram_percent": 45.5, - "ram_used_gb": 28.928115844726562, - "gpu_memory_used": 1112.0, - "relative_time": 136.66557025909424 - }, - { - "timestamp": "2025-01-06T02:01:39.150774", - "cpu_percent": 71.04, - "ram_percent": 45.4, - "ram_used_gb": 28.89849853515625, - "gpu_memory_used": 1112.0, - "relative_time": 137.79479718208313 - }, - { - "timestamp": "2025-01-06T02:01:40.286829", - "cpu_percent": 72.61, - "ram_percent": 45.4, - "ram_used_gb": 28.89031219482422, - "gpu_memory_used": 1111.0, - "relative_time": 138.92065834999084 - }, - { - "timestamp": "2025-01-06T02:01:41.411672", - "cpu_percent": 71.29, - "ram_percent": 45.4, - "ram_used_gb": 28.862991333007812, - "gpu_memory_used": 1111.0, - "relative_time": 140.02822160720825 - }, - { - "timestamp": "2025-01-06T02:01:42.514155", - "cpu_percent": 73.17, - "ram_percent": 45.4, - "ram_used_gb": 28.872455596923828, - "gpu_memory_used": 1112.0, - "relative_time": 141.16313910484314 - }, - { - "timestamp": "2025-01-06T02:01:43.647529", - "cpu_percent": 79.79, - "ram_percent": 45.4, - "ram_used_gb": 28.88043212890625, - "gpu_memory_used": 1107.0, - "relative_time": 142.30250215530396 - }, - { - "timestamp": "2025-01-06T02:01:44.783347", - "cpu_percent": 80.66, - "ram_percent": 45.4, - "ram_used_gb": 28.876739501953125, - "gpu_memory_used": 1107.0, - "relative_time": 143.39962935447693 - }, - { - "timestamp": "2025-01-06T02:01:45.887340", - "cpu_percent": 53.24, - "ram_percent": 45.4, - "ram_used_gb": 28.876079559326172, - "gpu_memory_used": 1107.0, - "relative_time": 144.51881790161133 - }, - { - "timestamp": "2025-01-06T02:01:47.006876", - "cpu_percent": 76.76, - "ram_percent": 45.4, - "ram_used_gb": 28.878875732421875, - "gpu_memory_used": 1106.0, - "relative_time": 145.62467503547668 - }, - { - "timestamp": "2025-01-06T02:01:48.112999", - "cpu_percent": 87.83, - "ram_percent": 45.4, - "ram_used_gb": 28.910343170166016, - "gpu_memory_used": 1109.0, - "relative_time": 146.79565000534058 - }, - { - "timestamp": "2025-01-06T02:01:49.275639", - "cpu_percent": 78.68, - "ram_percent": 45.5, - "ram_used_gb": 28.941452026367188, - "gpu_memory_used": 1112.0, - "relative_time": 147.9065911769867 - }, - { - "timestamp": "2025-01-06T02:01:50.392507", - "cpu_percent": 92.35, - "ram_percent": 45.5, - "ram_used_gb": 28.93001937866211, - "gpu_memory_used": 1111.0, - "relative_time": 149.0513255596161 - }, - { - "timestamp": "2025-01-06T02:01:51.531827", - "cpu_percent": 74.92, - "ram_percent": 45.5, - "ram_used_gb": 28.972396850585938, - "gpu_memory_used": 1111.0, - "relative_time": 150.190589427948 - }, - { - "timestamp": "2025-01-06T02:01:52.681259", - "cpu_percent": 83.39, - "ram_percent": 45.5, - "ram_used_gb": 28.97113800048828, - "gpu_memory_used": 1109.0, - "relative_time": 151.35132122039795 - }, - { - "timestamp": "2025-01-06T02:01:53.835701", - "cpu_percent": 76.66, - "ram_percent": 45.6, - "ram_used_gb": 29.026382446289062, - "gpu_memory_used": 1109.0, - "relative_time": 152.50396990776062 - }, - { - "timestamp": "2025-01-06T02:01:54.984694", - "cpu_percent": 90.89, - "ram_percent": 45.6, - "ram_used_gb": 28.993335723876953, - "gpu_memory_used": 1117.0, - "relative_time": 153.65559840202332 - }, - { - "timestamp": "2025-01-06T02:01:56.136412", - "cpu_percent": 87.1, - "ram_percent": 45.6, - "ram_used_gb": 28.996482849121094, - "gpu_memory_used": 1114.0, - "relative_time": 154.75753116607666 - }, - { - "timestamp": "2025-01-06T02:01:57.248956", - "cpu_percent": 94.48, - "ram_percent": 45.6, - "ram_used_gb": 29.01409912109375, - "gpu_memory_used": 1114.0, - "relative_time": 155.8850748538971 - }, - { - "timestamp": "2025-01-06T02:01:58.363971", - "cpu_percent": 68.38, - "ram_percent": 45.6, - "ram_used_gb": 29.0169677734375, - "gpu_memory_used": 1114.0, - "relative_time": 156.99017000198364 - }, - { - "timestamp": "2025-01-06T02:01:59.482082", - "cpu_percent": 66.96, - "ram_percent": 45.6, - "ram_used_gb": 28.99835205078125, - "gpu_memory_used": 1112.0, - "relative_time": 158.11476922035217 - }, - { - "timestamp": "2025-01-06T02:02:00.599761", - "cpu_percent": 72.88, - "ram_percent": 45.6, - "ram_used_gb": 29.014999389648438, - "gpu_memory_used": 1111.0, - "relative_time": 159.23513007164001 - }, - { - "timestamp": "2025-01-06T02:02:01.724187", - "cpu_percent": 83.27, - "ram_percent": 45.6, - "ram_used_gb": 28.979934692382812, - "gpu_memory_used": 1109.0, - "relative_time": 160.3893871307373 - }, - { - "timestamp": "2025-01-06T02:02:02.874035", - "cpu_percent": 84.25, - "ram_percent": 45.6, - "ram_used_gb": 29.00198745727539, - "gpu_memory_used": 1110.0, - "relative_time": 161.4882082939148 - }, - { - "timestamp": "2025-01-06T02:02:03.982069", - "cpu_percent": 46.27, - "ram_percent": 45.7, - "ram_used_gb": 29.048011779785156, - "gpu_memory_used": 1108.0, - "relative_time": 162.579003572464 - }, - { - "timestamp": "2025-01-06T02:02:05.067839", - "cpu_percent": 23.25, - "ram_percent": 45.7, - "ram_used_gb": 29.041690826416016, - "gpu_memory_used": 1102.0, - "relative_time": 163.69112300872803 - }, - { - "timestamp": "2025-01-06T02:02:06.170967", - "cpu_percent": 68.39, - "ram_percent": 45.6, - "ram_used_gb": 29.02823257446289, - "gpu_memory_used": 1102.0, - "relative_time": 164.8040177822113 - }, - { - "timestamp": "2025-01-06T02:02:07.291947", - "cpu_percent": 67.11, - "ram_percent": 45.6, - "ram_used_gb": 29.026527404785156, - "gpu_memory_used": 1102.0, - "relative_time": 165.9222002029419 - }, - { - "timestamp": "2025-01-06T02:02:08.407847", - "cpu_percent": 75.64, - "ram_percent": 45.7, - "ram_used_gb": 29.04144287109375, - "gpu_memory_used": 1101.0, - "relative_time": 167.02184295654297 - }, - { - "timestamp": "2025-01-06T02:02:09.510086", - "cpu_percent": 66.88, - "ram_percent": 45.6, - "ram_used_gb": 28.97747039794922, - "gpu_memory_used": 1101.0, - "relative_time": 168.15068435668945 - }, - { - "timestamp": "2025-01-06T02:02:10.642415", - "cpu_percent": 69.38, - "ram_percent": 45.6, - "ram_used_gb": 28.98614501953125, - "gpu_memory_used": 1101.0, - "relative_time": 169.26578378677368 - }, - { - "timestamp": "2025-01-06T02:02:11.757629", - "cpu_percent": 75.01, - "ram_percent": 45.6, - "ram_used_gb": 28.98458480834961, - "gpu_memory_used": 1101.0, - "relative_time": 170.3762068748474 - }, - { - "timestamp": "2025-01-06T02:02:12.861200", - "cpu_percent": 65.3, - "ram_percent": 45.6, - "ram_used_gb": 28.996482849121094, - "gpu_memory_used": 1101.0, - "relative_time": 171.49599766731262 - }, - { - "timestamp": "2025-01-06T02:02:13.979027", - "cpu_percent": 70.03, - "ram_percent": 45.6, - "ram_used_gb": 29.004348754882812, - "gpu_memory_used": 1101.0, - "relative_time": 172.63518261909485 - }, - { - "timestamp": "2025-01-06T02:02:15.127058", - "cpu_percent": 74.56, - "ram_percent": 45.6, - "ram_used_gb": 28.98357391357422, - "gpu_memory_used": 1100.0, - "relative_time": 173.78076720237732 - }, - { - "timestamp": "2025-01-06T02:02:16.269797", - "cpu_percent": 83.33, - "ram_percent": 45.6, - "ram_used_gb": 29.013275146484375, - "gpu_memory_used": 1102.0, - "relative_time": 174.92202520370483 - }, - { - "timestamp": "2025-01-06T02:02:17.403955", - "cpu_percent": 66.92, - "ram_percent": 45.6, - "ram_used_gb": 28.988414764404297, - "gpu_memory_used": 1102.0, - "relative_time": 175.9813597202301 - }, - { - "timestamp": "2025-01-06T02:02:18.464105", - "cpu_percent": 65.91, - "ram_percent": 45.6, - "ram_used_gb": 28.997055053710938, - "gpu_memory_used": 1102.0, - "relative_time": 177.0992946624756 - }, - { - "timestamp": "2025-01-06T02:02:19.584316", - "cpu_percent": 64.77, - "ram_percent": 45.6, - "ram_used_gb": 28.98046875, - "gpu_memory_used": 1102.0, - "relative_time": 178.22958946228027 - }, - { - "timestamp": "2025-01-06T02:02:20.719166", - "cpu_percent": 74.76, - "ram_percent": 45.6, - "ram_used_gb": 29.008087158203125, - "gpu_memory_used": 1100.0, - "relative_time": 179.40734457969666 - }, - { - "timestamp": "2025-01-06T02:02:21.901171", - "cpu_percent": 73.14, - "ram_percent": 45.6, - "ram_used_gb": 28.98965072631836, - "gpu_memory_used": 1101.0, - "relative_time": 180.55075860023499 - }, - { - "timestamp": "2025-01-06T02:02:23.037921", - "cpu_percent": 75.12, - "ram_percent": 45.6, - "ram_used_gb": 29.005779266357422, - "gpu_memory_used": 1101.0, - "relative_time": 181.6956079006195 - }, - { - "timestamp": "2025-01-06T02:02:24.190508", - "cpu_percent": 68.59, - "ram_percent": 45.6, - "ram_used_gb": 28.99420166015625, - "gpu_memory_used": 1103.0, - "relative_time": 182.8263657093048 - }, - { - "timestamp": "2025-01-06T02:02:25.318426", - "cpu_percent": 71.09, - "ram_percent": 45.5, - "ram_used_gb": 28.96915054321289, - "gpu_memory_used": 1103.0, - "relative_time": 183.96249556541443 - }, - { - "timestamp": "2025-01-06T02:02:26.452298", - "cpu_percent": 67.23, - "ram_percent": 45.5, - "ram_used_gb": 28.9666748046875, - "gpu_memory_used": 1103.0, - "relative_time": 185.08496713638306 - }, - { - "timestamp": "2025-01-06T02:02:27.575091", - "cpu_percent": 75.85, - "ram_percent": 45.5, - "ram_used_gb": 28.963436126708984, - "gpu_memory_used": 1100.0, - "relative_time": 186.21436381340027 - }, - { - "timestamp": "2025-01-06T02:02:28.698623", - "cpu_percent": 76.0, - "ram_percent": 45.5, - "ram_used_gb": 28.966102600097656, - "gpu_memory_used": 1101.0, - "relative_time": 187.35916352272034 - }, - { - "timestamp": "2025-01-06T02:02:29.850943", - "cpu_percent": 72.59, - "ram_percent": 45.5, - "ram_used_gb": 28.966720581054688, - "gpu_memory_used": 1101.0, - "relative_time": 188.48684239387512 - }, - { - "timestamp": "2025-01-06T02:02:30.973391", - "cpu_percent": 68.84, - "ram_percent": 45.6, - "ram_used_gb": 28.99535369873047, - "gpu_memory_used": 1109.0, - "relative_time": 189.58790564537048 - }, - { - "timestamp": "2025-01-06T02:02:32.080413", - "cpu_percent": 75.34, - "ram_percent": 45.6, - "ram_used_gb": 28.983352661132812, - "gpu_memory_used": 1109.0, - "relative_time": 190.70177555084229 - }, - { - "timestamp": "2025-01-06T02:02:33.181584", - "cpu_percent": 82.01, - "ram_percent": 45.6, - "ram_used_gb": 29.014503479003906, - "gpu_memory_used": 1109.0, - "relative_time": 191.79927110671997 - }, - { - "timestamp": "2025-01-06T02:02:34.289805", - "cpu_percent": 61.24, - "ram_percent": 45.6, - "ram_used_gb": 28.986770629882812, - "gpu_memory_used": 1108.0, - "relative_time": 192.92647433280945 - }, - { - "timestamp": "2025-01-06T02:02:35.408364", - "cpu_percent": 71.77, - "ram_percent": 45.6, - "ram_used_gb": 28.976062774658203, - "gpu_memory_used": 1108.0, - "relative_time": 194.02294254302979 - }, - { - "timestamp": "2025-01-06T02:02:36.509112", - "cpu_percent": 67.74, - "ram_percent": 45.5, - "ram_used_gb": 28.970108032226562, - "gpu_memory_used": 1108.0, - "relative_time": 195.13911247253418 - }, - { - "timestamp": "2025-01-06T02:02:37.630544", - "cpu_percent": 73.08, - "ram_percent": 45.6, - "ram_used_gb": 28.982391357421875, - "gpu_memory_used": 1108.0, - "relative_time": 196.25971221923828 - }, - { - "timestamp": "2025-01-06T02:02:38.751707", - "cpu_percent": 73.63, - "ram_percent": 45.5, - "ram_used_gb": 28.947036743164062, - "gpu_memory_used": 1108.0, - "relative_time": 197.3975374698639 - }, - { - "timestamp": "2025-01-06T02:02:39.890645", - "cpu_percent": 72.01, - "ram_percent": 45.5, - "ram_used_gb": 28.969207763671875, - "gpu_memory_used": 1108.0, - "relative_time": 198.50470900535583 - }, - { - "timestamp": "2025-01-06T02:02:40.995583", - "cpu_percent": 70.7, - "ram_percent": 45.5, - "ram_used_gb": 28.956737518310547, - "gpu_memory_used": 1108.0, - "relative_time": 199.64927458763123 - }, - { - "timestamp": "2025-01-06T02:02:42.130610", - "cpu_percent": 72.67, - "ram_percent": 45.6, - "ram_used_gb": 28.981361389160156, - "gpu_memory_used": 1108.0, - "relative_time": 200.74797558784485 - }, - { - "timestamp": "2025-01-06T02:02:43.233560", - "cpu_percent": 70.15, - "ram_percent": 45.6, - "ram_used_gb": 28.97726821899414, - "gpu_memory_used": 1108.0, - "relative_time": 201.86889624595642 - }, - { - "timestamp": "2025-01-06T02:02:44.354710", - "cpu_percent": 68.13, - "ram_percent": 45.6, - "ram_used_gb": 28.976673126220703, - "gpu_memory_used": 1108.0, - "relative_time": 202.98869681358337 - }, - { - "timestamp": "2025-01-06T02:02:45.472846", - "cpu_percent": 71.0, - "ram_percent": 45.5, - "ram_used_gb": 28.96390151977539, - "gpu_memory_used": 1108.0, - "relative_time": 204.10803961753845 - }, - { - "timestamp": "2025-01-06T02:02:46.590739", - "cpu_percent": 73.99, - "ram_percent": 45.5, - "ram_used_gb": 28.964736938476562, - "gpu_memory_used": 1107.0, - "relative_time": 205.21335530281067 - }, - { - "timestamp": "2025-01-06T02:02:47.693619", - "cpu_percent": 88.54, - "ram_percent": 45.7, - "ram_used_gb": 29.073219299316406, - "gpu_memory_used": 1096.0, - "relative_time": 206.3404152393341 - }, - { - "timestamp": "2025-01-06T02:02:48.831178", - "cpu_percent": 70.85, - "ram_percent": 45.7, - "ram_used_gb": 29.051692962646484, - "gpu_memory_used": 1111.0, - "relative_time": 207.6542706489563 - }, - { - "timestamp": "2025-01-06T02:02:50.135974", - "cpu_percent": 81.68, - "ram_percent": 44.9, - "ram_used_gb": 28.559471130371094, - "gpu_memory_used": 1047.0, - "relative_time": 208.75140976905823 - }, - { - "timestamp": "2025-01-06T02:02:51.237222", - "cpu_percent": 68.7, - "ram_percent": 44.8, - "ram_used_gb": 28.52242660522461, - "gpu_memory_used": 1044.0, - "relative_time": 209.86320424079895 - }, - { - "timestamp": "2025-01-06T02:02:52.355393", - "cpu_percent": 75.6, - "ram_percent": 44.9, - "ram_used_gb": 28.53069305419922, - "gpu_memory_used": 1044.0, - "relative_time": 210.99385690689087 - }, - { - "timestamp": "2025-01-06T02:02:53.488242", - "cpu_percent": 72.12, - "ram_percent": 44.8, - "ram_used_gb": 28.527976989746094, - "gpu_memory_used": 1045.0, - "relative_time": 212.15026259422302 - }, - { - "timestamp": "2025-01-06T02:02:54.635771", - "cpu_percent": 75.99, - "ram_percent": 44.8, - "ram_used_gb": 28.52718734741211, - "gpu_memory_used": 1047.0, - "relative_time": 213.26202607154846 - }, - { - "timestamp": "2025-01-06T02:02:55.743075", - "cpu_percent": 71.68, - "ram_percent": 44.8, - "ram_used_gb": 28.513080596923828, - "gpu_memory_used": 1047.0, - "relative_time": 214.37045192718506 - }, - { - "timestamp": "2025-01-06T02:02:56.860009", - "cpu_percent": 73.16, - "ram_percent": 44.8, - "ram_used_gb": 28.4967041015625, - "gpu_memory_used": 1046.0, - "relative_time": 215.45900750160217 - }, - { - "timestamp": "2025-01-06T02:02:57.948354", - "cpu_percent": 66.73, - "ram_percent": 44.8, - "ram_used_gb": 28.50171661376953, - "gpu_memory_used": 1039.0, - "relative_time": 216.58656573295593 - }, - { - "timestamp": "2025-01-06T02:02:59.070787", - "cpu_percent": 66.33, - "ram_percent": 44.8, - "ram_used_gb": 28.485286712646484, - "gpu_memory_used": 1030.0, - "relative_time": 217.7198555469513 - }, - { - "timestamp": "2025-01-06T02:03:00.210441", - "cpu_percent": 76.69, - "ram_percent": 44.8, - "ram_used_gb": 28.50790786743164, - "gpu_memory_used": 1041.0, - "relative_time": 218.82320833206177 - }, - { - "timestamp": "2025-01-06T02:03:01.312747", - "cpu_percent": 69.7, - "ram_percent": 44.8, - "ram_used_gb": 28.496017456054688, - "gpu_memory_used": 1042.0, - "relative_time": 219.93000483512878 - }, - { - "timestamp": "2025-01-06T02:03:02.415826", - "cpu_percent": 67.86, - "ram_percent": 44.8, - "ram_used_gb": 28.504711151123047, - "gpu_memory_used": 1042.0, - "relative_time": 221.04683136940002 - }, - { - "timestamp": "2025-01-06T02:03:03.526977", - "cpu_percent": 69.49, - "ram_percent": 44.8, - "ram_used_gb": 28.508529663085938, - "gpu_memory_used": 1041.0, - "relative_time": 222.13029503822327 - }, - { - "timestamp": "2025-01-06T02:03:04.618586", - "cpu_percent": 64.46, - "ram_percent": 44.8, - "ram_used_gb": 28.50463104248047, - "gpu_memory_used": 1041.0, - "relative_time": 223.2292401790619 - }, - { - "timestamp": "2025-01-06T02:03:05.723362", - "cpu_percent": 68.49, - "ram_percent": 44.8, - "ram_used_gb": 28.49188232421875, - "gpu_memory_used": 1041.0, - "relative_time": 224.3517026901245 - }, - { - "timestamp": "2025-01-06T02:03:06.831388", - "cpu_percent": 76.89, - "ram_percent": 44.8, - "ram_used_gb": 28.49285888671875, - "gpu_memory_used": 1041.0, - "relative_time": 225.4667146205902 - }, - { - "timestamp": "2025-01-06T02:03:07.951993", - "cpu_percent": 67.04, - "ram_percent": 44.8, - "ram_used_gb": 28.50518798828125, - "gpu_memory_used": 1041.0, - "relative_time": 226.58504056930542 - }, - { - "timestamp": "2025-01-06T02:03:09.069126", - "cpu_percent": 70.29, - "ram_percent": 44.8, - "ram_used_gb": 28.47330093383789, - "gpu_memory_used": 1041.0, - "relative_time": 227.708820104599 - }, - { - "timestamp": "2025-01-06T02:03:10.191425", - "cpu_percent": 71.98, - "ram_percent": 44.8, - "ram_used_gb": 28.495555877685547, - "gpu_memory_used": 1041.0, - "relative_time": 228.845196723938 - }, - { - "timestamp": "2025-01-06T02:03:11.325421", - "cpu_percent": 80.36, - "ram_percent": 44.8, - "ram_used_gb": 28.496841430664062, - "gpu_memory_used": 1031.0, - "relative_time": 230.06402707099915 - }, - { - "timestamp": "2025-01-06T02:03:12.552864", - "cpu_percent": 88.18, - "ram_percent": 44.9, - "ram_used_gb": 28.545429229736328, - "gpu_memory_used": 1032.0, - "relative_time": 231.31830477714539 - }, - { - "timestamp": "2025-01-06T02:03:13.798580", - "cpu_percent": 97.64, - "ram_percent": 44.8, - "ram_used_gb": 28.51227569580078, - "gpu_memory_used": 1032.0, - "relative_time": 232.4179232120514 - }, - { - "timestamp": "2025-01-06T02:03:14.901645", - "cpu_percent": 74.59, - "ram_percent": 44.8, - "ram_used_gb": 28.48064422607422, - "gpu_memory_used": 1032.0, - "relative_time": 233.52263975143433 - }, - { - "timestamp": "2025-01-06T02:03:16.008349", - "cpu_percent": 70.1, - "ram_percent": 44.8, - "ram_used_gb": 28.487651824951172, - "gpu_memory_used": 1027.0, - "relative_time": 234.65311098098755 - }, - { - "timestamp": "2025-01-06T02:03:17.141096", - "cpu_percent": 73.53, - "ram_percent": 44.8, - "ram_used_gb": 28.48447036743164, - "gpu_memory_used": 1027.0, - "relative_time": 235.7638533115387 - }, - { - "timestamp": "2025-01-06T02:03:18.244142", - "cpu_percent": 72.79, - "ram_percent": 44.8, - "ram_used_gb": 28.493579864501953, - "gpu_memory_used": 1027.0, - "relative_time": 236.8931279182434 - }, - { - "timestamp": "2025-01-06T02:03:19.380511", - "cpu_percent": 76.48, - "ram_percent": 44.8, - "ram_used_gb": 28.48853302001953, - "gpu_memory_used": 1032.0, - "relative_time": 238.01199889183044 - }, - { - "timestamp": "2025-01-06T02:03:20.498808", - "cpu_percent": 65.27, - "ram_percent": 44.7, - "ram_used_gb": 28.432437896728516, - "gpu_memory_used": 1031.0, - "relative_time": 239.09615564346313 - }, - { - "timestamp": "2025-01-06T02:03:21.590026", - "cpu_percent": 61.61, - "ram_percent": 44.7, - "ram_used_gb": 28.455894470214844, - "gpu_memory_used": 1034.0, - "relative_time": 240.2461507320404 - }, - { - "timestamp": "2025-01-06T02:03:22.737841", - "cpu_percent": 80.38, - "ram_percent": 44.8, - "ram_used_gb": 28.4761962890625, - "gpu_memory_used": 1035.0, - "relative_time": 241.36590909957886 - }, - { - "timestamp": "2025-01-06T02:03:23.859319", - "cpu_percent": 65.25, - "ram_percent": 44.7, - "ram_used_gb": 28.453296661376953, - "gpu_memory_used": 1035.0, - "relative_time": 242.49047112464905 - }, - { - "timestamp": "2025-01-06T02:03:24.975950", - "cpu_percent": 73.18, - "ram_percent": 44.8, - "ram_used_gb": 28.474166870117188, - "gpu_memory_used": 1036.0, - "relative_time": 243.60454726219177 - }, - { - "timestamp": "2025-01-06T02:03:26.096454", - "cpu_percent": 83.42, - "ram_percent": 44.7, - "ram_used_gb": 28.43814468383789, - "gpu_memory_used": 1037.0, - "relative_time": 244.7518711090088 - }, - { - "timestamp": "2025-01-06T02:03:27.240727", - "cpu_percent": 78.22, - "ram_percent": 44.7, - "ram_used_gb": 28.463787078857422, - "gpu_memory_used": 1037.0, - "relative_time": 245.88417291641235 - }, - { - "timestamp": "2025-01-06T02:03:28.374545", - "cpu_percent": 77.39, - "ram_percent": 44.7, - "ram_used_gb": 28.448078155517578, - "gpu_memory_used": 1037.0, - "relative_time": 246.99420261383057 - }, - { - "timestamp": "2025-01-06T02:03:29.481376", - "cpu_percent": 72.67, - "ram_percent": 44.7, - "ram_used_gb": 28.44430923461914, - "gpu_memory_used": 1037.0, - "relative_time": 248.13829016685486 - }, - { - "timestamp": "2025-01-06T02:03:30.629026", - "cpu_percent": 71.77, - "ram_percent": 44.7, - "ram_used_gb": 28.44107437133789, - "gpu_memory_used": 1036.0, - "relative_time": 249.24768805503845 - }, - { - "timestamp": "2025-01-06T02:03:31.732614", - "cpu_percent": 80.17, - "ram_percent": 44.8, - "ram_used_gb": 28.484886169433594, - "gpu_memory_used": 1042.0, - "relative_time": 250.38990330696106 - }, - { - "timestamp": "2025-01-06T02:03:32.882436", - "cpu_percent": 86.8, - "ram_percent": 44.8, - "ram_used_gb": 28.48214340209961, - "gpu_memory_used": 1043.0, - "relative_time": 251.52275609970093 - }, - { - "timestamp": "2025-01-06T02:03:34.004573", - "cpu_percent": 67.43, - "ram_percent": 44.8, - "ram_used_gb": 28.495872497558594, - "gpu_memory_used": 1044.0, - "relative_time": 252.6218719482422 - }, - { - "timestamp": "2025-01-06T02:03:35.102105", - "cpu_percent": 74.36, - "ram_percent": 44.7, - "ram_used_gb": 28.462890625, - "gpu_memory_used": 1044.0, - "relative_time": 253.7364068031311 - }, - { - "timestamp": "2025-01-06T02:03:36.229096", - "cpu_percent": 77.13, - "ram_percent": 44.8, - "ram_used_gb": 28.49734878540039, - "gpu_memory_used": 1042.0, - "relative_time": 254.84465980529785 - }, - { - "timestamp": "2025-01-06T02:03:37.328927", - "cpu_percent": 79.02, - "ram_percent": 44.7, - "ram_used_gb": 28.464012145996094, - "gpu_memory_used": 1038.0, - "relative_time": 255.96682238578796 - }, - { - "timestamp": "2025-01-06T02:03:38.448134", - "cpu_percent": 68.85, - "ram_percent": 44.8, - "ram_used_gb": 28.466339111328125, - "gpu_memory_used": 1035.0, - "relative_time": 257.09385228157043 - }, - { - "timestamp": "2025-01-06T02:03:39.582470", - "cpu_percent": 78.73, - "ram_percent": 44.7, - "ram_used_gb": 28.43946075439453, - "gpu_memory_used": 1035.0, - "relative_time": 258.2170786857605 - }, - { - "timestamp": "2025-01-06T02:03:40.696501", - "cpu_percent": 93.81, - "ram_percent": 44.8, - "ram_used_gb": 28.479244232177734, - "gpu_memory_used": 1043.0, - "relative_time": 259.3122980594635 - }, - { - "timestamp": "2025-01-06T02:03:41.801375", - "cpu_percent": 75.19, - "ram_percent": 44.8, - "ram_used_gb": 28.469676971435547, - "gpu_memory_used": 1043.0, - "relative_time": 260.4192705154419 - }, - { - "timestamp": "2025-01-06T02:03:42.902513", - "cpu_percent": 71.84, - "ram_percent": 44.8, - "ram_used_gb": 28.466243743896484, - "gpu_memory_used": 1043.0, - "relative_time": 261.4810314178467 - }, - { - "timestamp": "2025-01-06T02:03:43.966522", - "cpu_percent": 65.83, - "ram_percent": 44.8, - "ram_used_gb": 28.46680450439453, - "gpu_memory_used": 1043.0, - "relative_time": 262.59079146385193 - }, - { - "timestamp": "2025-01-06T02:03:45.071077", - "cpu_percent": 73.81, - "ram_percent": 44.7, - "ram_used_gb": 28.444644927978516, - "gpu_memory_used": 1043.0, - "relative_time": 263.69037771224976 - }, - { - "timestamp": "2025-01-06T02:03:46.177191", - "cpu_percent": 76.08, - "ram_percent": 44.7, - "ram_used_gb": 28.449535369873047, - "gpu_memory_used": 1044.0, - "relative_time": 264.811972618103 - }, - { - "timestamp": "2025-01-06T02:03:47.297543", - "cpu_percent": 69.21, - "ram_percent": 44.7, - "ram_used_gb": 28.455825805664062, - "gpu_memory_used": 1038.0, - "relative_time": 265.930095911026 - }, - { - "timestamp": "2025-01-06T02:03:48.419566", - "cpu_percent": 89.1, - "ram_percent": 44.8, - "ram_used_gb": 28.472156524658203, - "gpu_memory_used": 1042.0, - "relative_time": 267.05133056640625 - }, - { - "timestamp": "2025-01-06T02:03:49.545977", - "cpu_percent": 77.62, - "ram_percent": 44.8, - "ram_used_gb": 28.473480224609375, - "gpu_memory_used": 1043.0, - "relative_time": 268.1589620113373 - }, - { - "timestamp": "2025-01-06T02:03:50.649988", - "cpu_percent": 65.12, - "ram_percent": 44.8, - "ram_used_gb": 28.477100372314453, - "gpu_memory_used": 1043.0, - "relative_time": 269.270804643631 - }, - { - "timestamp": "2025-01-06T02:03:51.761398", - "cpu_percent": 72.83, - "ram_percent": 44.8, - "ram_used_gb": 28.466938018798828, - "gpu_memory_used": 1043.0, - "relative_time": 270.4037699699402 - }, - { - "timestamp": "2025-01-06T02:03:52.899078", - "cpu_percent": 69.92, - "ram_percent": 44.8, - "ram_used_gb": 28.516891479492188, - "gpu_memory_used": 1043.0, - "relative_time": 271.52356004714966 - }, - { - "timestamp": "2025-01-06T02:03:54.009357", - "cpu_percent": 64.24, - "ram_percent": 44.8, - "ram_used_gb": 28.478267669677734, - "gpu_memory_used": 1037.0, - "relative_time": 272.606009721756 - }, - { - "timestamp": "2025-01-06T02:03:55.099524", - "cpu_percent": 70.53, - "ram_percent": 44.8, - "ram_used_gb": 28.46905517578125, - "gpu_memory_used": 1037.0, - "relative_time": 273.72701716423035 - }, - { - "timestamp": "2025-01-06T02:03:56.206900", - "cpu_percent": 68.34, - "ram_percent": 44.7, - "ram_used_gb": 28.465599060058594, - "gpu_memory_used": 1037.0, - "relative_time": 274.8694078922272 - }, - { - "timestamp": "2025-01-06T02:03:57.354260", - "cpu_percent": 72.55, - "ram_percent": 44.8, - "ram_used_gb": 28.492900848388672, - "gpu_memory_used": 1036.0, - "relative_time": 275.9902341365814 - }, - { - "timestamp": "2025-01-06T02:03:58.471175", - "cpu_percent": 82.03, - "ram_percent": 44.8, - "ram_used_gb": 28.485233306884766, - "gpu_memory_used": 1035.0, - "relative_time": 277.10426807403564 - }, - { - "timestamp": "2025-01-06T02:03:59.587394", - "cpu_percent": 73.04, - "ram_percent": 44.8, - "ram_used_gb": 28.481033325195312, - "gpu_memory_used": 1036.0, - "relative_time": 278.22713327407837 - }, - { - "timestamp": "2025-01-06T02:04:00.719395", - "cpu_percent": 98.57, - "ram_percent": 44.8, - "ram_used_gb": 28.480728149414062, - "gpu_memory_used": 1044.0, - "relative_time": 279.353773355484 - }, - { - "timestamp": "2025-01-06T02:04:01.842768", - "cpu_percent": 71.4, - "ram_percent": 44.8, - "ram_used_gb": 28.491016387939453, - "gpu_memory_used": 1045.0, - "relative_time": 280.5646574497223 - }, - { - "timestamp": "2025-01-06T02:04:03.051785", - "cpu_percent": 89.75, - "ram_percent": 44.8, - "ram_used_gb": 28.49317169189453, - "gpu_memory_used": 1045.0, - "relative_time": 281.7272472381592 - }, - { - "timestamp": "2025-01-06T02:04:04.219136", - "cpu_percent": 73.74, - "ram_percent": 44.8, - "ram_used_gb": 28.491840362548828, - "gpu_memory_used": 1045.0, - "relative_time": 282.850713968277 - }, - { - "timestamp": "2025-01-06T02:04:05.332164", - "cpu_percent": 90.68, - "ram_percent": 44.8, - "ram_used_gb": 28.500041961669922, - "gpu_memory_used": 1046.0, - "relative_time": 283.98440742492676 - }, - { - "timestamp": "2025-01-06T02:04:06.464951", - "cpu_percent": 86.44, - "ram_percent": 44.8, - "ram_used_gb": 28.515422821044922, - "gpu_memory_used": 1046.0, - "relative_time": 285.0827331542969 - }, - { - "timestamp": "2025-01-06T02:04:07.566047", - "cpu_percent": 86.21, - "ram_percent": 44.8, - "ram_used_gb": 28.5203857421875, - "gpu_memory_used": 1045.0, - "relative_time": 286.2305998802185 - }, - { - "timestamp": "2025-01-06T02:04:08.713374", - "cpu_percent": 71.91, - "ram_percent": 45.0, - "ram_used_gb": 28.612117767333984, - "gpu_memory_used": 1044.0, - "relative_time": 287.33052921295166 - }, - { - "timestamp": "2025-01-06T02:04:09.815028", - "cpu_percent": 69.27, - "ram_percent": 45.1, - "ram_used_gb": 28.676231384277344, - "gpu_memory_used": 1044.0, - "relative_time": 288.4558057785034 - }, - { - "timestamp": "2025-01-06T02:04:10.950295", - "cpu_percent": 82.52, - "ram_percent": 45.2, - "ram_used_gb": 28.753101348876953, - "gpu_memory_used": 1043.0, - "relative_time": 289.58639645576477 - }, - { - "timestamp": "2025-01-06T02:04:12.077013", - "cpu_percent": 82.08, - "ram_percent": 45.3, - "ram_used_gb": 28.822429656982422, - "gpu_memory_used": 1038.0, - "relative_time": 290.69340085983276 - }, - { - "timestamp": "2025-01-06T02:04:13.177952", - "cpu_percent": 71.31, - "ram_percent": 45.5, - "ram_used_gb": 28.919525146484375, - "gpu_memory_used": 1038.0, - "relative_time": 291.77725172042847 - }, - { - "timestamp": "2025-01-06T02:04:14.261590", - "cpu_percent": 64.09, - "ram_percent": 45.6, - "ram_used_gb": 28.983028411865234, - "gpu_memory_used": 1038.0, - "relative_time": 292.8957669734955 - }, - { - "timestamp": "2025-01-06T02:04:15.378908", - "cpu_percent": 75.32, - "ram_percent": 45.6, - "ram_used_gb": 29.025554656982422, - "gpu_memory_used": 1037.0, - "relative_time": 294.02492356300354 - }, - { - "timestamp": "2025-01-06T02:04:16.510116", - "cpu_percent": 77.4, - "ram_percent": 45.7, - "ram_used_gb": 29.070560455322266, - "gpu_memory_used": 1031.0, - "relative_time": 295.1259517669678 - }, - { - "timestamp": "2025-01-06T02:04:17.615075", - "cpu_percent": 76.31, - "ram_percent": 45.8, - "ram_used_gb": 29.118003845214844, - "gpu_memory_used": 1027.0, - "relative_time": 296.2284073829651 - }, - { - "timestamp": "2025-01-06T02:04:18.721955", - "cpu_percent": 78.09, - "ram_percent": 45.3, - "ram_used_gb": 28.807754516601562, - "gpu_memory_used": 1028.0, - "relative_time": 297.38829493522644 - }, - { - "timestamp": "2025-01-06T02:04:19.874178", - "cpu_percent": 87.58, - "ram_percent": 45.2, - "ram_used_gb": 28.77080535888672, - "gpu_memory_used": 1028.0, - "relative_time": 298.5273485183716 - }, - { - "timestamp": "2025-01-06T02:04:21.030928", - "cpu_percent": 80.39, - "ram_percent": 45.2, - "ram_used_gb": 28.74532699584961, - "gpu_memory_used": 1029.0, - "relative_time": 299.648122549057 - }, - { - "timestamp": "2025-01-06T02:04:22.137059", - "cpu_percent": 69.39, - "ram_percent": 45.2, - "ram_used_gb": 28.74283218383789, - "gpu_memory_used": 1036.0, - "relative_time": 300.7701139450073 - }, - { - "timestamp": "2025-01-06T02:04:23.259177", - "cpu_percent": 68.48, - "ram_percent": 45.2, - "ram_used_gb": 28.735374450683594, - "gpu_memory_used": 1036.0, - "relative_time": 301.8923716545105 - }, - { - "timestamp": "2025-01-06T02:04:24.376291", - "cpu_percent": 77.94, - "ram_percent": 45.2, - "ram_used_gb": 28.745147705078125, - "gpu_memory_used": 1035.0, - "relative_time": 303.0271394252777 - }, - { - "timestamp": "2025-01-06T02:04:25.511434", - "cpu_percent": 72.12, - "ram_percent": 45.1, - "ram_used_gb": 28.703075408935547, - "gpu_memory_used": 1035.0, - "relative_time": 304.14817214012146 - }, - { - "timestamp": "2025-01-06T02:04:26.633643", - "cpu_percent": 79.6, - "ram_percent": 45.1, - "ram_used_gb": 28.69623565673828, - "gpu_memory_used": 1035.0, - "relative_time": 305.25277972221375 - }, - { - "timestamp": "2025-01-06T02:04:27.737495", - "cpu_percent": 45.9, - "ram_percent": 45.2, - "ram_used_gb": 28.770511627197266, - "gpu_memory_used": 1035.0, - "relative_time": 306.34188294410706 - }, - { - "timestamp": "2025-01-06T02:04:28.847601", - "cpu_percent": 32.22, - "ram_percent": 45.3, - "ram_used_gb": 28.79547119140625, - "gpu_memory_used": 1040.0, - "relative_time": 307.4505376815796 + "timestamp": "2025-01-30T05:03:26.422469", + "cpu_percent": 0.0, + "ram_percent": 18.5, + "ram_used_gb": 5.2551727294921875, + "gpu_memory_used": 1988.0, + "relative_time": 0.14498639106750488 + }, + { + "timestamp": "2025-01-30T05:03:27.568319", + "cpu_percent": 13.42, + "ram_percent": 18.6, + "ram_used_gb": 5.267307281494141, + "gpu_memory_used": 2025.0, + "relative_time": 1.1970372200012207 + }, + { + "timestamp": "2025-01-30T05:03:28.620098", + "cpu_percent": 12.89, + "ram_percent": 18.6, + "ram_used_gb": 5.267337799072266, + "gpu_memory_used": 3071.0, + "relative_time": 2.254074811935425 + }, + { + "timestamp": "2025-01-30T05:03:29.677030", + "cpu_percent": 12.43, + "ram_percent": 18.6, + "ram_used_gb": 5.29168701171875, + "gpu_memory_used": 2555.0, + "relative_time": 3.306957244873047 + }, + { + "timestamp": "2025-01-30T05:03:30.729971", + "cpu_percent": 12.47, + "ram_percent": 18.6, + "ram_used_gb": 5.292213439941406, + "gpu_memory_used": 3345.0, + "relative_time": 4.3373119831085205 + }, + { + "timestamp": "2025-01-30T05:03:31.760463", + "cpu_percent": 13.71, + "ram_percent": 18.7, + "ram_used_gb": 5.30987548828125, + "gpu_memory_used": 2549.0, + "relative_time": 5.368744850158691 + }, + { + "timestamp": "2025-01-30T05:03:32.791904", + "cpu_percent": 12.16, + "ram_percent": 18.7, + "ram_used_gb": 5.308803558349609, + "gpu_memory_used": 3358.0, + "relative_time": 6.418949842453003 + }, + { + "timestamp": "2025-01-30T05:03:33.842039", + "cpu_percent": 11.5, + "ram_percent": 18.7, + "ram_used_gb": 5.309070587158203, + "gpu_memory_used": 3349.0, + "relative_time": 7.4437031745910645 + }, + { + "timestamp": "2025-01-30T05:03:34.866692", + "cpu_percent": 15.38, + "ram_percent": 18.7, + "ram_used_gb": 5.2960205078125, + "gpu_memory_used": 3034.0, + "relative_time": 8.472418069839478 + }, + { + "timestamp": "2025-01-30T05:03:35.895656", + "cpu_percent": 13.44, + "ram_percent": 18.7, + "ram_used_gb": 5.294971466064453, + "gpu_memory_used": 3315.0, + "relative_time": 9.498533248901367 + }, + { + "timestamp": "2025-01-30T05:03:36.921589", + "cpu_percent": 12.64, + "ram_percent": 18.7, + "ram_used_gb": 5.297389984130859, + "gpu_memory_used": 3314.0, + "relative_time": 10.565555095672607 + }, + { + "timestamp": "2025-01-30T05:03:37.994149", + "cpu_percent": 8.32, + "ram_percent": 18.7, + "ram_used_gb": 5.305477142333984, + "gpu_memory_used": 1958.0, + "relative_time": 11.616873502731323 } ], - "test_duration": 310.5054519176483 + "test_duration": 14.051392793655396 } \ No newline at end of file diff --git a/examples/assorted_checks/benchmarks/output_data/cpu_benchmark_stats_rtf.txt b/examples/assorted_checks/benchmarks/output_data/cpu_benchmark_stats_rtf.txt index 7c8023a..37f1cfb 100644 --- a/examples/assorted_checks/benchmarks/output_data/cpu_benchmark_stats_rtf.txt +++ b/examples/assorted_checks/benchmarks/output_data/cpu_benchmark_stats_rtf.txt @@ -1,23 +1,23 @@ === Benchmark Statistics (with correct RTF) === -Total tokens processed: 1800 -Total audio generated (s): 568.53 -Total test duration (s): 306.02 -Average processing rate (tokens/s): 5.75 -Average RTF: 0.55 -Average Real Time Speed: 1.81 +Total tokens processed: 1500 +Total audio generated (s): 427.90 +Total test duration (s): 10.84 +Average processing rate (tokens/s): 133.35 +Average RTF: 0.02 +Average Real Time Speed: 41.67 === Per-chunk Stats === -Average chunk size (tokens): 600.00 -Min chunk size (tokens): 300 -Max chunk size (tokens): 900 -Average processing time (s): 101.89 -Average output length (s): 189.51 +Average chunk size (tokens): 300.00 +Min chunk size (tokens): 100 +Max chunk size (tokens): 500 +Average processing time (s): 2.13 +Average output length (s): 85.58 === Performance Ranges === -Processing rate range (tokens/s): 5.30 - 6.26 -RTF range: 0.51x - 0.59x -Real Time Speed range: 1.69x - 1.96x +Processing rate range (tokens/s): 102.04 - 159.74 +RTF range: 0.02x - 0.03x +Real Time Speed range: 33.33x - 50.00x diff --git a/examples/assorted_checks/benchmarks/output_data/gpu_benchmark_results_rtf.json b/examples/assorted_checks/benchmarks/output_data/gpu_benchmark_results_rtf.json index 89db38c..6b3e68a 100644 --- a/examples/assorted_checks/benchmarks/output_data/gpu_benchmark_results_rtf.json +++ b/examples/assorted_checks/benchmarks/output_data/gpu_benchmark_results_rtf.json @@ -2,616 +2,240 @@ "results": [ { "tokens": 150, - "processing_time": 2.36, - "output_length": 45.9, - "rtf": 0.05, - "elapsed_time": 2.44626 + "processing_time": 1.18, + "output_length": 43.7, + "rtf": 0.03, + "elapsed_time": 1.20302 }, { "tokens": 300, - "processing_time": 4.94, - "output_length": 96.425, - "rtf": 0.05, - "elapsed_time": 7.46073 + "processing_time": 2.27, + "output_length": 86.75, + "rtf": 0.03, + "elapsed_time": 3.49958 }, { "tokens": 450, - "processing_time": 8.94, - "output_length": 143.1, - "rtf": 0.06, - "elapsed_time": 16.55036 + "processing_time": 3.49, + "output_length": 125.9, + "rtf": 0.03, + "elapsed_time": 7.03862 }, { "tokens": 600, - "processing_time": 19.78, - "output_length": 188.675, - "rtf": 0.1, - "elapsed_time": 36.69352 + "processing_time": 4.64, + "output_length": 169.325, + "rtf": 0.03, + "elapsed_time": 11.71062 }, { "tokens": 750, - "processing_time": 19.89, - "output_length": 236.7, - "rtf": 0.08, - "elapsed_time": 56.77695 + "processing_time": 5.07, + "output_length": 212.3, + "rtf": 0.02, + "elapsed_time": 16.83186 }, { "tokens": 900, - "processing_time": 16.83, - "output_length": 283.425, - "rtf": 0.06, - "elapsed_time": 73.8079 + "processing_time": 6.66, + "output_length": 258.0, + "rtf": 0.03, + "elapsed_time": 23.54135 } ], "system_metrics": [ { - "timestamp": "2025-01-06T00:43:20.888295", - "cpu_percent": 36.92, - "ram_percent": 68.6, - "ram_used_gb": 43.6395263671875, - "gpu_memory_used": 7022.0, - "relative_time": 0.09646010398864746 + "timestamp": "2025-01-30T05:06:38.733338", + "cpu_percent": 0.0, + "ram_percent": 18.6, + "ram_used_gb": 5.284908294677734, + "gpu_memory_used": 1925.0, + "relative_time": 0.039948463439941406 }, - { - "timestamp": "2025-01-06T00:43:21.983741", - "cpu_percent": 22.29, - "ram_percent": 68.6, - "ram_used_gb": 43.642677307128906, - "gpu_memory_used": 7021.0, - "relative_time": 1.1906661987304688 - }, - { - "timestamp": "2025-01-06T00:43:23.078293", - "cpu_percent": 27.39, - "ram_percent": 68.6, - "ram_used_gb": 43.61421203613281, - "gpu_memory_used": 7190.0, - "relative_time": 2.264479160308838 - }, - { - "timestamp": "2025-01-06T00:43:24.151445", - "cpu_percent": 20.28, - "ram_percent": 68.6, - "ram_used_gb": 43.65406036376953, - "gpu_memory_used": 7193.0, - "relative_time": 3.349093198776245 - }, - { - "timestamp": "2025-01-06T00:43:25.237021", - "cpu_percent": 23.03, - "ram_percent": 68.6, - "ram_used_gb": 43.647274017333984, - "gpu_memory_used": 7191.0, - "relative_time": 4.413560628890991 - }, - { - "timestamp": "2025-01-06T00:43:26.300255", - "cpu_percent": 23.62, - "ram_percent": 68.6, - "ram_used_gb": 43.642295837402344, - "gpu_memory_used": 7185.0, - "relative_time": 5.484973430633545 - }, - { - "timestamp": "2025-01-06T00:43:27.377319", - "cpu_percent": 46.04, - "ram_percent": 68.7, - "ram_used_gb": 43.7291374206543, - "gpu_memory_used": 7178.0, - "relative_time": 6.658120632171631 - }, - { - "timestamp": "2025-01-06T00:43:28.546053", - "cpu_percent": 29.79, - "ram_percent": 68.7, - "ram_used_gb": 43.73202133178711, - "gpu_memory_used": 7177.0, - "relative_time": 7.725939035415649 - }, - { - "timestamp": "2025-01-06T00:43:29.613327", - "cpu_percent": 18.19, - "ram_percent": 68.8, - "ram_used_gb": 43.791343688964844, - "gpu_memory_used": 7177.0, - "relative_time": 8.800285577774048 - }, - { - "timestamp": "2025-01-06T00:43:30.689097", - "cpu_percent": 22.29, - "ram_percent": 68.9, - "ram_used_gb": 43.81514358520508, - "gpu_memory_used": 7176.0, - "relative_time": 9.899119853973389 - }, - { - "timestamp": "2025-01-06T00:43:31.786443", - "cpu_percent": 32.59, - "ram_percent": 68.9, - "ram_used_gb": 43.834510803222656, - "gpu_memory_used": 7189.0, - "relative_time": 11.042734384536743 - }, - { - "timestamp": "2025-01-06T00:43:32.929720", - "cpu_percent": 42.48, - "ram_percent": 68.8, - "ram_used_gb": 43.77507019042969, - "gpu_memory_used": 7192.0, - "relative_time": 12.117269277572632 - }, - { - "timestamp": "2025-01-06T00:43:34.004481", - "cpu_percent": 26.33, - "ram_percent": 68.8, - "ram_used_gb": 43.77891159057617, - "gpu_memory_used": 7192.0, - "relative_time": 13.19830870628357 - }, - { - "timestamp": "2025-01-06T00:43:35.086024", - "cpu_percent": 26.53, - "ram_percent": 68.8, - "ram_used_gb": 43.77515411376953, - "gpu_memory_used": 7192.0, - "relative_time": 14.29457426071167 - }, - { - "timestamp": "2025-01-06T00:43:36.183496", - "cpu_percent": 40.33, - "ram_percent": 68.9, - "ram_used_gb": 43.81095886230469, - "gpu_memory_used": 7192.0, - "relative_time": 15.402768850326538 - }, - { - "timestamp": "2025-01-06T00:43:37.290635", - "cpu_percent": 43.6, - "ram_percent": 69.0, - "ram_used_gb": 43.87236022949219, - "gpu_memory_used": 7190.0, - "relative_time": 16.574281930923462 - }, - { - "timestamp": "2025-01-06T00:43:38.462164", - "cpu_percent": 85.74, - "ram_percent": 69.0, - "ram_used_gb": 43.864280700683594, - "gpu_memory_used": 6953.0, - "relative_time": 17.66074824333191 - }, - { - "timestamp": "2025-01-06T00:43:39.548295", - "cpu_percent": 23.88, - "ram_percent": 68.8, - "ram_used_gb": 43.75236129760742, - "gpu_memory_used": 4722.0, - "relative_time": 18.739423036575317 - }, - { - "timestamp": "2025-01-06T00:43:40.626692", - "cpu_percent": 59.24, - "ram_percent": 68.7, - "ram_used_gb": 43.720741271972656, - "gpu_memory_used": 4723.0, - "relative_time": 19.846031665802002 - }, - { - "timestamp": "2025-01-06T00:43:41.733597", - "cpu_percent": 41.74, - "ram_percent": 68.4, - "ram_used_gb": 43.53546142578125, - "gpu_memory_used": 4722.0, - "relative_time": 20.920310020446777 - }, - { - "timestamp": "2025-01-06T00:43:42.808191", - "cpu_percent": 35.43, - "ram_percent": 68.3, - "ram_used_gb": 43.424468994140625, - "gpu_memory_used": 4726.0, - "relative_time": 22.00457763671875 - }, - { - "timestamp": "2025-01-06T00:43:43.891669", - "cpu_percent": 43.81, - "ram_percent": 68.2, - "ram_used_gb": 43.38311004638672, - "gpu_memory_used": 4727.0, - "relative_time": 23.08402943611145 - }, - { - "timestamp": "2025-01-06T00:43:44.971246", - "cpu_percent": 58.13, - "ram_percent": 68.0, - "ram_used_gb": 43.27970886230469, - "gpu_memory_used": 4731.0, - "relative_time": 24.249765396118164 - }, - { - "timestamp": "2025-01-06T00:43:46.137626", - "cpu_percent": 66.76, - "ram_percent": 68.0, - "ram_used_gb": 43.23844528198242, - "gpu_memory_used": 4731.0, - "relative_time": 25.32853865623474 - }, - { - "timestamp": "2025-01-06T00:43:47.219723", - "cpu_percent": 27.95, - "ram_percent": 67.8, - "ram_used_gb": 43.106136322021484, - "gpu_memory_used": 4734.0, - "relative_time": 26.499221563339233 - }, - { - "timestamp": "2025-01-06T00:43:48.386913", - "cpu_percent": 73.13, - "ram_percent": 67.7, - "ram_used_gb": 43.049781799316406, - "gpu_memory_used": 4736.0, - "relative_time": 27.592528104782104 - }, - { - "timestamp": "2025-01-06T00:43:49.480407", - "cpu_percent": 50.63, - "ram_percent": 67.6, - "ram_used_gb": 43.007415771484375, - "gpu_memory_used": 4736.0, - "relative_time": 28.711266040802002 - }, - { - "timestamp": "2025-01-06T00:43:50.599220", - "cpu_percent": 92.36, - "ram_percent": 67.5, - "ram_used_gb": 42.9685173034668, - "gpu_memory_used": 4728.0, - "relative_time": 29.916289567947388 - }, - { - "timestamp": "2025-01-06T00:43:51.803667", - "cpu_percent": 83.07, - "ram_percent": 67.5, - "ram_used_gb": 42.96232986450195, - "gpu_memory_used": 4724.0, - "relative_time": 31.039498805999756 - }, - { - "timestamp": "2025-01-06T00:43:52.927208", - "cpu_percent": 90.61, - "ram_percent": 67.5, - "ram_used_gb": 42.96202850341797, - "gpu_memory_used": 5037.0, - "relative_time": 32.2381911277771 - }, - { - "timestamp": "2025-01-06T00:43:54.128135", - "cpu_percent": 89.47, - "ram_percent": 67.5, - "ram_used_gb": 42.94692611694336, - "gpu_memory_used": 5085.0, - "relative_time": 33.35147500038147 + { + "timestamp": "2025-01-30T05:06:39.774003", + "cpu_percent": 13.37, + "ram_percent": 18.6, + "ram_used_gb": 5.2852630615234375, + "gpu_memory_used": 3047.0, + "relative_time": 1.0883615016937256 }, - { - "timestamp": "2025-01-06T00:43:55.238967", - "cpu_percent": 60.01, - "ram_percent": 67.4, - "ram_used_gb": 42.88222122192383, - "gpu_memory_used": 5085.0, - "relative_time": 34.455963373184204 - }, - { - "timestamp": "2025-01-06T00:43:56.344164", - "cpu_percent": 62.12, - "ram_percent": 67.3, - "ram_used_gb": 42.81411361694336, - "gpu_memory_used": 5083.0, - "relative_time": 35.549962282180786 - }, - { - "timestamp": "2025-01-06T00:43:57.437566", - "cpu_percent": 53.56, - "ram_percent": 67.3, - "ram_used_gb": 42.83011245727539, - "gpu_memory_used": 5078.0, - "relative_time": 36.66783380508423 - }, - { - "timestamp": "2025-01-06T00:43:58.554923", - "cpu_percent": 80.27, - "ram_percent": 67.3, - "ram_used_gb": 42.79304504394531, - "gpu_memory_used": 5069.0, - "relative_time": 37.77330660820007 + { + "timestamp": "2025-01-30T05:06:40.822449", + "cpu_percent": 13.68, + "ram_percent": 18.7, + "ram_used_gb": 5.303462982177734, + "gpu_memory_used": 3040.0, + "relative_time": 2.12058687210083 }, - { - "timestamp": "2025-01-06T00:43:59.660456", - "cpu_percent": 72.33, - "ram_percent": 67.2, - "ram_used_gb": 42.727474212646484, - "gpu_memory_used": 5079.0, - "relative_time": 38.885955810546875 + { + "timestamp": "2025-01-30T05:06:41.854375", + "cpu_percent": 15.39, + "ram_percent": 18.7, + "ram_used_gb": 5.306262969970703, + "gpu_memory_used": 3326.0, + "relative_time": 3.166278600692749 }, - { - "timestamp": "2025-01-06T00:44:00.773867", - "cpu_percent": 59.29, - "ram_percent": 66.9, - "ram_used_gb": 42.566131591796875, - "gpu_memory_used": 5079.0, - "relative_time": 39.99704432487488 - }, - { - "timestamp": "2025-01-06T00:44:01.884399", - "cpu_percent": 43.52, - "ram_percent": 66.5, - "ram_used_gb": 42.32980728149414, - "gpu_memory_used": 5079.0, - "relative_time": 41.13008522987366 - }, - { - "timestamp": "2025-01-06T00:44:03.018905", - "cpu_percent": 84.46, - "ram_percent": 66.5, - "ram_used_gb": 42.28911590576172, - "gpu_memory_used": 5087.0, - "relative_time": 42.296770095825195 - }, - { - "timestamp": "2025-01-06T00:44:04.184606", - "cpu_percent": 88.27, - "ram_percent": 66.3, - "ram_used_gb": 42.16263961791992, - "gpu_memory_used": 5091.0, - "relative_time": 43.42832589149475 + { + "timestamp": "2025-01-30T05:06:42.900882", + "cpu_percent": 14.19, + "ram_percent": 18.8, + "ram_used_gb": 5.337162017822266, + "gpu_memory_used": 2530.0, + "relative_time": 4.256956577301025 }, - { - "timestamp": "2025-01-06T00:44:05.315967", - "cpu_percent": 80.91, - "ram_percent": 65.9, - "ram_used_gb": 41.9491081237793, - "gpu_memory_used": 5089.0, - "relative_time": 44.52496290206909 + { + "timestamp": "2025-01-30T05:06:43.990792", + "cpu_percent": 12.63, + "ram_percent": 18.8, + "ram_used_gb": 5.333805084228516, + "gpu_memory_used": 3331.0, + "relative_time": 5.2854602336883545 }, - { - "timestamp": "2025-01-06T00:44:06.412298", - "cpu_percent": 41.68, - "ram_percent": 65.6, - "ram_used_gb": 41.72716522216797, - "gpu_memory_used": 5090.0, - "relative_time": 45.679444313049316 + { + "timestamp": "2025-01-30T05:06:45.019134", + "cpu_percent": 14.14, + "ram_percent": 18.8, + "ram_used_gb": 5.334297180175781, + "gpu_memory_used": 3332.0, + "relative_time": 6.351738929748535 }, - { - "timestamp": "2025-01-06T00:44:07.566964", - "cpu_percent": 73.02, - "ram_percent": 65.5, - "ram_used_gb": 41.64710998535156, - "gpu_memory_used": 5091.0, - "relative_time": 46.81710481643677 + { + "timestamp": "2025-01-30T05:06:46.085997", + "cpu_percent": 12.78, + "ram_percent": 18.8, + "ram_used_gb": 5.351467132568359, + "gpu_memory_used": 2596.0, + "relative_time": 7.392607688903809 }, - { - "timestamp": "2025-01-06T00:44:08.704786", - "cpu_percent": 75.38, - "ram_percent": 65.4, - "ram_used_gb": 41.59475326538086, - "gpu_memory_used": 5097.0, - "relative_time": 47.91444158554077 + { + "timestamp": "2025-01-30T05:06:47.127113", + "cpu_percent": 14.7, + "ram_percent": 18.9, + "ram_used_gb": 5.367542266845703, + "gpu_memory_used": 3341.0, + "relative_time": 8.441826343536377 }, - { - "timestamp": "2025-01-06T00:44:09.802745", - "cpu_percent": 42.21, - "ram_percent": 65.2, - "ram_used_gb": 41.45526885986328, - "gpu_memory_used": 5111.0, - "relative_time": 49.04095649719238 + { + "timestamp": "2025-01-30T05:06:48.176033", + "cpu_percent": 13.47, + "ram_percent": 18.9, + "ram_used_gb": 5.361263275146484, + "gpu_memory_used": 3339.0, + "relative_time": 9.500520706176758 }, - { - "timestamp": "2025-01-06T00:44:10.928231", - "cpu_percent": 65.65, - "ram_percent": 64.4, - "ram_used_gb": 40.93437957763672, - "gpu_memory_used": 5111.0, - "relative_time": 50.14311861991882 + { + "timestamp": "2025-01-30T05:06:49.234332", + "cpu_percent": 15.84, + "ram_percent": 18.9, + "ram_used_gb": 5.3612213134765625, + "gpu_memory_used": 3339.0, + "relative_time": 10.53744649887085 + }, + { + "timestamp": "2025-01-30T05:06:50.271159", + "cpu_percent": 14.89, + "ram_percent": 18.9, + "ram_used_gb": 5.379688262939453, + "gpu_memory_used": 3646.0, + "relative_time": 11.570110321044922 + }, + { + "timestamp": "2025-01-30T05:06:51.303841", + "cpu_percent": 15.71, + "ram_percent": 19.0, + "ram_used_gb": 5.390773773193359, + "gpu_memory_used": 3037.0, + "relative_time": 12.60651707649231 }, { - "timestamp": "2025-01-06T00:44:12.036249", - "cpu_percent": 28.51, - "ram_percent": 64.1, - "ram_used_gb": 40.749881744384766, - "gpu_memory_used": 5107.0, - "relative_time": 51.250269651412964 - }, + "timestamp": "2025-01-30T05:06:52.340383", + "cpu_percent": 15.46, + "ram_percent": 19.0, + "ram_used_gb": 5.389518737792969, + "gpu_memory_used": 3319.0, + "relative_time": 13.636165380477905 + }, { - "timestamp": "2025-01-06T00:44:13.137586", - "cpu_percent": 52.99, - "ram_percent": 64.2, - "ram_used_gb": 40.84278869628906, - "gpu_memory_used": 5104.0, - "relative_time": 52.34805965423584 - }, + "timestamp": "2025-01-30T05:06:53.370342", + "cpu_percent": 13.12, + "ram_percent": 19.0, + "ram_used_gb": 5.391136169433594, + "gpu_memory_used": 3320.0, + "relative_time": 14.67578935623169 + }, { - "timestamp": "2025-01-06T00:44:14.235248", - "cpu_percent": 34.55, - "ram_percent": 64.1, - "ram_used_gb": 40.7873420715332, - "gpu_memory_used": 5097.0, - "relative_time": 53.424301862716675 - }, + "timestamp": "2025-01-30T05:06:54.376175", + "cpu_percent": 14.98, + "ram_percent": 19.0, + "ram_used_gb": 5.390045166015625, + "gpu_memory_used": 3627.0, + "relative_time": 15.70747685432434 + }, { - "timestamp": "2025-01-06T00:44:15.311386", - "cpu_percent": 39.07, - "ram_percent": 64.2, - "ram_used_gb": 40.860008239746094, - "gpu_memory_used": 5091.0, - "relative_time": 54.50679922103882 - }, + "timestamp": "2025-01-30T05:06:55.441172", + "cpu_percent": 13.45, + "ram_percent": 19.0, + "ram_used_gb": 5.394947052001953, + "gpu_memory_used": 1937.0, + "relative_time": 16.758784770965576 + }, { - "timestamp": "2025-01-06T00:44:16.393626", - "cpu_percent": 31.02, - "ram_percent": 64.3, - "ram_used_gb": 40.884307861328125, - "gpu_memory_used": 5093.0, - "relative_time": 55.57431173324585 - }, + "timestamp": "2025-01-30T05:06:56.492442", + "cpu_percent": 17.03, + "ram_percent": 18.9, + "ram_used_gb": 5.361682891845703, + "gpu_memory_used": 3041.0, + "relative_time": 17.789713144302368 + }, { - "timestamp": "2025-01-06T00:44:17.461449", - "cpu_percent": 24.53, - "ram_percent": 64.3, - "ram_used_gb": 40.89955520629883, - "gpu_memory_used": 5070.0, - "relative_time": 56.660638093948364 - }, + "timestamp": "2025-01-30T05:06:57.523536", + "cpu_percent": 13.76, + "ram_percent": 18.9, + "ram_used_gb": 5.360996246337891, + "gpu_memory_used": 3321.0, + "relative_time": 18.838542222976685 + }, { - "timestamp": "2025-01-06T00:44:18.547558", - "cpu_percent": 19.93, - "ram_percent": 64.3, - "ram_used_gb": 40.92641830444336, - "gpu_memory_used": 5074.0, - "relative_time": 57.736456871032715 - }, + "timestamp": "2025-01-30T05:06:58.572158", + "cpu_percent": 15.94, + "ram_percent": 18.9, + "ram_used_gb": 5.3652801513671875, + "gpu_memory_used": 3323.0, + "relative_time": 19.86689043045044 + }, { - "timestamp": "2025-01-06T00:44:19.624478", - "cpu_percent": 15.63, - "ram_percent": 64.3, - "ram_used_gb": 40.92564392089844, - "gpu_memory_used": 5082.0, - "relative_time": 58.81701683998108 - }, + "timestamp": "2025-01-30T05:06:59.600551", + "cpu_percent": 15.67, + "ram_percent": 18.9, + "ram_used_gb": 5.363399505615234, + "gpu_memory_used": 3630.0, + "relative_time": 20.89712619781494 + }, { - "timestamp": "2025-01-06T00:44:20.705184", - "cpu_percent": 29.86, - "ram_percent": 64.4, - "ram_used_gb": 40.935394287109375, - "gpu_memory_used": 5082.0, - "relative_time": 59.88701677322388 - }, + "timestamp": "2025-01-30T05:07:00.631315", + "cpu_percent": 15.37, + "ram_percent": 18.9, + "ram_used_gb": 5.3663482666015625, + "gpu_memory_used": 3629.0, + "relative_time": 22.01374316215515 + }, { - "timestamp": "2025-01-06T00:44:21.775463", - "cpu_percent": 43.55, - "ram_percent": 64.4, - "ram_used_gb": 40.9350471496582, - "gpu_memory_used": 5080.0, - "relative_time": 60.96005439758301 - }, - { - "timestamp": "2025-01-06T00:44:22.847939", - "cpu_percent": 26.66, - "ram_percent": 64.4, - "ram_used_gb": 40.94179916381836, - "gpu_memory_used": 5076.0, - "relative_time": 62.02673673629761 - }, - { - "timestamp": "2025-01-06T00:44:23.914337", - "cpu_percent": 22.46, - "ram_percent": 64.4, - "ram_used_gb": 40.9537467956543, - "gpu_memory_used": 5076.0, - "relative_time": 63.10581707954407 - }, - { - "timestamp": "2025-01-06T00:44:24.993313", - "cpu_percent": 28.07, - "ram_percent": 64.4, - "ram_used_gb": 40.94577407836914, - "gpu_memory_used": 5076.0, - "relative_time": 64.18998432159424 - }, - { - "timestamp": "2025-01-06T00:44:26.077028", - "cpu_percent": 26.1, - "ram_percent": 64.4, - "ram_used_gb": 40.98012161254883, - "gpu_memory_used": 5197.0, - "relative_time": 65.28782486915588 - }, - { - "timestamp": "2025-01-06T00:44:27.175228", - "cpu_percent": 35.17, - "ram_percent": 64.6, - "ram_used_gb": 41.0831184387207, - "gpu_memory_used": 5422.0, - "relative_time": 66.37566781044006 - }, - { - "timestamp": "2025-01-06T00:44:28.265025", - "cpu_percent": 55.14, - "ram_percent": 64.9, - "ram_used_gb": 41.25740432739258, - "gpu_memory_used": 5512.0, - "relative_time": 67.48023676872253 - }, - { - "timestamp": "2025-01-06T00:44:29.367776", - "cpu_percent": 53.84, - "ram_percent": 65.0, - "ram_used_gb": 41.36682891845703, - "gpu_memory_used": 5616.0, - "relative_time": 68.57096815109253 - }, - { - "timestamp": "2025-01-06T00:44:30.458301", - "cpu_percent": 33.42, - "ram_percent": 65.3, - "ram_used_gb": 41.5602912902832, - "gpu_memory_used": 5724.0, - "relative_time": 69.66709041595459 - }, - { - "timestamp": "2025-01-06T00:44:31.554329", - "cpu_percent": 50.81, - "ram_percent": 65.5, - "ram_used_gb": 41.66044616699219, - "gpu_memory_used": 5827.0, - "relative_time": 70.75874853134155 - }, - { - "timestamp": "2025-01-06T00:44:32.646414", - "cpu_percent": 34.34, - "ram_percent": 65.6, - "ram_used_gb": 41.739715576171875, - "gpu_memory_used": 5843.0, - "relative_time": 71.86718988418579 - }, - { - "timestamp": "2025-01-06T00:44:33.754223", - "cpu_percent": 44.32, - "ram_percent": 66.0, - "ram_used_gb": 42.005794525146484, - "gpu_memory_used": 5901.0, - "relative_time": 72.95793795585632 - }, - { - "timestamp": "2025-01-06T00:44:34.848852", - "cpu_percent": 48.36, - "ram_percent": 66.5, - "ram_used_gb": 42.3160514831543, - "gpu_memory_used": 5924.0, - "relative_time": 74.35109186172485 - }, - { - "timestamp": "2025-01-06T00:44:36.240235", - "cpu_percent": 58.06, - "ram_percent": 67.5, - "ram_used_gb": 42.95722198486328, - "gpu_memory_used": 5930.0, - "relative_time": 75.47581958770752 - }, - { - "timestamp": "2025-01-06T00:44:37.363208", - "cpu_percent": 46.82, - "ram_percent": 67.6, - "ram_used_gb": 42.97764587402344, - "gpu_memory_used": 6364.0, - "relative_time": 76.58708119392395 - }, + "timestamp": "2025-01-30T05:07:01.747500", + "cpu_percent": 13.79, + "ram_percent": 18.9, + "ram_used_gb": 5.367362976074219, + "gpu_memory_used": 3620.0, + "relative_time": 23.05113124847412 + }, { - "timestamp": "2025-01-06T00:44:38.474408", - "cpu_percent": 50.93, - "ram_percent": 67.9, - "ram_used_gb": 43.1597900390625, - "gpu_memory_used": 6426.0, - "relative_time": 77.6842532157898 + "timestamp": "2025-01-30T05:07:02.784828", + "cpu_percent": 10.16, + "ram_percent": 19.1, + "ram_used_gb": 5.443946838378906, + "gpu_memory_used": 1916.0, + "relative_time": 24.08937978744507 } ], - "test_duration": 82.49591493606567 + "test_duration": 26.596059799194336 } \ No newline at end of file diff --git a/examples/assorted_checks/benchmarks/output_data/gpu_benchmark_stats_rtf.txt b/examples/assorted_checks/benchmarks/output_data/gpu_benchmark_stats_rtf.txt index 7876062..a62888f 100644 --- a/examples/assorted_checks/benchmarks/output_data/gpu_benchmark_stats_rtf.txt +++ b/examples/assorted_checks/benchmarks/output_data/gpu_benchmark_stats_rtf.txt @@ -1,23 +1,23 @@ === Benchmark Statistics (with correct RTF) === Total tokens processed: 3150 -Total audio generated (s): 994.22 -Total test duration (s): 73.81 -Average processing rate (tokens/s): 49.36 -Average RTF: 0.07 -Average Real Time Speed: 15.00 +Total audio generated (s): 895.98 +Total test duration (s): 23.54 +Average processing rate (tokens/s): 133.43 +Average RTF: 0.03 +Average Real Time Speed: 35.29 === Per-chunk Stats === Average chunk size (tokens): 525.00 Min chunk size (tokens): 150 Max chunk size (tokens): 900 -Average processing time (s): 12.12 -Average output length (s): 165.70 +Average processing time (s): 3.88 +Average output length (s): 149.33 === Performance Ranges === -Processing rate range (tokens/s): 30.33 - 63.56 -RTF range: 0.05x - 0.10x -Real Time Speed range: 10.00x - 20.00x +Processing rate range (tokens/s): 127.12 - 147.93 +RTF range: 0.02x - 0.03x +Real Time Speed range: 33.33x - 50.00x diff --git a/examples/assorted_checks/benchmarks/output_plots/cpu_processing_time_rtf.png b/examples/assorted_checks/benchmarks/output_plots/cpu_processing_time_rtf.png index 2317c52..402e64a 100644 Binary files a/examples/assorted_checks/benchmarks/output_plots/cpu_processing_time_rtf.png and b/examples/assorted_checks/benchmarks/output_plots/cpu_processing_time_rtf.png differ diff --git a/examples/assorted_checks/benchmarks/output_plots/cpu_realtime_factor_rtf.png b/examples/assorted_checks/benchmarks/output_plots/cpu_realtime_factor_rtf.png index 919c53b..3eb6012 100644 Binary files a/examples/assorted_checks/benchmarks/output_plots/cpu_realtime_factor_rtf.png and b/examples/assorted_checks/benchmarks/output_plots/cpu_realtime_factor_rtf.png differ diff --git a/examples/assorted_checks/benchmarks/output_plots/cpu_system_usage_rtf.png b/examples/assorted_checks/benchmarks/output_plots/cpu_system_usage_rtf.png index 5143bda..ce25636 100644 Binary files a/examples/assorted_checks/benchmarks/output_plots/cpu_system_usage_rtf.png and b/examples/assorted_checks/benchmarks/output_plots/cpu_system_usage_rtf.png differ diff --git a/examples/assorted_checks/benchmarks/output_plots/gpu_processing_time_rtf.png b/examples/assorted_checks/benchmarks/output_plots/gpu_processing_time_rtf.png index ebf7bcf..28eb2c4 100644 Binary files a/examples/assorted_checks/benchmarks/output_plots/gpu_processing_time_rtf.png and b/examples/assorted_checks/benchmarks/output_plots/gpu_processing_time_rtf.png differ diff --git a/examples/assorted_checks/benchmarks/output_plots/gpu_realtime_factor_rtf.png b/examples/assorted_checks/benchmarks/output_plots/gpu_realtime_factor_rtf.png index bcdacda..601ee43 100644 Binary files a/examples/assorted_checks/benchmarks/output_plots/gpu_realtime_factor_rtf.png and b/examples/assorted_checks/benchmarks/output_plots/gpu_realtime_factor_rtf.png differ diff --git a/examples/assorted_checks/benchmarks/output_plots/gpu_system_usage_rtf.png b/examples/assorted_checks/benchmarks/output_plots/gpu_system_usage_rtf.png index a6a6ea5..f30c8b3 100644 Binary files a/examples/assorted_checks/benchmarks/output_plots/gpu_system_usage_rtf.png and b/examples/assorted_checks/benchmarks/output_plots/gpu_system_usage_rtf.png differ