However, a crucial distinction must be made. As of August 2023, the GGML format has been officially superseded by a newer, more advanced format called GGUF (GPT-Generated Unified Format). Projects like llama.cpp , which spearheaded the GGML revolution, no longer support the legacy GGML format and exclusively use GGUF.
+-------------------------------------------------------------+ | OpenAI Whisper PyTorch Model | | (769M Parameters) | +-------------------------------------------------------------+ │ ▼ (via convert-pt-to-ggml.py) +-------------------------------------------------------------+ | ggml-medium.bin | | - Binary Tensor Weights - Optimized Layout | | - Quantized (optional) - Standalone Resource | +-------------------------------------------------------------+ │ ▼ +-------------------------------------------------------------+ | whisper.cpp Inference Engine | | - C/C++ Execution - CPU/GPU Acceleration | +-------------------------------------------------------------+ The Whisper Blueprint ggmlmediumbin work
When running a "medium" sized model (roughly 3B to 13B parameters), the memory bandwidth is the bottleneck, not the math itself. However, a crucial distinction must be made
The most critical decision when working with GGML/GGUF models is which quantization level to choose for your specific task. The table below summarizes the key trade-offs for a typical 7B parameter model: the memory bandwidth is the bottleneck
Are you looking to run this from the , or integrate it into a GUI/app ?
