I would like to request a feature to enable the loading of GGUF models from local storage. Currently, GGUF models can be loaded seamlessly from Hugging Face, but there is no option to load GGUF models that are stored locally. Adding this feature would greatly enhance flexibility and usability for various development environments, especially in offline or secure environments where accessing Hugging Face repositories may be restricted.
Additionally, if there is an existing method or workaround to load GGUF models locally that I may have overlooked, please provide any relevant documentation or instructions. This would be invaluable for developers looking to leverage GGUF models efficiently without needing an internet connection or external repository access.
Thank you for considering this request and for any guidance you can provide on this matter.
- Below is Intel's material related to Accelerating GGUF Models with Transformers.
Link Copied
