New open-source utility tracks GPU memory for local AI models

Granular GPU monitoring

The gputaskman utility has been released into the public domain to provide detailed monitoring of video cards and the processes using them. While standard tools often show general load, this program helps users see which specific applications are consuming graphics resources. It displays dedicated video memory use alongside system RAM availability.

The tool provides a suite of indicators that include temperature, power consumption, and graphics unit operation. It also tracks frequencies and the activity of individual processes in real time. This level of detail is designed to help users manage hardware more effectively when running demanding software.

The interface operates directly within the terminal and includes the ability to view graphs of these indicators as they change. This makes it a useful choice for users who prefer a command-line environment over a heavy graphical interface. The project is currently available as version 0.3.3 under the MIT license.

Focus on local neural networks

A primary focus of gputaskman is the monitoring of local neural networks. The utility can detect various software environments, including technologies like CUDA, HIP, and Vulkan. It is specifically designed to recognize models launched through popular platforms such as Ollama, LM Studio, and llama.cpp.

For these compatible models, the program evaluates how memory is distributed across different components. It looks at the space taken by neural network weights, the KV cache, and compute buffers. The KV cache is a specific area that stores data from previous tokens to help the model process new requests more efficiently.

By providing this breakdown, the tool helps users understand why a local language model occupies a specific amount of video memory. It also allows users to see how resource consumption changes as the context of a conversation increases. This is particularly helpful for troubleshooting performance issues during long interactions with an AI.

How the data is collected

The developer notes that some of the information provided by gputaskman is approximated rather than directly measured. The program relies on model metadata and program logs to estimate memory distribution. It does not directly read the actual contents of the video memory, which means it cannot identify every individual texture or compute buffer.

Because it uses approximations, the developer describes the project as experimental. Users are encouraged to distinguish these estimates from directly measured hardware indicators. Despite this, the tool provides a significant amount of context that standard monitors typically omit for AI workloads.

The utility is designed to be accessible across multiple operating systems. It supports Windows with NVIDIA and AMD cards, as well as Linux, WSL, and Apple Silicon computers running macOS. However, the specific metrics available to a user may vary depending on their specific drivers and platform.

Installation and compatibility

There are several ways for users to get the tool running on their machines. For Windows users, there are Portable x64 and ARM64 builds that do not require a Python installation. For those on Linux, ready-to-use archives are available for quick deployment.

Users who prefer a standard installation can also get the program via PyPI, provided they have Python 3.11 or later installed. This flexibility ensures that a wide range of hardware configurations can access the monitoring features. It is built to handle systems with multiple graphics adapters as well.

The tool includes features that allow users to select individual video cards in multi-GPU setups. It also allows for saving monitoring results in JSON format. This capability lets users compare the status of their system before and after launching specific applications to see the exact impact on resources.

Current limitations and goals

While the tool is functional, the developer acknowledges that some features still have limitations. Precise model detection is not always perfect, and memory distribution among multiple GPUs is still being refined. These are noted as areas for future development as the project matures.

The goal of gputaskman is to provide a clearer picture of the relationship between software demands and hardware limits. By showing how memory scales with context, it helps users make informed decisions about which models they can run locally. It serves as a bridge between raw hardware stats and the specific needs of AI software.

As the project moves forward, the availability of individual metrics will continue to depend on the underlying drivers. Users interested in the project can find the latest updates in the open-source repository. The tool remains a specialized utility for those who need to see beyond the surface level of GPU usage.