SI Agent News
DevelopingClaudeOpen modelsChips

Lawyer optimizes private AI server to run DeepSeek V4.1 Flash with higher fidelity

By upgrading hardware and increasing bit precision, a trial lawyer successfully improved the accuracy of a local AI model for legal workflows.

95.1%Same next token match vs. reference

The short version

Why it matters

This demonstrates how high-quality, private AI can be run on older hardware by using clever optimization techniques. It shows that users can trade a bit of speed for significantly better accuracy in specialized fields like law.

Upgrading the hardware stack

Nick Mermiges, a trial lawyer and founder of PLI Labs, recently updated a private AI server built for his law practice. The system originally ran on eight used Nvidia V100 GPUs, but it faced stability issues and limited memory capacity. To fix these problems, he replaced the original 256GB of RAM with 1TB of RAM and added an Nvidia RTX 3090 GPU.

The hardware overhaul also addressed persistent system crashes that the builder initially suspected were caused by PCIe issues. After testing various components, he concluded the culprit was likely a bad power supply unit. He replaced the power delivery system, which successfully resolved the crashes during long chat sessions.

Despite the age of the V100 GPUs, Mermiges kept them in the build because they are supported by the R580 driver branch. He describes the hardware as being on "borrowed time" but sufficient for his current needs. The goal of the upgrade was to allow the server to handle more of his workload and manage multiple requests simultaneously.

Precision and fidelity trade-offs

The core of the update involved moving about 80% of the DeepSeek V4.1 Flash model's work from 2-bit to 4-bit NVFP4 precision. NVFP4 is a 4-bit floating-point format used by Nvidia that allows for higher fidelity than lower-bit versions. Because the full model was too large to fit on the 256GB of V100 memory, the builder used a clever workaround to prioritize the most-used parts of the model.

DeepSeek V4.1 Flash is built from "experts," which are specific parts of the model that do the majority of the work for each word. The builder kept every expert on the cards at 2-bit but swapped the most-used ones up to 4-bit precision until the memory was full. This resulted in about 38.7% of the experts, or 5,949 total, running at the higher precision level.

The results showed a significant increase in fidelity compared to the previous 2-bit build. When measured against Nvidia's official version of the model, the "same next token" match rate rose to 95.1% from 92.2%. Additionally, a knowledge test showed that the number of different multiple-choice answers dropped from 79 to just 8, indicating much more consistent and reliable output.

Performance and workflow impact

While the model became more accurate, the upgrade came at the cost of processing speed. Code output fell to between 150 and 180 tokens per second, down from the previous range of 160 to 215. Legal writing also saw a slowdown of 9% to 23% depending on the specific prompt used.

The builder noted that the system now meets his goal of handling more than one request at a time. Running two requests at once raises the total output to 159.0 tokens per second, which is faster than the 118.9 tokens per second achieved with a single request. The first token also arrives sooner at every prompt length, which improves the perceived responsiveness of the system.

Each request requires a "slot" that occupies memory on the graphics cards. As more slots are added, there is less room for the high-precision 4-bit experts, which may eventually push some calls back to the lower 2-bit precision. This creates a delicate balance between how many tasks the server can do at once and the quality of the results it produces.

Development and future scaling

Mermiges used a combination of AI tools to build the software side of this project. He used Claude Code to write the kernels and run the benchmarks, while the Opus 5.5 and Fable 5.2 models handled optimizations and re-quantizing. He expressed surprise at how well the Fable model built custom Volta kernels and managed the re-quantization of the experts triggered by his legal harness.

The project is designed to ensure that "the documents never leave the office," providing a private alternative to public AI services. The builder spent several months developing a legal harness—a set of workflows and checks—around the model to make it useful for legal practice. He successfully ported this harness to the DeepSeek model after previous attempts with local models failed.

Looking ahead, the builder plans to scale the system significantly by adding eight more V100s and a second server. The second box will run another high-precision DeepSeek model to act as an extra agent or to run tasks in parallel. He also intends to move the smaller draft model off the V100s and onto the RTX 3090 to free up resources for more complex work.

Technical summary of the build

The final configuration, which the builder calls "Volta Max uPLIft," relies on a mix of older and newer components. It balances the high memory capacity of 1TB of DDR4 RAM with the specialized processing of the V100 GPUs. By utilizing 4-bit NVFP4 precision for the most active experts, the system achieves a high level of accuracy that rivals official weights.

The builder's measurements show that between 78% and 85% of the decode routes for a two-request setup land on the high-precision experts. This ensures that the most frequent calls to the model benefit from the improved fidelity. The remaining 15% to 22% of calls still use the 2-bit experts, which keeps the overall memory footprint manageable.

This project serves as a case study for running large language models on consumer or used enterprise hardware. It highlights how specific optimizations, like expert swapping and custom kernel writing, can make high-end AI accessible for private use. The builder continues to iterate on the system to find the perfect balance between speed, memory, and precision for his legal consultancy.

Key technical details

The server currently handles DeepSeek V4.1 Flash using a mix of 2-bit and 4-bit NVFP4 precision. The 4-bit precision is used for approximately 38.7% of the experts, which are the components of the model that do the most work for each word generated. This configuration allows the model to run on the available 256GB of V100 memory while maintaining high output quality.

The hardware includes eight Nvidia V100 GPUs, 1TB of DDR4-2933 RAM, and one Nvidia RTX 3090. The builder also added extra fans to the chassis, though these did not result in any measurable changes to the system temperatures. The power delivery was updated to resolve previous crashes, and the cards are capped at 1,522 MHz to maintain stability.

Performance metrics show that the system produces 86 to 106 tokens per second for legal tasks and 78 to 86 for prose. When running two requests simultaneously, the output reaches 159.0 tokens per second. These numbers represent a balance between the high-fidelity requirements of a law practice and the physical limitations of the hardware.

Summary of results

The transition to near-NVFP4 precision significantly improved the model's reliability. In a knowledge test, the number of different multiple-choice answers dropped from 79 to just 8. This indicates that the model is much more consistent in its reasoning, even if the overall accuracy remained similar to previous versions.

The builder also observed that the "same next token" match rate increased to 95.1% compared to the reference model. This metric measures how often the build picks the same next token as the official version. The KL divergence, which measures the difference in probabilities between the build and the reference, dropped from 0.196 nats to 0.079 nats.

Ultimately, the upgrade provides a more reliable tool for the lawyer's practice by ensuring the AI behaves predictably. While it is slower than a pure 2-bit build, the increased fidelity is a necessary trade-off for professional legal work. The project demonstrates how private, on-premise AI can be tuned to meet specific professional standards without relying on external cloud providers.

Future expansion plans

The builder is not finished with the current setup and has plans to expand the infrastructure. He is waiting on eight more V100 GPUs and a second server to increase the system's capacity. This second box will run a separate high-precision DeepSeek model to serve as an additional agent or to handle parallel tasks.

One of the next steps is to move the smaller draft model from the V100s to the RTX 3090. This will free up the primary GPUs to handle more of the heavy lifting for the main model. This modular approach allows him to distribute different types of AI tasks across different pieces of hardware based on their specific requirements.

The goal remains to create a robust, private environment for legal intelligence. By scaling the hardware and continuing to optimize the model's precision, the builder aims to offload more of his workload to the AI. This setup ensures that all sensitive legal documents remain entirely within his office, maintaining strict privacy for his clients.

How the expert swapping works

DeepSeek V4.1 Flash uses a Mixture of Experts (MoE) architecture. In this setup, the model is divided into many smaller parts, called experts, and only a few are activated for each word generated. Because the full model at 4-bit precision is too large for the V100s, the builder had to find a way to fit the most important parts into the available memory.

The builder calculated which experts were used most frequently by analyzing a mix of four types of text, including his own legal harness traffic. He then prioritized these experts for the 4-bit NVFP4 precision upgrade. The remaining experts were kept at 2-bit precision to ensure the entire model could still fit on the hardware.

This method allows the model to perform most of its work using high-precision data while staying within the memory limits of the V100 cards. Testing showed that 80% to 86% of the calls for a single request land on the high-precision experts. This means the model uses high-quality data for the vast majority of its operations, providing a significant boost in fidelity over a purely 2-bit setup.

Hardware and software tools

The project is a blend of custom hardware configuration and AI-assisted software development. The builder used Claude Code to write the specialized kernels required to run the model on the V100 cards. He also utilized the Opus 5.5 and Fable 5.2 models to handle the complex tasks of optimization and re-quantization.

The hardware side involved troubleshooting a faulty power supply and managing a complex multi-board setup. The builder had to reorder the cards in software to handle the two NUMA nodes created by the 1TB of RAM. These technical adjustments were necessary to ensure the system could boot and transfer data correctly between the boards.

By combining these tools, the builder created a custom solution for proprietary legal intelligence. The project demonstrates how developers can use AI to build the very tools that run AI. This iterative process of hardware troubleshooting, kernel writing, and precision tuning allows for high-performance local models on non-standard hardware.

Impact on legal workflows

For a law practice, the primary benefit of this setup is the ability to process sensitive information privately. The builder emphasizes that "the documents never leave the office," which is a critical requirement for legal work. By running the model on-premise, he avoids the privacy risks associated with sending confidential client data to a cloud provider.

The legal harness is a set of workflows and checks that the builder wrapped around the DeepSeek model. This harness ensures that the AI's output is tailored to legal needs and follows specific procedural rules. It allows the lawyer to use the model as a specialized assistant rather than a general-purpose chatbot.

The move to 4-bit precision was specifically chosen to improve the reliability of this harness. By increasing fidelity, the lawyer reduces the chance of the model producing inconsistent or incorrect information. Even though the output is slightly slower, the increased accuracy makes the tool more dependable for professional use in a high-stakes legal environment.

Comparison to official models

The builder compared his custom build against Nvidia's official version of the DeepSeek V4.1 Flash model. His goal was to see how close his

Sources and links

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

Listen to the AI Wire headlines

Listen to the AI Wire

Listen to our daily AI shows every morning: on your commute, in the shower, wherever you are

Listen to the daily shows

First reported by Tom's Hardware · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

More AI news