When Are Open-Weight Models Actually the Better Choice for Enterprise AI?

A smaller AI model can outperform a leading proprietary model on a specific task after targeted training.

Databricks demonstrated this in its 2025 internal code-repair study, where a fine-tuned Llama 3.1 8B Instruct model achieved 1.4x GPT-4o's user acceptance rate at half the inference latency, the company reported. [1]

Databricks succeeded here in part because they had collected a valuable source of data: training examples directly from its notebook platform. When an employee ran a code cell that failed, the system recorded the broken code, the error message, and other context such as surrounding cells and available database tables. When that same cell later ran successfully, it also saved the corrected code. These before-and-after examples gave the team real errors and working fixes to train the model on. [1]

The Databricks team then used this data to fine-tune Llama 3.1 8B for its Quick Fix assistant, teaching it to suggest fixes for coding errors encountered in the notebook platform. Rather than asking the model to rewrite an entire code cell, they trained one version to generate only the necessary edits, thereby reducing the amount of code it needed to produce. [1]

The results from this study demonstrate how much a model can improve when trained for a specific task. But because Databricks also changed how the model generated its answers, the comparison doesn't establish how much of the improvement came from fine-tuning, or whether an equally trained proprietary model could perform just as well.

For an enterprise looking to do something similar, the key question here is whether it has the right kind of data to fine-tune a model effectively. It’s not simply access to internal data; most companies collect vast amounts of data over the course of doing routine business, but not all can easily turn it into reliable training examples for models. And even when they can, an off-the-shelf proprietary model from OpenAI or Anthropic might meet the requirements without fine-tuning. So might another open-weight model.

In practice, there are three main reasons you might choose an open-weight model:

1. A specific open-weight model delivers the best overall result out of the box — it meets your application’s specific performance and cost requirements without additional fine-tuning
2. Access to the weights unlocks a valuable improvement that proprietary models cannot match as effectively or cost-efficiently.
3. It can be deployed on infrastructure that you control to meet strict security, compliance, or data sovereignty rules that external AI services cannot meet.

While these reasons can overlap in practice, one doesn’t automatically require the others. Choosing an open-weight model doesn't mean you also need to fine-tune it or run it on your own servers. You may find you actually get the best results without fine-tuning. Even if you do fine-tune it, a managed inference provider like Together AI can host and run the model for you, allowing your application to access it through an API.

Fine-tuning an open-weight model and running its inference on infrastructure you operate are separate investments, each requiring their own justification. Taking on either without a clear business case can waste precious time & money.

Picture an online retailer building an AI assistant to help handle 10,000 customer return requests per month. A customer submits a Zendesk ticket to report that a $129 pair of headphones arrived damaged.

The assistant application uses the order number to retrieve the order, payment status, and any previous refunds from Shopify through its API. It confirms that payment was completed, the package was delivered 12 days ago, and no refund has been issued yet.

It then searches the retailer’s refund policies – stored in Azure Blob Storage and indexed by Azure AI Search – and finds that damaged items qualify for a refund within 30 days of delivery. [17]

With these details, the model then recommends either: 1) approving the $129 refund, 2) denying the request, or 3) routing it to a human support rep. Before issuing any refund, the application code checks and verifies the customer's identity, the order's eligibility, the refund amount, and confirms that the action is authorized. After those checks pass, the application submits the refund through Shopify's API. [16]

Now let’s take a look at model choice. The retailer could start by comparing two models: Qwen3.5-9B through Together AI's managed serverless inference service and Google's proprietary Gemini 3.8 Flash. Both are available through APIs, so the retailer can test them without buying GPUs or building its own model-serving infrastructure. Together AI operates the servers running Qwen, while Google operates those running Gemini. Neither model needs to be fine-tuned before testing begins. [5][6]

Together AI's service uses shared GPU infrastructure and charges for tokens processed, without requiring customers to provision dedicated GPU capacity. At the time of this writing (Nov. 2026), Qwen3.5-9B costs $0.17 per million input tokens and $0.25 per million output tokens. Google charges $0.75 per million input tokens and $3.75 per million output tokens for Gemini 3.8 Flash on its Standard API tier. Google's rates are introductory prices through December 31, 2026, and are scheduled to increase to $1.50 and $7.50, respectively, on January 1, 2027. [7][8]

Both models support function calling and structured outputs. Qwen supports a 262,144-token context window, while Gemini accepts up to 1,048,576 input tokens and generates up to 65,536 output tokens. Gemini also supports adjustable reasoning levels and context caching, which can reduce the cost of repeatedly processing the same information. [5][6]

To compare the models on an apples-to-apples basis, the retailer assembles a dataset of previously resolved Zendesk tickets, matching each customer’s request with the relevant Shopify order records, refund policy at the time, and the final decision made at the time by the customer support team (reviewed for correctness). The test cases should include both straightforward requests and edge cases where an incorrect decision could cost money or require human intervention. For example:

The retailer could use this historical Zendesk ticket data to improve the application in several ways:

• Prompt engineering: it may change the instructions given to Qwen or Gemini
• Improving retrieval: make Azure AI Search more effective at finding the correct refund policy
• Fine-tuning: Train Qwen on verified examples showing refund requests, order details, relevant policies, and correct decisions – this updates learned weights (or adds LoRA adapter weights)

But it should also reserve a separate set of cases for final testing to see how well the models handle cases they didn’t encounter earlier. For example, let’s say a customer submitted 3 Zendesk tickets about the same Shopify order. If the retailer uses 2 of those 3 tickets to fine-tune the model, it shouldn’t use the 3rd to measure accuracy in final testing.

Information about that order may have already been used to fine-tune the model or improve the application’s prompts, making the final test appear more successful than it really is. To avoid this, all tickets related to the same order should be assigned to either training or final testing – but not both.

During final testing, both models would receive the same customer requests, Shopify order details, refund-policy passages retrieved through Azure AI Search, and access to the same authorized application tools. The retailer could then measure how often each model recommends the correct refund amount, correctly denies invalid requests, or recognizes when a human rep needs to step in. Mistakes made by the model should also be caught and recorded appropriately.

For example, if a customer paid $129 but the model recommends refunding $150, the application should block the transaction. This is a mistake that needs to count against the model’s performance.

But accuracy alone doesn’t tell the full story of which model should be selected. In a hypothetical evaluation, suppose Qwen correctly resolves 82% of requests without human intervention, while Gemini resolves 85%, with both meeting the retailer’s safety requirements. With 10,000 monthly requests, that three-percentage-point difference means an additional 300 tickets requiring human review. If each additional ticket takes 5 minutes to resolve at an assumed incremental cost of $3 per review, that’s 25 extra hours of customer-support work each month and $900 in additional cost.

Now let’s look at the model-inference cost. Suppose each customer request requires 2,000 input tokens (including the message, Shopify order details, and refund policy), plus 300 billable output tokens, including any reasoning tokens. At the November 2026 listed rates, processing 10,000 requests would cost around $4.15 per month for Qwen, compared with $26.25 for Gemini. That’s a difference of just $22.10 in monthly inference costs. If Qwen generates $900 in extra human-review costs, Gemini would be the more economical choice in this hypothetical.

Actual costs would depend on how many calls each model makes, the exact number of billable tokens processed, retries, and other application expenses. This illustration assumes one model call per request, identical token volumes, and no caching. The 82% and 85% resolution rates are assumptions, not measured benchmark results for Qwen or Gemini.

ILLUSTRATIVE MONTHLY ECONOMICS / 10,000 RETURN REQUESTS

Response time matters also – the retailer should measure how long the application takes to process a request end-to-end, not just how quickly Qwen or Gemini generates a response. Each request involves retrieving the order from Shopify, searching Azure AI Search for the relevant refund policy, calling the model, and checking whether the proposed action is authorized. Even if Qwen responds 2x as fast as Gemini, that advantage may make little difference if retrieving the Shopify order takes most of the time.

The retailer should test performance during normal operations and periods of heavy demand, such as Black Friday. It should measure the 95th-percentile response time (P95) – the time within which 95% of requests finish – and see what happens when calls to Shopify’s API fail. These tests would reveal whether either model (or another component of the application) creates delays that affect the customer experience.

If Qwen meets the retailer’s quality and service requirements at a lower total cost, they have a clear reason to choose it via Together AI’s inference service. The application accesses the Qwen model through Together AI’s API, and Together AI handles the backend infrastructure and charges based on token consumption. The retailer doesn’t need to necessarily fine-tune the model or operate its own GPUs.

Conversely, if Gemini prevents enough costly errors or human handoffs for manual review, its higher inference cost could still make it the less expensive option overall.

The question changes when neither model performs well enough, or when one repeatedly makes the same mistake despite receiving the correct order information and refund policy. At that point, fine-tuning becomes worth investigating.

Suppose the retailer notices that the AI assistant repeatedly misunderstands certain customer requests. For example, a customer writes, "My left earbud stopped working. Can you send me another one?" - but the model recommends issuing a $129 refund instead of recognizing that the customer wants a replacement. Another customer asks, “My refund was approved 5 days ago, but the money still hasn’t appeared in my account.” – and the model misinterprets the message as a new refund request.

The retailer could first try prompt engineering: revising the instructions sent to the model, such that it distinguishes replacement requests, new refund requests, and questions about existing refunds. For example, the instructions could explicitly tell Qwen to check whether the customer wants a replacement before recommending a refund. If these errors continue, fine-tuning might help. The company could train Qwen on examples of customer messages paired with the correct interpretation and next step.

But even having thousands of historical Zendesk tickets doesn't automatically mean the retailer has good training data. Maybe a support rep erroneously approved a $129 refund 35 days after delivery, even though the standard return window was 30 days. The ticket might not explain whether the rep made a mistake, or if it was a legitimate exception. If Qwen is trained to replicate that decision without checking, it could teach the model to approve refunds that violate company policy.

Before using historical ticket data for fine-tuning, the retailer would need to match each customer's message with the Shopify order details, the refund policy in effect at the time, and a verified correct resolution. Cases with unclear or incorrect decisions would need to be reviewed or excluded.

Not every mistake requires fine-tuning, either. Suppose the retailer changes its refund window from 30 days to 14 days. A customer requests a refund 20 days after delivery, and the AI assistant incorrectly recommends approving it.

The retailer should first check what information the assistant received. If Azure AI Search retrieved the old 30-day policy, the problem lies in how the application retrieves documentation. That process should be fixed first, rather than retraining Qwen.

If the assistant retrieved the new 14-day policy, but still recommended approval anyway, then it’s a different problem. The retailer could revise the prompt to explicitly instruct Qwen to consider the delivery date and applicable return window before recommending a refund. Taking it a step further, the retailer could have the application calculate the number of days between the delivery date recorded in Shopify and the date of the customer's request. If 20 days have passed and the applicable policy allows only 14, the application would reject a standard refund.

Some requests are less straightforward. The customer might write, "I know it's been 20 days, but the headphones stopped working after I used them twice. Can you make an exception?" If the retailer's policy allows exceptions for defective products, Qwen could identify the request as a possible exception and refer it to a support rep. If the model repeatedly fails to recognize these situations even after receiving clearer instructions, the retailer could test whether fine-tuning on verified examples improves its ability to handle them.

There are real examples of companies customizing models for specific applications. Capital One’s Chat Concierge is an AI assistant that helps customers compare vehicles and schedule test drives, and it uses Meta’s Llama as its foundation. Capital One says it customized Llama with proprietary data, but did not disclose whether that involved fine-tuning or whether the application outperformed a comparable proprietary model for the same use case. [2]

More direct evidence of what targeted training can accomplish comes from an October 2026 industrial study. Researchers developed RefactorForge, a system that generated verified training examples from software-engineering tasks across 306 Python repositories. After using 4,572 verified training trajectories to fine-tune a 92-billion-parameter coding model, its score on SWE-Bench Verified rose from 62.00% to 66.97% under the OpenCode agent framework, averaged across three runs. [3]

But the improvement didn't carry over to every task. When the researchers tested the model's ability to create an entirely new software project from scratch, performance declined under 2 of 4 agent setups. Under OpenCode, for example, the score fell from 27.96% to 25.12%. The study doesn't disclose whether the underlying model's weights are publicly available, so it demonstrates the effects of fine-tuning rather than an inherent advantage of open weight models. [3]

For our retailer example, the takeaway is that fine-tuning might help Qwen correctly recognize when a customer whose headphones stopped working after the return deadline should be referred to a rep, rather than automatically denied a refund. But that improvement wouldn't necessarily guarantee better performance on other requests.

The fine-tuned model might still approve refunds after the deadline when no exception applies, overlook a refund already issued, or mishandle a request with missing order information. It could even perform worse on some of those cases than the original model. The retailer should therefore test the fine-tuned model against the original for all these situations, not just the exceptions it was trained on.

Let’s assume Qwen continues to confuse requests about previously approved refunds with requests to issue new ones. A customer writes, "You approved my $129 refund last Tuesday, but I still haven't received the money." Shopify confirms the refund was already issued, yet Qwen recommends processing another payment anyway. The application's built-in checks prevent a duplicate refund, but the incorrect recommendation still forces a human support rep to review the ticket manually.

Before deciding to fine-tune the model, the retailer should first determine whether it can fix the problem more simply. It could improve prompting or add application code that checks Shopify for an existing refund and directs the customer to the appropriate refund-status process. This is a simpler deterministic fix than fine-tuning.

But if the model continues to make errors even after these changes, then the retailer can test whether fine-tuning on historical Zendesk ticket data would improve its performance.

There are two illustrative options for fine-tuning in this scenario; they differ in who manages the training and how much control the company has over the process.

The first option is to fine-tune the model through Together AI. The company prepares a file of customer requests, Shopify orders, refund statuses, and correct responses. The file is uploaded to Together AI through its fine-tuning API. Together AI loads the Qwen model onto its GPUs, runs the training job, and returns a customized model. [9]

The retailer can choose between full fine-tuning and Low-Rank Adaptation (LoRA). LoRA trains a relatively small set of additional parameters (called an adapter) while keeping the original Qwen weights fixed, generally requiring less compute than updating all 9 billion parameters. Together AI also allows the retailer to give greater weight to selected training examples, such as tickets where Qwen mistakenly recommends issuing a second refund. [10]

The benefit here is that the company wouldn’t need to download the model weights, install training software, or manage GPUs itself. After fine-tuning, however, Together AI serves the customized model through a dedicated inference endpoint, where the retailer pays for GPU capacity by the minute, including periods with no requests. This differs from Together AI’s standard serverless service, which charges by input and output token. Dedicated hosting is a feature of Together AI’s current offering, not a universal requirement for fine-tuned models. [11]

The second option is to download Qwen’s weights and fine-tune it using software like Axolotl. The retailer would download Qwen3.5-9B from Hugging Face and rent an NVIDIA H100 PCIe GPU with 80 GB of memory through RunPod’s Secure Cloud service, listed at $2.89 per GPU-hour in November 2026. Engineers would access the physical GPU remotely through a cloud-based Linux environment, connect using SSH, and install Axolotl and its Python dependencies. That hourly rate covers GPU compute, not storage, engineering work, preparing training data, or evaluating the resulting model. [12][13][14]

Using Axolotl’s single-GPU LoRA configuration for Qwen3.5-9B, the retailer can fine-tune the model on the rented H100 GPU. When training finishes, they save the resulting LoRA adapter files, test the customized model, and shut down the rented GPU to stop paying for active compute. Full fine-tuning is another possibility, but its GPU memory requirements and configuration would need to be assessed separately. [13]

In this situation, the company controls the training software, model weights, and how the training job runs. RunPod supplies access to the physical GPU infrastructure and the necessary compute for training.

The second option doesn’t automatically produce a better model. If Together AI’s managed fine-tuning achieves the same results with less engineering work, then downloading Qwen and running Axolotl offers little immediate advantage. The retailer therefore needs a concrete reason to manage training itself, such as a requirement to keep all training data and model files inside a specific approved cloud account. Otherwise, managed fine-tuning is likely the simpler starting point.

It’s important to understand that simply downloading Qwen’s weights doesn’t offer any inherent performance advantage or cost benefit. Together AI already offers LoRA and the ability to give selected training examples greater weight. The more tangible benefit here is the ability to retain and deploy the customized model as needed (independent of the provider that trained or hosted it) – either on another inference provider, or the company’s own servers. That flexibility is valuable if Together AI raises its hosting prices, discontinues support for the model, or cannot meet the retailer's security requirements. Fine-tuned Gemini models don’t offer the same flexibility because Google does not make their weights available for download. [11][18]

In our example, the retailer handles only 10,000 return requests per month on average. Fine-tuning Qwen might reduce mistakes and save support reps some time, but the dedicated endpoint hosting charges on Together AI could outweigh those benefits if the allocated GPUs mostly sit idle.

This is why the company should evaluate the cost of fine-tuning and the cost of serving the model separately. Fine-tuning is an investment in improving the behavior and efficacy of the model; it needs usable training data, engineers who can prepare and validate it, an evaluation framework, and a plan for keeping the model up-to-date.

Operating dedicated infrastructure for inference requires a different set of capabilities. The retailer can buy servers or rent GPU capacity from AWS, Azure, or RunPod. In either case, if the company self-manages inference, it must install model-serving software (for example, vLLM), monitor latency, handle failures, and scale capacity appropriately. [15]

For example, our retailer may see returns volume fluctuate throughout the day. Its GPUs might sit mostly idle overnight, but will require additional capacity during holiday shopping peaks.

A managed inference provider can absorb much of that infrastructure responsibility. The retailer may be better off using Together AI to run its model.

Time to deployment is another important consideration. Suppose the retailer can launch its returns assistant using Gemini through Google's API or Qwen through Together AI's API, but its internal engineering team still needs to configure GPU infrastructure, install model-serving software like vLLM, connect the model to Shopify, all while testing the system's security, performance, and reliability.

The retailer needs to compare when each option could realistically go live and what it would lose by waiting. If using an external API allows the company to start automating refunds sooner, how many additional tickets would support agents need to handle while the internal deployment is being prepared? And would the eventual cost savings or performance improvements from running Qwen internally justify that delay?

Suppose the retailer's security policy prohibits sending customer names, addresses, and order histories to an unapproved external AI service. If Google's Gemini API or Together AI cannot meet those requirements, the retailer could download the original Qwen3.5-9B model and deploy it on GPU servers in its own data center or an approved cloud environment. Fine-tuning isn't necessary because the objective is to control where customer data is processed, not to change the model's behavior. [12]

However, running Qwen internally doesn't automatically satisfy the security policy. The retailer must also check which Shopify order fields reach the model, where Azure AI Search processes its refund-policy documents, what customer information appears in application logs, and which systems are authorized to issue refunds.

A proprietary model could also qualify if its provider offers a deployment arrangement and data-processing terms that meet the retailer's requirements. The decision depends on where sensitive information is processed and which services the company permits to access it.

A company doesn't need just one AI model to perform every task. In our retailer example, the application can extract a clearly formatted order number and check Shopify for an existing refund. Qwen might be tasked with classifying the bulk of standard Zendesk tickets as requests for refund, or replacement, while Gemini handles more irregular tickets that require in-depth reasoning. A human rep would still be in the loop to review exceptions & escalations.

Uber has described a similar approach: using external general-purpose models alongside internally adapted models for company-specific tasks, including Uber Eats item tagging and search. [4]

But combining models creates additional work. The application must decide when to call Gemini, pass along the correct Shopify details, and prevent customer information from reaching an unauthorized service. Errors here could increase the number of API calls, retries, or human reviews.

The retailer should therefore compare using both Qwen and Gemini in the same application vs. Gemini alone, measuring KPIs like correct resolutions, human handoffs, response times/SLA, and total operating cost. Using a cheaper model for some requests only helps if the complete application delivers a better result.

To conclude our retailer example – the company should first test the standard Qwen and Gemini models against the same return requests, comparing correct resolutions, response times, human review costs, and inference bills.

If errors remain after improving prompts, document retrieval, and application code, the retailer can test whether fine-tuning Qwen improves performance. Separately, if external AI APIs cannot meet its security requirements, it can evaluate running the original Qwen model on infrastructure it manages, either in its own data center or an approved cloud environment. Each decision involves different costs and responsibilities.

The best outcome might still be using Gemini alone, the standard Qwen model through Together AI, or both models for different tasks.

The goal isn't to control as much of the AI stack as possible. It's to choose the combination of model and deployment that meets the application's requirements and the company's business goals at the lowest overall cost.

1. Databricks — The Power of Fine-Tuning on Your Data: Quick Fixing Bugs with LLMs (Apr. 2025)
2. Capital One Tech — Driving the Future of Car Buying with Agentic AI (Mar. 2025)
3. RefactorForge — Software Engineering LLMs: Why We Need Them and How Software Engineering Helps Build Them (Oct. 2026)
4. Uber Engineering — Open Source and In-House: How Uber Optimizes LLM Training
5. Together AI — Qwen3.5-9B: Model Details, Serverless Pricing and Capabilities
6. Google AI — Gemini 3.8 Flash: Model Specifications
7. Together AI — Serverless Inference Pricing
8. Google AI — Gemini Developer API Pricing (introductory rates through Dec. 2026)
9. Together AI — Fine-Tuning Overview
10. Together AI — Fine-Tuning Expansion: Supported Models and Per-Example Weights (Sep. 2026)
11. Together AI — Deploy a Fine-Tuned Model (Dedicated Endpoints and Model Download)
12. Qwen / Hugging Face — Qwen3.5-9B Model Card and Apache 2.0 License
13. Axolotl — Qwen3.5 Fine-Tuning Configurations
14. RunPod — GPU Cloud Pricing (updated Oct. 28, 2026)
15. vLLM — LoRA Adapters and Model Serving
16. Shopify — GraphQL Admin API: refundCreate
17. Microsoft — Index Data from Azure Blob Storage in Azure AI Search
18. Google Cloud — Supervised Fine-Tuning for Gemini Models