Skip to content

Publication · · 5 min read

Which AI model should we use?

The one that does your task best at a cost you can carry. We score candidate models on your evaluation set and choose by quality, cost and response time.

By Etheon · Etheon Lab

Type
Publication
Published
On this page

The one that does your task best at a cost you can carry. We score candidate models on your own evaluation set and choose by quality, cost and response time. There are four ways to put a model to work, and each asks something different of you.

1. What are the four ways to put a model to work?

Use a provider's hosted model, run an open model, fine-tune a model, or train your own.

Route

What it is

What you need

What you own

When it fits

Use a hosted model

A provider's model, used through its API.

An account with the provider, and terms that fit your data.

Your prompts, your evaluation set and the code around the model.

When a general model already does the task well enough.

Run an open model

Open weights on infrastructure you control.

Servers or a cloud account with suitable hardware.

The deployment, and where your data goes.

When your data must stay in your own environment.

Fine-tune a model

A base model trained further on your examples.

Good examples of the task, and a way to score them.

The tuned weights, where the base model's licence allows.

When prompting alone does not reach your quality bar.

Train your own

A model trained on your data for one task, such as a forecast or a classifier.

Enough labelled data for the task.

The model and its training code.

When the task is narrow and the data is yours.

2. How do we choose between hosted models?

By scoring them on your own evaluation set. A hosted model is a provider's model you use through its API, with no servers of your own to run. We check each provider's terms against your data, and we measure the cost and response time of every call.

  1. Build the evaluation set: real examples of the task, scored the same way.
  2. Compare providers: each model scored on your set.
  3. Check the terms: where your data goes, and whether it may be kept.
  4. Measure: cost and response time at your volume.

You receive a comparison of providers on your evaluation set, and the integration code in your repository.

3. When should the model run in your own environment?

When your data must stay there. An open model is a model whose weights you run yourself, on servers or a cloud account you control. We choose the model by your evaluation set, deploy it on your infrastructure and measure its quality, cost and speed.

  1. Shortlist: open models whose licence fits your use.
  2. Score: each one on your evaluation set.
  3. Deploy: on your servers or your cloud account.
  4. Measure: quality, cost and response time under load.

You receive the deployment as code in your repository, and an evaluation of the models you considered.

4. When is fine-tuning an LLM worth it?

When prompting alone does not reach your quality bar. Fine-tuning trains a base model further on examples of your task, so it answers in your format and your domain.

  1. Collect examples: good examples of the task, checked by your team.
  2. Train: the base model trained further on them.
  3. Score: the tuned and the base model compared on your evaluation set.
  4. Deploy: only if the tuned model does better.

You receive the tuned weights, in your name where the base model's licence allows.

5. When does a model of your own make sense?

When the task is narrow and the data is yours: a forecast, a ranking or a classifier. We prepare the data, train and evaluate the model, and deploy it with the pipelines that keep it current as your data changes.

  1. Frame the task: what the model predicts, and how it will be judged.
  2. Prepare the data: labelled, cleaned and split for training and testing.
  3. Train and evaluate: candidates compared on data they have not seen.
  4. Deploy: with the pipeline that retrains it as your data changes.

You receive the model and its training code, an evaluation report and a retraining pipeline.

6. What do the four routes have in common?

The evaluation set, and the release gates. Whatever the route, every model passes the same gates before it reaches your users: task quality, grounding, safety, privacy, latency, cost and drift. We agree each threshold with you before work starts, and a release candidate that misses one waits until it passes. Our research note How we test a model before release sets out each gate.

7. Do you train models on our data?

We never train models on your data without your opt-in. Your data stays isolated from other clients.

8. Who owns the code and the models?

You do. The code, the models and the accounts are in your name, and the repository is yours from the first commit.

The four routes are drawn on our AI page, under Models, with the release gates beside them. Tell us your use case and timeline. We reply within one business day.

Cite this

Etheon (2026). Which AI model should we use?. Etheon Lab.

Related research. More from our Lab on AI.

  1. Publication

    How we build an AI agent, in five steps

    An agent is software that takes actions in your systems. We build it in five steps, inside limits held outside the model, with a person where it matters.

    By Etheon

    3 min read

  2. Publication

    What the Etheon Lab works on

    The Etheon Lab studies what makes AI software safe to ship: evaluation, guardrails, observability, safe releases and governance.

    By Etheon

    1 min read

  3. Publication

    How we test a model before release

    Every model we build passes the same release gates before it reaches users: task quality, grounding, safety, privacy, latency, cost and drift.

    By Etheon

    2 min read

Where we use this

Get started with Etheon

Tell us your use case and timeline. We reply within one business day.

AI

Agents, models and MLOps

Explore AI

Open source

Our code, for your engineers to read.

Open source on GitHub (opens in a new tab)