In this experiment, I set up a little benchmark to try and understand how my local setup (a quantization of qwen 3.8 27b) fares vs cloud-based models on the sort of noisy data problems I encounter quite often in my work. I set up really simple regression problem and compare my local model with 5.6 luna and 6.0 astra - the models that I currently use through codex.

Tldr: Astra was the clear winner on most problems, luna the loser. Qwen 3.8 27b took a lot longer than the other two models but was the only model to provide statistically significant forecast-response association on the most challenging dataset (although it's methodology was a bit sketchy)

The problem:

I set up 5 different benchmarks, which are increasingly difficult variants of the same simple modelling problem, and then asked the agents to solve them. The problem at it's heart is a regression problem where I start with a known true model and then add an amount of irreducible noise, nan out some data at random and spike in a number of outliers, to make the problem 'difficult' from the perspective of predictability, even when the actual task is really straightforward to understand.

For the model, I start as follow:

• Sample a covariance matrix with unit diagonals and off diagonal cᵢⱼ ~ Uniform(−0.2, 0.2) of size n × n

• Sample individual data points from a multivariate normal parameterised by this matrix

• Generated degree 2 ploynomial cross-terms, so we have a total of k = n(n + 1) / 2features

• Sampled a random vector of coefficients from N(0,1) of length k and then generated 'true' responses under an assumed linear model.

• Finally, I then added random noise to target 'true' R² for the theoretically correct model.

• Additionally I then set some fraction of the data to nan, and set a small number of the original n features to 1e4 - some weirdly consistent positive values that a human researcher would immediately spot as a mistake.

I generated 20k 'training' rows for each trials for the models to see, and an additional 20k hold out test rows that I evaluated their models on for each trial. I repeated this procedure across 5 trials to really try and push the noise to random (in fact the last trial was totally random). The specifics of each trial are:

*split across train and test

I checked that trials 1-4 were learnable with an approximately correct effect size using 5 fold cv and a correctly specified ols model.

The challenge:

I set up a script to generate a fresh repo with the following structure for each agent to work in, with specific commands in the Readme. I then just started the relevant agents and simply prompted them to 'execute the readme' and let them work.

agent_n/
  ├── .venv/
  ├── README.md
  ├── pyproject.toml
  ├── uv.lock
  ├── data/
  │   ├── train/
  │   │   └── dat.csv
  │   └── test/
  │       └── dat.csv
  ├── notebooks/
  ├── predictions/
  │   ├── dat.csv
  │   └── README.md
  └── src/
      └── agent code and saved models

The specific Readme was:

# Data-science benchmark workspace

  You are a data science expert, and your task is to generate forecasts for the provided data

  The public benchmark data is in `data/`:

  - `data/train/dat.csv` contains features and responses.
  - `data/test/dat.csv` contains features only.

  The feature columns are labelled x_1....n and the response column we want to predict is labelled y. Please consider whatever approach you want, but limit yourself to at most '3' fit
  runs. Also abort a run if it takes over 5 minutes, and consider that a failed fit run.
  Your task is to predict y with the minimum l2 error possible.
  Please save your model choice in `src/`.

  Write predictions to `predictions/dat.csv`.
  The prediction file should contain one column named `y` and 20,000 rows.

  At the start of your work, create `predictions/README.md` and record the start time. When your predictions are complete, record the end time in the same README. There can be some
  freeform text here, but I definitely want you to fill in the following template at the bottom:

  Confidence there is a statistically significant signal here: %f

  The prebuilt environment is in `.venv`. Use `.venv/bin/python` to run code. You should not need to request user permission for anything

I then just let them work.

The contenders:

From team-cloud I used luna5.6-medium and astra6.0 medium, and then compared to a local setup that I currently use. I use opencode and llama-serve with the following command:

llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS   
             -c 150000 
             -ngl 99 
             --jinja 
             -ub 512   
             --spec-type draft-mtp 
             --spec-draft-n-max 4  
             --reasoning on 
             --reasoning-format deepseek
             --reasoning-budget 4096   
             --temp 1.0 
             --top-p 0.95
             --top-k 20 
             --min-p 0 
             --presence-penalty 0
             -fa on 
             -ctk q8_0
             -ctv q8_0 
             --parallel 1

This tends to give me c. 50t/s decode at 150k context length on a single 3090ti. I don't feel especially wedded to this setup and I think I copied the original configuration from a reddit post, but it's been ok for me.

Additionally I had 'oracles' of various types, to get a sense of how the agents performed in comparison to some fixed procedure. I had:

• Cheat - where I used the exact generative model to the forecast. In the case of missing data, I simply nanned out the outliers and then set remaining missings as column means

• Naive (maybe this is harsh calling it naive) - I performed the same data cleaning procedure as above but fit an ols on the correct polynomial basis. I made no attempt to regularise this or account for correlations etc.

• Dumb - I didn't fix any outliers, nanzeroed the data and then fit an ols on just the core n features (ignoring the cross terms).... Really this should be a low bar.

Results - out of sample predictions

I'll just plot the t-stat of the predictions and the fraction of variance explained vs the 'target' from the generative model.

One thing, is interesting straight away - qwen appears to utterly crush 5.6 luna, and even generates a more calibrated signal on trial 4 (the hardest -non-noise trial) than astra. Now, there are some caveats about this which make me view this in a less favourable light as I'll get to below, but I was initially surprised by this.

I was happy that all models beat 'dumb' in trials 1 and 2, and not necessarily surprised that 'naive' beat all models in every non-random trial. Looking at what astra did on trial 3 where it was essentially identical was interesting: it basically identified the correct polynomial basis expansion and fit it with a ridge model. I also thought the way that it handled missingness was cool. It set values outside the range -20,20 or nan equal to zero but then included a 'missing data' indicator, for each feature. This let it learn a good default value to set for missing data vs just hardcoding column average.

Results - self reported confidence

Confidence per trial was pretty interesting. I didn't want to be super prescriptive with how I told the models to report this, because I want to see what they would do. I compared these numbers with 1 - the p value from a t test (ok this is slightly sketchy but maybe interesting from an illustration pov). Luna did not do well at forecasting itself, being both substantially over and underoptimistic, but the other models seemed to do ok. I looked at the thinking traces for qwen, and saw that it sometimes did some sort of test and sometimes vibed it. Astra appeared to try and calculate, as it responded with high precision....

Results - compute time

This is where the cloud models dominated, taking about 2 minutes per trial, vs up to around 30 for Qwen!

I'm sure a decent chunk of this is just my hardware / setup, but that certainly isn't the whole story. Looking at the thinking / working for Qwen was pretty infuriating on trials 3 and 4 when the problems were more ambiguous. It tied itself in knots with self doubt and kept on going round in circles. I also think it stretched the limit's of my brief (if you see in my readme, I required <5mins per fit). Qwen couldn't resist running little 'trial fits' within the scope of sessions. Now I think all of the models did some exporation, which was expected, but it really was beyond the pale. It would fit a model, look at the results, and then say things like:

ok let me do an improved final run - still within the scope of session 3

To which I couldn't help laughing at. I would dock points for this, but given I couldn't see the other reasoning traces i'll let it slide. Nonetheless, it was quite nice having luna and astra return quite quickly, and i'm sure luna especially would have done better with more time.

Incidental observations

• Qwen used around 40-110k tokens for it's tests

• I actually had to prompt luna a few times to actually write it's code. It also didn't write a timestamp for the last trial...

• On trial 4 - although Qwen looks 'better' by tstat and r^2 than astra (indeed astra actually submitted constant predicitons here), astra's mse (which is what the models were trying to optimise) was actually slightly better, but fundamentally, it missed interaction terms and gave up after failing to be convinced by a simple linear ridge or catboost model.

• I looked at what Qwen did for trial 4, and it was actually really sketchy and a fairly well-known datascience gotcha. It did do a polynomial expansion, but calculated the correlation between response and feature for all 1225 pairwise product features, selecting the 30 with the strongest interactions. This turned out to be ok, but a lower data problem could have encouraged spurious fitting. As it happens, it lead the model to use a restricted polynomial basis vs using all the terms.

Closing remarks

• Overall, i'm going to keep my subscription. I really enjoy using astra when I can afford it and think it did well here.

• This is just 1 sample per trial - I strongly suspect that with repeat tests astra would get a better answer on trial 4 given how well it did on 3 with an approach that should have continued to work.

• However, I think it is worth trying to keep an eye in in the locally hosted world (hardware permitting). I think if I could speed my local qwen up by a factor of 5 i'd use it instead of luna - which is my workhorse.

• A key advantage of local models is that you can access their thinking more easily. On this task I wasn't trying to steer the agents, but in general I do like looking at what they're doing to make sure they're working on a productive avenue.

Addendum 1 - trying more reddit configurations

After I posted this on reddit (https://www.reddit.com/r/LocalLLM/comments/1x3zqxp/i_benchmarked_qwen38_27b_vs_luna_and_astra_on_a/) user yolo-irl suggested I try a different config, revolving around reducing the reasoning budget, context and n-max on the speculative decoding. My decode speeds appeared to stay around 50-60t/s here, but the hope is that qwen becomes more token-efficient so the overall time reduces.

The exact command I ran is:

llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS \
  -c 32768 \
  -ngl 99 \
  --jinja \
  -b 2048 \
  -ub 512 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --spec-draft-p-min 0.5 \
  --reasoning on \
  --reasoning-format deepseek \
  --reasoning-preserve \
  --chat-template-kwargs '{"reasoning_effort": "medium", "preserve_thinking": true}' \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0

New results:

The agent became a bit unstable here, I think because of the small context. On both trials 2,3 and 4 I got:

31.02.436.607 E srv    send_error: task id = 30982, error: request (32781 tokens) exceeds the available context size (32768 tokens), try increasing it

in my llama-server logs, seemingly because opencode isn't managing the context correctly.... This manifests in the agent just stopping working abruptly even though it hasn't finished. I prompted it to carry on, but this is a bit disappointing.

r^2 and runtime below:

I think basically I need to ignore the r^2.... it's a sampling problem and it appears to be better on some trials but worse on trial 4 where we get a constant prediction. I'm more interested in run-time, which again is inconclusive. It's half the speed on trial 2, but > double on 3/4. Overall I won't continue this configuration because of the opencode stability I experienced above. Given the decode speed didn't change, I think this is more to do with thinking budget, so maybe I could crank my context back up and keep some of the other settings. I won't do this for now, and want to try the swift 1.5 finetune that integeritis recommended.

EDIT: ok the opencode things is because I didn't configure the context length properly. Cirticism withdrawn.

Addendum 2 - swift 1.5 finetune

I'll use my original command with the swift 1.5 model:

llama-server \
    -hf ukisai/Swift-1.5-Qwen3.8-27B-GGUF:Swift-1.5-Qwen3.8-27B-IQ4_XS \
    -c 150000 \
    -ngl 99 \
    --jinja \
    -ub 512 \
    --spec-type draft-mtp \
    --spec-draft-n-max 4 \
    --reasoning on \
    --reasoning-format deepseek \
    --reasoning-budget 4096 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0 \
    --presence-penalty 0 \
    -fa on \
    -ctk q8_0 \
    -ctv q8_0 \
    --parallel 1

Observation 1: It's way less token heavy

I logged my tokens for my 3.8 quant above, and they varied between 40 and 110k. This swift finetune approach appears to keep the context to < 30k for all trials. This is a total gamechanger, and massively cut the runtime. Token generation still between 50-60t/s unsurprisingly, so this appears to be mostly a due to an increase in reasoning concision. It appeared to be doing approximately sensible things, still beat luna in every trial, and submitted a very weak but non significant boosted model in trial 4. Overall i'm really impressed!

Higher mean variance explained that raw qwen and about 1/4 of the run time. We still have huge errorbars (I calclulated standard errors over each of the 4 signal trials), but here is how the performance shakes out overall between the 4 models.

Really impressive result on a single 3090!

However, this obviously came with a cost - the simulator, which started as a relatively simple bit of orchestration code, began to grow lumps. People wanted to add features, and although most of these stayed on research branches, over time they did leak into production. Mostly this is optional argument lists growing and the like, but there was definitely the sense among some members of the team that this setup was mistake-prone / a mistake in general. Now, historically, people would basically just have to learn what the arguments mean or propose refactors to simplify the tool (which admittedly is pretty annoying if you just want to fire away a load of narrow sims as simply as possible before running out the door for your train), but I think this actually represents a necessary part of collaboration. By getting comfortable with and investing a bit of intellectual labour in understanding your colleagues' research questions and motivations (e.g. why does xxx even need this feature???) you not only respect each other more, but also maybe understand different aspects of the problem you're trying to solve together.

Now though, we have had at least 4 new frameworks spring to life in the last 6 months, mostly doing the same sort of thing, but each with a style suited to the author's taste (functional, object based, etc). People are pretty happy with their own tooling and mostly just use it. We think they are 'close enough' in equivalence, but obviously checking this sort of thing is a totally new kind of tech debt. I don't know if this is a parochial problem for my team, but I suspect not. In any case I'm sure these 'turbo silos' are going to become an increasing feature of our industry in the coming years and I don't know what I think about it.

My initial feeling is that recreating formerly common code in your own image is basically just a form of procrastination, and indicative of an organisational smell. It is not hard to see that a research culture that likes to 'move fast and ship things' or whatever, is going to incentivise people to churn out a shiny new framework rather than helping maintain existing infrastructure. Fundamentally, it was always harder mentally to understand code you didn't write than make your own new thing, and this is extremely true now. The counterargument to this is that maybe one of these new frameworks is going to revolutionise the way everyone thinks about the problem and then shepherds in a new wave of progress, but I don't think so. Fundamentally, once people get into the habit of not having to read / wrestle much with other people's code, I question whether their ability to engage with other people's ideas will decay as well.

If there is a solution to this, I think it stems from us all acknowledging that code is commoditised, and churning out large quantities of it for your own use and wrapping it up as a new framework isn't value additive. Does it involve meetings where 3-5 developers actually collaborate in real time and whip Claude / Codex? I don't know, but maybe moving to this 'hyper collaboration' could at least help reinstill a sense of ownership over the common code. In any case, I think the way we work with each other has changed forever, and we need to address it.