Rendered at 12:04:23 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jrockway 2 days ago [-]
There was an article on HN a month or so about benchmarking Opus 5.5 every day and recording / publishing the results. I recall this being a github repo, but my searches turned this up instead: https://www.bridgebench.ai/nerf-bench/claude-opus-5-5
So it seems like ... it is not getting dumber. I find myself using Fable a lot more, though. Opus 5.5 was never as fun as Fable, but it's pretty good.
I’ll ask the same question as when it came out: if you don’t trust the provider how are you trust them not to look for and cheat on repeat benchmarks like this.
jahala 2 days ago [-]
For me, Opus 5.5 has been consistently much better than Fable 5.1- in all aspects, but especially reasoning and "attention span"
jrockway 2 days ago [-]
Opus 5.5 is definitely good. I use both all day and it's really hard to say which is which from looking at the outputs; I can't read a PR and say "this was Fable" or "this was Opus" like I could pre-5.5.
Fable is better at playing along with my "You are a dog" instruction in CLAUDE.md. It is willing to be fun and playful, whereas Opus just wants to finish the job and go home. Friend vs. coworker, basically. For example, whenever I reward Fable with a dog treat (actually a <cookie emoji>, which I tell it is a chocolate-free chocolate chip cookie) it keeps a log in memories. Opus just says "crunch! tail wagging!" and moves on.
None of this matters but I think it's fun to have some fun :)
jahala 2 days ago [-]
Aha, I haven't looked into such aspects much - I'm "output" driven. But its good to know about that, if I feel like trying it- thanks.
iLemming 10 hours ago [-]
I can't even use Fable anymore, because it constantly corners itself into: "The model declined to generate this response (safety classifier refusal, category: cyber)" and I feel like flipping the table, "da fuk you're talking 'bout bitch? I'm just asking you to fuking explain this piece of code that has nothing, nothing to do with networking, encryption or auth..."
Was that not from the guy having analysed the anthropic models to incredible deephts and exposted them nerving it some while ago?
jrockway 1 days ago [-]
This is the one I wanted to link, thank you!
itopaloglu83 2 days ago [-]
I think something has changed, maybe about two weeks ago, because there’s this fixation from the model where if you ever need to make a correction, it simply becomes disinterested with your instructions.
nikolay 1 days ago [-]
Yes, it's about the same time when I noticed degradation in the quality, too.
flynfish 17 hours ago [-]
I found your post because I searched "Did Opus 5.5 get worse today?". I experienced the same thing with it not following instructions well and just missing things.
iLemming 9 hours ago [-]
That's why it is very important for every developer to support work and research on local and open weight models. How can we ever trust a provider? Any provider. It may say "claude-opus-5-5", but what if it actually sending requests to a thing they internally call "adolph-mephistopheles-6-6-6"?
12 hours ago [-]
cromka 10 hours ago [-]
I have same experience with Fable, it's absolutely ridiculous. It's not one, it's like 10 transgressions an hour, doing really stupid stuff, hallucinating, ignoring commands, gold-fish memory.
They're nerfing them real hard.
74948494i4 2 days ago [-]
New model incoming
godot 6 hours ago [-]
Sometimes I read a thread like this and I feel like I don't know what people are doing or am I the only crazy one in the world. I'm still using Composer 2.5 (a low tier model released 5 months ago and about a Sonnet 4.x tier capability one) and being extremely productive with it and working on multiple projects on a $20/mo Cursor sub. I move to Grok 4.7 in some cases if I find Composer 2.5 not smart enough for something, or for company work when I'm not the one paying for tokens, but it's honestly not even needed most of the time. Composer 2.5 writes all kinds of code (JS, Rust, etc.) just fine and tests and compiles them and everything, and is super fast. What are people doing that constantly need Opus 5.5 and even having to move to Fable/Mythos?
nikolay 2 days ago [-]
Agreed. Yet, it munches tokens like there's not tomorrow. I don't get anything done with Claude in the past 2-3 weeks - just eating up its weekly quota within a day.
ValentineC 2 days ago [-]
Not sure about how dumb Opus 5.5 has become, but I was surprised at how quickly I was burning through my limits this past week too.
nikolay 1 days ago [-]
It is dumb. None of my threads were closed with shipped features. It keeps going into endless review cycles essentially burning all my quota within a couple of days. And I'm not talking about complex a features an OpenAPI-based integration and some enhancement of pytest traces building maps of touched files. Both these are in their second week of token burn!
jpardoe 1 days ago [-]
[dead]
lp92 20 hours ago [-]
Honestly depending on what you're doing, consider Gemini + Claud combo. Use Opus/Fable for design/architecture/complex work and have Gemini Flash 3.8 do the code gen and the rest of the simple stuff.
nikolay 8 hours ago [-]
Yeah, I've tried this as well, but via Antigravity - it was very effective. I wish to have some smart router that monitors the trends of your usage and has skills-based routing to the best model for the job.
By usage trend I mean something considering Codex almost predictable resets, your usage, and maybe even plan ahead using your backlog to know what you will need sooner.
For example, if you want to ship a lot of UI/UX this week, save Claude for it, and use other models to do the coding.
In fact, I completely revamped the way I do testing and even GitHub Actions, because I started to get hit with a lot of GitHub Actions costs. So, what I did was a smart test actions, which verify if the code has test attestations for local runs, so, they just skip burning GitHub Actions redundantly - the code has already been tested successfully locally.
Another huge burn from Actions came from squashed PRs - my attestations didn't work as the commit hash changes, so, I needed to put that into consideration as well.
In general, I'm completely revamping my release cycle, not allowing PRs to age, and main no longer represents Production. Agents keep merging as soon as possible and then another process assesses if there's a good enough release candidate having enough features that pass more extensive tests. There's no point to always test and always fix piece by piece if you're not going to release.
So, the RC candidate also generate a changelog, blog posts with the features, and so on.
jpbryan 2 days ago [-]
I started noticing at the end of last week. My colleagues started noticing on different days this week.
Bender 1 days ago [-]
Is it just me, or has Opus suddenly turned dumb as rocks
What I've seen is that if I let Opus 5.5 run for too many turns (not sure how to quantify that) it starts exhibiting signs of human dementia. It will forget it told me to run something and will then accuse me of misconfiguring something that it literally gave me a script to run that configures something. It also forgets the instructions in my project and has to go back and read them and apologizes for forgetting the instructions. Between this increasing context rot and the excessive shock collars applied to defensive security coding it's rare that I can even use Opus 5.5. For the past few weeks I have had to use Opus 4.8 and even that now has a sub-set of the shock collars applied [cyber] which forces me to start a new session. This has led me to researching the Abliterated 70B/120B (uncensored) models and required hardware.
So dumb as rocks? I am not seeing that, but the context rot is getting worse for me personally. I don't know if it makes a difference, I am on the pro 20x plan. Maybe people on enterprise plans do not see this which I assume is most of the people here using Claude.
neilunlimitess 5 hours ago [-]
[flagged]
ptkachykby 20 hours ago [-]
[flagged]
jaapz 2 days ago [-]
Fable 5.5 incoming?
1_1xdev1 1 days ago [-]
some of my coworkers said it got stupid verbose yesterday or the day before, all of a sudden. i haven't noticed any difference personally.
you, and they, might be getting a/b tested
vldszn 2 days ago [-]
nerfed, fable 5.5 coming soon
greatgib 2 days ago [-]
I also noticed that it started suddenly since end of last week maybe to make sentence that are non sense or very hard to read.
My best guess was also that in one or two days they will drop a new model that will be so much more better than the nerfed 5.5.
15 hours ago [-]
15 hours ago [-]
Theophane_Rame 21 hours ago [-]
Fable 5.5 incoming?
fractorial 1 days ago [-]
First time?
anitroves 1 days ago [-]
I am feeling the same got dumbest answers so I just changed the model
So it seems like ... it is not getting dumber. I find myself using Fable a lot more, though. Opus 5.5 was never as fun as Fable, but it's pretty good.
I’ll ask the same question as when it came out: if you don’t trust the provider how are you trust them not to look for and cheat on repeat benchmarks like this.
Fable is better at playing along with my "You are a dog" instruction in CLAUDE.md. It is willing to be fun and playful, whereas Opus just wants to finish the job and go home. Friend vs. coworker, basically. For example, whenever I reward Fable with a dog treat (actually a <cookie emoji>, which I tell it is a chocolate-free chocolate chip cookie) it keeps a log in memories. Opus just says "crunch! tail wagging!" and moves on.
None of this matters but I think it's fun to have some fun :)
https://github.com/ninjahawk/livenerf
They're nerfing them real hard.
By usage trend I mean something considering Codex almost predictable resets, your usage, and maybe even plan ahead using your backlog to know what you will need sooner.
For example, if you want to ship a lot of UI/UX this week, save Claude for it, and use other models to do the coding.
In fact, I completely revamped the way I do testing and even GitHub Actions, because I started to get hit with a lot of GitHub Actions costs. So, what I did was a smart test actions, which verify if the code has test attestations for local runs, so, they just skip burning GitHub Actions redundantly - the code has already been tested successfully locally.
Another huge burn from Actions came from squashed PRs - my attestations didn't work as the commit hash changes, so, I needed to put that into consideration as well.
In general, I'm completely revamping my release cycle, not allowing PRs to age, and main no longer represents Production. Agents keep merging as soon as possible and then another process assesses if there's a good enough release candidate having enough features that pass more extensive tests. There's no point to always test and always fix piece by piece if you're not going to release.
So, the RC candidate also generate a changelog, blog posts with the features, and so on.
What I've seen is that if I let Opus 5.5 run for too many turns (not sure how to quantify that) it starts exhibiting signs of human dementia. It will forget it told me to run something and will then accuse me of misconfiguring something that it literally gave me a script to run that configures something. It also forgets the instructions in my project and has to go back and read them and apologizes for forgetting the instructions. Between this increasing context rot and the excessive shock collars applied to defensive security coding it's rare that I can even use Opus 5.5. For the past few weeks I have had to use Opus 4.8 and even that now has a sub-set of the shock collars applied [cyber] which forces me to start a new session. This has led me to researching the Abliterated 70B/120B (uncensored) models and required hardware.
So dumb as rocks? I am not seeing that, but the context rot is getting worse for me personally. I don't know if it makes a difference, I am on the pro 20x plan. Maybe people on enterprise plans do not see this which I assume is most of the people here using Claude.
you, and they, might be getting a/b tested
My best guess was also that in one or two days they will drop a new model that will be so much more better than the nerfed 5.5.