Formatting problem in Kimi K2.5 Reasoning
The model concatenates letters and numbers. I told it about that problem and in response the model was thinking correctly but still concatenating. It didn’t seem to really influence the response, but given how often Kimi models are offline or have some other issues, there might be some problem on the moonshot’s end.
As far as I am aware, this is non-critical (as in actual reasoning is not poisoned). But it’s something to be aware and maybe tested? No pressure, I already acted like a jackass once or twice, I won’t do it again. Take care. Thank You.
Log in to comment and vote
Comments15
Mark Florkowski
It looks like there is a certain provider that is malforming the responses. I have reported the issue to OpenRouter, who we use for model routing. They’re usually pretty quick to get the providers to fix these issues or block them if they don’t
Sapphire Crab
Thank You very much. In the last two months there were a lot of small problems (I only use Kimi K2.5 - so I don’t know if it’s just issue with that model - in hindsight I assumed that’s the case) with Kimi K2.5. I assumed Moonshot had some issues, but of course it’s an open weight model - there are others who host it. So my title choice was bad. Sorry about that.
Mark Florkowski
Hey no worries, not on you to know all of the ins and outs of these things! Getting consistency across providers is challenging, but not impossible, I hope that the issues you have encountered have not beeen too egregious, and if they are, don’t hesitate to report them!
Sapphire Crab
That’s my prompt
Still persists:
https://t3.chat/chat/0fa82109-3d6b-4bca-a6be-4e5db11168c8
And it’s honestly annoying for now. The thing is that it does impact reasoning in small ways. When they make plan in numbers, but they re concatenated. I’ve seen the Reasoning that had 3 points number 2. I had no idea to which number 2 the reply was relevant even.
Mark Florkowski
Ok, the screenshot is definitely our markdown renderer hitting a false positive thinking that you are trying to render math - will see how we can avoid that.
> I’ve seen the Reasoning that had 3 points number 2
This, unfortunately, is not something I think we can do anything about, as that would just be bad output from the model, rather than a parsing error. If you have a chat where this happened that I can take a look at, I am happy to. If there is anything that can be done, I will!
Sapphire Crab
And normal responses are affected as well (previously it was reasoning only)
Sapphire Crab
Also when I mean that it makes many sub point that it then sometimes uses and sometimes don’t:
You can see 3 points here, even though only 1 is well enumerated. This is okayish. But if I told it to make a list and go point by point - coinflip whether it would work. Especially for longer ones. Somewhere I had loop of 8 points in 1 and 6,7,8 were ignored. I don’t have a link, but this is the link I give you examples from.
https://t3.chat/chat/3a797c74-55a2-453b-bc7e-8c7b3302d041
Mark Florkowski
This issue with missing spaces around numbers appears to be a model issue, we are looking into it with OpenRouter. The same issue occurs across multiple providers, as well as OpenRouter’s test chat as well.
Mark Florkowski
the
$ … $syntax is invalid, I am not sure why it keeps trying to do that. Is this an older thread?Sapphire Crab
Today’s thread… technically yesterday’s if we count time zones.
Sapphire Crab
Both Described as Yesterday. First one (Arithmetic, is the one that started this thread. The other one is new)
Sapphire Crab
Completely new thread. Same question
I was hopeful, because in other reasoning tasks it seemed like there was upwards trend towards sanity? But nah.
https://t3.chat/chat/190504bc-a761-4ab3-8d17-2ea201eb13eb
Sapphire Crab
FWIW in many other places there are way less dollar signs. It’s almost like you system prompt them out of existence. It even tries to not use them in places where it previously did. GJ
Mark Florkowski
We do guide the models to not use improper formatting, but it does seem that there is a portion of the training data used for these models that included this formatting, so it can be difficult to get it completely out.
Sapphire Crab
Thx. Tell Theo that I will sick my 64 subscribers on him, if he doesn’t give you a raise ;) /s /jk
On more serious note - is there any kind of QA I could do from my home? I’m a caretaker of 2 disabled people. I spend 18 hours a day in front of PC. Often 80 hours straight without sleep. I’m ML hobbyist. I would gladly look for some kind of issues, but I don’t know what kind of issues I as end user might be seeing. And this is not a pitch to be paid. I’m bored. Very bored. That way at least I could do something useful. And if for some ungodly reason you decide that my work is useful, I’m not closing the door. But for 35 years I’ve been proof reading text, being assistant editor. Even in English I know how to proof read EVERYTHING. It’s 2:30 am Polish time. Take care.