this is a super interesting pairs; thanks for linking it. I've skimmed through it, so apologies if I've misunderstood something, but although I find the call to better define what we mean by reasoning in the debate convincing, I don't find all of the arguments here compelling.
Briefly:
The paper privileges a pretty broad attainment-based definition of "reading" in the example it gives of separating the three dimensions, and does the same for "reasoning" later. I don't know very many people who would consider "can recite a page from memory" as "attainment on a reading task"; I think most would in fact call that performance of a very different task. If you are measuring "reading attainment" in such a way that memorization is a means to success, then your definition of reading attainment is much broader than typical, because of a focus on attainment alone to define the task. A shorter way of saying that is that means-differences and task differences blur together depending on how much emphasis one's task definition places on means. Those two axes aren't really separable. You give a nod to this when mentioning that psychologists do exclude certain means as "cheating" but IMO don't dig deep enough.
In the ML context we can view overfitting as both an attainment problem and a means-based problem. Using a statistical prediction system to accomplish the task in the first place permits overfitting and when we observe overfitting, it shows us that because of the means used, attainment will be limited (of course, in some cases it's possible to overcome overfitting). When overfitting is extreme and generalization is basically nonexistent, we should suspect instead a *task* issue, like when performance on cancer prediction turns out to have learned how to tell whether an X-ray was taken standing or sitting down and is using that as the basis of its prediction. That's not just a means problem, but a task problem (and your paper discusses exactly this kind of issue).
What the paper I described shows is not mere overfitting though: it's overfitting on a broader test domain even though overfitting was not observed in the narrower original test set. This implies that either intentionally or unintentionally, the original test set was not a good measure of the task because the model was performing well (high attainment on a too-broad assessment) merely due to having access to the test set ahead of time (assessment wasn't task-specific). The underlying problem has been well described in papers like this one about data cards; we don't know enough about the datasets used to train foundation models, so they could perform well on a wide variety of benchmarks simply through memorization: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C22&q=%22model+cards%22+training+set+%22data+cards%22&btnG=#d=gs_qabs&t=1775692189785&u=%23p%3DopSzYwlmudkJ
The paper I cited demonstrates exactly that, and a lot of the criticisms that your paper dismisses are about both task and means, not just means. I'm not sure I have a complete formal definition of "reasoning" right now, but I can clarify that when I say "LLMs don't reason" in part what I mean is that much of their observed attainment on various benchmarks that claim to measure reasoning is actually due to the correct answers being predicted because they (or very close analogues) appeared in the training data, and that the resulting system will therefore be fragile and perform much worse when given inputs not in (or not as frequent in) the training data. This is a critique about both means and task, because the model's approach does not involve inference in the reasoning domain at all (means difference) and performs well on the (badly-designed) benchmark only because it has a some level memorized the answers.
This critique is supported by the paper I cited because the obvious prediction from my critique is that unlike a system performing some kind of inference, a system that has merely memorized answers will degrade rapidly when small details of the questions are changed, whereas a system that actually used reasoning to solve the problem should not degrade or should only degrade slightly. The cited paper does exactly this experiment and then concludes correctly that there is a task difference between what the LLM is doing and what a reasoning system would have to do.
Another stark example of this is: prompt a chatbot with any simple reasoning question and pick new questions until it gets one right (often it will do so the first time). Now follow up with "that's wrong" (a lie) and watch it attempt to justify the opposite (incorrect) conclusion. If it were reasoning, by any definition In willing to admit, then the correctness of its first answer would be a proposition that it "believed in" and a naked assertion that it's incorrect would not alone be enough to immediately cause it to abandon that assertion. It might argue or otherwise reassert its original logic. However, it doesn't, because it's merely predicting likely answers and has no concept of reaching a conclusion or belief in propositions at all. It's internal states can affect future predictions, but this doesn't rise to the level of epistemic commitment, which I view as one precondition for reasoning (both a means aspect and an attainment aspect). Of course it's possible for a reasoning agent to abandon epistemic commitments out of extreme deference and exhibit similar behavior, and boosters might argue this is what the LLM is doing. But knowledge of the mechanisms at play shows that this isn't true. We can peer into the black box and see that there's nothing even approximating epistemic commitment inside.
I'm actually happy to admit that LLMs exhibit something than can be classified as a (narrow, idiosyncratic, dangerous) form of "intelligence" because I've got a broader-than usual idea of that term. But to claim that they "reason" only works under some truly unusual definitions of that term that pretty much only people with vested economic interests in the continued inflation of the AI bubble claim that subscribe to.
