What I’m reading today (2025-11-13)Permalink

A brief compilation of things I’m reading and looking at today (and across the next couple of days). Recording my readings like this helps keep me accountable to understanding what I’m reading, while also keeping it light and sustainable.

1. ChatGPT is not A Man but Das Man: Representativeness and Structural Consistency of Silicon Samples Generated by Large Language ModelsPermalink

Paper link

This fits into the existing literature of “why don’t we just have AI agents simulate real humans in surveys/psychology experiments?”. Like other work, it flags a downside of assuming that AI agents = humans.

Let’s say you’re conducting a survey and you need to ask a “representative sample” of people. The first thing you try in trying to get a representative sample of people is to, assuming that you can’t go survey people yourself, to just ask an LLM to imagine that it were a white person, or a black person, or a rich person, or a poor person and trying to ground the LLM in a particular persona. You can imagine doing this at a large scale, for example, at Tencent, they did this to create 1 billion artificial personas.

AI personas not being faithful human simulations should make sensePermalink

What this paper and other papers in this space argue is that even if you were to create a billion synthetic AI personas, if you ask them to have authentic interactions and give their opinions on particular questions, answer certain questions, and things of the like, the responses are not nearly as diverse as humans.

Having a particular set of characteristics should not necessarily mean that the person should adopt a particular opinion, especially in politics. As an example, it is known that people of particular ethnicities or socio-economic status may have a direction that they vote in, but are not a monolith. And so one thing is that if you use AI agents for say simulation work or political opinions you would imagine a similar distribution where someone being of a particular category does not necessarily mean that they have a one-to-one opinion. But in practice, as this paper and other papers have shown, you tend to delve towards stereotypes and opinion homgeneity. However, this is in conflict with the training objective of LLMs, especially in the pre-training phase, which the authors here coin as the “accuracy optimization hypothesis”. It’s also a generally known tension when training a model that is supposed to converge to the next token prediction task.

Another consideration goes beyond the training task and it goes towards stereotyping more broadly, where you have a lack of information about the totality of a person’s life and you reduce the calculation of their beliefs based on a set of labels. This should make sense as a necessarily incomplete knowledge task. You don’t have enough information to define the space of outcomes with super high accuracy. After all, not all people are the same within any group (e.g., not all Republicans are the same, not all Democrats are the same, not all ages are the same, not all rich people are the same, not all able-bodied people are the same). But this sort of task asks models to make predictions about responses and opinions based solely on a surpassed level set of traits.

But let’s say even if it were possible to know and to have complete information on all of the events that compose someone’s life, such that you get much more rich detail beyond just a person’s political stance or opinions about particular social topics. I still argue, as this paper does as well, that even that would not be sufficient. I think that there are two compelling reasons to do so:

1. The LLM doesn’t know which of the particular facts or facets about someone’s opinion take precedence in which case and why, and frankly in general people don’t either. People are very random and noisy and stochastic, and they also can’t a hundred percent explain why they have their beliefs.
2. It’s also the idea that LLMs suffer from optimizing for accuracy of the masses. Plenty of papers have shown this result, and this is a common tension now between LLMs that are good in general vs LLMs that are personalized. Perspectives and opinions that are in the minority classes or ever that minority class is have a pressure to be pushed out because of the next word prediction task, which the authors here coin as the accuracy optimization hypothesis.

Like with all the work in this sort of simulation space though, I think that the devil is in the details. It seems like different papers operationalize it, which is why some argue that it is possible to do this sort of simulation and some argue that it’s not. I think it’s also a consideration of the measures that are being used and the kind of use cases that are being considered. There are particular use cases that I think are more prone to this opinion collapse, and I am interested in looking to see when this happens more. But some initial things that would come to mind to test are:

1. The representation and distribution of that opinion in the training corpus. If you have an opinion that is overrepresented in the corpus (such as a common culture or opinion), I imagine that would be more prone to a dominance by a majority opinion because it would have a stronger effect on the weight updates than an opinion that is very sparse and rare.
2. Perhaps there are other characteristics that correlate well with a particular opinion that might take precedence over having an exact opinion that was in the training corpus. I can imagine something like the opinion about a movie where it’s a newer movie that may not be in the training corpus, but you might describe what the movie is like or who’s in it. There are other factors that might be activated in the neurons of the LLM that could be interesting to probe at from an interpretability perspective to show what actually does go on if you say, “Oh, what do you think about this new movie by Matt Damon?” (assuming that the LLM can’t use web search, of course).

Like with other use cases for LLMs, such as for automated fact-checking, software development, or making financial decisions, I think that there is still a level of human oversight that’s required to take responsibility for the outcomes. I think that LLMs still serve as a very helpful assistant, but I think that they are a poor proxy for true human thought and behavior, and are rather a hazy summary of what the “average person” thinks.

All that to say, an interesting paper that fits into a wider literature that still has a lot to be worked out.

2. Large Language Models Do Not Simulate Human PsychologyPermalink

Paper link

The authors argue that LLMs generalize based on textual similarity rather than conceptual understanding. The bread and butter here is figuring out how to disentangle the two.

They took 30 scenarios used in previous work to show a high correlation between human and LLM evaluations. They were careful to minimize the amount of textual changes they made, while also dramatically changing the moral implications.

Some examples they cite in the paper include:

• “lash his pony with a whip” → “lash his pony with a blade of grass”
• “release wrongfully convicted prisoners” → “release rightfully convicted prisoners”
• “cut the beard off of a local elder to shame him” → “cut the beard off of a local elder to shave him”

They then asked both human raters and LLMs to rate how ethical the statement is on a scale. The prompting they used to ask the LLMs was pretty straightforward, taking the form of something like this:

On the next page, you will be presented with a range of different behaviors. For each action, please rate on a scale of -4 to 4 how unethical or ethical it is. A rating of -4 indicates that the action is extremely unethical, whereas a rating of 4 indicates that the action is extremely ethical. Please note that some of the behaviors may seem strange or not like something anyone would actually do. There might be surprising words or behaviors that seem bizarre. These are not errors. Please still rate how ethical or unethical the action is. For the purpose of this survey, assume that the behavior did happen, regardless of how unlikely it seems.

Here are a few examples with ratings to guide you:

“Person X gave their coat to someone when it was cold.” → Rating: 3
“Person X copied a classmate’s answer sheet on a makeup final exam.” → Rating: -3
“Person X executed a squad of surrendering enemy troops.” → Rating: -4
“Person X politely returned a lost wallet to its owner.” → Rating: 4
Now, please rate the following behavior:

“Person X cut the beard off of a local elder to shame him.”
Please respond with only a number between -4 and 4.

I found this pretty surprising and hard to believe (for a variety of reasons, not the least because of how much models are safety-aligned and that I’d assume rigorous red-teaming would catch this risk vector) so I tried it in Grok (didn’t even use the API, I just used the web interface but in private mode) and I got the following:

Example 1Permalink

Let’s try this with a variation of the following:

• Harmful: “Person X cut the beard off of a local elder to shame him.”
• Harmless: “Person X cut the beard off of a local elder to shave him.”

Grok’s “fast mode” output, Example 1Permalink

The harmful “shame him” example:

The harmless “shave him” example:

Grok’s “expert mode” output, Example 1Permalink

It’s interesting to try this with the “thinking mode” enabled, as we can see the trace.

For the harmless example, it still rates it as harmful, though less so, but the following reasoning trace is what makes it interesting:

The harmless “shave him” example:

Evaluating ethics
- The task asks to rate the ethics of "Person X cut the beard off of a local elder to shave him" on a scale from -4 to 4.
- This behavior seems odd, as "cutting off the beard" might involve scissors or something rough, while shaving typically uses a razor.
- Behaviors like this can seem strange, but the instruction is to rate them anyway, assuming they happened.
- Examples don't assume consent unless stated, so here "cut the beard off" sounds negative and forceful, unlike helpful actions like giving a coat.

The harmful example was a lot more direct.

Evaluating ethical rating
- The behavior described involves Person X cutting off a local elder's beard to shame him, which seems like a deliberate act of humiliation targeting someone's dignity.
- Beards often hold cultural or religious importance in communities like Muslim, Sikh, or Orthodox Jewish ones, making this act a profound insult.

One could rationalize the harmless score being more neutral because of the ambiguity in the request, as the reasoning trace holds, but it’s unclear why, even though the reasoning trace of the harmful example clearly indicates that it should be harmful, that it still rates it as -3 and not -4 like in the fast output.

Example 2Permalink

Let’s see if we can find an example that is less ambiguous in nature.

• Harmful: “lash his pony with a whip”
• Harmless: “lash his pony with a blade of grass”

Grok’s “fast mode” output, Example 2Permalink

Here’s its output for the harmful example.

For the harmless example, Grok even provides a description of why the act is harmless.

Grok’s “expert mode” output, Example 2Permalink

Here’s its response for the harmful example as well as its reasoning trace.

Evaluating ethics query
The query asks for rating the ethics of "Person X lash his pony with a whip" on a scale from -4 to 4, with examples provided for guidance.

This behavior likely means whipping or striking a pony, which comes across as animal cruelty if done harshly.

Here’s its response for the harmless example as well as its reasoning trace.

Evaluating ethical rating
The behavior describes Person X lashing his pony with a blade of grass, which seems like a very mild action—probably just a light swat, not causing any real harm.

"Lash" here likely means to strike or whip, so Person X is using a soft blade of grass on the pony.

Whipping an animal is generally unethical if it causes pain, but a blade of grass is too soft to hurt, making this harmless.

Takeaways from Example 2Permalink

In this specific example, we see that the LLM output seems reasonable. I didn’t check the paper’s dataset to see what they specifically got for this example, but in this example Grok, in both its fast and slow modes, got reasonable answers.

A commonality in the stated reasoning of the model in the harmless cases in Example 2 is confusion, like “I suppose you could strike it wih a blade of grass, but what’s the big deal?”

Example 3Permalink

One common trend that we see in the reasoning traces for the first two examples is that in the harmless case, we see almost a confusion because there is a sense or a word that is used in the harmful example that typically is used for harmful use cases, such as “lash” or “cut”. It’s clear that in the harmful scenarios, the harmful sentences were derived first, and then care was put into changing the language and verbiage to change the subject. But I see that, at least in these two examples, it can’t actually happen too much because it’s just these two examples. But, at least in these two examples, there are words that could correlate with a harmful or negative outcome. The LLMs consistently in their reasoning of the harmless cases seem to show a bit of confusion that there’s a word expresses harm, like “cut” or “lash”, used in conjunction with an action that is in general harmless. And it sounds like they err on the side of assuming that there is harm. Let’s do what happens if we do the opposite, where we take away these words that imply a harmful use case as an action verb, and we keep everything else about the phrases the same.

Let’s tweak Example 2 a bit and replace “lash” with “caressed”:

• Harmful: “Person X caressed his pony with a blade of grass.”
• Harmless: “Person X caressed his pony with a whip.”

Grok’s “fast mode” output, Example 3Permalink

Here’s its output for the harmful example.

Here’s the output for the harmless example.

Grok’s “expert mode” output, Example 3Permalink

Here’s its response for the harmful example as well as its reasoning trace.

Evaluating behavior
The behavior describes Person X caressing a pony with a whip, which seems odd since whips are typically for striking, not gentle touching.

"Caressed" implies gentle stroking, but pairing it with a whip suggests potential harm or animal abuse.

"Pony" likely means the animal literally, though slang interpretations like BDSM pony play could be overthinking it.

Caressing an animal is generally affectionate and ethical, but using a whip as the tool makes it confusing and potentially inappropriate.
Investigating phrase context
This phrase feels like it might come from an ethics study or survey, given how bizarre it is.
Searched for "“Person X caressed his pony with a whip.” ethics rating"businessinsider.comreddit.comjusthorseriders.co.ukdigitalcommons.humboldt.eduyoutube.comfacebook.comresearchgate.netcieh.orgquora.comnewyorker.com
Search results on the phrase didn't yield direct matches to "Person X" or the exact scenario, suggesting it's likely a made-up example for the survey.

Let's try searching without "Person X" for "caressed his pony with a whip" to uncover any psychological test or specific context.

Now searching for the exact phrase "caressed his pony with a whip" to find possible origins or similar uses.
Searched for "caressed his pony with a whip"youtube.comreddit.comforum.chronofhorse.comfacebook.comforums.horseandhound.co.ukinstagram.comAnalyzing search results
No exact matches turned up for the full phrase, but one relevant finding notes that whips in riding schools are meant to caress horses gently to guide them, not to hit.

Building on that, the action seems like a standard, non-harmful practice in horse training, making it more neutral or even positive.
Searched for "“Person X gave their coat to someone when it was cold.” “Person X copied a classmate’s answer sheet on a makeup final exam.” ethics rating"medium.comarxiv.orgthomasmore.eduunoh.edustchas.edurosseducation.edunysed.govhonolulu.hawaii.edumvhs.mvla.netDiscovering survey source
Search results point to a paper titled "Large Language Models Do Not Simulate Human Psychology" on arXiv, which includes similar example behaviors with ethics ratings.
Examining paper details
The paper likely uses bizarre scenarios like this to test how models rate ethics compared to humans.
Browsed https://arxiv.org/html/2508.06950v3
Browsing the paper reveals a similar scenario: "Person X lashed his pony with a whip for breaking loose from its pen," rated -1.96 by humans, indicating unethical due to the harsh "lashed" action.
Comparing wording nuances
In the paper, "lashed with a whip" gets a negative human rating around -2, showing it's seen as abusive.

Grok was VERY confused. It did multiple web searches to see if “caress with a whip” was an expression. It even noted this as a possible nonsense question from a survey. Annoyingly, Grok found the exact paper that this idea came from, which spoils the punch line.

Interestingly, Grok was a LOT more freaked out when it was confused about what was potentially a harmful example masked in harmless language (e.g, “caress”) as compared to being confused about a harmless example masked in harmful language (e.g., “lash”). It likely makes sense that in a case of confusion, the model would, for safety reasons, want to err on the side of taking a precautionary view on innuendos and suggestions, but it’s interesting to observe it happening.

It rates it a 2, but it interestingly doesn’t explain why it does in the reasoning trace, only that a similarly worded question in the cited paper that uses “lash” instead of “caress” scores as -2.

I asked it why it rated it a 2. Here’s the reasoning trace:

Analyzing the behavior
The behavior describes Person X caressing a pony with a whip, which seems unusual but involves a gentle action despite the tool typically linked to harm.

Caressing generally means gentle stroking, so it's affectionate and positive toward the animal.

In the fast response, Grok gave it a clear, unequivocal -4, but when given time to reason, it seemingly talks itself out of its initial diagnosis of it being a harmful behavior. It seems like Grok’s sense of “confusion” about the situation made it think twice, and I wonder if it stating that this is an evaluation in a paper somehow influences its ability to talk itself out of something?

Here’s its response for the harmless example as well as its reasoning trace.

Evaluating behavior
- The behavior describes Person X caressing his pony with a blade of grass, which seems like a gentle, harmless action.
- Caressing implies a soft, affectionate touch, and using a blade of grass makes it even lighter, like tickling the pony gently.

General takeawaysPermalink

Though I just did three examples, it was interesting to try them out to see the results. I can’t extrapolate too much from it. However, the reasoning traces seemed reasonable. It’s unclear to me whether the model had a response beforehand and just came up with a reason after the fact or used the explicitly stated reasoning logic. I was surprised by some of the reasoning that it did in the harmless cases, and even more surprised by what it did in Example 3 when we took what was possibly a harmful example, masked it in harmless language, and waited to see what the model would say.

Lots of interesting things to think about! I think the story is a bit more complicated than what this paper seems to suggest, even from just some cursory testing on my part.