And from what I understand, all they “learn” to do is predict what letter goes next.

https://thegradient.pub/othello/

It can be difficult to tease out exactly how a neural network is modeling its training data, but claiming that it's solely predicting the next letter is reductive to the point of being wrong.

That aside, I also just think people are being silly. If an AI can write working code (or beat chess grand masters every time), then obviously something interesting is going on, and protestations that it's not really thinking and reasoning for realz in a real way, are just kinda obnoxious.