I completely agree with you that the internal stuff matters. However so often people say that people might experience red as blue or something like that which is a bad thing to say (and I hear often, and it is bad for multiple reasons, but especially that it shines the spot light away from the interesting bit which you focused on rightfully and introduces some tackled web to uncurl).

That said the physiology of bodies means and how light interacts with them means that two colours couldn't arbitrarily be switched in their internal mechanic. Cause they would be different experiences for the sensors our body got. Which sometimes gets ignored by some people engaging in philosophy who don't care about physical bodies and are too idealist and non-materialist.

You pointed to the correct thing though.