There are many things we have to take into consideration when trying to understand colour perception, such as colour constancy, but there's no point in me reciting my colour perception lecture on here.
This is known as Qualia, but the only way they can test this is by observing whether similar neurons fire at the same time for different people at the perception of a certain colour, but of course, this is more of a philosophical question.
DL