I agree that "feel" is somewhat subjective; but, in this case, most people would agree on the effect (I've just tried it myself and it's fairly clear-cut) if not the magnitude of it. To me, that suggests there is an underlying physical mechanism, related to the material being touched. My further thoughts on this are as follows:
There are two physical requirements for removal of heat from a finger, say, when it touches a solid surface.
The first is that the material in contact with the finger is capable of conducting the heat away. A better conductor will conduct the heat away better, so one up for thermal conductivity.
However, the second requirement is that there be a temperature difference between the material in contact with the finger and the rest of the material. If the local temperature of the solid doesn't rise, the heat will not flow away no matter how good the thermal conductivity. The local material will rise in temperature quicker and hence start to remove heat more quickly if it has a smaller specific heat capacity. One up for specific heat capacity.
Both properties are involved.
I'm tempted to construct a simple mathematical model of the process!