In general, intraclass correlation coefficients (ICC's) are designed to assess consistency or conformity between two or more quantitative measurements. They are claimed to handle a wide range of problems, including questions of reliability, reproducibility and validity. It is shown that care must be taken in choosing a suitable ICC with respect to the underlying sampling theory. For this purpose a decision tree is developed. It may be used to choose a coefficient which is appropriate for a specific study setting. We demonstrate that different ICC's may result in quite different values for the same data set, even under the same sampling theory. Other general limitations of ICC's are also addressed. Potential alternatives are presented and discussed, and some recommendations are given for the use of an appropriate method.