When you notice something interesting in data it is tempting to do a statistical test of this. However, there is a clear selection bias, by definition one only notices patterns that are in the data and it is therefore not so surprising that these patterns emerge as (apparently) statistically significant. One viewpoint is that one should *never* HARK, instead use the observation as a basis to frame a hypothesis for a fresh study or experiment. However, soemtimes this is not possible or practical, for example when analysing a pre-existing or very hard to replicate dataset. In such cases one has to be very careful to assess how nay other patterns would have been be equally or more surprising and factor this into your results..
Also known as HARK, HARKing
Links:
alandix.com: Doing it (making sense of statistics) – 8 – or worse- ACM Digital Library: HARK No More: On the Preregistration of CHI Experiments