How do people test whether something is statistically significant?

What are the odds that something happens by mere chance? This is called the null hypothesis and it relies on selecting the correct probability distribution in the first place. Then the probability of the result assuming that distribution, the p-value, is calculated and compared to a value. If it is less than that value the null hypothesis is rejected and the result is deemed to be too unlikely to have occurred due to mere chance.

The standard value to compare to is 5%. i.e. a one in twenty chance. This is quite high and is one possible reason why you see flip-flopping in reports that something is bad for you then, a while later, that it is actually good. More perniciously some researchers massage their testing to achieve a lower p-value in what is known as p hacking.

Having been embarrassed in the past, particle Physicists typically use a significance value equivalent to five standard deviations of the normal distribution. i.e. one in 3.5 million.