{"id":151,"date":"2017-05-31T13:48:59","date_gmt":"2017-05-31T13:48:59","guid":{"rendered":"https:\/\/alandix.com\/statistics\/?p=151"},"modified":"2017-05-31T13:53:16","modified_gmt":"2017-05-31T13:53:16","slug":"gaining-power-2","status":"publish","type":"post","link":"https:\/\/alandix.com\/statistics\/2017\/05\/31\/gaining-power-2\/","title":{"rendered":"gaining power (2) &#8211; the noise-effect-number triangle"},"content":{"rendered":"<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide06.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-127 alignright\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide06-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide06-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide06-768x576.jpg 768w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide06-1024x768.jpg 1024w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide06-400x300.jpg 400w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide06.jpg 1920w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a>The heart of gaining power in your studies is understanding the noise&#8211;effect&#8211;number triangle. \u00a0Power arises from a combination of the size of the effect you are trying to detect, the size of the study (number of trails\/participants) and the size of the &#8216;noise&#8217; (the random or uncontrolled factors). We can increase\u00a0power by addressing any one of these.<\/p>\n<p><iframe loading=\"lazy\" title=\"The noise-effect-number triangle\" width=\"584\" height=\"329\" src=\"https:\/\/www.youtube.com\/embed\/Zd2MVcX9gfM?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide07.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-128\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide07-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide07-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide07-768x576.jpg 768w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide07-1024x768.jpg 1024w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide07-400x300.jpg 400w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide07.jpg 1920w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Cast your mind back to your first statistics course, or when you first opened a book on statistics.<\/p>\n<p>The standard deviation (sd) is one of the most common ways to measure of the variability of a data point. This is often due to &#8216;noise&#8217;, or the things you can&#8217;t control or measure.<\/p>\n<p>For example, the average adult male height in the UK is about 5 foot 9 inches ( with a standard deviation of about 3 inches (7.5cm), most British men are between 5&#8242; 6&#8243; (165cm) and 6&#8242; (180cm) tall.<\/p>\n<p>However, if you take a random sample and look at the average (arithmetic mean), this varies less as typically your sample has some people higher than average, and some people shorter than average, and they tend to cancel out. The variability of this average is called the standard error of the mean (or just s.e.), and is often drawn as little &#8216;error bars&#8217; on graphs or histograms, to give you some idea of the accuracy of the average measure.<\/p>\n<p>You might also remember that, for many kinds of data the standard error of the mean is given by:<\/p>\n<p style=\"padding-left: 30px;\">s.e. = \u03c3 \/ \u221an\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 (or if \u03c3 is an estimate \u221an-1 )<\/p>\n<p>For example, of you have one hundred people, the variability of the average height is one tenth the variability of a single person.<\/p>\n<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide08.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-129\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide08-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide08-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide08-768x576.jpg 768w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide08-1024x768.jpg 1024w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide08-400x300.jpg 400w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide08.jpg 1920w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>The question you then have to ask yourself is how big an effect do you want to detect? Imagine I am about to visit Denmark. I have pretty good idea that Danish men are taller than British men and would like to check this.\u00a0\u00a0 If the average were a foot (30cm) I definitely want to know as I&#8217;ll end up with a sore neck looking up all the time, but if it is just half an inch (1.25cm) I probably don&#8217;t care.<\/p>\n<p>Let&#8217;s call this least difference that I care about \u03b4 (Greek letters, it&#8217;s a mathematician thing!), so in the example \u03b4 = 0.5 inch.<\/p>\n<p>If I took a sample of 100 British men and 100 Danes, the standard error of the mean would be about 0.3 inch (~1cm) for each, so it would be touch and go if I&#8217;d be able to detect the difference. However, if I took a sample of 900 of each, then the s.e. of each average would be about 0.1 inch, so I&#8217;d probably be easily able to detect differences of 0.5 inch.<\/p>\n<p>In general, we&#8217;d like the minimum difference we want to detect to be substantially bigger than the standard error of the mean in order to be able to detect the difference. That is:<\/p>\n<p style=\"padding-left: 30px;\">\u03b4\u00a0\u00a0 &gt;&gt; \u03c3 \/ \u221an<\/p>\n<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide09.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-130\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide09-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide09-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide09-768x576.jpg 768w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide09-1024x768.jpg 1024w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide09-400x300.jpg 400w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide09.jpg 1920w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Note the three elements here:<\/p>\n<ul>\n<li>the effect size<\/li>\n<li>the amount of noise or uncontrolled variation<\/li>\n<li>the number of participants, groups or trials<\/li>\n<\/ul>\n<p>Although the meanings of these vary between different kinds of data and different statistical methods, the basic triad is similar. This is even in data, such as network power-law, where the standard deviation is not well defined and other measures of spread or variation apply (Remember that this is a different use of the term \u2018power\u2019). In such data it is not the square root of participants that is the key factor, but still the general rule that you need a lot more participants to get greater accuracy in measures \u2026 only for power law data the \u2018more\u2019 is even greater than squaring!<\/p>\n<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide10.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-131\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide10-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide10-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide10-768x576.jpg 768w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide10-1024x768.jpg 1024w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide10-400x300.jpg 400w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide10.jpg 1920w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Once we understand that statistical power is about the relationship between these three factors, it becomes obvious that while increasing the number of subjects is one way to address power, it is not the only way. We can attempt to effect any one of the three, or indeed several while designing our user studies or experiments.<\/p>\n<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide11.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-132\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide11-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide11-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide11-768x576.jpg 768w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide11-1024x768.jpg 1024w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide11-400x300.jpg 400w, https:\/\/alandix.com\/statistics\/files\/2017\/05\/Slide11.jpg 1920w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Thinking of this we have three general strategies:<\/p>\n<ul>\n<li><strong><em>increase number<\/em><\/strong> \u2013 As mentioned several times, this is the standard approach, and the only one that many people think about. However, as we have seen, the square root means that we often need very lareg increase in the number of subjects or trials in order to reduce the variability of our results to acceptable level. Even when you have addressed other parts of the noise&#8211;effect&#8211;number triangle, you still have to ensure you have sufficient subjects, although hopefully less than you would need by a more na\u00efve approach.<\/li>\n<li><strong><em>reduce noise<\/em><\/strong> \u2013 Noise is about variation due to actors that you do not control or know about; so, we can attempt to attack either of these. First we can <em>control conditions<\/em> reducing the variability in our study; this is the approach usually take in physics and other sciences, using very pure substances, with very precise instruments in controlled environments. Alternatively, we can <em>measure other factors<\/em> and fit or model the effect of these, for example, we might ask the participants\u2019 age, prior experience, or other things we think may affect the results of our study.<\/li>\n<li><strong><em>increase effect size<\/em><\/strong> \u2013 Finally, we can attempt to <em>manipulate the sensitivity<\/em> of our study. A notable example of this is the photo from the back of the crowd at President Trump\u2019s inauguration. It was very hard to assess differences in crowd size at different events from the photos taken from the front of the crowd, but photos at the back are a far more sensitive. Your studies will probably be less controversial, but you can use the same technique. Of course, there is a corresponding danger of <em>false baselines<\/em>, in that we may end up with a misleading idea of the size of effects &#8212; as noted previously with power comes the responsibility to report fairly and accurately.<\/li>\n<\/ul>\n<p>In the following two posts, we will consider strategies that address the factors of the noise&#8211;effect&#8211;number triangle in different ways. We will concentrate first on the subjects, the users or participants in our studies, and then on the tasks we give them to perform.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The heart of gaining power in your studies is understanding the noise&#8211;effect&#8211;number triangle. \u00a0Power arises from a combination of the size of the effect you are trying to detect, the size of the study (number of trails\/participants) and the size &hellip; <a href=\"https:\/\/alandix.com\/statistics\/2017\/05\/31\/gaining-power-2\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_themeisle_gutenberg_block_has_review":false,"footnotes":""},"categories":[4],"tags":[3],"class_list":["post-151","post","type-post","status-publish","format-standard","hentry","category-chi-course-notes","tag-chi-2017"],"_links":{"self":[{"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/posts\/151","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/comments?post=151"}],"version-history":[{"count":5,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/posts\/151\/revisions"}],"predecessor-version":[{"id":172,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/posts\/151\/revisions\/172"}],"wp:attachment":[{"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/media?parent=151"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/categories?post=151"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/tags?post=151"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}