{"id":423,"date":"2018-03-28T13:40:27","date_gmt":"2018-03-28T13:40:27","guid":{"rendered":"https:\/\/alandix.com\/statistics\/?p=423"},"modified":"2018-03-30T18:07:29","modified_gmt":"2018-03-30T18:07:29","slug":"doing-it-2","status":"publish","type":"post","link":"https:\/\/alandix.com\/statistics\/2018\/03\/28\/doing-it-2\/","title":{"rendered":"Doing it (making sense of statistics) &#8211; 2 &#8211; probing the unknown"},"content":{"rendered":"<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide02.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignright size-medium wp-image-416\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide02-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide02-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide02.jpg 400w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a>You use statistics when there is something in the world you don&#8217;t&#8217; know, and want to get a level of quantified understanding of it based on some form of the measurement or sample.<\/p>\n<p>One key mathematical element of this shared by all techniques is the idea of conditional probability and likelihood; that is the probability of a specific measurement occurring assuming you know everything pertinent about the real world. Of course the whole point is that you don&#8217;t know what is true of the real world, but do know about the measurement, so you need to do back-to-front counterfactual reasoning, to go back from measurement to the world!<\/p>\n<p>Future videos will discuss three major kinds of statistical analysis methods:<\/p>\n<ul>\n<li>Hypothesis testing (the dreaded p!) \u2013 robust but confusing<\/li>\n<li>Confidence intervals \u2013 powerful but underused<\/li>\n<li>Bayesian stats \u2013 mathematically clean but fragile<\/li>\n<\/ul>\n<p>The first two use essentially the same theoretical approach, and the difference is more about the way you present results. Bayesian statistics takes a fundamentally different approach, with its own strengths and weaknesses.<\/p>\n<p><iframe loading=\"lazy\" title=\"Doing it! (making sense of statistics)  \u2013 probing the unknown\" width=\"584\" height=\"329\" src=\"https:\/\/www.youtube.com\/embed\/SKeNrMq3nXA?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<p>&nbsp;<\/p>\n<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide03.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-417 alignnone\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide03-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide03-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide03.jpg 400w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>First of all let&#8217;s recall the &#8216;<a rel=\"noopener\" href=\"https:\/\/alandix.com\/statistics\/2017\/08\/31\/the-job-of-statistics\/\" target=\"_blank\">job of statistics<\/a>&#8216;, which is an attempt to understand the fundamental properties of the real world based on measurements and samples. For example, you may have taken a dozen people (the sample), asked them to perform a task on a piece of software and a new version of the software. You have measured response times, satisfaction, error rate, etc., (the measurement) and want to know whether your new software will out perform the original software for the whole user group (the real world).<\/p>\n<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide04.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-418\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide04-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide04-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide04.jpg 400w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>We are dealing with data with a lot of randomness and so need to deal with probabilities, but in particular what is known as <em>conditional probability<\/em>.<\/p>\n<p>Imagine the main street of a local city. What is the probability that it is busy?<\/p>\n<p>Now imagine that you are standing in the same street but it is 4am on a Sunday morning: what is the probability it is busy <em>given this<\/em>?<\/p>\n<p>Although the overall probability of it being busy (at a random time of day) is high, the probability that it is busy <em>given<\/em> it is 4am on a Sunday is lower.<\/p>\n<p>Similarly think of a throwing single die.\u00a0\u00a0 What is the probability it is a six?\u00a0\u00a0 1 in 6.<\/p>\n<p>However, if I peek and tell you it is at least 4.\u00a0\u00a0 What now is the probability it is a six? The probability it is a six <em>given<\/em> it is four or greater is 1 in 3.<\/p>\n<p>When we have more information, then we change our assessment of the probability of events accordingly.\u00a0 This calculation of probability <em>given<\/em> some information is what mathematicians call <em>conditional probability.<\/em><\/p>\n<p>&nbsp;<\/p>\n<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide05.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-419\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide05-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide05-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide05.jpg 400w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Returning to the job of statistics, we are interested in the relationship between measurements of the real world and what is true of the real world. Although we may not know what is true of the world (what is the actual error rate of our new software going to be), we can often work out the probability of measurements <em>given<\/em> (the unknown) state of the world.<\/p>\n<p>For example, if the probability of a user making a particular error is 1 in 10, then the probability that exactly 3 make the error out of a sample of 5 is 7.29% (calculated from the Binomial distribution).<\/p>\n<p>This <em>conditional probability<\/em> of a measurement given the state of the world (or typically some specific parameters of the world) is what statisticians call <em>likelihood<\/em>.<\/p>\n<p>As another example the probability that six tosses of a coin will come out heads <em>given<\/em> the coin is fair is 1\/64, or in other words the <em>likelihood<\/em> that it is fair is 1\/64. If instead the coin were biased 2\/3 heads 1\/3 tails, the probability of 6 heads <em>given<\/em> this, likelihood fo the coin having this bias, is 64\/729 ~ 1\/ 11.<\/p>\n<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide06.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-420\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide06-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide06-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide06.jpg 400w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Note this likelihood is NOT the probability that the coin is fair or biased, we may have good reason to believe that most coins are fair. However, it does constitute evidence. The core difference between different kinds of statistics is the way this evidence is used.<\/p>\n<p>Effectively statistics tries to turn this round, to take the likelihood, the probability of the measurement given the unknown state of the world, and reverse this, use the fact that the measurement has occurred to tell us something about the world.<\/p>\n<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide07.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-421\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide07-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide07-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide07.jpg 400w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Going back again to the job of statistics, the measurements we have of the world are prone to all sorts of random effects. The likelihood models the impact of these random effects as probabilities.\u00a0\u00a0 The different types of statistics then use this to produce conclusions about the real world.<\/p>\n<p>However, crucially these are always uncertain conclusions. Although we can improve our ability to see through the fog of randomness, there is always the possibility that by shear chance things appear to suggest one conclusion even though it is not true.<\/p>\n<p><a href=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide08.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-422\" src=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide08-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" srcset=\"https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide08-300x225.jpg 300w, https:\/\/alandix.com\/statistics\/files\/2017\/12\/Slide08.jpg 400w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>We will look at three types of statistics.<\/p>\n<p><strong><em>Hypothesis testing<\/em><\/strong> is what you are most likely to have seen \u2013 the dreaded p! It was originally introduced as a form of &#8216;quick hack&#8217;, but has come to be the most widely used tool. Although it can be misused, deliberately or accidentally, in many ways, it is time-tested, robust and quite conservative. The downside is that understanding what it really says (not p&lt;5% means true!) can be slightly complex.<\/p>\n<p><strong><em>Confidence intervals<\/em><\/strong> use the same underlying mathematical methods as hypothesis testing, but instead of taking about whether there is evidence for or against a single value, or proposition, confidence intervals give a range of values. This is really powerful in giving a sense of the level of uncertainty around an estimate or prediction, but are woefully underused.<\/p>\n<p><strong><em>Bayesian statistics<\/em><\/strong> use the same underlying likelihood (although not called that!) but combine this with numerical estimates of the probability of the world. It is mathematically very clean, but can be fragile. One needs to be particularly careful to avoid conformation bias and when dealing with multiple sources of non-independent evidence.\u00a0 In addition, because the results are expressed as probabilities, this may give an impression of objectivity, but in most cases it is really about modifying one&#8217;s assessment of belief.<\/p>\n<p>We will look at each of these in more detail in coming videos.<\/p>\n<p>To some extent these techniques have been pretty much the same for the past 50 years, however computation has gradually made differences. Crucially, early statistics needed to relatively easy to compute by hand, whereas computer-based statistical analyses can use more complex methods. This has allowed more complex models based on theoretical distributions, and also simulation methods that use models where there is no &#8216;nice&#8217; mathematical solution.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>You use statistics when there is something in the world you don&#8217;t&#8217; know, and want to get a level of quantified understanding of it based on some form of the measurement or sample. One key mathematical element of this shared &hellip; <a href=\"https:\/\/alandix.com\/statistics\/2018\/03\/28\/doing-it-2\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_themeisle_gutenberg_block_has_review":false,"footnotes":""},"categories":[4],"tags":[3,9],"class_list":["post-423","post","type-post","status-publish","format-standard","hentry","category-chi-course-notes","tag-chi-2017","tag-doing-it"],"_links":{"self":[{"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/posts\/423","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/comments?post=423"}],"version-history":[{"count":4,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/posts\/423\/revisions"}],"predecessor-version":[{"id":529,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/posts\/423\/revisions\/529"}],"wp:attachment":[{"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/media?parent=423"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/categories?post=423"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/alandix.com\/statistics\/wp-json\/wp\/v2\/tags?post=423"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}