christopherroach.com

Running Jupyter Lab as a Desktop Application

2018-01-18T00:00:00-08:00

So, Jupyter Lab is starting to get really interesting as a day-to-day replacement for standard Jupyter Notebooks and as a python competitor to R’s RStudio IDE. But, while a Jupyter Notebook with its multi-page interface feels right at home in the browser, I feel that as a single page application, Jupyter Lab would work better as a standalone desktop app without all the unwanted “chrome” that comes with the standard web browser. Luckily, the Chrome browser has an application mode that allows it to run with all of the toolbars and unnecessary UI removed. Using this mode will give you back a good deal of screen real estate and make Jupyter Lab feel more like a native application rather than a website running inside of your browser.

To use the application mode of Chrome with Jupyter Lab, you simply need to run the Jupyter Lab server with the --no-browser option to prevent it from popping open the application in your default browser.

$ jupyter lab --no-browser

Then, copy the URL printed out to the terminal (example below).

...
[I 09:36:41.608 LabApp] The Jupyter Notebook is running at:
[I 09:36:41.608 LabApp] http://localhost:8888/?token=<token>
[I 09:36:41.608 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 09:36:41.609 LabApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=<token>

And, to open Chrome in application mode with the Jupyter Lab URL, you simply need to call it with the --app=<URL> option and pass the URL you just copied.

On a Mac, the command to do so would look like the following.

$ /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --app=http://localhost:8888/?token=<token>

Once you run the command, you should see something like the following window pop up.

Notice the distinct lack of any and all toolbars. In fact, the only toolbar that shows up is the set of menus in the Jupyter Lab application.

Making the change permanent

Now, if running two commands with a few extra options every time you want to open up Jupyter Lab happens to be a bit too much for you, you’re in luck. You can make this behavior permanent by simply modifying the config to change the default browser for Jupyter Lab.

If you’ve modified your default Jupyter Notebook configuration in the past, you can simply open up the configuration file that already exists (~/.jupyter/jupyter_notebook_config.py) and add the following line to it.

c.LabApp.browser = '/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --app=%s'

However, if this is your first time ever modifying your Jupyter Notebook configuation, you’ll probably need to generate it first. You can do so with the following command.

$ jupyter lab --generate-config

After generating the config, simply open up the newly created file and add the browser config line from above. Once you’ve modified the config, you can simply call jupyter lab and the app will open up in a pristine window devoid of all the typical bells and whistles.

Did Illegal Voting Cost Trump the Popular Vote?

2017-04-20T00:00:00-07:00

On the whole, politicians are not typically known for their honesty, but our recently elected 45th president, Mr. Donald J. Trump, has taken the practice of peddling "alternative facts" to all new levels. In fact, the pulitzer prize winning fact checking website, Politifact, had at one point during the campaign calculated that only a mere 4% of his statements could be considered completely true.

Given Trump's propensity for duplicity, there certainly is no shortage of fantastic claims for an enterprising citizen journalist to dig into. For my money though, his most interesting assertion to date still has to be that millions of illegal ballots cost him the popular vote this past November.

In addition to winning the Electoral College in a landslide, I won the popular vote if you deduct the millions of people who voted illegally
— Donald J. Trump (@realDonaldTrump) November 27, 2016

This particular assertion is of interest to me for a handful of reasons. First, I have yet to see a single source outside of the Trump camp treat this claim with even an ounce of credibility. Second, unlike many of his most bombastic claims, this one is based on actual research from reputable sources in some cases. And, finally, this particular claim can be easily proven, or disproven as the case may be, by simply looking at the data.

So, let's jump right in and examine the evidence both for and against Trump's claim of a popular vote win. And feel free to follow along with the analysis by downloading the jupyter notebook for this article and running it locally.

The Evidence for Trump¶

There seems to be three main sources of proof for Trump's proclamation of a popular vote win.

The first is a peer reviewed paper published in the journal Electoral Studies by Political Scientists Jesse T. Richman, Gulshan A. Chattha, and David C. Earnest. The second is a Pew Research report written by David Becker on the inefficiencies within our current voter registration system. The third, and most recent, is a yet to be seen piece of analysis from True the Vote, a non-profit dedicated to monitoring voter fraud through the use of the VoteStand mobile application for "quickly report[ing] suspected election illegalities."

The VoteStand Analysis¶

Let's start by quickly dispensing with the VoteStand analysis first since there's really nothing to discuss at this point. The story here is that the person behind the creation of the VoteStand application, Gregg Phillips, made the claim shortly after the election that upwards of 3 million votes were cast illegally in favor of Clinton.

Look forward to seeing final results of VoteStand. Gregg Phillips and crew say at least 3,000,000 votes were illegal. We must do better!
— Donald J. Trump (@realDonaldTrump) January 27, 2017

To date, however, Gregg and VoteStand have yet to share any of their findings or their methodologies with the general public, so all we currently have to go on are Gregg's initial claims, and even he seems to be hedging his bet a bit now. So, until we see some actual evidence from the VoteStand analysis, there's not much we can do with the unsubstantiated claims of its founder other than acknowledge them and move on.

The Pew Research Report¶

The other easily disputable piece of evidence is the Pew Research report. In one tweet, Trump stated that he would be calling for an investigation into voter fraud with a specific interest in voters registered in more than one state.

I will be asking for a major investigation into VOTER FRAUD, including those registered to vote in two states, those who are illegal and....
— Donald J. Trump (@realDonaldTrump) January 25, 2017

This idea has no doubt come out of his interpretation of the Pew Research findings. However, though the report found evidence of millions of invalid voter registrations (e.g., 24 million outdated records, 1.8 million deceased voters, and 2.4 million registered in multiple states), the author, David Becker, has since stated that no evidence of voter fraud was ever uncovered in the Pew Research study.

We found millions of out of date registration records due to people moving or dying, but found no evidence that voter fraud resulted.
— David Becker (@beckerdavidj) November 28, 2016

Bottom line, there's nothing in this report to suggest that any illegal voting activity occurred in this past election. On a somewhat snarky side note, however, in Trump's own administration there are at least five members that are actively registered to vote in more than one state.

Trump Team Members Registered to Vote in Multiple States¶

Chief White House strategist Stephen Bannon
Press Secretary Sean Spicer
Treasury Secretary Steven Mnuchin
Tiffany Trump, the president's youngest daughter
Senior White House Advisor Jared Kushner, and Trump's son-in-law

The question then, given Trump's interest in invalid voter registrations, is "are people such as Stephen Bannon, Sean Spicer, and even Trump's daughter Tiffany part of some conspiracy to illegally elect Hillary Clinton?" Oh, and I almost forget VoteStand's founder Gregg Phillips who, as it turns out, is registered to vote in three different states according to the Associated Press.

The point I'm trying to make, and this is confirmed in David Becker's tweet, is that, yes, we know our system isn't perfect; people move around and pass away and our voter records are not always kept up to date, but that's not proof that millions of people voted illegally.

The CCES Analysis¶

That brings me to the most interesting piece of evidence for Trump's claim: a peer reviewed paper titled "Do non-citizens vote in U.S. elections?". In this paper, three political scientists from Old Dominion University describe an analysis they performed on data from two large online, opt-in surveys conducted by the London based polling firm YouGov.

The surveys were given during the 2008 presidential election and the subsequent 2010 midterms, and though these surveys were designed to capture information from the adult citizen population, a relatively small number of non-citizens did participate in the surveys: 339 of 32,800 respondents in 2008 and 489 of 55,400 in 2010. The data from these small samples of non-citizen participants was then used to extrapolate the numbers by which Trump professes to have won the popular vote. Specifically, the authors of the paper conclude that up to an estimated 14.7% of non-citizens could have voted in the 2008 election, and as we'll see shortly, it is this overly-optimistic estimate that Trump is drawing upon to prove his November mandate.

Now, there are numerous reasons why Trump's use of this paper's findings as proof of his win could be somewhat misleading, but there are two reasons in particular that we will discuss in the following sections.

Bad Math and Misinterpretation¶

First, it's no secret that Trump doesn't particularly like to read, so it's quite possible that he just hasn't read the paper that his claim relies so heavily upon. If he had, he would have no doubt realized that the odds of Hillary winning because of a groundswell of support in the non-citizen community is relatively low at best, and is not supported by the results in the paper. But don't take my word for it, the math is simple, so let's run the numbers.

The Cook Political Report, an independent, non-partisan political newsletter, reported that Hillary Clinton won the popular vote by roughly 2.9 million votes. Within the same time period, the total number of non-citizens living in the U.S. was about 23 million, according to a report from the Kaiser Family Foundation.

In [1]:

# The number of votes Hillary won over Trump
popular_vote_margin = 2864974
# The total number of non-citizens living in the U.S. in 2016
total_non_citizens = 22984400
# Proportion of non-citizen votes needed for a Trump win
percent_needed_for_trump_win = round(popular_vote_margin / total_non_citizens * 100, 1)

So, for Trump to win the popular vote, he would have needed approximately 12.5% of non-citizens to not only vote, but to vote exclusively for Hillary. In the paper, Richman et al., conclude that at the high end, 11.3% of non-citizens may have actually voted in the 2008 election. Now, that falls a bit shy of the 12.5% that Trump needs for a win, but when you take into account the margin of error (see the calculation below), you get a range of 7.9 - 14.7% (with a confidence level of 95%), which would mean that a popular vote victory could, theoretically at least, be possible.

In [2]:

import math

def margin_of_error(p, n, z=1.96):
    """Returns the Margin of Error for the given proportion.
    
    Arguments:
    p -- the proportion of the sample with a characteristic
    n -- the size of the sample
    z -- z-score for the desired confidence interval
    
    """
    return z * math.sqrt(p * (1 - p)/n)

# The proportion (11.3%) of non-citizens that may have voted in 2008
p = 0.113 
# The total number (339) of non-citizens in the 2008 survey
n = 339
# Calculate the margin of error for the sample population of 339 non-citizens
moe = round(margin_of_error(p, n) * 100, 1)
# Calculate the 95% confidence interval
ci = (11.3 - moe, 11.3 + moe)

print('Margin of Error: {0:.1f}%'.format(moe))
print('Confidence Interval (95%): {0:.1f} - {1:.1f}%'.format(*ci))

Margin of Error: 3.4%
Confidence Interval (95%): 7.9 - 14.7%

So, what could possibly be wrong with this picture?

Well, for starters, the author of the paper himself has stated that the numbers simply don't add up to a victory for Trump. In a blog post on his personal website, Richman explains that the 12.5% that Trump needs for a win is not likely to have materialized. In fact, as Richman points out in his blog post, the authors of the original paper had already estimated that the most likely proportion of non-citizens who voted would have been closer to 6.4%. If Trump had read the entire paper, he would have noticed this number and realized that his claims just don't add up.

In [3]:

# Richman's paper concludes that 6.4% is a more 
# realistic estimate of non-citizen voters
adj_p = 0.064
illegal_voters = adj_p * total_non_citizens

Now, if we use the more realistic percentage to calculate the number of illegal voters, we get only 1.5 million, which is roughly half of the 2.9 million that Trump needs to declare a win.

Furthermore, based on voter behavior from the 2008 election, Richman calculates that it's likely that only 81.8% of those illegal voters would have voted for Clinton, the rest would have gone with Trump, with some small percentage going to third party candidates. Using this new information, we can get a more accurate picture of the number of votes that Clinton would have received from the non-citizen population.

In [4]:

# Of those illegal voters 81.8% would have voted for Clinton
clinton = round(illegal_voters * 0.818)
# and 17.5% would have gone to Trump
trump = round(illegal_voters * 0.175)
# Number of popular votes for Clinton minus non-citizens
adj_popular_vote_margin = popular_vote_margin - (clinton - trump)
# The proportion of non-citizens needed to vote to have lost the election
# as a result of illegal voting activity (assuming that 90% of non-citizens
# vote for Clinton and keeping the third party vote constant).
non_citizen_proportion_needed = popular_vote_margin/(total_non_citizens * (0.9 - 0.093))

With this new information taken into account, the most likely result would then be that Trump would have still lost the popular vote by 1.9 million votes.

In fact, even if we give Trump the benefit of the doubt and assume that an implausibly high proportion of non-citizens voted for Clinton (let's say 90%), Trump would then need 15.4% of all non-citizens in the U.S. to vote in order to lose the election solely due to illegal voting activity. That's a far cry from the 11.3% reported as the high end in the original paper, and even adjusting that number for the margin of error, the number Trump needs still falls outside of the possible range (7.9 - 14.7%).

Bottom line, even if the paper's results are to be believed, the math just doesn't add up. And, as we've just seen, the results of the paper suggest that the possibility of Trump winning the popular vote from illegal votes alone is so slim as to have little to no grounding in reality. Of course, this assumes that the results of the paper can be believed. However, according to another peer reviewed paper in the same journal—the authors of which are the PI and co-PI of the studies used in the Richman paper—that may not be the case.

Respondent Error¶

The main argument of "The Perils of Cherry Picking Low Frequency Events in Large Sample Surveys" is that, due to a unique characteristic of these large sample surveys (which we will cover shortly), the findings of Richman et al. can be completely explained by simple respondent error.

Respondent error, according to Wikipedia, "refers to any error introduced into the survey results due to respondents providing untrue or incorrect information" and simply put, it's a natural part of any survey. It could be that the participant didn't understand the question. Or, perhaps they haven't had that first cup of coffee, and in a state of caffeine-deprived grogginess, mistakenly checked the wrong box. The truth of the matter is that errors such as these happen, and the larger the survey the more chances for such errors to occur.

Now, I know that it may seem a little far-fetched to say that we can go from millions of illegal voters handing an illegitimate win to Clinton, to basically saying "there's nothing to see here folks" just because a handful of people checked the wrong box, but this may prove to be the case.

In the remaining sections I work through each of the main points made by Ansolabehere et al. in their short, but effective rebuttal.

A Real World Example¶

The authors of the paper begin by stepping through an example that illustrates how inferring population behavior from a relatively low-frequency event in a large sample can lead to misinterpretations. In this article, however, I'm going to bypass the fictional example and just jump straight into the real data.

The dataset that had the most impact with respect to the findings in Richman's paper was the 2008 presidential survey data. This dataset contains observations from 32,800 participants, of which 339 (1%) were labeled as non-citizens. Making the assumption that 99.9% of the time (a pretty high rate of success) the respondents answered the citizenship question correctly, we would expect that roughly 33 individuals (32,800 * (1 - 0.999)) would have incorrectly classified themselves as non-citizens. These incorrectly labeled individuals would then account for about 10% (33/339) of the non-citizen sample. If you then make the additional assumption that citizens vote with a much higher regularity than non-citizens (or, perhaps that non-citizens do not vote at all), the result is that the incorrectly labeled participants would be nearly completely responsible for any observed non-citizen voting behavior.

As an example, if we assume that 70% of citizens voted and all non-citizens abstained, then the behavior of the incorrectly labeled citizens alone would give the appearance that nearly 7% ((33 * 0.7)/339) of non-citizens voted illegally in the presidential election. If we were to take this one step further and apply this number to the total number of non-citizens in the US in 2016 (23 million), it would appear that roughly 1.6 million (7% of 23M non-citizens) illegal votes could have been cast in the most recent election. 1.6 million votes is quite a bit larger than the number of illegal votes estimated by Richman (800k), so it would appear that the conclusions reached in the original paper, if our assumptions above hold true, could be just a side effect of expected errors in the data.

The question then is, "just how accurate are our assumptions?"

Well, according to Ansolabehere, et al., the rate of respondent error and the citizen voter rates that we posited above (0.1% and 70% respectively) match the rates observed in a separate panel survey that was conducted during the 2010 and 2012 elections. However, rather than just taking this assertion at face value, we can go a step further and actually download the panel data and take a look for ourselves.

Step 1: Download, load, and clean the data.¶

I had a little trouble at first finding the panel data that was used in the paper. The yearly survey data can be easily found from the CCES homepage, but you have to dig a bit deeper to find the panel data. Luckily, the fine folks at Harvard who conduct the CCES surveys were quick to respond with a link to the panel data used in the paper.

Several formats of the panel data are provided via the download button on the website linked above, but since the original format of the dataset is a Stata file, and pandas has the handy read_stata function, I decided to go with that since it has the nice side effect of making sure that all of the attributes in the dataset have the correct type without any extra work on my part.

Note: If you're following along on your own machine and wish to run the code below, you'll first need to follow the link above to download the panel data before you can execute the next bit of code.

In [5]:

import pandas as pd

cces_panel = pd.read_stata('CCES1012PaneV2.dta')

The next problem you may run into is finding the right attributes in the data to analyze.

We're specifically trying to figure out the percentage of respondents that could have mistakenly labeled themselves incorrectly with respect to their citizenship, and so we'll need to grab the columns that represent the citizenship question from both years. Unfortunately, as of this writing, the guide that comes with the panel data refers to the columns incorrectly. After a quick email exchange with the PI of the survey, I was able to piece together which features to use. As it turns out, the names of the features we need actually match the names from the original 2010 and 2012 non-panel surveys, these are V263 and immstat respectively.

Now that we have the correct columns for our analysis, we can clean up the data a bit by removing any participants from the dataset that did not answer the citizenship question in both years.

In [6]:

cces_panel.dropna(subset=['V263', 'immstat'], inplace=True)

With the data loaded and properly scrubbed, we're ready to calculate the respondent error rate for the survey.

Step 2: Calculate the respondent error rate¶

We'll start our calculation by first calculating the contingency table for the survey year and citizenship features. We could do this by using the crosstab function that the pandas library provides, but we'll do it by hand instead so we can use the totals for further calculations later. The following code creates a subset of the data for each of the four possible groups of participants:

cc - Individuals who marked themselves as citizens in both years
cnc - Individuals who marked themselves as citizens in 2010 and non-citizens in 2012
ncc - Individuals who marked themselves as non-citizens in 2010 and citizens in 2012
ncnc - Individuals who marked themselves as non-citizens in both years

In [7]:

# Citizen in 2010 and Citizen in 2012
cc = cces_panel[(cces_panel.V263 != 'Immigrant non-citizen') & 
                (cces_panel.immstat != 'Immigrant non-citizen')]
# Citizen in 2010 and Non-citizen in 2012
cnc = cces_panel[(cces_panel.V263 != 'Immigrant non-citizen') & 
                 (cces_panel.immstat == 'Immigrant non-citizen')]
# Non-citizen in 2010 and Citizen in 2012
ncc = cces_panel[(cces_panel.V263 == 'Immigrant non-citizen') & 
                 (cces_panel.immstat != 'Immigrant non-citizen')]
# Non-citizen in 2010 and Non-citizen in 2012
ncnc = cces_panel[(cces_panel.V263 == 'Immigrant non-citizen') &                  
                  (cces_panel.immstat == 'Immigrant non-citizen')]

Once we have the individual groups created, we simply need to calculate each group's proportion of the entire sample population to produce our contingency table.

In [8]:

cc_percent = round(len(cc)/len(cces_panel) * 100, 2)
cnc_percent = round(len(cnc)/len(cces_panel) * 100, 2)
ncc_percent = round(len(ncc)/len(cces_panel) * 100, 2)
ncnc_percent = round(len(ncnc)/len(cces_panel) * 100, 2)

Response in 2010	Response in 2012	Number of Respondents	Percentage
Citizen	Citizen	18737	99.25
Citizen	Non-citizen	20	0.11
Non-citizen	Citizen	36	0.19
Non-citizen	Non-citizen	85	0.45

Using the contingency table above we can estimate the respondent error rate (conversely, the success rate) of the survey participants with respect to correctly reporting their citizenship status.

We can see that the overwhelming majority of participants gave matching answers in consecutive years: 99.25% labeled themselves as citizens in both years, while 0.45% identified themselves as non-citizens. It's probably pretty safe to assume that these individuals filled in this portion of the survey correctly, so we'll add them together to get an initial success rate of 99.7%. Of the two remaining groups, those who marked themselves as citizens in 2010 and non-citizens in 2012 are likely to have responded incorrectly since it is highly unlikely that someone would lose their citizenship in the two years between elections. On the other hand, it's very possible that a handful of non-citizens surveyed in 2010 would be granted citizenship in the intervening years, so we'll add that group to our success rate as well, which will bring the total to 99.9%. Conversely, we can say that our respondent error rate is equal to 0.1%, which perfectly matches the number we used in our example above.

Step 3: Calculate the citizen voter rate¶

The other number in question was the citizen voter rate of approximately 70%. Let's dig into the data once more to see if the number we used in the example above matches the real world.

Again, it took a little poking around to find the correct feature to analyze, but if we take a look at the VV_2010_gen field, we can see whether or not the participant voted as well as the method by which they did. To determine the number of people who did vote in the 2010 election, we will need to know the list of possible values that each record can take on for that field. The VV_2010_gen field is a category, so we can get a look at its list of possible values through the cc.cat.categories property.

In [9]:

cc.VV_2010_gen.cat.categories

Out[9]:

Index(['Absentee', 'Early', 'Mail', 'Voted unknown method',
       'Confirmed Non-voter', 'Unmatched'],
      dtype='object')

Of the possible values listed above, only the 'Absentee', 'Early', 'Mail', and 'Voted unknown method' options pertain to people who voted in the 2010 election. Therefore, we can calculate the percentage of voters by simply counting the number of records with one of those values and dividing that by the total number of survey participants.

In [10]:

voted = ['Absentee', 'Early', 'Mail', 'Voted unknown method']
citizen_voter_rate = sum(cc.VV_2010_gen.isin(voted))/len(cc)

We can then see that the voter rate among individuals registered as citizens in both surveys was 71.0%, which is roughly equivalent to the 70% number we used in the example above. Couple that with the 0.1% respondent error rate that we calculated previously, and it looks like the example we gave does in fact match reality.

In short, it does look like the results of the Richman paper could be completely attributed to response error and thusly discarded as proof of illegal voting activity swaying the popular vote in favor of Clinton.

Margin of Error (or, why is this only a problem with large datasets?)¶

As I mentioned earlier, drawing conclusions from relatively small samples pulled from larger sample surveys is only a problem when the original survey data is quite large, such as the 2008 and 2010 CCES surveys use by Richman et al. to come up with their estimates of possible illegal voting activity. The question then is, "why does this problem only turn up in large sample surveys?"

The answer to that question would be the margin of error.

A margin of error, according to Wikipedia, is "a statistic that expresses the amount of random sampling error in a survey's results." Essentially, it is a number that can be applied to a poll's reported results to calculate a range within which we have a given probabilistic chance of the real value falling. In short, the larger the margin of error, the wider the range, and consequently the lower the possibility of the poll's reported results actually matching the real world figures.

To make this concept a bit clearer, we can take a look at a short example.

Let's start with a small dataset, say n = 1000, and then calculate the number of citizens and non-citizens based on the breakdown we saw in the original 2008 CCES survey.

In [11]:

n = 1000
non_citizens_proportion = 339/32800
non_citizens = round(n * non_citizens_proportion)
citizens = n - non_citizens

Using the same breakdown that we saw in the 2008 survey, we end up with a sample of 990 citizens and 10 non-citizens.

Next, we'll use the same error rate that we calculated earlier to determine the number of citizens that would incorrectly identify themselves as non-citizens.

In [12]:

error_rate = 0.001
incorrectly_labeled_citizens = round(citizens * error_rate)

With a respondent error rate of 0.001, we would expect that 1 citizen would incorrectly identify themselves as a non-citizen. Given a sample size of 10, we would expect that one incorrectly labeled citizen to have a high impact on any numbers extrapolated from the sample since they would account for 10.0% of the entire non-citizen sample. We'll see just how big an impact that person has by calculating the expected illegal voter rate, and consequently the number of illegal votes, based solely on respondent error alone.

In [13]:

illegal_voter_rate = incorrectly_labeled_citizens/non_citizens * citizen_voter_rate
illegal_votes = total_non_citizens * illegal_voter_rate

Making the assumption that only the citizens that were incorrectly labeled as non-citizens voted, and that they voted at the normal citizen voter rate of 71.0%, we were able to calculate the number of illegal votes we would expect to see based on respondent error and nothing more. Doing so gives us an estimate of around 1.6 million illegal votes, which is exactly what we got in our example above.

If we then calculate the margin of error for our small sample of 10 non-citizens, we can calculate the confidence interval, or the range within which the actual number of illegal votes is likely to be found.

In [14]:

moe_small_survey = margin_of_error(illegal_voter_rate, non_citizens)
ci_small_survey_low = round((illegal_voter_rate - moe_small_survey) * total_non_citizens)
ci_small_survey_high = round((illegal_voter_rate + moe_small_survey) * total_non_citizens)

The small sample size gives us a relatively large margin of error of $\pm$15.9%. Using this margin of error to calculate a confidence interval, we get a range of between -2.0 and 5.3 million illegal votes cast. This is a huge range that dwarfs our initial estimate of 1.6 million votes and essentially renders our estimate useless. Thus, it's easy to see how extrapolating population behavior from a low-frequency event in a small sample typically would not happen simply because the margin of error is too great.

On the other hand, if we calculate the margin of error and confidence interval again using the larger sample size of 339 non-citizens that we saw in the 2008 survey, we get a much more acceptable margin of error.

In [15]:

moe_large_survey = margin_of_error(illegal_voter_rate, 339)
ci_large_survey_low = round((illegal_voter_rate - moe_large_survey) * total_non_citizens)
ci_large_survey_high = round((illegal_voter_rate + moe_large_survey) * total_non_citizens)

The larger sample size gives us a much smaller margin of error of $\pm$2.7%. Using this margin of error to calculate the new confidence interval, we get a range of between 1.0 and 2.3 million illegal votes cast. This is a much more reasonable range and it's easy to see how a researcher could be lulled into believing that they've just uncovered a significant finding in the data. However, as we've already seen, the danger with using findings from these low-frequency events to extrapolate behavior in the larger population is that these findings can be attributed solely to response error.

Conclusion¶

To recap, Trump has claimed, on several occasions, that not only did he win the electoral college vote, he also won the popular vote, if you discount all of the votes Hillary received from non-citizens residing within our borders. To the best of my knowledge, there are three main resources to which Trump has alluded that seem to support this claim.

The first, is a bit of analysis performed by the non-profit VoteStand that declared that close to 3 million non-citizens voted on behalf of Hillary in this past election. To date, however, this analysis has not been seen by anyone outside of the company, and as a result it can be easily discarded as proof.

The second piece of evidence is a Pew Research study that investigated issues with the current voter registration system and found evidence of millions of invalid voter registrations. Though the numbers in this study sound convincing, the author has since stated that no actual evidence of voter fraud was ever uncovered. Therefore, as an interesting look at some of the issues inherent in our voter registration system, this study is fantastic, but as proof of illegal voting activity, unfortunately it falls short.

Finally, the most interesting resource was a peer-reviewed paper from several political scientists with the Old Dominion University. In this paper, the authors used data from two different large sample surveys to extrapolate the number of possible illegal voters to be in the millions. The conclusions of this paper, however, have been disputed, by the original authors of the surveys, as being possibly a result of response error in the survey data. In addition, even if the results of the paper are to be believed, the original author of the paper has subsequently claimed that the expected numbers would still fall well below that needed for a Trump win of the popular vote.

In short, there is currently no evidence that supports Trump's claim that illegal voting activity cost him the popular vote. So, I'm sorry Mr. President, but it looks like you're just wrong on this one.

Statistics for Hackers

2017-01-18T00:00:00-08:00

Motivation¶

There's no shortage of absolutely magnificent material out there on the topics of data science and machine learning for an autodidact, such as myself, to learn from. In fact, so many great resources exist that an individual can be forgiven for not knowing where to begin their studies, or for getting distracted once they're off the starting block. I honestly can't count the number of times that I've started working through many of these online courses and tutorials only to have my attention stolen by one of the multitudes of amazing articles on data analysis with Python, or some great new MOOC on Deep Learning. But this year is different! This year, for one of my new year's resolutions, I've decided to create a personalized data science curriculum and stick to it. This year, I promise not to just casually sign up for another course, or start reading yet another textbook to be distracted part way through. This year, I'm sticking to the plan.

As part of my personalized program of study, I've chosen to start with Harvard's Data Science course. I'm currently on week 3 and one of the suggested readings for this week is Jake VanderPlas' talk from PyCon 2016 titled "Statistics for Hackers". As I was watching the video and following along with the slides, I wanted to try out some of the examples and create a set of notes that I could refer to later, so I figured why not create a Jupyter notebook. Once I'd finished, I realized I'd created a decently-sized resource that could be of use to others working their way through the talk. The result is the article you're reading right now, the remainder of which contains my notes and code examples for Jake's excellent talk.

So, enjoy the article, I hope you find this resource useful, and if you have any problems or suggestions of any kind, the full notebook can be found on github, so please send me a pull request, or submit an issue, or just message me directly on Twitter.

Preliminaries¶

In [1]:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

# Suppress all warnings just to keep the notebook nice and clean. 
# This must happen after all imports since numpy actually adds its
# RankWarning class back in.
import warnings
warnings.filterwarnings("ignore")

# Setup the look and feel of the notebook
sns.set_context("notebook", 
                font_scale=1.5, 
                rc={"lines.linewidth": 2.5})
sns.set_style('whitegrid')
sns.set_palette('deep')

# Create a couple of colors to use throughout the notebook
red = sns.xkcd_rgb['vermillion']
blue = sns.xkcd_rgb['dark sky blue']

from IPython.display import display

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

Warm-up¶

The talk starts off with a motivating example that asks the question "If you toss a coin 30 times and see 22 heads, is it a fair coin?"

We all know that a fair coin should come up heads roughly 15 out of 30 tosses, give or take, so it does seem unlikely to see so many heads. However, the skeptic might argue that even a fair coin could show 22 heads in 30 tosses from time-to-time. This could just be a chance event. So, the question would then be "how can you determine if you're tossing a fair coin?"

The Classic Method¶

The classic method would assume that the skeptic is correct and would then test the hypothesis (i.e., the Null Hypothesis) that the observation of 22 heads in 30 tosses could happen simply by chance. Let's start by first considering the probability of a single coin flip coming up heads and work our way up to 22 out of 30.

$$ P(H) = \frac{1}{2} $$

As our equation shows, the probability of a single coin toss turning up heads is exactly 50% since there is an equal chance of either heads or tails turning up. Taking this one step further, to determine the probability of getting 2 heads in a row with 2 coin tosses, we would need to multiply the probability of getting heads by the probability of getting heads again since the two events are independent of one another.

$$ P(HH) = P(H) \cdot P(H) = P(H)^2 = \left(\frac{1}{2}\right)^2 = \frac{1}{4} $$

From the equation above, we can see that the probability of getting 2 heads in a row from a total of 2 coin tosses is 25%. Let's now take a look at a slightly different scenario and calculate the probability of getting 2 heads and 1 tails with 3 coin tosses.

$$ P(HHT) = P(H)^2 \cdot P(T) = \left(\frac{1}{2}\right)^2 \cdot \frac{1}{2} = \left(\frac{1}{2}\right)^3 = \frac{1}{8} $$

The equation above tells us that the probability of getting 2 heads and 1 tails in 3 tosses is 12.5%. This is actually the exact same probability as getting heads in all three tosses, which doesn't sound quite right. The problem is that we've only calculated the probability for a single permutation of 2 heads and 1 tails; specifically for the scenario where we only see tails on the third toss. To get the actual probability of tossing 2 heads and 1 tails we will have to add the probabilities for all of the possible permutations, of which there are exactly three: HHT, HTH, and THH.

$$ P(2H,1T) = P(HHT) + P(HTH) + P(THH) = \frac{1}{8} + \frac{1}{8} + \frac{1}{8} = \frac{3}{8} $$

Another way we could do this is to calculate the total number of permutations and simply multiply that by the probability of each event happening. To get the total number of permutations we can use the binomial coefficient. Then, we can simply calculate the probability above using the following equation.

$$ P(2H,1T) = \binom{3}{2} \left(\frac{1}{2}\right)^{3} = 3 \left(\frac{1}{8}\right) = \frac{3}{8} $$

While the equation above works in our particular case, where each event has an equal probability of happening, it will run into trouble with events that have an unequal chance of taking place. To deal with those situations, you'll want to extend the last equation to take into account the differing probabilities. The result would be the following equation, where $N$ is number of coin flips, $N_H$ is the number of expected heads, $N_T$ is the number of expected tails, and $P_H$ is the probability of getting heads on each flip.

$$ P(N_H,N_T) = \binom{N}{N_H} \left(P_H\right)^{N_H} \left(1 - P_H\right)^{N_T} $$

Now that we understand the classic method, let's use it to test our null hypothesis that we are actually tossing a fair coin, and that this is just a chance occurrence. The following code implements the equations we've just discussed above.

In [2]:

def factorial(n):
    """Calculates the factorial of `n`
    """
    vals = list(range(1, n + 1))
    if len(vals) <= 0:
        return 1

    prod = 1
    for val in vals:
        prod *= val
        
    return prod
    
    
def n_choose_k(n, k):
    """Calculates the binomial coefficient
    """
    return factorial(n) / (factorial(k) * factorial(n - k))


def binom_prob(n, k, p):
    """Returns the probability of see `k` heads in `n` coin tosses
    
    Arguments:
    
    n - number of trials
    k - number of trials in which an event took place
    p - probability of an event happening
    
    """
    return n_choose_k(n, k) * p**k * (1 - p)**(n - k)

Now that we have a method that will calculate the probability for a specific event happening (e.g., 22 heads in 30 coin tosses), we can calculate the probability for every possible outcome of flipping a coin 30 times, and if we plot these values we'll get a visual representation of our coin's probability distribution.

In [3]:

# Calculate the probability for every possible outcome of tossing 
# a fair coin 30 times.
probabilities = [binom_prob(30, k, 0.5) for k in range(1, 31)]

# Plot the probability distribution using the probabilities list 
# we created above.
plt.step(range(1, 31), probabilities, where='mid', color=blue)
plt.xlabel('number of heads')
plt.ylabel('probability')
plt.plot((22, 22), (0, 0.1599), color=red);
plt.annotate('0.8%', 
             xytext=(25, 0.08), 
             xy=(22, 0.08), 
             multialignment='right',
             va='center',
             color=red,
             size='large',
             arrowprops={'arrowstyle': '<|-', 
                         'lw': 2, 
                         'color': red, 
                         'shrinkA': 10});

The visualization above shows the probability distribution for flipping a fair coin 30 times. Using this visualization we can now determine the probability of getting, say for example, 12 heads in 30 flips, which looks to be about 8%. Notice that we've labeled our example of 22 heads as 0.8%. If we look at the probability of flipping exactly 22 heads, it looks likes to be a little less than 0.8%, in fact if we calculate it using the binom_prob function from above, we get 0.5%

In [4]:

print("Probability of flipping 22 heads: %0.1f%%" % (binom_prob(30, 22, 0.5) * 100))

Probability of flipping 22 heads: 0.5%

So, then why do we have 0.8% labeled in our probability distribution above? Well, that's because we are showing the probability of getting at least 22 heads, which is also known as the p-value.

What's a p-value?¶

In statistical hypothesis testing we have an idea that we want to test, but considering that it's very hard to prove something to be true beyond doubt, rather than test our hypothesis directly, we formulate a competing hypothesis, called a null hypothesis, and then try to disprove it instead. The null hypothesis essentially assumes that the effect we're seeing in the data could just be due to chance.

In our example, the null hypothesis assumes we have a fair coin, and the way we determine if this hypothesis is true or not is by calculating how often flipping this fair coin 30 times would result in 22 or more heads. If we then take the number of times that we got 22 or more heads and divide that number by the total of all possible permutations of 30 coin tosses, we get the probability of tossing 22 or more heads with a fair coin. This probability is what we call the p-value.

The p-value is used to check the validity of the null hypothesis. The way this is done is by agreeing upon some predetermined upper limit for our p-value, below which we will assume that our null hypothesis is false. In other words, if our null hypothesis were true, and 22 heads in 30 flips could happen often enough by chance, we would expect to see it happen more often than the given threshold percentage of times. So, for example, if we chose 10% as our threshold, then we would expect to see 22 or more heads show up at least 10% of the time to determine that this is a chance occurrence and not due to some bias in the coin. Historically, the generally accepted threshold has been 5%, and so if our p-value is less than 5%, we can then make the assumption that our coin may not be fair.

The binom_prob function from above calculates the probability of a single event happening, so now all we need for calculating our p-value is a function that adds up the probabilities of a given event, or a more extreme event happening. So, as an example, we would need a function to add up the probabilities of getting 22 heads, 23 heads, 24 heads, and so on. The next bit of code creates that function and uses it to calculate our p-value.

In [5]:

def p_value(n, k, p):
    """Returns the p-value for the given the given set 
    """
    return sum(binom_prob(n, i, p) for i in range(k, n+1))

print("P-value: %0.1f%%" % (p_value(30, 22, 0.5) * 100))

P-value: 0.8%

Running the code above gives us a p-value of roughly 0.8%, which matches the value in our probability distribution above and is also less than the 5% threshold needed to reject our null hypothesis, so it does look like we may have a biased coin.

The Easier Method¶

That's an example of using the classic method for testing if our coin is fair or not. However, if you don't happen to have at least some background in statistics, it can be a little hard to follow at times, but luckily for us, there's an easier method...

Simulation!

The code below seeks to answer the same question of whether or not our coin is fair by running a large number of simulated coin flips and calculating the proportion of these experiments that resulted in at least 22 heads or more.

In [6]:

M = 0
n = 50000
for i in range(n):
    trials = np.random.randint(2, size=30)
    if (trials.sum() >= 22):
        M += 1
p = M / n

print("Simulated P-value: %0.1f%%" % (p * 100))

Simulated P-value: 0.8%

The result of our simulations is 0.8%, the exact same result we got earlier when we calculated the p-value using the classical method above. So, it definitely looks like it's possible that we have a biased coin since the chances of seeing 22 or more heads in 30 tosses of a fair coin is less than 1%.

Four Recipes for Hacking Statistics¶

We've just seen one example of how our hacking skills can make it easy for us to answer questions that typically only a statistician would be able to answer using the classical methods of statistical analysis. This is just one possible method for answering statistical questions using our coding skills, but Jake's talk describes four recipes in total for "hacking statistics", each of which is listed below. The rest of this article will go into each of the remaining techniques in some detail.

Direct Simulation
Shuffling
Bootstrapping
Cross Validation

In the Warm-up section above, we saw an example direct simulation, the first recipe in our tour of statistical hacks. The next example uses the Shuffling method to figure out if there's a statistically significant difference between two different sample populations.

Shuffling¶

In this example, we look at the Dr. Seuss story about the Star-belly Sneetches. In this Seussian world, a group of creatures called the Sneetches are divided into two groups: those with stars on their bellies, and those with no "stars upon thars". Over time, the star-bellied sneetches have come to think of themselves as better than the plain-bellied sneetches. As researchers of sneetches, it's our job to uncover whether or not star-bellied sneetches really are better than their plain-bellied cousins.

The first step in answering this question will be to create our experimental data. In the following code snippet we create a dataframe object that contains a set of test scores for both star-bellied and plain-bellied sneetches.

In [7]:

import pandas as pd

df = pd.DataFrame({'star':  [1, 1, 1, 1, 1, 1, 1, 1] + 
                            [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                   'score': [84, 72, 57, 46, 63, 76, 99, 91] +
                            [81, 69, 74, 61, 56, 87, 69, 65, 66, 44, 62, 69]})
df

Out[7]:

	score	star
0	84	1
1	72	1
2	57	1
3	46	1
4	63	1
5	76	1
6	99	1
7	91	1
8	81	0
9	69	0
10	74	0
11	61	0
12	56	0
13	87	0
14	69	0
15	65	0
16	66	0
17	44	0
18	62	0
19	69	0

If we then take a look at the average scores for each group of sneetches, we will see that there's a difference in scores of 6.6 between the two groups. So, on average, the star-bellied sneetches performed better on their tests than the plain-bellied sneetches. But, the real question is, is this a significant difference?

In [8]:

star_bellied_mean = df[df.star == 1].score.mean()
plain_bellied_mean = df[df.star == 0].score.mean()

print("Star-bellied Sneetches Mean: %2.1f" % star_bellied_mean)
print("Plain-bellied Sneetches Mean: %2.1f" % plain_bellied_mean)
print("Difference: %2.1f" % (star_bellied_mean - plain_bellied_mean))

Star-bellied Sneetches Mean: 73.5
Plain-bellied Sneetches Mean: 66.9
Difference: 6.6

To determine if this is a signficant difference, we could perform a t-test on our data to compute a p-value, and then just make sure that the p-value is less than the target 0.05. Alternatively, we could use simulation instead.

Unlike our first example, however, we don't have a generative function that we can use to create our probability distribution. So, how can we then use simulation to solve our problem?

Well, we can run a bunch of simulations where we randomly shuffle the labels (i.e., star-bellied or plain-bellied) of each sneetch, recompute the difference between the means, and then determine if the proportion of simulations in which the difference was at least as extreme as 6.6 was less than the target 5%. If so, we can conclude that the difference we see is, in fact, one that doesn't occur strictly by chance very often and so the difference is a significant one. In other words, if the proportion of simulations that have a difference of 6.6 or greater is less than 5%, we can conclude that the labels really do matter, and so we can conclude that star-bellied sneetches are "better" than their plain-bellied counterparts.

In [9]:

df['label'] = df['star']

num_simulations = 10000

differences = []
for i in range(num_simulations):
    np.random.shuffle(df['label'])
    star_bellied_mean = df[df.label == 1].score.mean()
    plain_bellied_mean = df[df.label == 0].score.mean()
    differences.append(star_bellied_mean - plain_bellied_mean)

Now that we've ran our simulations, we can calculate our p-value, which is simply the proportion of simulations that resulted in a difference greater than or equal to 6.6.

$$ p = \frac{N_{>6.6}}{N_{total}} = \frac{1512}{10000} = 0.15 $$

In [10]:

p_value = sum(diff >= 6.6 for diff in differences) / num_simulations
print("p-value: %2.2f" % p_value)

p-value: 0.16

The following code plots the distribution of the differences we found by running the simulations above. We've also added an annotation that marks where the difference of 6.6 falls in the distribution along with its corresponding p-value.

In [11]:

plt.hist(differences, bins=50, color=blue)
plt.xlabel('score difference')
plt.ylabel('number')
plt.plot((6.6, 6.6), (0, 700), color=red);
plt.annotate('%2.f%%' % (p_value * 100), 
             xytext=(15, 350), 
             xy=(6.6, 350), 
             multialignment='right',
             va='center',
             color=red,
             size='large',
             arrowprops={'arrowstyle': '<|-', 
                         'lw': 2, 
                         'color': red, 
                         'shrinkA': 10});

We can see from the histogram above---and from our simulated p-value, which was greater than 5%---that the difference that we are seeing between the populations can be explained by random chance, so we can effectively dismiss the difference as not statistically significant. In short, star-bellied sneetches are no better than the plain-bellied ones, at least not from a statistical point of view.

For further discussion on this method of simulation, check out John Rauser's keynote talk "Statistics Without the Agonizing Pain" from Strata + Hadoop 2014. Jake mentions that he drew inspiration from it in his talk, and it is a really excellent talk as well; I wholeheartedly recommend it.

Bootstrapping¶

In this example, we'll be using the story of Yertle the Turtle to explore the bootstrapping recipe. As the story goes, in the land of Sala-ma-Sond, Yertle the Turtle was the king of the pond and he wanted to be the most powerful, highest turtle in the land. To achieve this goal, he would stack turtles as high as he could in order to stand upon their backs. As observers of this curious behavior, we've recorded the heights of 20 turtle towers and we've placed them in a dataframe in the following bit of code.

In [12]:

df = pd.DataFrame({'heights': [48, 24, 51, 12, 21, 
                               41, 25, 23, 32, 61, 
                               19, 24, 29, 21, 23, 
                               13, 32, 18, 42, 18]})

The questions we want to answer in this example are: what is the mean height of Yertle's turtle stacks, and what is the uncertainty of this estimate?

The Classic Method¶

The classic method is simply to calculate the sample mean...

$$ \bar{x} = \frac{1}{N} \sum_{i=1}^{N} x_i = 28.9 $$

...and the standard error of the mean.

$$ \sigma_{\bar{x}} = \frac{1}{ \sqrt{N}}\sqrt{\frac{1}{N - 1} \sum_{i=1}^{N} (x_i - \bar{x})^2 } = 3.0 $$

But, being hackers, we'll be using simulation instead.

Just like in our last example, we are once again faced with the problem of not having a generative model, but unlike the last example, we're not comparing two groups, so we can't just shuffle around labels here, instead we'll use something called bootstrap resampling.

Bootstrap resampling is a method that simulates several random sample distributions by drawing samples from the current distribution with replacement, i.e., we can draw the same data point more than once. Luckily, pandas makes this super easy with its sample function. We simply need to make sure that we pass in True for the replace argument to sample from our dataset with replacement.

In [13]:

sample = df.sample(20, replace=True)
display(sample)
print("Mean: %2.2f" % sample.heights.mean())
print("Standard Error: %2.2f" % (sample.heights.std() / np.sqrt(len(sample))))

	heights
1	24
9	61
10	19
15	13
7	23
1	24
17	18
9	61
3	12
15	13
11	24
3	12
7	23
11	24
9	61
14	23
14	23
10	19
15	13
7	23

Mean: 25.65
Standard Error: 3.55

More than likely the mean and standard error from our freshly drawn sample above didn't exactly match the one that we calculated using the classic method beforehand. But, if we continue to resample several thousand times and take a look at the average (mean) of all those sample means and their standard deviation, we should have something that very closely approximates the mean and standard error derived from using the classic method above.

In [14]:

xbar = []
for i in range(10000):
    sample = df.sample(20, replace=True)
    xbar.append(sample.heights.mean())
    

print("Mean: %2.1f" % np.mean(xbar))
print("Standard Error: %2.1f" % np.std(xbar))

Mean: 28.8
Standard Error: 2.9

Cross Validation¶

For the final example, we dive into the world of the Lorax. In the story of the Lorax, a faceless creature sales an item that (presumably) all creatures need called a Thneed. Our job as consultants to Onceler Industries is to project Thneed sales. But, before we can get started forecasting the sales of Thneeds, we'll first need some data.

Lucky for you, I've already done the hard work of assembling that data in the code below by "eyeballing" the data in the scatter plot from the slides of the talk. So, it may not be exactly the same, but it should be close enough for our example analysis.

In [15]:

df = pd.DataFrame({
    'temp': [22, 36, 36, 38, 44, 45, 47,
             43, 44, 45, 47, 49,
             52, 53, 53, 53, 54, 55, 55, 55, 56, 57, 58, 59,
             60, 61, 61.5, 61.7, 61.7, 61.7, 61.8, 62, 62, 63.4, 64.6,
             65, 65.6, 65.6, 66.4, 66.9, 67, 67, 67.4, 67.5, 68, 69, 
             70, 71, 71, 71.5, 72, 72, 72, 72.7, 73, 73, 73, 73.3, 74, 75, 75, 
             77, 77, 77, 77.4, 77.9, 78, 78, 79,
             80, 82, 83, 84, 85, 85, 86, 87, 88,
             90, 90, 91, 93, 95, 97,
             102, 104],
    'sales': [660, 433, 475, 492, 302, 345, 337,
              479, 456, 440, 423, 269,
              331, 197, 283, 351, 470, 252, 278, 350, 253, 253, 343, 280,
              200, 194, 188, 171, 204, 266, 275, 171, 282, 218, 226, 
              187, 184, 192, 167, 136, 149, 168, 218, 298, 199, 268,
              235, 157, 196, 203, 148, 157, 213, 173, 145, 184, 226, 204, 250, 102, 176,
              97, 138, 226, 35, 190, 221, 95, 211,
              110, 150, 152, 37, 76, 56, 51, 27, 82,
              100, 123, 145, 51, 156, 99,
              147, 54]
})

Now that we have our sales data in a pandas dataframe, we can take a look to see if any trends show up. Plotting the data in a scatterplot, like the one below, reveals that a relationship does seem to exist between temperature and Thneed sales.

In [16]:

# Grab a reference to fig and axes object so we can reuse them
fig, ax = plt.subplots()

# Plot the Thneed sales data
ax.scatter(df.temp, df.sales)
ax.set_xlim(xmin=20, xmax=110)
ax.set_ylim(ymin=0, ymax=700)
ax.set_xlabel('temperature (F)')
ax.set_ylabel('thneed sales (daily)');

We can see what looks like a relationship between the two variables temperature and sales, but how can we best model that relationship so we can accurately predict sales based on temperature?

Well, one measure of a model's accuracy is the Root-Mean-Square Error (RMSE). This metric represents the sample standard deviation between a set of predicted values (from our model) and the actual observed values.

In [17]:

def rmse(predictions, targets):
    return np.sqrt(((predictions - targets)**2).mean())

We can now use our rmse function to measure how well our models' accurately represent the Thneed sales dataset. And, in the next cell, we'll give it a try by creating two different models and seeing which one does a better job of fitting our sales data.

In [18]:

# 1D Polynomial Fit
d1_model = np.poly1d(np.polyfit(df.temp, df.sales, 1))
d1_predictions = d1_model(range(111))
ax.plot(range(111), d1_predictions, 
        color=blue, alpha=0.7)

# 2D Polynomial Fit
d2_model = np.poly1d(np.polyfit(df.temp, df.sales, 2))
d2_predictions = d2_model(range(111))
ax.plot(range(111), d2_predictions, 
        color=red, alpha=0.5)

ax.annotate('RMS error = %2.1f' % rmse(d1_model(df.temp), df.sales),
             xy=(75, 650),
             fontsize=20,
             color=blue,
             backgroundcolor='w')

ax.annotate('RMS error = %2.1f' % rmse(d2_model(df.temp), df.sales),
             xy=(75, 580),
             fontsize=20,
             color=red,
             backgroundcolor='w')

display(fig);

In the figure above, we plotted our sales data along with the two models we created in the previous step. The first model (in blue) is a simple linear model, i.e., a first-degree polynomial. The second model (in red) is a second-degree polynomial, so rather than a straight line, we end up with a slight curve.

We can see from the RMSE values in the figure above that the second-degree polynomial performed better than the simple linear model. Of course, the question you should now be asking is, is this the best possible model that we can find?

To find out, let's take a look at the RMSE of a few more models to see if we can do any better.

In [19]:

rmses = []
for deg in range(15):
    model = np.poly1d(np.polyfit(df.temp, df.sales, deg))
    predictions = model(df.temp)
    rmses.append(rmse(predictions, df.sales))
    
plt.plot(range(15), rmses)
plt.ylim(45, 70)
plt.xlabel('number of terms in fit')
plt.ylabel('rms error')

plt.annotate('$y = a + bx$', 
             xytext=(14.2, 70), 
             xy=(1, rmses[1]), 
             multialignment='right',
             va='center',
             arrowprops={'arrowstyle': '-|>',
                         'lw': 1,
                         'shrinkA': 10,
                         'shrinkB': 3})

plt.annotate('$y = a + bx + cx^2$', 
             xytext=(14.2, 64), 
             xy=(2, rmses[2]), 
             multialignment='right',
             va='top',
             arrowprops={'arrowstyle': '-|>',
                         'lw': 1,
                         'shrinkA': 35,
                         'shrinkB': 3})


plt.annotate('$y = a + bx + cx^2 + dx^3$', 
             xytext=(14.2, 58), 
             xy=(3, rmses[3]), 
             multialignment='right',
             va='top',
             arrowprops={'arrowstyle': '-|>',
                         'lw': 1,
                         'shrinkA': 12,
                         'shrinkB': 3});

We can see, from the plot above, that as we increase the number of terms (i.e., the degrees of freedom) in our model we decrease the RMSE, and this behavior can continue indefinitely, or until we have as many terms as we do data points, at which point we would be fitting the data perfectly.

The problem with this approach though, is that as we increase the number of terms in our equation, we simply match the given dataset closer and closer, but what if our model were to see a data point that's not in our training dataset?

As you can see in the plot below, the model that we've created, though it has a very low RMSE, it has so many terms that it matches our current dataset too closely.

In [20]:

# Remove everything but the datapoints
ax.lines.clear()
ax.texts.clear()

# Changing the y-axis limits to match the figure in the slides
ax.set_ylim(0, 1000)

# 14 Dimensional Model
model = np.poly1d(np.polyfit(df.temp, df.sales, 14))
ax.plot(range(20, 110), model(range(20, 110)),
         color=sns.xkcd_rgb['sky blue'])

display(fig)

The problem with fitting the data too closely, is that our model is so finely tuned to our specific dataset, that if we were to use it to predict future sales, it would most likely fail to get very close to the actual value. This phenomenon of too closely modeling the training dataset is well known amongst machine learning practitioners as overfitting and one way that we can avoid it is to use cross-validation.

Cross-validation avoids overfitting by splitting the training dataset into several subsets and using each one to train and test multiple models. Then, the RMSE's of each of those models are averaged to give a more likely estimate of how a model of that type would perform on unseen data.

So, let's give it a try by splitting our data into two groups and randomly assigning data points into each one.

In [21]:

df_a = df.sample(n=len(df)//2)
df_b = df.drop(df_a.index)

We can get a look at the data points assigned to each subset by plotting each one as a different color.

In [22]:

plt.scatter(df_a.temp, df_a.sales, color='red')
plt.scatter(df_b.temp, df_b.sales, color='blue')
plt.xlim(0, 110)
plt.ylim(0, 700)
plt.xlabel('temprature (F)')
plt.ylabel('thneed sales (daily)');

Then, we'll find the best model for each subset of data. In this particular example, we'll fit a second-degree polynomial to each subset and plot both below.

In [23]:

# Create a 2-degree model for each subset of data
m1 = np.poly1d(np.polyfit(df_a.temp, df_a.sales, 2))
m2 = np.poly1d(np.polyfit(df_b.temp, df_b.sales, 2))

fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, 
                               sharex=False, sharey=True,
                               figsize=(12, 5))

x_min, x_max = 20, 110
y_min, y_max = 0, 700
x = range(x_min, x_max + 1)

# Plot the df_a group
ax1.scatter(df_a.temp, df_a.sales, color='red')
ax1.set_xlim(xmin=x_min, xmax=x_max)
ax1.set_ylim(ymin=y_min, ymax=y_max)
ax1.set_xlabel('temprature (F)')
ax1.set_ylabel('thneed sales (daily)')
ax1.plot(x, m1(x),
         color=sns.xkcd_rgb['sky blue'],
         alpha=0.7)

# Plot the df_b group
ax2.scatter(df_b.temp, df_b.sales, color='blue')
ax2.set_xlim(xmin=x_min, xmax=x_max)
ax2.set_ylim(ymin=y_min, ymax=y_max)
ax2.set_xlabel('temprature (F)')
ax2.plot(x, m2(x),
         color=sns.xkcd_rgb['rose'], 
         alpha=0.5);

Finally, we'll compare models across subsets by calculating the RMSE for each model using the training set for the other model. This will give us two RMSE scores which we'll then average to get a more accurate estimate of how well a second-degree polynomial will perform on any unseen data.

In [24]:

print("RMS = %2.1f" % rmse(m1(df_b.temp), df_b.sales))
print("RMS = %2.1f" % rmse(m2(df_a.temp), df_a.sales))
print("RMS estimate = %2.1f" % np.mean([rmse(m1(df_b.temp), df_b.sales),
                                        rmse(m2(df_a.temp), df_a.sales)]))

RMS = 50.5
RMS = 57.8
RMS estimate = 54.1

Then, we simply repeat this process for as long as we so desire.

The following code repeats the process described above for polynomials up to 14 degrees and plots the average RMSE for each one against the non-cross-validated RMSE's that we calculated earlier.

In [25]:

rmses = []
cross_validated_rmses = []
for deg in range(15):
    # df_a the model on the whole dataset and calculate its
    # RMSE on the same set of data
    model = np.poly1d(np.polyfit(df.temp, df.sales, deg))
    predictions = model(df.temp)
    rmses.append(rmse(predictions, df.sales))
    
    # Use cross-validation to create the model and df_a it
    m1 = np.poly1d(np.polyfit(df_a.temp, df_a.sales, deg))
    m2 = np.poly1d(np.polyfit(df_b.temp, df_b.sales, deg))
    
    p1 = m1(df_b.temp)
    p2 = m2(df_a.temp)
    
    cross_validated_rmses.append(np.mean([rmse(p1, df_b.sales), 
                                          rmse(p2, df_a.sales)]))

    
plt.plot(range(15), rmses, color=blue, 
         label='RMS')
plt.plot(range(15), cross_validated_rmses, color=red, 
         label='cross validated RMS')
plt.ylim(45, 70)
plt.xlabel('number of terms in fit')
plt.ylabel('rms error')
plt.legend(frameon=True)

plt.annotate('Best model minimizes the\ncross-validated error.', 
             xytext=(7, 60), 
             xy=(2, cross_validated_rmses[2]), 
             multialignment='center',
             va='top',
             color='blue',
             size=25,
             backgroundcolor='w',
             arrowprops={'arrowstyle': '-|>',
                         'lw': 3,
                         'shrinkA': 12,
                         'shrinkB': 3,
                         'color': 'blue'});

According to the graph above, going from a 1-degree to a 2-degree polynomial gives us quite a large improvement overall. But, unlike the RMSE that we calculated against the training set, when using cross-validation we can see that adding more degrees of freedom to our equation quickly reduces the effectiveness of the model against unseen data. This is overfitting in action! In fact, from the looks of the graph above, it would seem that a second-degree polynomial is actually our best bet for this particular dataset.

2-Fold Cross-Validation¶

Several different methods for performing cross-validation exist, the one we've just seen is called 2-fold cross-validation since the data is split into two subsets. Another close relative is a method called $k$-fold cross-validation. It differs slightly in that the original dataset is divided into $k$ subsets (instead of just 2), one of which is reserved strictly for testing and the other $k - 1$ subsets are used for training models. This is just one example of an alternate cross-validation method, but more do exist and each one has advantages and drawbacks that you'll need to consider when deciding which method to use.

Conclusion¶

Ok, so that covers nearly everything that Jake covered in his talk. The end of the talk contains a short overview of some important areas that he didn't have time to cover, and there's a nice set of Q&A at the end, but I'll simply direct you to the video for those parts of the talk. Hopefully, this article/notebook has been helpful to anyone working their way through Jake's talk, and if for some reason, you've read through this entire article and haven't watched the video of the talk yet, I encourage you to take 40 minutes out of your day and go watch it now---it really is a fantastic talk!

Why is Programming in Node So Different?

2013-03-24T00:00:00-07:00

I have a series of screencasts on Node.js that I've been doing now for quite some time through Nettuts+, and I recently received a question in reaction to some of the code in my latest episode. The code in question is pretty typical for anyone familiar with Node, but can cause some serious looks of consternation from anyone new to Node, especially if they have any experience with more traditional web frameworks and languages like Rails, Django, or PHP. The code that caused this question, and is thus the catalyst for this blog post, can be seen below:

function parseBody (req, callback) {
    var body = '';
    req.on('data', function(chunk) {
        body += chunk;
    });
    req.on('end', function() {
        callback(qs.parse(body));
    })
}

The code above shows the standard way to get form data from a POST request. The code registers two callback functions: one with the data event and the other with the end event. The data event is fired every time a new chunk of data associated with that particular request is received from the network. The callback, in this case, simply appends the new chunk to a buffer variable. The end event fires once all chunks for a request have been received, and at this point, our callback simply calls another callback function that was passed into our function, and it passes to it the full request body. Now all this probably sounds pretty complex, so it's no wonder that people new to Node find it somewhat odd and disconcerting. The reason it tends to cause so much confusion amongst those new to Node is that it seems to be an unnecessary over-complication of what is a relatively simple task--getting the data from a simple HTTP POST request. The problem with that statement is that there really is no such thing as a "simple" HTTP POST/GET operation in Node.

If you've used other, more traditional languages and frameworks, like I mentioned above, you're probably expecting to be able to do something like request.POST['name'] to get at the data from a POST request. Well, in those frameworks you can do that due to the fact that they're using a thread to handle each HTTP request. The reason that each request gets its very own thread is that HTTP is inherently asynchronous.

HTTP sits atop a packet-based communication protocol stack that, in turn, sits atop a dynamic network. This means that whenever you hit the Submit in a form in your browser, the data in that form is collected and divvied up into chunks of data (packets) and sent across the network piecemeal. Due to the dynamic nature of the network, these packets, at the lower levels, can be lost, sent in duplicate, or out of order. The higher level protocols take on the task of making sure all of the packets for a request are eventually received and stitched back together in the correct order. However, due to the non-deterministic manner in which these packets can arrive, it is impossible to know when, in what order, or even if, all of the packets that make up a single request will be delivered to the server. As a result, the server must be able to just wait around for all of the packets to come in, and there are two methods that we have devised for doing just that.

The first is to assign a thread to each HTTP connection, and this is what the traditional frameworks like Django and Rails do. Doing so allows a single connection to "wait" around for the remaining packets without blocking the server application from taking on new connections. The nice part about this is that it allows you to program as you would normally, i.e., with calls that block, ordered one after the other. The problem with this method, is that too many threads can quickly bog down your server and exhaust all of its available resources and essentially cause your application to become very slow and possibly even crash.

The second solution is to use an event-oriented mechanism and this is what Node is doing. In this scenario, there is only one thread (this is vastly oversimplified, but for our purposes this definition will suffice) that runs in a loop and calls functions--callbacks that we register with specific events--whenever an event occurs. Doing this allows all of the connections to be interlaced (i.e., multiplexed) onto a single thread. Each connection registers a callback for an event with the main loop and then exits--it's essentially done executing until the event it registered its callback with occurs. The beauty of this is that it doesn't block the main loop, instead it can continue to look for new events, call callbacks, and take new callback registrations. The advantage of this method is that you don't have a ton of threads bogging down the server, just a small set of threads in a thread pool for handling everything. In other words, it's extremely efficient. The downside of this approach though is that, the way you program it is a little unintuitive, as you've seen with the example code above. Also, the developer needs to be more aware of computationally expensive code since it could bring the main loop to a complete stop, blocking all other requests.

I want to delve into that last statement a bit further. By computationally expensive, I mean something that will keep the CPU engaged. While reading a large file from a disk can take a while, it's not computationally expensive since the CPU basically does nothing while a file is being read. The task of reading a file belongs to the disk drive hardware and the CPU only needs to do something once the drive completes its task and notifies the CPU of the completion. In the meantime, it could be off doing something else. An example of a computationally expensive task would be something more like performing a Fibonacci calculation, as this can be a very long task and it takes place solely in the CPU. The CPU must loop (or recurse), constantly calculating this number. So, imagine you receive an HTTP request that asks for the 1000th Fibonacci number (contrived, I know, but bear with me). This is going to take quite a while to complete. In the traditional example that's fine, as all the computation will take place on a separate thread that was created specifically for handling that HTTP request. In the meantime, the system will continue to accept and process new connections on a completely different thread.

In the event-oriented scenario though, this same request would bring the main loop to a screeching halt. In this scenario, the request would come into the main loop and it would call the callback function associated with receiving new requests. If that callback function was just reading a huge file, everything would be just fine since it could just register another callback function with the event for the "end of the file read" event and then exit returning control back to the main loop and allowing the system to continue processing more requests. However, in the case of our Fibonacci example, the callback function would begin calculating the 1000th Fibonacci number and the main loop would be stuck since the callback, rather than exiting immediately, would be crunching through a long Fibonacci calculation. What that would mean is that all incoming requests would also be stuck waiting for the main loop to handle them which, in turn, is stuck waiting for the Fibonacci request to finish its calculation.

Hopefully from this explanation it's now easy (well, at least, hopefully possible) to see that, when dealing with Node, you simply cannot create functions that do any sort of task that can take a long, or indeterminate amount of time. If you are going to do so, then you have to write it in such a way that the calling function can simply register a callback with it and exit, returning control back to its calling function and eventually back to the main loop. What this means with respect to our example code is that we cannot simply call a function or property on our request object and expect to receive an object back representing the data for that request. Doing so would mean blocking our main loop while we wait for the entire request body to be received, and thereby, losing the ability to handle new requests. So, instead, we're stuck with what has been (affectionately?) called "callback hell". While certainly not the most intuitive way to write code, it does offer us the benefit of being able to write, relatively, simple servers that can handle extreme amounts of traffic with little or no performance degradation compared to the more traditional methods.

I hope that helps anyone having trouble understanding why Node seems to do things so weird compared to other technologies. It may be ugly and hard to understand at times, but once you understand the reason for it, you come to see that there is definitely a method to its madness.

My First Sublime Text Plugin

2013-02-22T00:00:00-08:00

I'm very happy to anounce that my very first Sublime Text plugin just went live a couple of days ago!

It's a simple little plugin that automatically shows and hides the tab bar depending on how many files you currently have open. If you've used Safari or TextMate then you're already familiar with how this feature works. If you'd like to see it in action, checkout the video below, otherwise, if you'd like to just go ahead and install it and give it a try, you can do so through Will Bond's fantastic Package Manager by just searching for Hide Tabs, or you can download and install it yourself from its GitHub page. One more thing, the plugin was created to work with both Sublime Text 2 and new beta version of 3, so feel free to use it with either.

I hope you like it.

	score	star
0	84	1
1	72	1
2	57	1
3	46	1
4	63	1
5	76	1
6	99	1
7	91	1
8	81	0
9	69	0
10	74	0
11	61	0
12	56	0
13	87	0
14	69	0
15	65	0
16	66	0
17	44	0
18	62	0
19	69	0

	heights
1	24
9	61
10	19
15	13
7	23
1	24
17	18
9	61
3	12
15	13
11	24
3	12
7	23
11	24
9	61
14	23
14	23
10	19
15	13
7	23

	score	star
0	84	1
1	72	1
2	57	1
3	46	1
4	63	1
5	76	1
6	99	1
7	91	1
8	81	0
9	69	0
10	74	0
11	61	0
12	56	0
13	87	0
14	69	0
15	65	0
16	66	0
17	44	0
18	62	0
19	69	0

	heights
1	24
9	61
10	19
15	13
7	23
1	24
17	18
9	61
3	12
15	13
11	24
3	12
7	23
11	24
9	61
14	23
14	23
10	19
15	13
7	23

	score	star
0	84	1
1	72	1
2	57	1
3	46	1
4	63	1
5	76	1
6	99	1
7	91	1
8	81	0
9	69	0
10	74	0
11	61	0
12	56	0
13	87	0
14	69	0
15	65	0
16	66	0
17	44	0
18	62	0
19	69	0

	heights
1	24
9	61
10	19
15	13
7	23
1	24
17	18
9	61
3	12
15	13
11	24
3	12
7	23
11	24
9	61
14	23
14	23
10	19
15	13
7	23