{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "![](images/EscUpmPolit_p.gif \"UPM\")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "# Course Notes for Learning Intelligent Systems" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "Department of Telematic Engineering Systems, Universidad Politécnica de Madrid, © Carlos A. Iglesias" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "## [Introduction to Preprocessing](00_Intro_Preprocessing.ipynb)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Duplicated values\n", "\n", "Sometimes, data comes with messy data. \n", "\n", "We will use the package [dedupe](https://dedupe.io/) to eliminate duplicates. \n", "\n", "\n", "Some alternatives are the packages [recordlinkage](https://pypi.org/project/recordlinkage/) and [thefuzz](https://github.com/seatgeek/thefuzz).\n", "\n", "Instead of using directly the package dedupe, we are going to use **pandas-dedupe**:\n", "\n", "\n", "**pip install pandas-dedupe**\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "\n", "Let's start by loading messy data." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "import warnings\n", "warnings.filterwarnings('ignore') # Avoid warnings\n", "\n", "import pandas as pd\n", "import numpy as np\n", "import pandas_dedupe" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "df = pd.read_csv('https://raw.githubusercontent.com/dedupeio/dedupe-examples/master/csv_example/csv_example_messy_input.csv')" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Let's do some initial checking" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "(3337, 32)" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.shape" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IdSourceSite nameAddressZipPhoneFaxProgram NameLength of DayIDHS Provider ID...Executive DirectorCenter DirectorECE Available ProgramsNAEYC Valid UntilNAEYC Program IdEmail AddressOunce of Prevention DescriptionPurple binder service typeColumnColumn2
00CPS_Early_Childhood_Portal_scrape.csvSalvation Army - Temple / Salvation Army1 N Ogden AveNaN2262649.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
11CPS_Early_Childhood_Portal_scrape.csvSalvation Army - Temple / Salvation Army1 N Ogden AveNaN2262649.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
22CPS_Early_Childhood_Portal_scrape.csvNational Louis University - Dr. Effie O. Elli...10 S Kedzie AveNaN5339011.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
33CPS_Early_Childhood_Portal_scrape.csvNational Louis University - Dr. Effie O. Elli...10 S Kedzie AveNaN5339011.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
44CPS_Early_Childhood_Portal_scrape.csvBoard Trustees-City Colleges of Chicago - Oli...10001 S Woodlawn AveNaN2916100.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
55CPS_Early_Childhood_Portal_scrape.csvBoard Trustees-City Colleges of Chicago - Oli...10001 S Woodlawn AveNaN2916100.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
66CPS_Early_Childhood_Portal_scrape.csvEaster Seals Society of Metropolitan Chicago ...1001 W Roosevelt RdNaN9395115.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
77CPS_Early_Childhood_Portal_scrape.csvEaster Seals Society of Metropolitan Chicago ...1001 W Roosevelt RdNaN9395115.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
88CPS_Early_Childhood_Portal_scrape.csvHull House Association - Uptown Head Start / ...1020 W Bryn Mawr AveNaN7695753.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
99CPS_Early_Childhood_Portal_scrape.csvHull House Association - Child Dev. Central O...1030 W Van Buren StNaN9068600.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
\n", "

10 rows × 32 columns

\n", "
" ], "text/plain": [ " Id Source \\\n", "0 0 CPS_Early_Childhood_Portal_scrape.csv \n", "1 1 CPS_Early_Childhood_Portal_scrape.csv \n", "2 2 CPS_Early_Childhood_Portal_scrape.csv \n", "3 3 CPS_Early_Childhood_Portal_scrape.csv \n", "4 4 CPS_Early_Childhood_Portal_scrape.csv \n", "5 5 CPS_Early_Childhood_Portal_scrape.csv \n", "6 6 CPS_Early_Childhood_Portal_scrape.csv \n", "7 7 CPS_Early_Childhood_Portal_scrape.csv \n", "8 8 CPS_Early_Childhood_Portal_scrape.csv \n", "9 9 CPS_Early_Childhood_Portal_scrape.csv \n", "\n", " Site name Address \\\n", "0 Salvation Army - Temple / Salvation Army 1 N Ogden Ave \n", "1 Salvation Army - Temple / Salvation Army 1 N Ogden Ave \n", "2 National Louis University - Dr. Effie O. Elli... 10 S Kedzie Ave \n", "3 National Louis University - Dr. Effie O. Elli... 10 S Kedzie Ave \n", "4 Board Trustees-City Colleges of Chicago - Oli... 10001 S Woodlawn Ave \n", "5 Board Trustees-City Colleges of Chicago - Oli... 10001 S Woodlawn Ave \n", "6 Easter Seals Society of Metropolitan Chicago ... 1001 W Roosevelt Rd \n", "7 Easter Seals Society of Metropolitan Chicago ... 1001 W Roosevelt Rd \n", "8 Hull House Association - Uptown Head Start / ... 1020 W Bryn Mawr Ave \n", "9 Hull House Association - Child Dev. Central O... 1030 W Van Buren St \n", "\n", " Zip Phone Fax Program Name Length of Day IDHS Provider ID ... \\\n", "0 NaN 2262649.0 NaN Child Care EXTENDED DAY NaN ... \n", "1 NaN 2262649.0 NaN Child Care EXTENDED DAY NaN ... \n", "2 NaN 5339011.0 NaN Child Care EXTENDED DAY NaN ... \n", "3 NaN 5339011.0 NaN Child Care EXTENDED DAY NaN ... \n", "4 NaN 2916100.0 NaN Child Care EXTENDED DAY NaN ... \n", "5 NaN 2916100.0 NaN Child Care EXTENDED DAY NaN ... \n", "6 NaN 9395115.0 NaN Child Care EXTENDED DAY NaN ... \n", "7 NaN 9395115.0 NaN Child Care EXTENDED DAY NaN ... \n", "8 NaN 7695753.0 NaN Child Care EXTENDED DAY NaN ... \n", "9 NaN 9068600.0 NaN Child Care EXTENDED DAY NaN ... \n", "\n", " Executive Director Center Director ECE Available Programs NAEYC Valid Until \\\n", "0 NaN NaN NaN NaN \n", "1 NaN NaN NaN NaN \n", "2 NaN NaN NaN NaN \n", "3 NaN NaN NaN NaN \n", "4 NaN NaN NaN NaN \n", "5 NaN NaN NaN NaN \n", "6 NaN NaN NaN NaN \n", "7 NaN NaN NaN NaN \n", "8 NaN NaN NaN NaN \n", "9 NaN NaN NaN NaN \n", "\n", " NAEYC Program Id Email Address Ounce of Prevention Description \\\n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "2 NaN NaN NaN \n", "3 NaN NaN NaN \n", "4 NaN NaN NaN \n", "5 NaN NaN NaN \n", "6 NaN NaN NaN \n", "7 NaN NaN NaN \n", "8 NaN NaN NaN \n", "9 NaN NaN NaN \n", "\n", " Purple binder service type Column Column2 \n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "2 NaN NaN NaN \n", "3 NaN NaN NaN \n", "4 NaN NaN NaN \n", "5 NaN NaN NaN \n", "6 NaN NaN NaN \n", "7 NaN NaN NaN \n", "8 NaN NaN NaN \n", "9 NaN NaN NaN \n", "\n", "[10 rows x 32 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head(10)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Index(['Id', 'Source', 'Site name', 'Address', 'Zip', 'Phone', 'Fax',\n", " 'Program Name', 'Length of Day', 'IDHS Provider ID', 'Agency',\n", " 'Neighborhood', 'Funded Enrollment', 'Program Option',\n", " 'Number per Site EHS', 'Number per Site HS', 'Director',\n", " 'Head Start Fund', 'Eearly Head Start Fund', 'CC fund', 'Progmod',\n", " 'Website', 'Executive Director', 'Center Director',\n", " 'ECE Available Programs', 'NAEYC Valid Until', 'NAEYC Program Id',\n", " 'Email Address', 'Ounce of Prevention Description',\n", " 'Purple binder service type', 'Column', 'Column2'],\n", " dtype='object')\n" ] } ], "source": [ "print(df.columns)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Id int64\n", "Source object\n", "Site name object\n", "Address object\n", "Zip float64\n", "Phone float64\n", "Fax object\n", "Program Name object\n", "Length of Day object\n", "IDHS Provider ID object\n", "Agency object\n", "Neighborhood object\n", "Funded Enrollment object\n", "Program Option object\n", "Number per Site EHS object\n", "Number per Site HS object\n", "Director float64\n", "Head Start Fund float64\n", "Eearly Head Start Fund object\n", "CC fund object\n", "Progmod object\n", "Website object\n", "Executive Director object\n", "Center Director object\n", "ECE Available Programs object\n", "NAEYC Valid Until object\n", "NAEYC Program Id float64\n", "Email Address object\n", "Ounce of Prevention Description object\n", "Purple binder service type object\n", "Column float64\n", "Column2 object\n", "dtype: object\n" ] } ], "source": [ "print(df.dtypes)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/plain": [ "Id 0\n", "Source 0\n", "Site name 0\n", "Address 0\n", "Zip 1333\n", "Phone 146\n", "Fax 3299\n", "Program Name 2009\n", "Length of Day 2009\n", "IDHS Provider ID 3298\n", "Agency 3325\n", "Neighborhood 2754\n", "Funded Enrollment 2424\n", "Program Option 2800\n", "Number per Site EHS 3319\n", "Number per Site HS 3319\n", "Director 3337\n", "Head Start Fund 3337\n", "Eearly Head Start Fund 2881\n", "CC fund 2818\n", "Progmod 2818\n", "Website 2815\n", "Executive Director 3114\n", "Center Director 2874\n", "ECE Available Programs 2379\n", "NAEYC Valid Until 2968\n", "NAEYC Program Id 3337\n", "Email Address 3203\n", "Ounce of Prevention Description 3185\n", "Purple binder service type 3215\n", "Column 3337\n", "Column2 3018\n", "dtype: int64" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Missing values\n", "df.isnull().sum()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Check duplicates" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "0 False\n", "1 False\n", "2 False\n", "3 False\n", "4 False\n", " ... \n", "3332 False\n", "3333 False\n", "3334 False\n", "3335 False\n", "3336 False\n", "Length: 3337, dtype: bool" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.duplicated()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Remove duplicates" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IdSourceSite nameAddressZipPhoneFaxProgram NameLength of DayIDHS Provider ID...Executive DirectorCenter DirectorECE Available ProgramsNAEYC Valid UntilNAEYC Program IdEmail AddressOunce of Prevention DescriptionPurple binder service typeColumnColumn2
00CPS_Early_Childhood_Portal_scrape.csvSalvation Army - Temple / Salvation Army1 N Ogden AveNaN2262649.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
11CPS_Early_Childhood_Portal_scrape.csvSalvation Army - Temple / Salvation Army1 N Ogden AveNaN2262649.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
22CPS_Early_Childhood_Portal_scrape.csvNational Louis University - Dr. Effie O. Elli...10 S Kedzie AveNaN5339011.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
33CPS_Early_Childhood_Portal_scrape.csvNational Louis University - Dr. Effie O. Elli...10 S Kedzie AveNaN5339011.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
44CPS_Early_Childhood_Portal_scrape.csvBoard Trustees-City Colleges of Chicago - Oli...10001 S Woodlawn AveNaN2916100.0NaNChild CareEXTENDED DAYNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
\n", "

5 rows × 32 columns

\n", "
" ], "text/plain": [ " Id Source \\\n", "0 0 CPS_Early_Childhood_Portal_scrape.csv \n", "1 1 CPS_Early_Childhood_Portal_scrape.csv \n", "2 2 CPS_Early_Childhood_Portal_scrape.csv \n", "3 3 CPS_Early_Childhood_Portal_scrape.csv \n", "4 4 CPS_Early_Childhood_Portal_scrape.csv \n", "\n", " Site name Address \\\n", "0 Salvation Army - Temple / Salvation Army 1 N Ogden Ave \n", "1 Salvation Army - Temple / Salvation Army 1 N Ogden Ave \n", "2 National Louis University - Dr. Effie O. Elli... 10 S Kedzie Ave \n", "3 National Louis University - Dr. Effie O. Elli... 10 S Kedzie Ave \n", "4 Board Trustees-City Colleges of Chicago - Oli... 10001 S Woodlawn Ave \n", "\n", " Zip Phone Fax Program Name Length of Day IDHS Provider ID ... \\\n", "0 NaN 2262649.0 NaN Child Care EXTENDED DAY NaN ... \n", "1 NaN 2262649.0 NaN Child Care EXTENDED DAY NaN ... \n", "2 NaN 5339011.0 NaN Child Care EXTENDED DAY NaN ... \n", "3 NaN 5339011.0 NaN Child Care EXTENDED DAY NaN ... \n", "4 NaN 2916100.0 NaN Child Care EXTENDED DAY NaN ... \n", "\n", " Executive Director Center Director ECE Available Programs NAEYC Valid Until \\\n", "0 NaN NaN NaN NaN \n", "1 NaN NaN NaN NaN \n", "2 NaN NaN NaN NaN \n", "3 NaN NaN NaN NaN \n", "4 NaN NaN NaN NaN \n", "\n", " NAEYC Program Id Email Address Ounce of Prevention Description \\\n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "2 NaN NaN NaN \n", "3 NaN NaN NaN \n", "4 NaN NaN NaN \n", "\n", " Purple binder service type Column Column2 \n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "2 NaN NaN NaN \n", "3 NaN NaN NaN \n", "4 NaN NaN NaN \n", "\n", "[5 rows x 32 columns]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.drop_duplicates(inplace=True)\n", "df[0:5]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Remove 'real duplicates'\n", "\n", "The problem is that the records are not the same. \n", "\n", "Data is messy. \n", "\n", "We will use **dedupe**." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "scrolled": true, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Importing data ...\n", "Reading from dedupe_dataframe_learned_settings\n", "Clustering...\n", "# duplicate sets 871\n" ] } ], "source": [ "# canonalize for standardizing names in a cluster\n", "df_dedupe = pandas_dedupe.dedupe_dataframe(df, ['Source', 'Site name', 'Address', 'Zip', 'Phone', 'Email Address'], canonicalize=True)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "If you want to retrain, you should delete the settings and training files (the dedupe* and link_dataframes* files).\n", "\n", "\n", "Now, if you inspect the dataframe, you will see the duplicated records that have been clustered." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index(['Id', 'Source', 'Site name', 'Address', 'Zip', 'Phone', 'Fax',\n", " 'Program Name', 'Length of Day', 'IDHS Provider ID', 'Agency',\n", " 'Neighborhood', 'Funded Enrollment', 'Program Option',\n", " 'Number per Site EHS', 'Number per Site HS', 'Director',\n", " 'Head Start Fund', 'Eearly Head Start Fund', 'CC fund', 'Progmod',\n", " 'Website', 'Executive Director', 'Center Director',\n", " 'ECE Available Programs', 'NAEYC Valid Until', 'NAEYC Program Id',\n", " 'Email Address', 'Ounce of Prevention Description',\n", " 'Purple binder service type', 'Column', 'Column2', 'cluster id',\n", " 'confidence', 'canonical_Id', 'canonical_Source', 'canonical_Site name',\n", " 'canonical_Address', 'canonical_Zip', 'canonical_Phone',\n", " 'canonical_Fax', 'canonical_Program Name', 'canonical_Length of Day',\n", " 'canonical_IDHS Provider ID', 'canonical_Agency',\n", " 'canonical_Neighborhood', 'canonical_Funded Enrollment',\n", " 'canonical_Program Option', 'canonical_Number per Site EHS',\n", " 'canonical_Number per Site HS', 'canonical_Director',\n", " 'canonical_Head Start Fund', 'canonical_Eearly Head Start Fund',\n", " 'canonical_CC fund', 'canonical_Progmod', 'canonical_Website',\n", " 'canonical_Executive Director', 'canonical_Center Director',\n", " 'canonical_ECE Available Programs', 'canonical_NAEYC Valid Until',\n", " 'canonical_NAEYC Program Id', 'canonical_Email Address',\n", " 'canonical_Ounce of Prevention Description',\n", " 'canonical_Purple binder service type', 'canonical_Column',\n", " 'canonical_Column2'],\n", " dtype='object')" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_dedupe.columns" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IdSourceSite nameAddressZipPhoneFaxProgram NameLength of DayIDHS Provider ID...canonical_Executive Directorcanonical_Center Directorcanonical_ECE Available Programscanonical_NAEYC Valid Untilcanonical_NAEYC Program Idcanonical_Email Addresscanonical_Ounce of Prevention Descriptioncanonical_Purple binder service typecanonical_Columncanonical_Column2
33273327purple_binder_early_childhood.csvprecious infants & tots learning center624 e 47th street60653.02682685.0NoneNoneNoneNone...early head start
33003300purple_binder_early_childhood.csvywca metropolitan chicago360 n michigan avenue60601.03726600.0NoneNoneNoneNone...child care
32993299purple_binder_early_childhood.csvymca west side5080 w harrison street60644.09553100.0NoneNoneNoneNone...child care
32853285purple_binder_early_childhood.csvwoodlawn organization6040 s harper avenue60637.02885840.0NoneNoneNoneNone...child care
32813281purple_binder_early_childhood.csvurban family and community centers4241 w washington boulevard60624.07228333.0NoneNoneNoneNone...child care
..................................................................
18371837chapin_dfss_providers_2011_070212.csvnorth avenue day nursery fcch-carolyn price2020 w jackson60612.0NoneNoneNoneNoneNone...5833ip(ehs collaboration enhanced home ip), 58...www.crcl.netcpbetty lee05/31/14723127youngt@crcl.netchild care
26432643ece chicago find a school scrape.csvmary crane north 0-32905 n. leavitt60618.03485528.0NoneNoneNoneNone...4374it(child care it center), 4374ps(child car...www.marycrane.orglavetter terrymartuice williams08/01/16722999info@marycrane.orgchild care
32283228purple_binder_early_childhood.csvmary crane center north2905 n leavitt street60618.09753322.0NoneNoneNoneNone...4374it(child care it center), 4374ps(child car...www.marycrane.orglavetter terrymartuice williams08/01/16722999info@marycrane.orgchild care
32293229purple_binder_early_childhood.csvmary crane family and day care center2905 n clybourn avenue60618.03485528.0NoneNoneNoneNone...4374it(child care it center), 4374ps(child car...www.marycrane.orglavetter terrymartuice williams08/01/16722999info@marycrane.orgchild care
32413241purple_binder_early_childhood.csvour lady of guadalupe early childhood center9129 s burley avenue60617.09785320.0NoneNoneNoneNone...2488it(child care it center), 7030ps(hs collab...www.catholiccharities.netlaura riosdeborah o'brien01/31/13486949pgutierr@catholiccharities.netchild care
\n", "

3337 rows × 66 columns

\n", "
" ], "text/plain": [ " Id Source \\\n", "3327 3327 purple_binder_early_childhood.csv \n", "3300 3300 purple_binder_early_childhood.csv \n", "3299 3299 purple_binder_early_childhood.csv \n", "3285 3285 purple_binder_early_childhood.csv \n", "3281 3281 purple_binder_early_childhood.csv \n", "... ... ... \n", "1837 1837 chapin_dfss_providers_2011_070212.csv \n", "2643 2643 ece chicago find a school scrape.csv \n", "3228 3228 purple_binder_early_childhood.csv \n", "3229 3229 purple_binder_early_childhood.csv \n", "3241 3241 purple_binder_early_childhood.csv \n", "\n", " Site name \\\n", "3327 precious infants & tots learning center \n", "3300 ywca metropolitan chicago \n", "3299 ymca west side \n", "3285 woodlawn organization \n", "3281 urban family and community centers \n", "... ... \n", "1837 north avenue day nursery fcch-carolyn price \n", "2643 mary crane north 0-3 \n", "3228 mary crane center north \n", "3229 mary crane family and day care center \n", "3241 our lady of guadalupe early childhood center \n", "\n", " Address Zip Phone Fax Program Name \\\n", "3327 624 e 47th street 60653.0 2682685.0 None None \n", "3300 360 n michigan avenue 60601.0 3726600.0 None None \n", "3299 5080 w harrison street 60644.0 9553100.0 None None \n", "3285 6040 s harper avenue 60637.0 2885840.0 None None \n", "3281 4241 w washington boulevard 60624.0 7228333.0 None None \n", "... ... ... ... ... ... \n", "1837 2020 w jackson 60612.0 None None None \n", "2643 2905 n. leavitt 60618.0 3485528.0 None None \n", "3228 2905 n leavitt street 60618.0 9753322.0 None None \n", "3229 2905 n clybourn avenue 60618.0 3485528.0 None None \n", "3241 9129 s burley avenue 60617.0 9785320.0 None None \n", "\n", " Length of Day IDHS Provider ID ... \\\n", "3327 None None ... \n", "3300 None None ... \n", "3299 None None ... \n", "3285 None None ... \n", "3281 None None ... \n", "... ... ... ... \n", "1837 None None ... \n", "2643 None None ... \n", "3228 None None ... \n", "3229 None None ... \n", "3241 None None ... \n", "\n", " canonical_Executive Director \\\n", "3327 \n", "3300 \n", "3299 \n", "3285 \n", "3281 \n", "... ... \n", "1837 5833ip(ehs collaboration enhanced home ip), 58... \n", "2643 4374it(child care it center), 4374ps(child car... \n", "3228 4374it(child care it center), 4374ps(child car... \n", "3229 4374it(child care it center), 4374ps(child car... \n", "3241 2488it(child care it center), 7030ps(hs collab... \n", "\n", " canonical_Center Director canonical_ECE Available Programs \\\n", "3327 \n", "3300 \n", "3299 \n", "3285 \n", "3281 \n", "... ... ... \n", "1837 www.crcl.net cp \n", "2643 www.marycrane.org lavetter terry \n", "3228 www.marycrane.org lavetter terry \n", "3229 www.marycrane.org lavetter terry \n", "3241 www.catholiccharities.net laura rios \n", "\n", " canonical_NAEYC Valid Until canonical_NAEYC Program Id \\\n", "3327 \n", "3300 \n", "3299 \n", "3285 \n", "3281 \n", "... ... ... \n", "1837 betty lee \n", "2643 martuice williams \n", "3228 martuice williams \n", "3229 martuice williams \n", "3241 deborah o'brien \n", "\n", " canonical_Email Address canonical_Ounce of Prevention Description \\\n", "3327 \n", "3300 \n", "3299 \n", "3285 \n", "3281 \n", "... ... ... \n", "1837 05/31/14 723127 \n", "2643 08/01/16 722999 \n", "3228 08/01/16 722999 \n", "3229 08/01/16 722999 \n", "3241 01/31/13 486949 \n", "\n", " canonical_Purple binder service type canonical_Column canonical_Column2 \n", "3327 early head start \n", "3300 child care \n", "3299 child care \n", "3285 child care \n", "3281 child care \n", "... ... ... ... \n", "1837 youngt@crcl.net child care \n", "2643 info@marycrane.org child care \n", "3228 info@marycrane.org child care \n", "3229 info@marycrane.org child care \n", "3241 pgutierr@catholiccharities.net child care \n", "\n", "[3337 rows x 66 columns]" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_sorted = df_dedupe.sort_values(['confidence', 'cluster id'], ascending=False)\n", "df_dedupe.sort_values(['confidence', 'cluster id'], ascending=False)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Idcluster idconfidenceSourceZipAddresscanonical_Executive Director
332733278701.000000purple_binder_early_childhood.csv60653.0624 e 47th street
330033008691.000000purple_binder_early_childhood.csv60601.0360 n michigan avenue
329932998681.000000purple_binder_early_childhood.csv60644.05080 w harrison street
328532858671.000000purple_binder_early_childhood.csv60637.06040 s harper avenue
328132818661.000000purple_binder_early_childhood.csv60624.04241 w washington boulevard
........................
18371837400.133246chapin_dfss_providers_2011_070212.csv60612.02020 w jackson5833ip(ehs collaboration enhanced home ip), 58...
26432643310.130475ece chicago find a school scrape.csv60618.02905 n. leavitt4374it(child care it center), 4374ps(child car...
32283228310.130475purple_binder_early_childhood.csv60618.02905 n leavitt street4374it(child care it center), 4374ps(child car...
32293229310.130474purple_binder_early_childhood.csv60618.02905 n clybourn avenue4374it(child care it center), 4374ps(child car...
32413241270.058419purple_binder_early_childhood.csv60617.09129 s burley avenue2488it(child care it center), 7030ps(hs collab...
\n", "

3337 rows × 7 columns

\n", "
" ], "text/plain": [ " Id cluster id confidence Source \\\n", "3327 3327 870 1.000000 purple_binder_early_childhood.csv \n", "3300 3300 869 1.000000 purple_binder_early_childhood.csv \n", "3299 3299 868 1.000000 purple_binder_early_childhood.csv \n", "3285 3285 867 1.000000 purple_binder_early_childhood.csv \n", "3281 3281 866 1.000000 purple_binder_early_childhood.csv \n", "... ... ... ... ... \n", "1837 1837 40 0.133246 chapin_dfss_providers_2011_070212.csv \n", "2643 2643 31 0.130475 ece chicago find a school scrape.csv \n", "3228 3228 31 0.130475 purple_binder_early_childhood.csv \n", "3229 3229 31 0.130474 purple_binder_early_childhood.csv \n", "3241 3241 27 0.058419 purple_binder_early_childhood.csv \n", "\n", " Zip Address \\\n", "3327 60653.0 624 e 47th street \n", "3300 60601.0 360 n michigan avenue \n", "3299 60644.0 5080 w harrison street \n", "3285 60637.0 6040 s harper avenue \n", "3281 60624.0 4241 w washington boulevard \n", "... ... ... \n", "1837 60612.0 2020 w jackson \n", "2643 60618.0 2905 n. leavitt \n", "3228 60618.0 2905 n leavitt street \n", "3229 60618.0 2905 n clybourn avenue \n", "3241 60617.0 9129 s burley avenue \n", "\n", " canonical_Executive Director \n", "3327 \n", "3300 \n", "3299 \n", "3285 \n", "3281 \n", "... ... \n", "1837 5833ip(ehs collaboration enhanced home ip), 58... \n", "2643 4374it(child care it center), 4374ps(child car... \n", "3228 4374it(child care it center), 4374ps(child car... \n", "3229 4374it(child care it center), 4374ps(child car... \n", "3241 2488it(child care it center), 7030ps(hs collab... \n", "\n", "[3337 rows x 7 columns]" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_sorted[['Id', 'cluster id', 'confidence', 'Source', 'Zip', 'Address', 'canonical_Executive Director']]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Matching / Linking records\n", "Another problem is matching / linking records from different sources." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Let's load two datasets from FEBRl (Freely extensible biomedical record linkage)." ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "#load dataframes\n", "dfa = pd.read_csv('data/dataset1-febrl.csv')\n", "dfb = pd.read_csv('data/dataset2-febrl.csv')" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "We cannot have missing values for applying record matching with this library, so we fill them.\n", "\n", "The problem is that many values are ' ' (not NaN). So, we first convert to NaN, and then we drop them." ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "dfa.replace(['', ' '], np.nan, inplace=True)\n", "dfb.replace(['', ' '], np.nan, inplace=True)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "dfa.dropna(inplace=True)\n", "dfb.dropna(inplace=True)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
rec_idgiven_namesurnamestreet_numberaddress_1address_2suburbpostcodestatedate_of_birthsoc_sec_id
1rec-122-orglachlanberry69giblin streetkillarneybittern4814qld199902197364009
2rec-373-orgdeakinsondergeld48goldfinch circuitkooltuocanterbury2776vic196002102635962
4rec-227-orglukepurdon23ramsay placemiranigarbutt2260vic198310248099933
7rec-294-orgwilliambishop21neworra placeapmnt 65worongary6225qld194901309773843
10rec-81-dup-0abbeyfit13kosciusko avenuethe wharf complexyass2594nsw198705107661096
11rec-34-orgisabellalodder156messenger streettongbong sanctuarybayswater4870vic196507142790666
12rec-478-organthonybeazley12birubi placecurrandinaflemington2477qld197309246558077
13rec-225-orgaliastreich74maranoa streetrocky bendrowville6152vic197904181975340
15rec-452-orgalissakilmartin37reveley crescentcrown allotwolumla6210nsw190411187994055
16rec-67-orgjacoblyden25haddon streetglenviewwoodville north2226qld199104246426415
\n", "
" ], "text/plain": [ " rec_id given_name surname street_number address_1 \\\n", "1 rec-122-org lachlan berry 69 giblin street \n", "2 rec-373-org deakin sondergeld 48 goldfinch circuit \n", "4 rec-227-org luke purdon 23 ramsay place \n", "7 rec-294-org william bishop 21 neworra place \n", "10 rec-81-dup-0 abbey fit 13 kosciusko avenue \n", "11 rec-34-org isabella lodder 156 messenger street \n", "12 rec-478-org anthony beazley 12 birubi place \n", "13 rec-225-org alia streich 74 maranoa street \n", "15 rec-452-org alissa kilmartin 37 reveley crescent \n", "16 rec-67-org jacob lyden 25 haddon street \n", "\n", " address_2 suburb postcode state date_of_birth \\\n", "1 killarney bittern 4814 qld 19990219 \n", "2 kooltuo canterbury 2776 vic 19600210 \n", "4 mirani garbutt 2260 vic 19831024 \n", "7 apmnt 65 worongary 6225 qld 19490130 \n", "10 the wharf complex yass 2594 nsw 19870510 \n", "11 tongbong sanctuary bayswater 4870 vic 19650714 \n", "12 currandina flemington 2477 qld 19730924 \n", "13 rocky bend rowville 6152 vic 19790418 \n", "15 crown allot wolumla 6210 nsw 19041118 \n", "16 glenview woodville north 2226 qld 19910424 \n", "\n", " soc_sec_id \n", "1 7364009 \n", "2 2635962 \n", "4 8099933 \n", "7 9773843 \n", "10 7661096 \n", "11 2790666 \n", "12 6558077 \n", "13 1975340 \n", "15 7994055 \n", "16 6426415 " ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dfa.head(10)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
rec_idgiven_namesurnamestreet_numberaddress_1address_2suburbpostcodestatedate_of_birthsoc_sec_id
0rec-2778-orgsarahbruhn44forbes streetwintersloekellerberrin4510vic193002137535316
1rec-712-dup-0jacoblanyon5milne covewellwodbeaconsfield upper2602vic190807129497788
2rec-1321-orgbrinleyefthimiou35sturdee crescenttremearnescarborough5211qld199403196814956
3rec-3004-orgaleishahobson54oliver streetinglewoodtoowoomba3175qld192904275967384
4rec-1384-orgethangazzola49sheaffe streetbimby valeport pirie3088sa196312253832742
5rec-3981-orgaliciahope100mansfield placesunsetbyford6061sa194212017934773
6rec-916-orgbenjaminkolosche78keenan streetwingararaymond terrace3212sa194509185698873
8rec-63-dup-0oliviawhite55duffy streetshopping villagemirrabooka2260vic190001064996142
10rec-112-orgjoshuarudd78max henry crescentbrentwood vlgeport douglas2315vic199511251697892
11rec-3297-orgrachaellomman37carlile streetclonturklebronte2177nsw199102289462397
\n", "
" ], "text/plain": [ " rec_id given_name surname street_number address_1 \\\n", "0 rec-2778-org sarah bruhn 44 forbes street \n", "1 rec-712-dup-0 jacob lanyon 5 milne cove \n", "2 rec-1321-org brinley efthimiou 35 sturdee crescent \n", "3 rec-3004-org aleisha hobson 54 oliver street \n", "4 rec-1384-org ethan gazzola 49 sheaffe street \n", "5 rec-3981-org alicia hope 100 mansfield place \n", "6 rec-916-org benjamin kolosche 78 keenan street \n", "8 rec-63-dup-0 olivia white 55 duffy street \n", "10 rec-112-org joshua rudd 78 max henry crescent \n", "11 rec-3297-org rachael lomman 37 carlile street \n", "\n", " address_2 suburb postcode state date_of_birth \\\n", "0 wintersloe kellerberrin 4510 vic 19300213 \n", "1 wellwod beaconsfield upper 2602 vic 19080712 \n", "2 tremearne scarborough 5211 qld 19940319 \n", "3 inglewood toowoomba 3175 qld 19290427 \n", "4 bimby vale port pirie 3088 sa 19631225 \n", "5 sunset byford 6061 sa 19421201 \n", "6 wingara raymond terrace 3212 sa 19450918 \n", "8 shopping village mirrabooka 2260 vic 19000106 \n", "10 brentwood vlge port douglas 2315 vic 19951125 \n", "11 clonturkle bronte 2177 nsw 19910228 \n", "\n", " soc_sec_id \n", "0 7535316 \n", "1 9497788 \n", "2 6814956 \n", "3 5967384 \n", "4 3832742 \n", "5 7934773 \n", "6 5698873 \n", "8 4996142 \n", "10 1697892 \n", "11 9462397 " ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dfb.head(10)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Check the two datasets have the same columns." ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Index(['rec_id', ' given_name', ' surname', ' street_number', ' address_1',\n", " ' address_2', ' suburb', ' postcode', ' state', ' date_of_birth',\n", " ' soc_sec_id'],\n", " dtype='object')\n", "Index(['rec_id', ' given_name', ' surname', ' street_number', ' address_1',\n", " ' address_2', ' suburb', ' postcode', ' state', ' date_of_birth',\n", " ' soc_sec_id'],\n", " dtype='object')\n" ] } ], "source": [ "print(dfa.columns)\n", "print(dfb.columns)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Let's match..." ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Importing data ...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : reeve\n", " surname : quilliam\n", " street_number : 2\n", " address_1 : renwick street\n", " address_2 : yarrabee\n", " suburb : barwon heads\n", " postcode : 2340\n", " state : nsw\n", " date_of_birth : 19810406\n", " soc_sec_id : 1066923\n", "\n", " given_name : jessica\n", " surname : reid\n", " street_number : 280\n", " address_1 : medley street\n", " address_2 : warra creek\n", " suburb : ballarat\n", " postcode : 3149\n", " state : nsw\n", " date_of_birth : 19830907\n", " soc_sec_id : 1067529\n", "\n", "0/10 positive, 0/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Starting active labeling...\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : daniel\n", " surname : couzens\n", " street_number : 37\n", " address_1 : coventry close\n", " address_2 : cressbrook\n", " suburb : mount eliza\n", " postcode : 5073\n", " state : nsw\n", " date_of_birth : 19881127\n", " soc_sec_id : 6934299\n", "\n", " given_name : dante\n", " surname : dakin\n", " street_number : 3\n", " address_1 : chuculba crescent\n", " address_2 : greenpatch\n", " suburb : forbes\n", " postcode : 5072\n", " state : nsw\n", " date_of_birth : 19481028\n", " soc_sec_id : 7288639\n", "\n", "0/10 positive, 1/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : lachlan\n", " surname : jukic\n", " street_number : 2\n", " address_1 : morgan crescent\n", " address_2 : parklea\n", " suburb : raymond terrace\n", " postcode : 2250\n", " state : nsw\n", " date_of_birth : 19780702\n", " soc_sec_id : 4027998\n", "\n", " given_name : meg\n", " surname : feil\n", " street_number : 17\n", " address_1 : biraban place\n", " address_2 : hughloch lincoln red stud\n", " suburb : hawthorne\n", " postcode : 3429\n", " state : vic\n", " date_of_birth : 19060812\n", " soc_sec_id : 4027997\n", "\n", "0/10 positive, 2/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : jacob\n", " surname : lyen\n", " street_number : 25\n", " address_1 : haddon srteet\n", " address_2 : glenvie w\n", " suburb : woodville north\n", " postcode : 2226\n", " state : qld\n", " date_of_birth : 19910424\n", " soc_sec_id : 6426415\n", "\n", " given_name : zac\n", " surname : white\n", " street_number : 26\n", " address_1 : companion crescent\n", " address_2 : glenview\n", " suburb : toronto\n", " postcode : 2226\n", " state : sa\n", " date_of_birth : 19431117\n", " soc_sec_id : 3437945\n", "\n", "0/10 positive, 3/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : kydan\n", " surname : mccarthy\n", " street_number : 67\n", " address_1 : clemenger street\n", " address_2 : the points holsteins\n", " suburb : fairlawn\n", " postcode : 6415\n", " state : nsw\n", " date_of_birth : 19720518\n", " soc_sec_id : 6527653\n", "\n", " given_name : daniel\n", " surname : mccarthy\n", " street_number : 6\n", " address_1 : brunton street\n", " address_2 : tall pines\n", " suburb : fairlight\n", " postcode : 3155\n", " state : nsw\n", " date_of_birth : 19760107\n", " soc_sec_id : 8093038\n", "\n", "0/10 positive, 4/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : brooklyn\n", " surname : manson\n", " street_number : 27\n", " address_1 : clive steele avenue\n", " address_2 : port fairy road\n", " suburb : mount low\n", " postcode : 3450\n", " state : vic\n", " date_of_birth : 19710727\n", " soc_sec_id : 4493900\n", "\n", " given_name : ruby\n", " surname : mason\n", " street_number : 7\n", " address_1 : clive steele avenue\n", " address_2 : kooyong\n", " suburb : botany\n", " postcode : 3636\n", " state : vic\n", " date_of_birth : 19730913\n", " soc_sec_id : 4397223\n", "\n", "0/10 positive, 5/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : emiily\n", " surname : coleman\n", " street_number : 108\n", " address_1 : chewings street\n", " address_2 : berkeley vlge\n", " suburb : wellington point\n", " postcode : 2550\n", " state : nsw\n", " date_of_birth : 19421221\n", " soc_sec_id : 7206933\n", "\n", " given_name : emiily\n", " surname : went\n", " street_number : 18\n", " address_1 : glenmaggie street\n", " address_2 : berkeley vlge\n", " suburb : blue haven\n", " postcode : 6051\n", " state : vic\n", " date_of_birth : 19521205\n", " soc_sec_id : 8530937\n", "\n", "0/10 positive, 6/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : jacob\n", " surname : white\n", " street_number : 5\n", " address_1 : findlay street\n", " address_2 : booroopki park rmb 596\n", " suburb : robina\n", " postcode : 2197\n", " state : vic\n", " date_of_birth : 19170205\n", " soc_sec_id : 4702928\n", "\n", " given_name : talia\n", " surname : reid\n", " street_number : 147\n", " address_1 : sid barnes crescent\n", " address_2 : tathra\n", " suburb : berowra heights\n", " postcode : 2170\n", " state : vic\n", " date_of_birth : 19230203\n", " soc_sec_id : 4712927\n", "\n", "0/10 positive, 7/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : carla\n", " surname : amiet\n", " street_number : 50\n", " address_1 : carstensz street\n", " address_2 : oaklodge\n", " suburb : blackmans bay\n", " postcode : 3180\n", " state : nsw\n", " date_of_birth : 19790801\n", " soc_sec_id : 9646483\n", "\n", " given_name : cameron\n", " surname : coleman\n", " street_number : 10\n", " address_1 : edwards street\n", " address_2 : broadbridge manor\n", " suburb : blackmans bay\n", " postcode : 3630\n", " state : nsw\n", " date_of_birth : 19871030\n", " soc_sec_id : 5502408\n", "\n", "0/10 positive, 8/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : lara\n", " surname : matthews\n", " street_number : 6\n", " address_1 : beaumaris street\n", " address_2 : sunforest therapy centre\n", " suburb : the basin\n", " postcode : 4179\n", " state : nsw\n", " date_of_birth : 19911006\n", " soc_sec_id : 2164704\n", "\n", " given_name : dominic\n", " surname : matthews\n", " street_number : 67\n", " address_1 : campbell street\n", " address_2 : narraburra lodge\n", " suburb : coonabarabran\n", " postcode : 3174\n", " state : nsw\n", " date_of_birth : 19470226\n", " soc_sec_id : 3115384\n", "\n", "0/10 positive, 9/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : jack\n", " surname : matthews\n", " street_number : 17\n", " address_1 : herron crescent\n", " address_2 : broadmere\n", " suburb : highton\n", " postcode : 2035\n", " state : vic\n", " date_of_birth : 19081119\n", " soc_sec_id : 5613395\n", "\n", " given_name : alexandra\n", " surname : matthews\n", " street_number : 174\n", " address_1 : port jackson circuit\n", " address_2 : old timers south\n", " suburb : whaleback\n", " postcode : 2830\n", " state : vic\n", " date_of_birth : 19261017\n", " soc_sec_id : 2919332\n", "\n", "0/10 positive, 10/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : emiily\n", " surname : kowald\n", " street_number : 18\n", " address_1 : daglish street\n", " address_2 : oakdale\n", " suburb : avalon\n", " postcode : 3030\n", " state : vic\n", " date_of_birth : 19250313\n", " soc_sec_id : 1590627\n", "\n", " given_name : emiily\n", " surname : went\n", " street_number : 18\n", " address_1 : glenmaggie street\n", " address_2 : berkeley vlge\n", " suburb : blue haven\n", " postcode : 6051\n", " state : vic\n", " date_of_birth : 19521205\n", " soc_sec_id : 8530937\n", "\n", "0/10 positive, 11/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : jaden\n", " surname : humphreys\n", " street_number : 14\n", " address_1 : euree street\n", " address_2 : moorillyah\n", " suburb : thornlie\n", " postcode : 3116\n", " state : wa\n", " date_of_birth : 19700116\n", " soc_sec_id : 9382782\n", "\n", " given_name : isabelle\n", " surname : jolly\n", " street_number : 166\n", " address_1 : hodges street\n", " address_2 : bosmit\n", " suburb : thornlie\n", " postcode : 3163\n", " state : wa\n", " date_of_birth : 19050126\n", " soc_sec_id : 2719590\n", "\n", "0/10 positive, 12/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : brooklyn\n", " surname : manson\n", " street_number : 27\n", " address_1 : clive steele avenue\n", " address_2 : port fairy road\n", " suburb : yagoona\n", " postcode : 3450\n", " state : vic\n", " date_of_birth : 19710727\n", " soc_sec_id : 4493900\n", "\n", " given_name : laura\n", " surname : campbell\n", " street_number : 152\n", " address_1 : clive steele avenue\n", " address_2 : irrigation farm\n", " suburb : yagoona\n", " postcode : 3350\n", " state : vic\n", " date_of_birth : 19160610\n", " soc_sec_id : 6214635\n", "\n", "0/10 positive, 13/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : jacob\n", " surname : white\n", " street_number : 5\n", " address_1 : findlay street\n", " address_2 : booroopki park rmb 596\n", " suburb : robina\n", " postcode : 2197\n", " state : vic\n", " date_of_birth : 19170205\n", " soc_sec_id : 4702928\n", "\n", " given_name : jakob\n", " surname : menzies\n", " street_number : 33\n", " address_1 : coverdale street\n", " address_2 : bundong\n", " suburb : worongary\n", " postcode : 2190\n", " state : vic\n", " date_of_birth : 19140610\n", " soc_sec_id : 4557295\n", "\n", "0/10 positive, 14/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : benjamin\n", " surname : kirchener\n", " street_number : 12\n", " address_1 : stuart street\n", " address_2 : wurrami\n", " suburb : theodore\n", " postcode : 2620\n", " state : wa\n", " date_of_birth : 19751110\n", " soc_sec_id : 1766048\n", "\n", " given_name : max\n", " surname : rees\n", " street_number : 10\n", " address_1 : waite street\n", " address_2 : rosedown\n", " suburb : nambour\n", " postcode : 2620\n", " state : wa\n", " date_of_birth : 19751230\n", " soc_sec_id : 1361900\n", "\n", "0/10 positive, 15/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " n\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : brydon\n", " surname : webb\n", " street_number : 11\n", " address_1 : walker crescent\n", " address_2 : bugoren\n", " suburb : charlestown\n", " postcode : 6258\n", " state : nsw\n", " date_of_birth : 19190528\n", " soc_sec_id : 4191569\n", "\n", " given_name : bradley\n", " surname : haberfield\n", " street_number : 11\n", " address_1 : carumbo place\n", " address_2 : bungarra\n", " suburb : canley heights\n", " postcode : 2758\n", " state : vic\n", " date_of_birth : 19190528\n", " soc_sec_id : 5039500\n", "\n", "0/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " y\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : brydon\n", " surname : webb\n", " street_number : 11\n", " address_1 : walker crescent\n", " address_2 : bugoren\n", " suburb : charlestown\n", " postcode : 6258\n", " state : nsw\n", " date_of_birth : 19190528\n", " soc_sec_id : 4191569\n", "\n", " given_name : bradley\n", " surname : haberfield\n", " street_number : 11\n", " address_1 : carumbi place\n", " address_2 : bungarra\n", " suburb : canley heights\n", " postcode : 2758\n", " state : vic\n", " date_of_birth : 19190528\n", " soc_sec_id : 5039500\n", "\n", "1/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " y\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : lochlan\n", " surname : savidge\n", " street_number : 29\n", " address_1 : pohlman street\n", " address_2 : moline village\n", " suburb : jingili\n", " postcode : 3071\n", " state : vic\n", " date_of_birth : 19140228\n", " soc_sec_id : 1498207\n", "\n", " given_name : jayme\n", " surname : parr\n", " street_number : 2\n", " address_1 : clive steele avenue\n", " address_2 : henry kendall hostel\n", " suburb : hoskinstown\n", " postcode : 2770\n", " state : nsw\n", " date_of_birth : 19140228\n", " soc_sec_id : 5840194\n", "\n", "2/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " y\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : charlie\n", " surname : headon\n", " street_number : 17\n", " address_1 : sheehy street\n", " address_2 : hawkins masonic vlge\n", " suburb : warrandyte\n", " postcode : 3073\n", " state : vic\n", " date_of_birth : 19880814\n", " soc_sec_id : 7871445\n", "\n", " given_name : william\n", " surname : hislop\n", " street_number : 17\n", " address_1 : deane street\n", " address_2 : sunbury\n", " suburb : cedar creek\n", " postcode : 3073\n", " state : qld\n", " date_of_birth : 19830819\n", " soc_sec_id : 6153593\n", "\n", "3/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " y\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : mitchell\n", " surname : scrbak\n", " street_number : 950\n", " address_1 : holyman street\n", " address_2 : berkeley vlge\n", " suburb : safety bay\n", " postcode : 4300\n", " state : qld\n", " date_of_birth : 19060811\n", " soc_sec_id : 3592109\n", "\n", " given_name : hannah\n", " surname : beams\n", " street_number : 9\n", " address_1 : light street\n", " address_2 : castle hill farm\n", " suburb : sale\n", " postcode : 3221\n", " state : vic\n", " date_of_birth : 19560811\n", " soc_sec_id : 7444484\n", "\n", "4/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " y\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : finley\n", " surname : haeusler\n", " street_number : 27\n", " address_1 : noarlunga crescent\n", " address_2 : spring ridge\n", " suburb : nambour\n", " postcode : 3180\n", " state : vic\n", " date_of_birth : 19660711\n", " soc_sec_id : 3025217\n", "\n", " given_name : elki\n", " surname : trent\n", " street_number : 27\n", " address_1 : wray place\n", " address_2 : the mews royal hotel bldg\n", " suburb : gawler east\n", " postcode : 3152\n", " state : nsw\n", " date_of_birth : 19600211\n", " soc_sec_id : 5679502\n", "\n", "5/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " y\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : india\n", " surname : negrean\n", " street_number : 1\n", " address_1 : barringer street\n", " address_2 : sherwood\n", " suburb : parkinson\n", " postcode : 3168\n", " state : nsw\n", " date_of_birth : 19860923\n", " soc_sec_id : 2097928\n", "\n", " given_name : logan\n", " surname : selth\n", " street_number : 147\n", " address_1 : goyder street\n", " address_2 : rivonia\n", " suburb : queenscliff\n", " postcode : 2120\n", " state : tas\n", " date_of_birth : 19860921\n", " soc_sec_id : 4161322\n", "\n", "6/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " y\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : annablle\n", " surname : kounis\n", " street_number : 121\n", " address_1 : calder place\n", " address_2 : brambletye vinyard\n", " suburb : joondkalup\n", " postcode : 3120\n", " state : nsw\n", " date_of_birth : 19640907\n", " soc_sec_id : 1612956\n", "\n", " given_name : claurdia\n", " surname : clelland\n", " street_number : 12\n", " address_1 : box hill a venue\n", " address_2 : st francis vlge\n", " suburb : old beach\n", " postcode : 3127\n", " state : wa\n", " date_of_birth : 19640902\n", " soc_sec_id : 9508954\n", "\n", "7/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " y\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : troy\n", " surname : reid\n", " street_number : 1\n", " address_1 : allan street\n", " address_2 : townview\n", " suburb : page\n", " postcode : 2774\n", " state : qld\n", " date_of_birth : 19250727\n", " soc_sec_id : 3580821\n", "\n", " given_name : william\n", " surname : tossell\n", " street_number : 1\n", " address_1 : lutana street\n", " address_2 : nara cnsa\n", " suburb : craigmore\n", " postcode : 2509\n", " state : nsw\n", " date_of_birth : 19250116\n", " soc_sec_id : 5322906\n", "\n", "8/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " y\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : isaac\n", " surname : quilliam\n", " street_number : 11\n", " address_1 : namatjira drive\n", " address_2 : delaware\n", " suburb : geelong west\n", " postcode : 3072\n", " state : nsw\n", " date_of_birth : 19930926\n", " soc_sec_id : 1556150\n", "\n", " given_name : bailey\n", " surname : clarke\n", " street_number : 1\n", " address_1 : hetherington circuit\n", " address_2 : gundaline\n", " suburb : harden\n", " postcode : 2077\n", " state : vic\n", " date_of_birth : 19930416\n", " soc_sec_id : 6134615\n", "\n", "9/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " y\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : james\n", " surname : blake\n", " street_number : 19\n", " address_1 : sturt avenue\n", " address_2 : laloki\n", " suburb : carnegie\n", " postcode : 2218\n", " state : nsw\n", " date_of_birth : 19050716\n", " soc_sec_id : 7830672\n", "\n", " given_name : finn\n", " surname : kapoor\n", " street_number : 1994\n", " address_1 : sturt avenue\n", " address_2 : john flynn medical centre\n", " suburb : mullumbimby\n", " postcode : 2262\n", " state : vic\n", " date_of_birth : 19880816\n", " soc_sec_id : 8680815\n", "\n", "10/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " y\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " given_name : harry\n", " surname : ryn\n", " street_number : 5\n", " address_1 : kellway street\n", " address_2 : rowethorpe\n", " suburb : toowong\n", " postcode : 3931\n", " state : nsw\n", " date_of_birth : 19220503\n", " soc_sec_id : 7228670\n", "\n", " given_name : samantha\n", " surname : grierson\n", " street_number : 5\n", " address_1 : kennedy street\n", " address_2 : tantallon\n", " suburb : oakleigh\n", " postcode : 3034\n", " state : vic\n", " date_of_birth : 19210114\n", " soc_sec_id : 4683164\n", "\n", "11/10 positive, 16/10 negative\n", "Do these records refer to the same thing?\n", "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ " f\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Finished labeling\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Clustering...\n" ] }, { "ename": "BlockingError", "evalue": "No records have been blocked together. Is the data you are trying to match like the data you trained on? If so, try adding more training data.", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mBlockingError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[31], line 2\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;66;03m#initiate matching\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m df_final \u001b[38;5;241m=\u001b[39m pandas_dedupe\u001b[38;5;241m.\u001b[39mlink_dataframes(dfa, dfb, [\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m given_name\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m surname\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m street_number\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m address_1\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m address_2\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m suburb\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m postcode\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m state\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m date_of_birth\u001b[39m\u001b[38;5;124m'\u001b[39m,\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m soc_sec_id\u001b[39m\u001b[38;5;124m'\u001b[39m])\n", "File \u001b[0;32m~/anaconda3/lib/python3.11/site-packages/pandas_dedupe/link_dataframes.py:112\u001b[0m, in \u001b[0;36mlink_dataframes\u001b[0;34m(dfa, dfb, field_properties, config_name, n_cores)\u001b[0m\n\u001b[1;32m 100\u001b[0m \u001b[38;5;66;03m# ## Blocking\u001b[39;00m\n\u001b[1;32m 101\u001b[0m \n\u001b[1;32m 102\u001b[0m \u001b[38;5;66;03m# ## Clustering\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 108\u001b[0m \u001b[38;5;66;03m# If we had more data, we would not pass in all the blocked data into\u001b[39;00m\n\u001b[1;32m 109\u001b[0m \u001b[38;5;66;03m# this function but a representative sample.\u001b[39;00m\n\u001b[1;32m 111\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mClustering...\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m--> 112\u001b[0m linked_records \u001b[38;5;241m=\u001b[39m linker\u001b[38;5;241m.\u001b[39mjoin(data_1, data_2, \u001b[38;5;241m0\u001b[39m)\n\u001b[1;32m 114\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m# duplicate sets\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;28mlen\u001b[39m(linked_records))\n\u001b[1;32m 117\u001b[0m \u001b[38;5;66;03m#Convert linked records into dataframe\u001b[39;00m\n", "File \u001b[0;32m~/anaconda3/lib/python3.11/site-packages/dedupe/api.py:549\u001b[0m, in \u001b[0;36mRecordLinkMatching.join\u001b[0;34m(self, data_1, data_2, threshold, constraint)\u001b[0m\n\u001b[1;32m 543\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m constraint \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mone-to-one\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmany-to-one\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmany-to-many\u001b[39m\u001b[38;5;124m\"\u001b[39m}, (\n\u001b[1;32m 544\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m is an invalid constraint option. Valid options include \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 545\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mone-to-one, many-to-one, or many-to-many\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m%\u001b[39m constraint\n\u001b[1;32m 546\u001b[0m )\n\u001b[1;32m 548\u001b[0m pairs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mpairs(data_1, data_2)\n\u001b[0;32m--> 549\u001b[0m pair_scores \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mscore(pairs)\n\u001b[1;32m 551\u001b[0m links: Links\n\u001b[1;32m 552\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m constraint \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mone-to-one\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n", "File \u001b[0;32m~/anaconda3/lib/python3.11/site-packages/dedupe/api.py:125\u001b[0m, in \u001b[0;36mIntegralMatching.score\u001b[0;34m(self, pairs)\u001b[0m\n\u001b[1;32m 116\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 117\u001b[0m \u001b[38;5;124;03mScores pairs of records. Returns pairs of tuples of records id and\u001b[39;00m\n\u001b[1;32m 118\u001b[0m \u001b[38;5;124;03massociated probabilities that the pair of records are match\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 122\u001b[0m \n\u001b[1;32m 123\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 124\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 125\u001b[0m matches \u001b[38;5;241m=\u001b[39m core\u001b[38;5;241m.\u001b[39mscoreDuplicates(\n\u001b[1;32m 126\u001b[0m pairs, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdata_model\u001b[38;5;241m.\u001b[39mdistances, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mclassifier, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mnum_cores\n\u001b[1;32m 127\u001b[0m )\n\u001b[1;32m 128\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m:\n\u001b[1;32m 129\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\n\u001b[1;32m 130\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 131\u001b[0m \u001b[38;5;124;03m You need to either turn off multiprocessing or protect\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 134\u001b[0m \u001b[38;5;124;03m https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods\"\"\"\u001b[39;00m\n\u001b[1;32m 135\u001b[0m )\n", "File \u001b[0;32m~/anaconda3/lib/python3.11/site-packages/dedupe/core.py:126\u001b[0m, in \u001b[0;36mscoreDuplicates\u001b[0;34m(record_pairs, featurizer, classifier, num_cores)\u001b[0m\n\u001b[1;32m 124\u001b[0m first, record_pairs \u001b[38;5;241m=\u001b[39m peek(record_pairs)\n\u001b[1;32m 125\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m first \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 126\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m BlockingError(\n\u001b[1;32m 127\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mNo records have been blocked together. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 128\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIs the data you are trying to match like \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 129\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mthe data you trained on? If so, try adding \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 130\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmore training data.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 131\u001b[0m )\n\u001b[1;32m 133\u001b[0m record_pairs_queue: _Queue \u001b[38;5;241m=\u001b[39m Queue(\u001b[38;5;241m2\u001b[39m)\n\u001b[1;32m 134\u001b[0m exception_queue: _Queue \u001b[38;5;241m=\u001b[39m Queue()\n", "\u001b[0;31mBlockingError\u001b[0m: No records have been blocked together. Is the data you are trying to match like the data you trained on? If so, try adding more training data." ] } ], "source": [ "#initiate matching\n", "df_final = pandas_dedupe.link_dataframes(dfa, dfb, [' given_name', ' surname', ' street_number', ' address_1', ' address_2', ' suburb', ' postcode', ' state', ' date_of_birth',' soc_sec_id'])\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Exercise\n", "Try to deduplicate the data of the visitors of the White House.\n", "You can find the data [here](https://obamawhitehouse.archives.gov/goodgovernment/tools/visitor-records)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "# References\n", "* [Cleaning and Prepping Data with Python for Data Science — Best Practices and Helpful Packages](https://medium.com/@rrfd/cleaning-and-prepping-data-with-python-for-data-science-best-practices-and-helpful-packages-af1edfbe2a3), DeFilippi, 2019, \n", "* [Data Preprocessing for Machine learning in Python, GeeksForGeeks](https://www.geeksforgeeks.org/data-preprocessing-machine-learning-python/)\n", "* [Dedupe](https://dedupe.io/) package\n", "* [pandas-dedupe](https://pypi.org/project/pandas-dedupe/) package" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "## Licence\n", "The notebook is freely licensed under under the [Creative Commons Attribution Share-Alike license](https://creativecommons.org/licenses/by/2.0/). \n", "\n", "© Carlos A. Iglesias, Universidad Politécnica de Madrid." ] } ], "metadata": { "celltoolbar": "Slideshow", "datacleaner": { "position": { "height": "158.667px", "left": "400px", "right": "20px", "top": "50px", "width": "700px" }, "python": { "varRefreshCmd": "try:\n print(_datacleaner.dataframe_metadata())\nexcept:\n print([])" }, "window_display": false }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.7" }, "latex_envs": { "LaTeX_envs_menu_present": true, "autocomplete": true, "bibliofile": "biblio.bib", "cite_by": "apalike", "current_citInitial": 1, "eqLabelWithNumbers": true, "eqNumInitial": 1, "hotkeys": { "equation": "Ctrl-E", "itemize": "Ctrl-I" }, "labels_anchors": false, "latex_user_defs": false, "report_style_numbering": false, "user_envs_cfg": false } }, "nbformat": 4, "nbformat_minor": 4 }