Slice & Dice Data Analysis using Pandas

Transcriptie

1 Slice & Dice Data Analysis using Pandas Guido PyGrunn May 9th, 2014

2 $ who am i

3 $ who am i gkoller ttys001 May 09 14:35

4 $ who am i

5 $ who am i Freelance Software Developer Python whenever I can Though I ve done my share of Perl, Java & C# Living in Amsterdam (for now)

6 Pandas

7 What is Pandas? A data analysis library for Python that provides rich data structures and functions designed to make working with structured data fast, easy, and expressive. and... combines the high performance array-computing features of NumPy with the flexible data manipulation capabilities of spreadsheets and relational databases Python for Data Analysis - Wes McKinney

8 What is NumPy? NumPy is an extension to the Python programming language, adding support for large, multidimensional arrays and matrices, along with a large library of high-level mathematical functions to operate on these arrays.

9 Installing Pandas

10 Installing Pandas in 8 minutes

11 Installing Pandas in 8 minutes $ pyvenv-3.4 env $ source env/bin/activate $ pip -v install pandas \ ipython[all] \ matplotlib

12 Why IPython? IPython Notebooks web-based interactive computational environment combines code execution, text, mathematics, plots and rich media into a single document A must-have for Pandas development

13 Starting IPython $ ipython notebook

14 Starting IPython $ ipython notebook --pylab=inline

15 IPython Notebook

16 Pandas Data Structures Series DataFrame Panel (won t cover)

17 Pandas Data Structures Series DataFrame Panel (won t cover)

18 Common Imports import pandas as pd from pandas import Series, DataFrame

19 Series Array like data structure Backed by a NumPy array

20 Series - Creation s = Series(randn(5))!! dtype: float64

21 Series - Creation With a specified index s = Series(randn(5), index=list('abcde'))!! a b c d e dtype: float64

22 Series - Creation Using a dictionary s = Series(dict(one=1, two=2, three=3, four=4, five=5))! five 5 four 4 one 1 three 3 two 2 dtype: int64

23 Series - Selection s['one'] # as a dictionary 1! s[0] # as a list 5! s[:2] # slice operation five 5 four 4 dtype: int64! s[s > 3] # boolean based indexing five 5 four 4 dtype: int64

24 Series - Operations s.min(), s.max(), s.mean(), s.sum() (1, 5, 3.0, 15)! s * 2 # vector based operations five 10 four 8 one 2 three 6 two 4 dtype: int64

25 Series - Operations User defined import string s.apply(lambda x: string.ascii_letters[x])! five f four e one b three d two c dtype: object

26 Series - Operations User defined import string s.apply(lambda x: string.ascii_letters[x])! five f four e one b three d two c dtype: object

27 Series - Operations Vector based u = Series(randn(5)) v = Series(randn(7)) u + v! NaN 6 NaN dtype: float64

30 DataFrame 2D array like Labelled rows and columns Heterogeneously typed

31 DF - Creation df = DataFrame(dict(foo=[1,2,3,4], bar=[5.,6.,8.,9.])) df.dtypes bar float64 foo int64 dtype: object

32 DF - Creation With a specified index df1 = DataFrame(dict(foo=[1,2,3,4], bar=[5.,6.,8.,9.]), index=['one','two','three','four'])

33 DF - Creation Using dictionaries df2 = DataFrame(dict(u=u, v=v)) # dict of Series

34 DF - Creation From a CSV file df3 = pd.read_csv('04. Inschrijvingen wo_tcm csv', sep=';', encoding= latin-1') df3.head()

35 DF - Selection df1['bar'] # column selection df1.bar # column selection via attribute one 5 two 6 three 8 four 9 Name: bar, dtype: float64

36 DF - Selection df1.loc['one'] # row selection by label df1.iloc[0] # row selection by integer bar 5 foo 1 Name: one, dtype: float64! df1[1:3] # slice rows

37 DF - Selection df1[df1['bar'] > 6] # select rows by boolean vector

38 DF - Addition/Deletion df1['foobar'] = df1['foo'] + df1['bar']

39 DF - Addition/Deletion df1['zero'] = 0 # assign a scalar value to column del df1['foobar']

40 DF - Merging/Joining # straight from the pandas' documentation left = DataFrame({'key1': ['foo', 'foo', 'bar'], 'key2': ['one', 'two', 'one'], 'lval': [1, 2, 3]})!! right = DataFrame({'key1': ['foo', 'foo', 'bar', 'bar'], 'key2': ['one', 'one', 'one', 'two'], 'rval': [4, 5, 6, 7]})

41 DF - Merging/Joining left right

42 DF - Merging/Joining pd.merge(left, right, how='outer') pd.merge(left, right, how='inner')

43 Some Simple Data Analysis

44 Some Simple Data Analysis

45 Number of Students wo = df3 # wo -> Wetenschappelijk Onderwijs wo.rename(columns=str.lower, inplace=true) wo.columns! Index(['provincie', 'gemeentenummer', 'gemeentenaam', 'brin nummer actueel', 'instellingsnaam actueel', 'croho onderdeel', 'croho subonderdeel', 'opleidingscode actueel', 'opleidingsnaam actueel', 'opleidingsvorm', 'opleidingsfase actueel', '2008 man', '2008 vrouw', '2009 man', '2009 vrouw', '2010 man', '2010 vrouw', '2011 man', '2011 vrouw', '2012 man', '2012 vrouw'], dtype=object)

46 Number of Students wo = df3 # wo -> Wetenschappelijk Onderwijs wo.rename(columns=str.lower, inplace=true) wo.columns! Index(['provincie', 'gemeentenummer', 'gemeentenaam', 'brin nummer actueel', 'instellingsnaam actueel', 'croho onderdeel', 'croho subonderdeel', 'opleidingscode actueel', 'opleidingsnaam actueel', 'opleidingsvorm', 'opleidingsfase actueel', '2008 man', '2008 vrouw', '2009 man', '2009 vrouw', '2010 man', '2010 vrouw', '2011 man', '2011 vrouw', '2012 man', '2012 vrouw'], dtype=object)

47 Select Columns men = [c for c in wo.columns.tolist() if c.endswith('man')] women = [c for c in wo.columns.tolist() if c.endswith( vrouw )]! # croho -> Centraal Register Opleidingen Hoger Onderwijs uni = wo[['instellingsnaam actueel', 'croho onderdeel'] + men + women]

48 Slice & Dice

49 Group By & Sum # math operations ignore nuisance columns (non-numeric cols) uni_size = uni.groupby('instellingsnaam actueel').sum()

50 (Men + Women) / Year uni_size.index.name = Instelling! uni_size['2008'] = uni_size['2008 man'] + uni_size['2008 vrouw'] uni_size['2009'] = uni_size['2009 man'] + uni_size['2009 vrouw'] uni_size['2010'] = uni_size['2010 man'] + uni_size['2010 vrouw'] uni_size['2011'] = uni_size['2011 man'] + uni_size['2011 vrouw'] uni_size['2012'] = uni_size['2012 man'] + uni_size['2012 vrouw ]! # select only relevant columns uni_size = uni_size[[ 2008','2009','2010','2011','2012']]

51 Sort uni_size.sort(columns='2012', ascending=false).head()

52 Sort uni_size.sort(columns='2012', ascending=false).head()

53 Sort # axis=0 -> by row, axis=1 -> by column uni_size.sort(axis=1, ascending=false)

54 Sort # axis=0 -> by row, axis=1 -> by column uni_size.sort(axis=1, ascending=false)

55 uni_size.sort( axis=1, ascending=false).sort( columns='2012', ascending=false).plot( kind= barh', figsize=[10,10])

56 Quick One

57 Men/Women Diffs opl = wo.groupby('opleidingsnaam actueel').sum() opl = opl[['2012 man', '2012 vrouw']] opl['diff'] = opl['2012 man'] - opl['2012 vrouw'] sorted_opl = opl.sort(columns='diff', ascending=false)! top5_max = sorted_opl[:5] top5_min = sorted_opl[-5:] top5 = pd.concat([top5_max, top5_min])! top5['diff'].plot(kind='barh')

58 Men/Women Diffs

59 CS like studies cs_crit = wo['opleidingsnaam actueel ].\ str.contains('informatica') cs = wo[cs_crit] cs['opleidingsnaam actueel ].value_counts()! B Informatica 12 B Technische Informatica 8 Technische Informatica 6 Informatica 4 B Economie en Informatica 2 M Informatica 2 M Lerarenopleiding Informatica 1 dtype: int64

60 Pivot Tables pv = pd.pivot_table(wo, values=['2012 man', '2012 vrouw'], rows=['instellingsnaam actueel'], cols=['croho onderdeel'], fill_value=0, aggfunc=np.sum)

61 Pivot Tables

62 Multi-level index/columns Pivot Tables

63 Stack

64 Stack

65 Stack

66 Stack pv.stack(1)

67 Unstack pv.loc['universiteit van Amsterdam']! croho onderdeel 2012 man economie 2650 gedrag en maatschappij 2727 gezondheidszorg 1248 landbouw en natuurlijke omgeving 0 natuur 2152 onderwijs 125 recht 1573 sectoroverstijgend 365 taal en cultuur vrouw economie 1448 gedrag en maatschappij 5703 gezondheidszorg 2077 landbouw en natuurlijke omgeving 0 natuur 1344 onderwijs 124 recht 2305 sectoroverstijgend 497 taal en cultuur 4725 Name: Universiteit van Amsterdam, dtype: int64

68 Unstack Multi-level index pv.loc['universiteit van Amsterdam']! croho onderdeel 2012 man economie 2650 gedrag en maatschappij 2727 gezondheidszorg 1248 landbouw en natuurlijke omgeving 0 natuur 2152 onderwijs 125 recht 1573 sectoroverstijgend 365 taal en cultuur vrouw economie 1448 gedrag en maatschappij 5703 gezondheidszorg 2077 landbouw en natuurlijke omgeving 0 natuur 1344 onderwijs 124 recht 2305 sectoroverstijgend 497 taal en cultuur 4725 Name: Universiteit van Amsterdam, dtype: int64

69 Unstack! croho onderdeel 2012 man economie 2650 gedrag en maatschappij 2727 gezondheidszorg 1248 landbouw en natuurlijke omgeving 0 natuur 2152 onderwijs 125 recht 1573 sectoroverstijgend 365 taal en cultuur vrouw economie 1448 gedrag en maatschappij 5703 gezondheidszorg 2077 landbouw en natuurlijke omgeving 0 natuur 1344 onderwijs 124 recht 2305 sectoroverstijgend 497 taal en cultuur 4725 Name: Universiteit van Amsterdam, dtype: int64

72 Unstack uva = pv.loc['universiteit van Amsterdam'].unstack()

73 Transpose uva.t

74 Lots more Reording levels within multi-level indices Time Series (with up- & downsampling) Linear Regression (via statsmodels)

75 Want to learn more?