[데이터전처리_3] Category Data 처리

아래 내용은 인프런 : 밑바닥부터 시작하는 머신러닝 입문 과정의 최성철 교수님 강. 의를 수강하고, 나름대로 한번 정리를 하여 더 오래 기억하고자 작성한 사항입니다.

일부 추가, 삭제, 수정한 사항들도 있습니다.

1. Category Data 처리

• 다음과 같은 데이터를 어떻게 처리할까? (Green, Blue, Yellow)

그 방법은 One-Hot Encoding . 
{Green, Blue, Yellow} 의 데이터 집합이 있을 때 데이터 set 의 크기만큼 Binary Feature 를 생성

• {Green} -> [1, 0, 0]
• {Blue} -> [0, 1, 0]
• {Yellow} -> [0, 0, 1]

In [41]:

import pandas as pd
import numpy as np

In [42]:

edges = pd.DataFrame({'source':[0, 1, 2],
                      'target':[2, 2, 3],
                      'weight':[3, 4, 5],
                        'color': ['red', 'blue', 'blue']})
edges

Out[42]:

	color	source	target	weight
0	red	0	2	3
1	blue	1	2	4
2	blue	2	3	5

In [43]:

edges.dtypes

Out[43]:

color     object
source     int64
target     int64
weight     int64
dtype: object

In [44]:

edges["color"]

Out[44]:

0     red
1    blue
2    blue
Name: color, dtype: object

In [45]:

pd.get_dummies(edges) # get_dummies 를 사용하면 문자열 특성만 인코딩되며 숫자 특성은 바뀌지 않음

Out[45]:

	source	target	weight	color_blue	color_red
0	0	2	3	0	1
1	1	2	4	1	0
2	2	3	5	1	0

In [46]:

pd.get_dummies(edges["color"])

Out[46]:

	blue	red
0	0	1
1	1	0
2	1	0

In [47]:

pd.get_dummies(edges[["color"]]) ## [] 를 하나 더 사용하면 column 명에 _ 가 붙음 

Out[47]:

	color_blue	color_red
0	0	1
1	1	0
2	1	0

weight(Ordinary data) 을 One Hot Encoding 으로 변경해보자

In [48]:

weight_dict = {3:"M", 4:"L", 5:"XL"}
edges["weight_sign"] = edges["weight"].map(weight_dict)
edges

Out[48]:

	color	source	target	weight	weight_sign
0	red	0	2	3	M
1	blue	1	2	4	L
2	blue	2	3	5	XL

In [49]:

weight_sign = pd.get_dummies(edges["weight_sign"])
weight_sign

Out[49]:

	L	M	XL
0	0	1	0
1	1	0	0
2	0	0	1

In [50]:

pd.concat([edges, weight_sign], axis=1)

Out[50]:

	color	source	target	weight	weight_sign	L	M	XL
0	red	0	2	3	M	0	1	0
1	blue	1	2	4	L	1	0	0
2	blue	2	3	5	XL	0	0	1

In [51]:

edges

Out[51]:

	color	source	target	weight	weight_sign
0	red	0	2	3	M
1	blue	1	2	4	L
2	blue	2	3	5	XL

In [52]:

pd.get_dummies(edges)

Out[52]:

	source	target	weight	color_blue	color_red	weight_sign_L	weight_sign_M	weight_sign_XL
0	0	2	3	0	1	0	1	0
1	1	2	4	1	0	1	0	0
2	2	3	5	1	0	0	0	1

In [53]:

pd.get_dummies(edges).values

Out[53]:

array([[0, 2, 3, 0, 1, 0, 1, 0],
       [1, 2, 4, 1, 0, 1, 0, 0],
       [2, 3, 5, 1, 0, 0, 0, 1]], dtype=int64)

2. Data Binning

• pd.cut 함수를 이용. 예시는 아래와 같다.

In [54]:

raw_data = {
    'regiment': ['Nighthawks', 'Nighthawks', 'Nighthawks', 'Nighthawks', 'Dragoons', 'Dragoons', 'Dragoons', 'Dragoons', 'Scouts', 'Scouts', 'Scouts', 'Scouts'],
        'company': ['1st', '1st', '2nd', '2nd', '1st', '1st', '2nd', '2nd','1st', '1st', '2nd', '2nd'],
        'name': ['Miller', 'Jacobson', 'Ali', 'Milner', 'Cooze', 'Jacon', 'Ryaner', 'Sone', 'Sloan', 'Piger', 'Riani', 'Ali'],
        'preTestScore': [4, 24, 31, 2, 3, 4, 24, 31, 2, 3, 2, 3],
        'postTestScore': [25, 94, 57, 62, 70, 25, 94, 57, 62, 70, 62, 70]
}
df = pd.DataFrame(raw_data, columns = ['regiment', 'company', 'name', 'preTestScore', 'postTestScore'])
df

Out[54]:

	regiment	company	name	preTestScore	postTestScore
0	Nighthawks	1st	Miller	4	25
1	Nighthawks	1st	Jacobson	24	94
2	Nighthawks	2nd	Ali	31	57
3	Nighthawks	2nd	Milner	2	62
4	Dragoons	1st	Cooze	3	70
5	Dragoons	1st	Jacon	4	25
6	Dragoons	2nd	Ryaner	24	94
7	Dragoons	2nd	Sone	31	57
8	Scouts	1st	Sloan	2	62
9	Scouts	1st	Piger	3	70
10	Scouts	2nd	Riani	2	62
11	Scouts	2nd	Ali	3	70

In [55]:

bins = [0, 25, 50, 75, 100] # 구간 정의 0~25, 25~50, 50~75, 75~100
group_name = ['Low', 'Okay', 'Good', 'Great']
categories = pd.cut(df['postTestScore'], bins, labels=group_name)
categories

Out[55]:

0       Low
1     Great
2      Good
3      Good
4      Good
5       Low
6     Great
7      Good
8      Good
9      Good
10     Good
11     Good
Name: postTestScore, dtype: category
Categories (4, object): [Low < Okay < Good < Great]

In [56]:

df['categories'] = pd.cut(df['postTestScore'], bins, labels=group_name)

In [57]:

pd.value_counts(df['categories'])

Out[57]:

Good     8
Great    2
Low      2
Okay     0
Name: categories, dtype: int64

In [58]:

df

Out[58]:

	regiment	company	name	preTestScore	postTestScore	categories
0	Nighthawks	1st	Miller	4	25	Low
1	Nighthawks	1st	Jacobson	24	94	Great
2	Nighthawks	2nd	Ali	31	57	Good
3	Nighthawks	2nd	Milner	2	62	Good
4	Dragoons	1st	Cooze	3	70	Good
5	Dragoons	1st	Jacon	4	25	Low
6	Dragoons	2nd	Ryaner	24	94	Great
7	Dragoons	2nd	Sone	31	57	Good
8	Scouts	1st	Sloan	2	62	Good
9	Scouts	1st	Piger	3	70	Good
10	Scouts	2nd	Riani	2	62	Good
11	Scouts	2nd	Ali	3	70	Good

In [59]:

pd.get_dummies(df)

Out[59]:

	preTestScore	postTestScore	regiment_Dragoons	regiment_Nighthawks	regiment_Scouts	company_1st	company_2nd	name_Ali	name_Cooze	name_Jacobson	...	name_Milner	name_Piger	name_Riani	name_Ryaner	name_Sloan	name_Sone	categories_Low	categories_Okay	categories_Good	categories_Great
0	4	25	0	1	0	1	0	0	0	0	...	0	0	0	0	0	0	1	0	0	0
1	24	94	0	1	0	1	0	0	0	1	...	0	0	0	0	0	0	0	0	0	1
2	31	57	0	1	0	0	1	1	0	0	...	0	0	0	0	0	0	0	0	1	0
3	2	62	0	1	0	0	1	0	0	0	...	1	0	0	0	0	0	0	0	1	0
4	3	70	1	0	0	1	0	0	1	0	...	0	0	0	0	0	0	0	0	1	0
5	4	25	1	0	0	1	0	0	0	0	...	0	0	0	0	0	0	1	0	0	0
6	24	94	1	0	0	0	1	0	0	0	...	0	0	0	1	0	0	0	0	0	1
7	31	57	1	0	0	0	1	0	0	0	...	0	0	0	0	0	1	0	0	1	0
8	2	62	0	0	1	1	0	0	0	0	...	0	0	0	0	1	0	0	0	1	0
9	3	70	0	0	1	1	0	0	0	0	...	0	1	0	0	0	0	0	0	1	0
10	2	62	0	0	1	0	1	0	0	0	...	0	0	1	0	0	0	0	0	1	0
11	3	70	0	0	1	0	1	1	0	0	...	0	0	0	0	0	0	0	0	1	0

12 rows × 22 columns

3. Label encoding by sklearn

In [60]:

raw_example = df.as_matrix()
raw_example[:3]

Out[60]:

array([['Nighthawks', '1st', 'Miller', 4, 25, 'Low'],
       ['Nighthawks', '1st', 'Jacobson', 24, 94, 'Great'],
       ['Nighthawks', '2nd', 'Ali', 31, 57, 'Good']], dtype=object)

In [61]:

data = raw_example.copy()
data

Out[61]:

array([['Nighthawks', '1st', 'Miller', 4, 25, 'Low'],
       ['Nighthawks', '1st', 'Jacobson', 24, 94, 'Great'],
       ['Nighthawks', '2nd', 'Ali', 31, 57, 'Good'],
       ['Nighthawks', '2nd', 'Milner', 2, 62, 'Good'],
       ['Dragoons', '1st', 'Cooze', 3, 70, 'Good'],
       ['Dragoons', '1st', 'Jacon', 4, 25, 'Low'],
       ['Dragoons', '2nd', 'Ryaner', 24, 94, 'Great'],
       ['Dragoons', '2nd', 'Sone', 31, 57, 'Good'],
       ['Scouts', '1st', 'Sloan', 2, 62, 'Good'],
       ['Scouts', '1st', 'Piger', 3, 70, 'Good'],
       ['Scouts', '2nd', 'Riani', 2, 62, 'Good'],
       ['Scouts', '2nd', 'Ali', 3, 70, 'Good']], dtype=object)

In [62]:

from sklearn import preprocessing
le = preprocessing.LabelEncoder()

In [63]:

raw_example[:,0]

Out[63]:

array(['Nighthawks', 'Nighthawks', 'Nighthawks', 'Nighthawks', 'Dragoons',
       'Dragoons', 'Dragoons', 'Dragoons', 'Scouts', 'Scouts', 'Scouts',
       'Scouts'], dtype=object)

In [64]:

le.fit(raw_example[:,0])

Out[64]:

LabelEncoder()

In [65]:

le.classes_

Out[65]:

array(['Dragoons', 'Nighthawks', 'Scouts'], dtype=object)

In [66]:

le.transform(raw_example[:,0])

Out[66]:

array([1, 1, 1, 1, 0, 0, 0, 0, 2, 2, 2, 2], dtype=int64)

In [67]:

data[:,0] =le.transform(raw_example[:,0])
data[:3]

Out[67]:

array([[1, '1st', 'Miller', 4, 25, 'Low'],
       [1, '1st', 'Jacobson', 24, 94, 'Great'],
       [1, '2nd', 'Ali', 31, 57, 'Good']], dtype=object)

In [68]:

label_column = [0, 1, 2, 5]
label_encoder_list = []
for column_index in label_column:
    le = preprocessing.LabelEncoder()
    le.fit(raw_example[:, column_index])
    data[:, column_index] = le.transform(raw_example[:,column_index ])
    label_encoder_list.append(le)
    del le
    
data[:3]

Out[68]:

array([[1, 0, 4, 4, 25, 2],
       [1, 0, 2, 24, 94, 1],
       [1, 1, 0, 31, 57, 0]], dtype=object)

In [69]:

label_encoder_list[0].transform(raw_example[:10, 0])

Out[69]:

array([1, 1, 1, 1, 0, 0, 0, 0, 2, 2], dtype=int64)

4. One-hot encoding by sklearn

In [70]:

one_hot_enc = preprocessing.OneHotEncoder()
data[:,0].reshape(-1,1)

Out[70]:

array([[1],
       [1],
       [1],
       [1],
       [0],
       [0],
       [0],
       [0],
       [2],
       [2],
       [2],
       [2]], dtype=object)

In [71]:

one_hot_enc.fit(data[:,0].reshape(-1,1))

Out[71]:

OneHotEncoder(categorical_features='all', dtype=<class 'numpy.float64'>,
       handle_unknown='error', n_values='auto', sparse=True)

In [72]:

one_hot_enc.n_values_

Out[72]:

array([3])

In [73]:

one_hot_enc.active_features_

Out[73]:

array([0, 1, 2], dtype=int64)

In [74]:

onehotlabels = one_hot_enc.transform(data[:,0].reshape(-1,1)).toarray()

In [75]:

onehotlabels

Out[75]:

array([[ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 1.,  0.,  0.],
       [ 1.,  0.,  0.],
       [ 1.,  0.,  0.],
       [ 1.,  0.,  0.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.]])

In [ ]: