---
title: "Pandas use case"
description: "Data preprocessing and visualisation of a credit scoring dataset"
date: "2024-02-07"
jupyter: python3
execute: 
  eval: false
---


We'll work on a dataset `gro.csv` for **credit scoring** that was proposed some years ago as a data challenge on some data challenge website.
It is a realistic and somewhat messy dataset that contains a lot of missing values, several types of features (dates, categories, continuous features), so that serious data cleaning and formating is required.
This dataset contains the following columns:

| Column name          | Description |
|:---------------------|:------------|
| BirthDate            | Date of birth of the client |
| Customer_Open_Date   | Creation date of the client's first account at the bank |
| Customer_Type        | Type of client (existing / new) | 
| Educational_Level    | Highest diploma |
| Id_Customer          | Id of the client |
| Marital_Status       | Family situation |
| Nb_Of_Products       | Number of products held by the client |
| Net_Annual_Income    | Annual revenue |
| Number_Of_Dependant  | Number of dependents |
| P_Client             | Non-disclosed feature |
| Prod_Category        | Product category |
| Prod_Closed_Date     | Closing date of the last product |
| Prod_Decision_Date   | Decision date of the last agreement for a financing product |
| Prod_Sub_Category    | Sub-category of the product |
| Source               | Financing source (Branch or Sales) |
| Type_Of_Residence    | Residential situation |
| Y                    | Credit was granted (yes / no) |
| Years_At_Business    | Number of year at the current job position |
| Years_At_Residence   | Number of year at the current housing |

# Your job

Read the `gro.csv` dataset and work on it using `pandas`, `matplotlib` and `seaborn`.
- The column separator in the CSV file is not `,` but `;` so you need to use the `sep` option in `pd.read_csv`
- The categorical columns must be imported as `category` type 
- Something weird is going on with the `Net_Annual_Income` column... Try to  understand what is going on and try to correct the problem
- Several columns are empty, we need to remove them (or not even read them)
- Dates must be imported as dates and not strings
- Remove rows with missing values

Many of these things can be done right from the beginning, when reading the CSV file, through some options to the `pd.read_csv` function. You might need to read carefully its documentation in order to understand some useful options. Once you are happy with your importation and cleaning of the data, you can:
- Use `matplotlib` and `pandas` to perform data visualization...
- ... in order to understand visually the impact of some features on `Y` (credit was granted or not). For this, you need to decide on the plots that make sense for this and produce them

We will provide thorough explanations and code that performs all of this in subsequent sessions.


# A quick and easy (but actually bad) import

Let's import the data into a pandas dataframe, as simply as possible
The only thing we care about for now is the fact that the column separator 
is `';'` and not `','` as it should be in a `.csv` file.

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:41:56.108900Z', start_time: '2022-01-26T08:41:56.049443Z'}
import requests
import os

# The path containing your notebook
path_data = './'
# The name of the file
filename = 'gro.csv.gz'

if os.path.exists(os.path.join(path_data, filename)):
    print('The file %s already exists.' % os.path.join(path_data, filename))
else:
    url = 'https://stephanegaiffas.github.io/big_data_course/data/gro.csv.gz'
    r = requests.get(url)
    with open(os.path.join(path_data, filename), 'wb') as f:
        f.write(r.content)
    print('Downloaded file %s.' % os.path.join(path_data, filename))
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:41:59.632842Z', start_time: '2022-01-26T08:41:59.362267Z'}
#| scrolled: false
import numpy as np
import pandas as pd

filename = "gro.csv.gz"
df = pd.read_csv(filename, sep=';')
df.head(n=5)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:42:21.832814Z', start_time: '2022-01-26T08:42:21.827652Z'}
df.columns
```

**Remark**. There are weird columns in the end, they look empty. 
They don't appear in the description of the data.

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:43:22.688396Z', start_time: '2022-01-26T08:43:22.668087Z'}
df.info()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:44:01.929033Z', start_time: '2022-01-26T08:44:01.923127Z'}
df["BirthDate"].head()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:44:31.953941Z', start_time: '2022-01-26T08:44:31.949306Z'}
type(df.loc[0, 'BirthDate'])
```

This means that dates are indeed imported as a strings...

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:44:35.422848Z', start_time: '2022-01-26T08:44:35.416701Z'}
#| scrolled: false
df['Prod_Sub_Category'].head()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:44:49.621150Z', start_time: '2022-01-26T08:44:49.616182Z'}
#| scrolled: true
type(df.loc[0, 'Prod_Sub_Category'])
```

Categorical variables are imported as a strings as well

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:44:49.915041Z', start_time: '2022-01-26T08:44:49.908977Z'}
#| scrolled: false
df['Net_Annual_Income'].head(n=10)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:44:58.913595Z', start_time: '2022-01-26T08:44:58.908912Z'}
#| scrolled: true
type(df.loc[0, 'Net_Annual_Income'])
```

Net actual income is a string as well ! While it is clearly a number !!!

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:45:22.503815Z', start_time: '2022-01-26T08:45:22.414339Z'}
#| scrolled: false
df.describe(include='all')
```

## Let's assess what we did

It appears that we have to work a little bit more for a correct import of the data.
Here is a list of the problems we face.
- The last three columns are empty
- Dates are actually `str` (python's **string** type)
- There is a lot of missing values
- Categorial features are `str`
- The `Net_Annual_Income` is imported as a string

By looking at the column names, the descriptions of the columns and using some basic, we infer the type of features that we have.
There are dates features, continuous features, categorical features, and
some features that could be either treated as categorical or continuous.

- There are **many** missing values, that need to be handled.
- The annual net income is imported as a string, we need to understand why.
- We really need to treat dates as dates and not strings (because we want to compute the age of a client based on its birth year for instance).

Here is a tentative structure of the features

**Continuous features**

- `Years_At_Residence`
- `Net_Annual_Income`
- `Years_At_Business`

**Features to be decided**

- `Number_Of_Dependant`
- `Nb_Of_Products`

**Categorical features**

- `Customer_Type`
- `P_Client`
- `Educational_Level`
- `Marital_Status`
- `Prod_Sub_Category`
- `Source`
- `Type_Of_Residence`
- `Prod_Category`

**Date features**

- `BirthDate`
- `Customer_Open_Date`
- `Prod_Decision_Date`
- `Prod_Closed_Date`

# A closer look at the import problems

Let's find solutions to all these import problems.

## The last three columns are weird and empty 

It seems to come from the fact that the data always ends with several `';'` characters. 
We can remove them simply using the `usecols` option from `read_csv`.

## Dates are actually `str`

We need to specify which columns must be encoded as dates using the `parse_dates` option from `read_csv`.
Fortunately enough, `pandas` is clever enough to interpret the date format.

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:47:15.127321Z', start_time: '2022-01-26T08:47:15.122017Z'}
#| scrolled: true
type(df.loc[0, 'BirthDate'])
```

## There is a lot of missing values 

We'll see below that actually a single column mostly contain missing values.

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:47:17.216136Z', start_time: '2022-01-26T08:47:17.202815Z'}
#| scrolled: false
df.isnull().sum()
```

The column `Prod_Closed_Date` contains mostly missing values !

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:47:59.141586Z', start_time: '2022-01-26T08:47:59.133465Z'}
df[['Prod_Closed_Date']].head(5)
```

Let's remove the useless columns and check the remaining missing values

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:49:35.453574Z', start_time: '2022-01-26T08:49:35.369144Z'}
df.drop(['Prod_Closed_Date', 'Unnamed: 19', 
         'Unnamed: 20', 'Unnamed: 21'], axis="columns", inplace=True)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:50:26.872707Z', start_time: '2022-01-26T08:50:26.852940Z'}
#| scrolled: true
df.head()
```

Let's display the rows with missing values and let's highlight them

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:50:29.129812Z', start_time: '2022-01-26T08:50:28.848546Z'}
#| scrolled: false
df[df.isnull().any(axis="columns")].style.highlight_null()
```

## Categorial features are `str`

We need to say the dtype we want to use for some columns using the `dtype` option of `read_csv`.

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:51:23.307753Z', start_time: '2022-01-26T08:51:23.302513Z'}
#| scrolled: true
type(df.loc[0, 'Prod_Sub_Category'])
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:51:23.431694Z', start_time: '2022-01-26T08:51:23.426862Z'}
#| scrolled: false
df['Prod_Sub_Category'].unique()
```

## The annual net income is imported as a string

This problem comes from the fact that the decimal separator is in European notation: it's a `','` and not a `'.'`, so we need to specify it using the `decimal` option to `read_csv`. (Data is French, pardon my French...) 

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:51:31.734757Z', start_time: '2022-01-26T08:51:31.729725Z'}
#| scrolled: true
type(df.loc[0, 'Net_Annual_Income'])
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:51:32.594904Z', start_time: '2022-01-26T08:51:32.589396Z'}
#| scrolled: false
df['Net_Annual_Income'].head(n=10)
```

# A correct import of the data

- We build a dict that specifies the dtype to use for each column 
and pass it to `read_csv` using the `dtype` option
- We also specify the `decimal`, `usecols` and `parse_dates` options

**Very pro remark.** Some columns could be imported as `int`. 
However, `pandas` (actually its `numpy`) does not support columns 
with integer dtype and missing values.

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:53:26.355301Z', start_time: '2022-01-26T08:53:26.305085Z'}
gro_dtypes = {
    'Years_At_Residence': np.int64,
    'Net_Annual_Income' : np.float64,
    'Years_At_Business': np.float64,
    'Number_Of_Dependant': np.float64,
    'Nb_Of_Products': np.int64,
    'Customer_Type': 'category',
    'P_Client': 'category',
    'Educational_Level': 'category',
    'Marital_Status': 'category',
    'Prod_Sub_Category': 'category',
    'Source': 'category',
    'Type_Of_Residence': 'category',
    'Prod_Category': 'category',
}

df = pd.read_csv(
    # Filename
    "gro.csv.gz",
    # Column separator
    sep=';',
    # Decimal separator
    decimal=',',
    # Range of the columns to keep (remove the last three ones)
    usecols=range(19),
    # Which columns should be parsed as dates
    parse_dates=['BirthDate', 'Customer_Open_Date', 'Prod_Decision_Date', 
                 'Prod_Closed_Date'],
    # Specify some dtypes
    dtype=gro_dtypes
)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:53:28.223129Z', start_time: '2022-01-26T08:53:28.191202Z'}
#| scrolled: false
df.info()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:54:35.304338Z', start_time: '2022-01-26T08:54:35.297788Z'}
df['Prod_Sub_Category'].head()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:55:27.982853Z', start_time: '2022-01-26T08:55:27.979592Z'}
truc = df['Prod_Sub_Category'].head()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:54:35.618441Z', start_time: '2022-01-26T08:54:35.613698Z'}
df.loc[0, 'BirthDate']
```

Let's remove `Prod_Closed_Date` (mostly contains missing values)

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:58:20.302851Z', start_time: '2022-01-26T08:58:20.296797Z'}
prod_closed_date = df.pop('Prod_Closed_Date')
df.shape
```

And remove the remaining rows with missing values

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:58:20.747526Z', start_time: '2022-01-26T08:58:20.736836Z'}
#| scrolled: true
print(df.shape)
df = df.dropna()
print(df.shape)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:58:43.264816Z', start_time: '2022-01-26T08:58:43.044315Z'}
# Now we save the cleaned dataset into a CSV file
df.to_csv("gro_cleaned.csv")
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T08:59:40.899844Z', start_time: '2022-01-26T08:59:40.543853Z'}
ls -al gro_*
```

## Comment on file formats

You can use other methods starting with `.to_XX` to save in another format.
Here are some main examples

- OK to use `csv` for "small" datasets (several MB)
- Use `pickle` for more compressed and faster format (limited to 4GB). It's the standard binary serialization format of `Python`
- `feather` is another fast and lightweight file format for storing data frames. A very popular exchange format. 
- `parquet` is a format for big distributed data (works nicely with `Spark`)

among several others...

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:00:24.678383Z', start_time: '2022-01-26T09:00:24.451613Z'}
df.to_pickle("gro_cleaned.pkl")
df.to_parquet("gro_cleaned.parquet")
df.reset_index().to_feather("gro_cleaned.feather")
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:00:24.684721Z', start_time: '2022-01-26T09:00:24.680205Z'}
df.index
```

And you can read again using the corresponding `read_XX` function

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:00:26.948108Z', start_time: '2022-01-26T09:00:26.920795Z'}
df = pd.read_pickle("gro_cleaned.pkl")
df.head()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:00:53.060529Z', start_time: '2022-01-26T09:00:52.739375Z'}
ls -alh gro_cleaned*
```

## The net income columns is very weird

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:00:54.182160Z', start_time: '2022-01-26T09:00:54.169740Z'}
#| scrolled: true
income = df['Net_Annual_Income']
income.describe()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:01:30.007015Z', start_time: '2022-01-26T09:01:30.001154Z'}
(income <= 100).sum(), (income > 100).sum()
```

Most values are smaller than 100, while some are much much larger...

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:01:34.268341Z', start_time: '2022-01-26T09:01:34.264165Z'}
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline

sns.set_context("notebook", font_scale=1.2)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:01:36.544112Z', start_time: '2022-01-26T09:01:36.314127Z'}
sns.displot(x='Net_Annual_Income', 
            data=df, 
            bins=20,
            height=4, 
            aspect=1.5)
```

This is annoying, we don't really see much...

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:01:39.935025Z', start_time: '2022-01-26T09:01:39.401260Z'}
#| scrolled: false
sns.displot(x='Net_Annual_Income', 
            data=df, 
            bins=20, 
            height=4, 
            aspect=1.5, 
            log_scale=(False, True))
```

Distribution for less than 100K revenue

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:01:45.765082Z', start_time: '2022-01-26T09:01:45.595609Z'}
sns.displot(x='Net_Annual_Income', 
            data=df[df['Net_Annual_Income'] < 100], 
            bins=15, 
            height=4, 
            aspect=1.5)
```

Distribution for less than 400K revenue

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:01:48.520303Z', start_time: '2022-01-26T09:01:48.332497Z'}
#| scrolled: false
sns.displot(x='Net_Annual_Income', data=df[df['Net_Annual_Income'] < 400], 
            bins=15, height=4, aspect=1.5)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:02:09.418067Z', start_time: '2022-01-26T09:02:09.412979Z'}
#| scrolled: true
(df['Net_Annual_Income'] == 36.0).sum()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:02:09.612302Z', start_time: '2022-01-26T09:02:09.590381Z'}
#| scrolled: false
income_counts = (
    pd.DataFrame({
        "income_category": df['Net_Annual_Income'].astype("category"),
        "income": df['Net_Annual_Income']
    })
    .groupby("income_category")
    .count()
    .reset_index()
    .rename(columns={"income": "#customers"})
    .sort_values(by="#customers", axis="index", ascending=False)
)

income_counts["%cummulative clients"] \
    = income_counts["#customers"].cumsum() / income_counts["#customers"].sum()

income_counts.iloc[:20].style.bar(subset=["%cummulative clients"], vmin=0, vmax=1)
```

- We have some overrepresented values (many possible explanations for this)
- To clean the data, we can, for instance, keep only the revenues between [10, 200], or leave it as such

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:02:15.637841Z', start_time: '2022-01-26T09:02:15.462548Z'}
#| scrolled: false
df = df[(df['Net_Annual_Income'] >= 10) & (df['Net_Annual_Income'] <= 200)]

sns.displot(x='Net_Annual_Income', data=df, bins=15, height=4, aspect=1.5)
```

# Some data visualization with `pandas` + `seaborn`

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:02:19.682028Z', start_time: '2022-01-26T09:02:19.329145Z'}
#| scrolled: false
plt.figure(figsize=(8, 5))
sns.stripplot(x='Educational_Level', 
              y='Net_Annual_Income', 
              hue='Y', 
              jitter=True, 
              data=df)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:02:19.999101Z', start_time: '2022-01-26T09:02:19.684915Z'}
plt.figure(figsize=(12, 6))
sns.boxplot(x='Educational_Level', y='Net_Annual_Income', 
            hue='Y', data=df)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:02:20.253119Z', start_time: '2022-01-26T09:02:20.030215Z'}
#| scrolled: false
plt.figure(figsize=(12, 6))
sns.violinplot(x='Marital_Status', y='Net_Annual_Income', 
               hue='Y', split=True, data=df)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:02:22.371294Z', start_time: '2022-01-26T09:02:22.187081Z'}
plt.figure(figsize=(10, 5))
sns.countplot(x='Marital_Status', hue='Y', data=df)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:02:38.731741Z', start_time: '2022-01-26T09:02:37.647614Z'}
#| scrolled: false
fig, axes = plt.subplots(nrows=4, ncols=3, figsize=(16, 16))
columns = ['Customer_Type', 'P_Client', 'Educational_Level', 
           'Number_Of_Dependant', 'Marital_Status', 'Prod_Sub_Category',
           'Source', 'Type_Of_Residence', 'Nb_Of_Products', 
           'Prod_Category', 'Y']

for i, colname in enumerate(columns):
    sns.countplot(x=colname, data=df, ax=fig.axes[i])
plt.tight_layout()
```

# Final preparation of the dataset

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:04:00.912103Z', start_time: '2022-01-26T09:04:00.908216Z'}
# First we make lists of continuous, categorial and date features

cnt_featnames = [
    'Years_At_Residence',
    'Net_Annual_Income',
    'Years_At_Business',
    'Number_Of_Dependant'
]

cat_featnames = [
    'Customer_Type',
    'P_Client',
    'Educational_Level',
    'Marital_Status',
    'Prod_Sub_Category',
    'Source',
    'Type_Of_Residence',
    'Prod_Category',
    'Nb_Of_Products'
]

date_featnames = [
    'BirthDate',
    'Customer_Open_Date',
    'Prod_Decision_Date'
    #'Prod_Closed_Date'
]
```

## Creation of the features matrix

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:04:01.678608Z', start_time: '2022-01-26T09:04:01.666246Z'}
#| scrolled: false
df[cnt_featnames].head()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:04:16.173481Z', start_time: '2022-01-26T09:04:16.143985Z'}
#| scrolled: false
bin_features = pd.get_dummies(df[cat_featnames],
                              prefix_sep='#', drop_first=True)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:04:17.233373Z', start_time: '2022-01-26T09:04:17.214560Z'}
#| scrolled: false
bin_features.head()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:04:26.356194Z', start_time: '2022-01-26T09:04:26.344810Z'}
cnt_features = df[cnt_featnames]
cnt_features.head()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:08:33.786380Z', start_time: '2022-01-26T09:08:33.771973Z'}
#| scrolled: true
from pandas import Timestamp

def age(x):
    today = Timestamp.today()
    return (today - x).dt.days

date_features = df[date_featnames].apply(age, axis="index")
date_features.head()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:09:00.929864Z', start_time: '2022-01-26T09:09:00.925726Z'}
today = Timestamp.today()
today
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:09:51.560767Z', start_time: '2022-01-26T09:09:51.556643Z'}
tt = (today - df["BirthDate"]).loc[0]
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:10:28.879136Z', start_time: '2022-01-26T09:10:28.871614Z'}
(today - df["BirthDate"]).dt.days
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:09:53.715142Z', start_time: '2022-01-26T09:09:53.711322Z'}
tt
```

## Final features matrix

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:10:40.419444Z', start_time: '2022-01-26T09:10:40.413677Z'}
all_features = pd.concat([bin_features, cnt_features, date_features], axis=1)
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:10:40.839744Z', start_time: '2022-01-26T09:10:40.834595Z'}
#| scrolled: true
all_features.columns
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:10:41.033968Z', start_time: '2022-01-26T09:10:41.011917Z'}
all_features.head()
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:14:19.156063Z', start_time: '2022-01-26T09:14:19.151924Z'}
df_debile = pd.DataFrame({"nom etudiant": ["yiyang", "jaouad", "mokhtar", "massil", "simon"], 
              "portable": [True, True, None, True, False]})
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:14:23.985250Z', start_time: '2022-01-26T09:14:23.976807Z'}
df_debile
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:15:05.220160Z', start_time: '2022-01-26T09:15:05.214989Z'}
df_debile.index
```

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:15:27.978917Z', start_time: '2022-01-26T09:15:27.972342Z'}
df_debile.dropna().index
```


```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:14:40.099302Z', start_time: '2022-01-26T09:14:40.088308Z'}
df_debile.info()
```



**VERY IMPORTANT**: we removed lines of data that contained missing values. The index of the dataframe is
    therefore not contiguous anymore

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:11:59.090408Z', start_time: '2022-01-26T09:11:59.085196Z'}
all_features.index.max()
```

This could be a problem for later. So let's reset the index to get a contiguous one

```{python}
#| ExecuteTime: {end_time: '2022-01-26T09:11:59.457045Z', start_time: '2022-01-26T09:11:59.451256Z'}
all_features.shape
```

```{python}
#| ExecuteTime: {end_time: '2021-01-15T10:06:59.865987Z', start_time: '2021-01-15T10:06:59.862161Z'}
all_features.reset_index(inplace=True, drop=True)
```

```{python}
#| ExecuteTime: {end_time: '2021-01-15T10:06:59.894460Z', start_time: '2021-01-15T10:06:59.869361Z'}
all_features.head()
```

## Let's save the data using `pickle`

```{python}
#| ExecuteTime: {end_time: '2021-01-15T10:06:59.915443Z', start_time: '2021-01-15T10:06:59.896682Z'}
import pickle as pkl

X = all_features
y = df['Y']

# Let's put eveything in a dictionary
df_pkl = {}
# The features and the labels
df_pkl['features'] = X
df_pkl['labels'] = y
# And also the list of columns we built above
df_pkl['cnt_featnames'] = cnt_featnames
df_pkl['cat_featnames'] = cat_featnames
df_pkl['date_featnames'] = date_featnames

with open("gro_training.pkl", 'wb') as f:
    pkl.dump(df_pkl, f)
```

```{python}
#| ExecuteTime: {end_time: '2021-01-15T10:07:00.206231Z', start_time: '2021-01-15T10:06:59.917845Z'}
ls -al gro*
```

The preprocessed data is saved in a pickle file called `gro_training.pkl`.

