Understand the 'DATA SET' before doing any machine learning project


  1. Take a look at the raw data.
  2. Check the dimensions of the data set.
  3. Review the datatypes of attributes in the data.
  4. check the class distribution.
  5. Review the descriptive statistics of the data.
  6. Understand the relationships in the data using correlations.
  7. Review the skew of the distributions of each attribute

Looking at the raw data can reveal some insights into the data. Here we print the first 
20 rows of our data using the head( ) function on the Pandas DataFrame. 







   2. Check the dimensions of the data set.

We can review the shape and size of our dataset using the shape property of the Pandas DataFrame. We can check the following by checking the dimensions.


  • We can check whether or not there is enough training data and also
  • Too much training data.
  • Too many features.
  • Less features.




3. Review the data types of the attributes in the data.

The type of the attribute is very important as we may need to do any conversion like string to float etc.,







4. Check the class distribution

We can quickly check get an idea of the distribution of the class attribute in Pandas






5. Review the statistics of the data

We can get a great insight of each attribute like count, Mean, Standard deviation etc.,








6. Understanding the relationships in the data using correlations

Correlation refers to the relationship between the two variables and how they will change together.
  • 0 indicates no correlation
  • close to 1 indicates positive correlation
  • close to -1 indicates negative correlation
It is evident that pregnancy has a relationship with age factor.







7. Review the skew of the distribution for each attribute
Skew refers to the distribution which is assumed Gaussian






Comments

Post a Comment

Popular posts from this blog

Image
Feature selection for Machine Learning The data features used to train the machine learning model have a great impact on the ultimate performance. Irrelevant or partially relevant feature can negatively influence the model. The various automatic feature selection techniques are: Univariate Selection Recursive feature elimination Principal Component Analysis Feature Importance   Benefits of feature selection techniques: Reduces overfitting  Improves accuracy  Reduces training time Univariate Selection: This selection can be used to select the features that have the strongest relationship with the output variable. The example below uses scikit-learn which provides SeleceKBest class that can be used combinely with chi-squared (chi2) statistical test for non-negative features to select 4 of the best features from the dataset(Pima Indian). Recursive Feature Elimination: REF works by recursively removing the attributes a...
Read more
Image
Knowing sample dataset for use in Machine Learning The Pima Indians Diabetic Dataset is used to illustrate the Machine Learning concepts. The dataset describes the medical records for Pima Indians and whether or not each patient will have an onset of diabetes within five years. # Columns description    Pregnancies(preg) Number of times pregnant Glucose(plas) Plasma glucose concentration a 2 hours in an oral glucose tolerance test BloodPressure(pres) Diastolic blood pressure (mm Hg) SkinThickness(skin) Triceps skin fold thickness (mm) Insulin(test) 2-Hour serum insulin (mu U/ml) BMI(mass) Body mass index (weight in kg/(height in m)^2) DiabetesPedigreeFunction(pedi) Diabetes pedigree function Age(age) Age (years) Outcome(class) Class variable (0 or 1) 268 of 768 are 1, the others are 0 ...
Read more
Image
How to load Machine Learning Data The most common format to load machine learning data is CSV files. The most common ways to load the data in Python: Load CSV files using Python Standard Library. Load CSV files with NumPy. Load CSV files with Pandas.
Read more