How to Detect and Fill Missing Values in Pandas (Python)

Channel:
United States DataDaft
Subscribers:
53,100
Published on ● Video Link: https://www.youtube.com/watch?v=AbBZYHNYFaY


Category:
Guide
Duration: 4:02
6,193 views
116

This video shows how to detect and fill missing values such as NaN, NA, None and the empty string in Pandas data frames. Detecting, counting and filling missing values or other odd values is a basic data exploration and cleaning step that is going to be necessary with all but the cleanest real world data sets.

If you find this video useful, like, share and subscribe to support the channel!
► Subscribe: https://www.youtube.com/c/DataDaft?sub_confirmation=1


Code used in this Python Code Clip:

import numpy as np
import pandas as pd

import statsmodels.api as sm #(To access mtcars dataset)
mtcars = sm.datasets.get_rdataset("mtcars", "datasets", cache=True).data

mtcars.iloc[1:4, 2:3] = np.NaN

mtcars.iloc[1:4, 3:4] = "NA"

mtcars.iloc[1:4, 4:5] = ""

mtcars["None_col"] = None

mtcars.head()

# Detect NaN and None with df.isnull() or df.isna()

null = pd.isnull(mtcars)
null.head()

# Count the total number of missing values

pd.isnull(mtcars).sum().sum()

# Detect a list of missing values with df.isin()

missing_vals = ["NA", "", None, np.NaN]
missing = mtcars.isin(missing_vals)
missing.head()

# Fill null values (NaN and None) with a given value:

mtcars.fillna(0).head()

# Fill a list of missing values with a given value:

missing_vals = ["NA", "", None, np.NaN]
missing = mtcars.isin(missing_vals) # Detect missing vals
mtcars.mask(missing, "missing").head() # Fill missing with df.mask()




* Note: YouTube does not allow greater than or less than symbols in the text description, so the code above will not be exactly the same as the code shown in the video! I will use Unicode large ＜ and ＞ symbols in place of the standard sized ones. .


⭐ Kite is a free AI-powered coding assistant that integrates with popular editors and IDEs to give you smart code completions and docs while you’re typing. It is a cool application of machine learning that can also help you code faster! Check it out here: https://www.kite.com/get-kite/?utm_medium=referral&utm_source=youtube&utm_campaign=datadaft&utm_content=description-only



Other Videos By DataDaft


2020-10-23How to Sort a Data Frame by a Column in Pandas
2020-10-22How to Spread Columns in Pandas (Python)
2020-10-21How to Gather Columns in Pandas (Python)
2020-10-19How to Remove Duplicate Rows From a Data Frame in Pandas (Python)
2020-10-06Python Regex: How To Remove Punctuation
2020-10-05Python Regex: How To Match All Whitespace
2020-10-02Python Regex: How To Replace All Substrings In a String
2020-10-01Python Regex: How To Split a String On Multiple Characters
2020-09-30Python Regex: How Find a Substring in a String
2020-09-21Fastest To OVER 9000: Battle of the GOATs
2020-09-17How to Detect and Fill Missing Values in Pandas (Python)
2020-09-11How to Use where() in Numpy and Pandas (Python)
2020-09-10How to Do Elementwise Math in Python
2020-09-04How to Add a Row To a Data Frame in Pandas (Python)
2020-09-03How to Remove a Row From a Data Frame in Pandas (Python)
2020-09-02How to Remove a Column From a Data Frame in Pandas (Python)
2020-08-29Python Programming Practice: LeetCode #104 -- Maximum Depth of Binary Tree
2020-08-28Python Programming Practice: LeetCode #136 -- Single Number
2020-08-27Python Programming Practice: LeetCode #283 -- Move Zeroes
2020-08-26Python Programming Practice: LeetCode #169 -- Majority Element
2020-08-25Python Programming Practice: LeetCode #70 -- Climbing Stairs


Tags:
detect missing values in pandas
fill missing values in pandas
detect na in pandas
detect nan in pandas
fill na in pandas
fill nan in pandas
detect na values in python
find na in pandas
find null values in pandas
fill null values in pandas
detect na in python
pandas data frame fill na
pandas data frame fill missing values
pandas count na
pandas count nan
pandas count missing values
pandas
python
nan
none
missing values
empty string
NA
data cleaning