We can define a Recommendation system as a collection of a bunch of algorithms that are used to recommend items to the users based on the information taken from the user. These systems have become very common and are widely used and can be commonly seen in online stores, movies databases and job finders.
A very simple representation of a content based recommendation system for movies looks like this
In this blog, we will explore Content-based recommendation systems and implement a simple version of one using Python and the Pandas library.
Please run these 2 commands to download and unzip the data
!wget -O moviedataset.zip https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/moviedataset.zip
('and the command 2 is ...')
!unzip -o -j moviedataset.zip
Now you're ready to start working with the data!
First, let's get all of the imports out of the way:
# Pandas - Dataframe manipulation library
import pandas as pd
# Math - Math functions, we'll only need the sqrt function so let's import only that
from math import sqrt
# numpy - for all numeric operations
import numpy as np
# matplotlib - to plot charts
import matplotlib.pyplot as plt
%matplotlib inline
Now let's read each file into their Dataframes:
# Storing the movie information into a pandas dataframe
dataframe_movies = pd.read_csv('movies.csv')
# Storing the user information into a pandas dataframe
rating_scores_df = pd.read_csv('rating_scores.csv')
# Head is a function that gets the first N rows of a dataframe. N's default is 5.
dataframe_movies.head()
From the name we also need to remove the column "year". We will use pandas' replace function & store it in a new column
dataframe_movies['year'] = dataframe_movies.name.str.extract('(\(\d\d\d\d\))',expand=False)
# Removing paranthesis{}
dataframe_movies['year'] = dataframe_movies.year.str.extract('(\d\d\d\d)',expand=False)
# Removing the years from the 'name' column
dataframe_movies['name'] = dataframe_movies.name.str.replace('(\(\d\d\d\d\))', '')
#Applying the strip function to get rid of any ending whitespace characters that may have appeared
dataframe_movies['name'] = dataframe_movies['name'].apply(lambda x: x.strip())
dataframe_movies.head()
Now as we have performed some preprocessing on our data, let us also split the values according to categorys_class column into a list of categorys_class to simplify future use. This can be achieved by applying Python's split string function on the correct column.
#Every category is separated by a | so we simply have to call the split function on |
dataframe_movies['categorys_class'] = dataframe_movies.categorys_class.str.split('|')
dataframe_movies.head()
We will use a technique named One Hot Encoding to convert the list of categorys_class to a vector where each column corresponds to one possible value of the feature because keeping categorys_class in a list format isn't optimal for a recommendation system which is content-based. We will input the categorical data to one hot encoding like we will our data of categorys_class in columns which represents either 0 or . 0 represents that a movie does not have that category and 1 represents that it does.
#Copying the movie dataframe into a new one since we won't need to use the category information in our first case.
moviesWithcategorys_class_df = dataframe_movies.copy()
#For every row in the dataframe, iterate through the list of categorys_class and place a 1 into the corresponding column
for index, row in dataframe_movies.iterrows():
for category in row['categorys_class']:
moviesWithcategorys_class_df.at[index, category] = 1
#Filling in the NaN values with 0 to show that a movie doesn't have that column's category
moviesWithcategorys_class_df = moviesWithcategorys_class_df.fillna(0)
moviesWithcategorys_class_df.head()
Next, let's look at the rating_scores dataframe.
rating_scores_df.head()
As we do not require it, we will be removing timestamp from df
#Drop removes a specified row or column from a dataframe
rating_scores_df = rating_scores_df.drop('timestamp', 1)
rating_scores_df.head()
Now, let's take a look at how to implement Content-Based or Item-Item recommendation systems. This technique attempts to figure out what a user's favourite aspects of an item is, and then recommends items that present those aspects. In our case, we're going to try to figure out the input's favorite categorys_class from the movies and rating_scores given.
Let's begin by creating an input user to recommend movies to:
Notice: To add more movies, simply increase the amount of elements in the userInput. Feel free to add more in! Just be sure to write it in with capital letters and if a movie starts with a "The", like "The Matrix" then write it in like this: 'Matrix, The' .
userInput = [
{'name':'Dinner', 'rating_score':6},
{'name':'Titanic', 'rating_score':5},
{'name':'Notebook', 'rating_score':2},
{'name':"Inception", 'rating_score':5},
{'name':'Fast2Furious', 'rating_score':4}
]
movie_input = pd.DataFrame(userInput)
movie_input
Add Id_movie to input user
With the input finalised, let us extract the ID of the input movie from the dataframe df and add the ID into it.
We can now achieve this by 1st filtering out the rows which contain the input_movie's name and then merging this subset with the input df dataframe. Also, we can drop not-required columns from the input to save some memory.
# searching movies by name
inputId = dataframe_movies[dataframe_movies['name'].isin(movie_input['name'].tolist())]
# merging it to get the Id_movie.
movie_input = pd.merge(inputId, movie_input)
# Deleting columns that are not required
movie_input = movie_input.drop('categorys_class', 1).drop('year', 1)
#Final input dataframe
movie_input
We're going to start by learning the input's preferences, so let's get the subset of movies that the input has watched from the Dataframe containing categorys_class defined with binary values.
#Filtering out the movies from the input
userMovies = moviesWithcategorys_class_df[moviesWithcategorys_class_df['Id_movie'].isin(movie_input['Id_movie'].tolist())]
userMovies
We'll only need the actual category table, so let's clean this up a bit by resetting the index and dropping the Id_movie, name, categorys_class and year columns.
#Resetting the index to avoid future issues
userMovies = userMovies.reset_index(drop=True)
#Dropping unnecessary issues due to save memory and to avoid issues
usertable_category = userMovies.drop('Id_movie', 1).drop('name', 1).drop('categorys_class', 1).drop('year', 1)
usertable_category
Now we're ready to start learning the input's preferences!
To do this, we're going to transform all category into respective weights. We can do this by using the input's reviews and multiplying them into the input's category table and then summing up the resulting table by column. This operation is actually a dot product between a matrix and a vector, so we can simply accomplish by calling Pandas's "dot" function.
movie_input['rating_score']
#Dot produt to get weights
userProfile = usertable_category.transpose().dot(movie_input['rating_score'])
#The user profile
userProfile
After this step, we have weights of each of the user's movie choices, we term this as the User's Profile. By utilizing this information, we can suggest movies that satisfy the user's preferences.
Let's start by extracting the category table from the original dataframe:
#Now let's get the categorys_class of every movie in our original dataframe
table_category = moviesWithcategorys_class_df.set_index(moviesWithcategorys_class_df['Id_movie'])
#And drop the unnecessary information
table_category = table_category.drop('Id_movie', 1).drop('name', 1).drop('categorys_class', 1).drop('year', 1)
table_category.head()
table_category.shape
As we have the profile of the input and the complete list of movies and their categorys_class in hand, we are not going to take the weighted average of each movie with respect to the input profile and recommend the top twenty movies that most satisfy it.
#Multiply the categorys_class by the weights and then take the weighted average
table_df_recomendation = ((table_category*userProfile).sum(axis=1))/(userProfile.sum())
table_df_recomendation.head()
# Now we will sort our recommendations in descending order
table_df_recomendation = table_df_recomendation.sort_values(ascending=False)
#Just a peek at the values
table_df_recomendation.head()
Now here's the recommendation table!
#The final recommendation table
dataframe_movies.loc[dataframe_movies['Id_movie'].isin(table_df_recomendation.head(20).keys())]
Advantages and Disadvantages of Content-Based Filtering
Advantages
- Learns user's preferences
- Highly personalized for the user
Disadvantages
- Doesn't take into account what others think of the item, so low quality item recommendations might happen
- Extracting data is not always intuitive
- Determining what characteristics of the item the user dislikes or likes is not always obvious