One-Hot Encoding

One hot encoding is a binary representation of categorical data. This became popular after deep learning came into practice because categorical data can’t be used directly with many ML (Machine learning) algorithms. It is very simple and one can understand it as follows. Let’s have three colors, ‘red’, ‘green’ and ‘blue’.

• We will first convert these to integers. For example, red->1, green->2 and blue->3.
• In one hot encoding, each word is represented by a vector of the same length as other samples. The vector length will be the number of colors and only one value of any color’s vector will be one corresponding to its integer value. All others will be zero. Our three colors can be represented as follows:
  • red-> [1, 0, 0],
  • green-> [0, 1, 0],
  • blue-> [0, 0, 1].

Label Encoding

Two popular methods for converting categorical characteristics into a numerical representation suitable for machine learning models are label encoding and one hot encoding. In a categorical feature, label encoding gives each category a distinct integer value. This is an easy-to-use strategy that can be helpful when the categories’ order matters. However, because of the allocated integer values, it could create unintentional linkages between categories.

Label encoding, for instance, might assign the values 0, 1, and 2 to category features that are “small,” “medium,” and “large,” respectively. This would suggest that “big” is twice as significant as “small,” which is probably untrue. In a categorical feature, one hot encoding generates a binary column for every category. The category-corresponding column is given a value of 1, and the remaining columns are given a value of 0. When the order of the categories is not significant, this method can help prevent the introduction of unintentional links between them. However, it can result in a lot of columns, which could affect the model’s performance and memory utilization.

In conclusion, one hot encoding is helpful when the number of categories is not too great and the order of categories is not crucial, whereas label encoding can be helpful when the number of categories is small.

Count Encoding

A useful and effective technique for turning categorical variables into numerical values according to how frequently they occur in the dataset is count encoding. It is especially helpful for features with high cardinality, but it should be utilized with awareness of its limits, as it may create bias or fail to take into account the relationship between the target variable.

The Operation of Count Encoding:

1. Determine the Frequency: Determine the number of times (frequency) that each category appears in the dataset for each category in a categorical feature.
2. Replace Categorical Values: Put the corresponding frequency count for each category in place.

Target Encoding

By utilizing the target variable, target encoding is a potent technique for converting categorical information into numerical values. When applied properly, with the right strategies to avoid overfitting and data leakage, it can improve model performance and is especially helpful for high-cardinality features. Using this strategy, the target variable for each category is replaced with a summary statistic, usually the mean.

The Operation of Target Encoding:

1. Compute the Statistic: Determine the summary statistic (mean, median, etc.) of the target variable for each category in a categorical feature.
2. Change Categorical Values: Change each category to the statistic value that corresponds to it.

Leave-One-Out Encoding

A potent technique for converting categorical data into numerical values that leverages the target variable and lowers the chance of overfitting is leave-one-out encoding. When applied appropriately, it can enhance model performance and is especially helpful in scenarios involving high cardinality.

The Operation of Leave-One-Out Encoding:

1. Determine the Average Goal: Determine the target variable’s mean for each group.
2. Leave-Out the Current Observation: The target value of the current instance is subtracted from the category mean target for every instance in the dataset. Using the goal value that it is attempting to predict, keeps the model from overfitting.
3. Replace Categorical Value: This mean target, which was determined in the preceding phase, should be used in place of the categorical value.

CatBoost Encoding

Target-based statistics and order-sensitive approaches are used in CatBoost encoding, an effective and sophisticated method for managing categorical data in machine learning, which enhances model performance. Yandex created a machine learning technique called CatBoost (Categorical Boosting), which works especially well with categorical data. CatBoost encoding describes how CatBoost manages categorical features, which enables it to handle categorical variables with high cardinality.

• Treatment of categorical features: Unlike many other machine learning methods that call for explicit encoding (e.g., one-hot encoding, label encoding), CatBoost supports categorical information natively.
• Encoding based on targets: Target-based encoding, also known as mean encoding, is a technique used by CatBoost in which the target statistics—that is, the mean of the target variable—are used to translate the category values into numerical values. This facilitates the understanding of the connection between the target variable and the category variable.
• Encrypt with order sensitivity: CatBoost uses an order-sensitive encoding process to guarantee robust encoding and avoid overfitting. It uses just the data points that come before it to compute the target-based encoding for each categorical value after processing the dataset in a random order.