Transformers and the transform() method
In scikit-learn, transformers are tools that modify data by applying transformations such as scaling, normalization, or encoding to prepare it for modeling. Each transformer follows a consistent interface, using the fit() method to learn any necessary parameters from the data and the transform() method to apply those transformations. For instance, StandardScaler() calculates the mean and standard deviation during fit() and uses those values to transform the data by scaling it (as you may recall from high school statistics, this transformed value is called a z-score).
Figure 1.1 – Data transformation in the context of scikit-learn’s Pipeline() class
Data transformations provide several benefits when applied to ML scenarios. First, many models presuppose data to be normally distributed, free of outliers, and so on. Second, most real-world datasets do not come in this neat-and-tidy format and require some massaging before modeling occurs:
from sklearn.preprocessing import StandardScaler import numpy as np # Example data X = np.array([[1, 2], [3, 4], [5, 6]]) # Create a StandardScaler instance scaler = StandardScaler() # Fit the scaler on the data scaler.fit(X) # Transform the data X_scaled = scaler.transform(X) print(X_scaled) # Output: [[-1.22474487 -1.22474487] [ 0. 0. ] [ 1.22474487 1.22474487]]
Another common shortcut that we saw previously, fit_transform(), allows users to perform both steps in one command, making preprocessing workflows more efficient. Again, when to use fit_transform() and fit() with transform() separately depends on the task at hand. Typically, we should apply the fit_transform() method to our training data if we want to transform our data immediately based on the calculated transformation, something the fit() method can’t achieve by itself. However, when applying transformations to our test dataset, we wouldn’t want to reapply the fit() method; this would impose a potentially different data transformation, as our test data will be slightly different from our training data. Remember, our test dataset is meant to be treated exactly like our training data for model consistency purposes, so implementing a separate fit() method on it could potentially alter our test data and make our model predictions unreliable when applied in a real-world scenario:
from sklearn.preprocessing import StandardScaler import numpy as np # Example data X = np.array([[1, 2], [3, 4], [5, 6]]) # Create a StandardScaler instance and # fit_transform the data in one step scaler = StandardScaler() X_scaled = scaler.fit_transform(X) print(X_scaled) # Output: [[-1.22474487 -1.22474487] [ 0. 0. ] [ 1.22474487 1.22474487]]
This consistency across all transformers allows them to be integrated seamlessly into ML pipelines, ensuring that the same transformation is applied to both the training and test data, something that becomes significantly important when implementing production-level models.
We will explore various transformers, including StandardScaler(), MinMaxScaler(), and OneHotEncoder(), in Chapter 2 to demonstrate how they can be used to prepare data for ML models using the fit(), transform(), and fit_transform() methods. Practical examples will be provided to illustrate how you can integrate transformers into workflows to ensure your data is preprocessed consistently.
