Feature engineering

Feature engineering is preparing data so that a prediction model can learn from it: choosing which facts to give the model and turning them into numbers it can work with. A shop that forecasts daily sales might turn its till receipts into "day of the week", "public holiday" and "sales on the same day a week earlier". Good features often decide whether such a model works at all.

Why does it matter?

"Easily the most important factor is the features used," Pedro Domingos wrote in A Few Useful Things to Know About Machine Learning (2012). Raw data often has to be reshaped before a model can learn from it, and according to Domingos, constructing those features is typically where most of the effort in a project goes. It is also harder than choosing the algorithm, because features depend on the domain while learning algorithms are largely general purpose.

Google's Machine Learning Crash Course explains why: a model does not read the rows of your spreadsheet directly. It reads a feature vector, a row of floating-point numbers, and feature engineering decides how the raw values become those numbers.

How does it work?

• Converting numbers. Normalisation puts features on a similar scale, for example 0 to 1. Log scaling suits values where a few are huge and most are small, such as the number of ratings per film. Binning turns numbers into ranges, such as age groups.
• Encoding categories. A model needs numbers, so a value such as a colour becomes a one-hot vector: one position per category, with a 1 for the category that applies and 0 elsewhere. With plain numbering, the model would treat the numbers as amounts: in Google's example it would consider "purple" six times more likely than "orange".
• Creating features. Derive new values from existing ones: the weekday from a date, the days since a customer's last order, or a feature cross that combines two features into one.
• Selecting features. Drop what adds nothing. scikit-learn offers methods that remove features that never vary, score each feature's link with the outcome (for example with mutual information), or retrain a model on ever smaller sets of features (recursive feature elimination).

And deep learning?

Deep learning learns many features by itself: in the review by LeCun, Bengio and Hinton, the layers of features "are not designed by human engineers". That changed work on images, speech and text more than work on tables. Grinsztajn et al. (2022) compared both across 45 tabular datasets and found that tree-based models such as XGBoost and random forests still outperformed deep learning on medium-sized data of about 10,000 rows. For spreadsheet-style business data, features plus a classic model remain a sensible start.

Where does it go wrong?

The classic mistake is data leakage: a feature that uses information you will not have when the model makes its prediction. According to scikit-learn, leakage gives overly optimistic test results and poorer performance in production. It also happens when a preprocessing step, such as normalisation, learns from the test data. Feature stores such as Feast serve the same feature definitions for training and for live predictions, and build "point-in-time correct" training sets so that future values do not leak into training.

Example

Suppose a bakery wants to predict how many loaves each of its shops will sell tomorrow. The raw data is its till receipts, one line per sale with a timestamp. Useful features: the day of the week, whether tomorrow is a public holiday, the weather forecast, sales on the same weekday a week earlier and the average over the four weeks before. Tomorrow's actual weather cannot be a feature, because nobody knows it when the prediction is made; the forecast can. Whether these features help is tested on past months that the model did not see during training.