4  What’s in the data?

A column called Renewal_Likelihood looks like an answer. Before fitting a model, ask who created it, from which information, and at what point in the customer’s history. A label assigned by a person or business rule is different from an observed renewal. Predicting that label may reproduce the assessor’s judgement without telling you who will actually leave.

That distinction changes the whole analysis. In this chapter, the target is explicitly the Low-renewal label, not future churn. The complete twelve-row synthetic dataset is in Appendix G. The larger Tessera site remains an optional extension; its files must be inspected on their own terms.

4.1 Two questions, two kinds of answer

Supervised learning asks: given examples with labels, can a rule predict the label on another example? Here the inputs include tenure and satisfaction, and the target is Low or High. A model trained on those rows learns an association with that target. It does not turn the label into an objective measure of loyalty.

Unsupervised learning asks: what structure appears when we group examples without supplying the target? You might cluster customers by tenure and satisfaction, then inspect the groups. The algorithm has not discovered “loyal customers” until someone supplies and tests that interpretation. It has grouped points under a chosen representation and distance measure.

Both can help an analyst. Neither decides the business meaning of a result. A useful first page in your dossier states the question, the unit of analysis, the target if any, and what the data cannot answer.

4.2 Inspect before modelling

Start with counts and definitions. In lab-v1, each row represents one fictional customer. There are no names or observed departures. Tenure is measured in months; satisfaction is an invented score. The development/test designation is part of the exercise and must not become a predictor.

Six of twelve rows have Low labels. South has four Low labels among five rows, Metro two among four, and North none among three. Those fractions are easy to verify by hand. They describe label prevalence. They do not show that a model treats South customers badly, or that North customers are safe.

Write the denominators beside the percentages. “Zero out of three” leaves the uncertainty visible in a way that “0% risk” does not. Also ask what is absent: a date, a sampling method, a definition of actual renewal, and a history of interventions. Those absences limit the conclusions before a model enters the picture.

On a real dataset, check duplicated customers, missing values, inconsistent units, changing definitions, and whether information recorded after an outcome has slipped into the inputs. Preserve the original file and a record of every transformation.

4.3 A rule you can trace

Use the supplied Rule A: predict Low when satisfaction is 6 or below; otherwise predict High. It is a predefined teaching rule, not a fitted model or a claim about real customer behaviour.

On the eight development rows, it matches every label. This is exactly the sort of result that can make a demo feel finished. Now cover the test labels, write down the four predictions, and uncover them. T01 and T02 are wrong. The rule gets two of four right, detects one of the two Low labels, and makes one false alarm.

T01 has satisfaction 7 but a Low label; T02 has satisfaction 5 but a High label. A rule fitted neatly to the development examples does not necessarily handle a different example. Whether those exceptions arise from noise, a missing variable, or a different labelling process cannot be determined from these rows.

A decision tree generalises this form: it learns a sequence of splits rather than taking the single split as given. A small tree may be easy to trace. A deep tree may fit incidental details so closely that it generalises poorly. Choosing its depth is part of model development, not a reason to keep consulting the final test labels.

4.4 The trap: association is not an intervention

Suppose a tree repeatedly splits on satisfaction. You can report that satisfaction helps reproduce the supplied labels. You cannot conclude that increasing the score will cause a customer to renew.

The score might reflect an underlying service problem. It might have helped someone assign the label in the first place. It might have been recorded after the renewal decision. A promotional intervention could change the score without changing the service experience. Each possibility calls for different evidence.

A useful conversation looks like this:

Draft: “Low satisfaction causes churn, so offer every dissatisfied customer a discount.”

Check: “The target is a renewal-risk label, not an observed departure. Where is the evidence that the discount changes renewal?”

Owned conclusion: “The label is associated with satisfaction in this teaching sample. We need dated outcomes and an appropriate intervention study before recommending a retention policy.”

The correction changes what the team is entitled to do.

4.5 Clustering without naming the answer in advance

For a paper exercise, set the label column aside and arrange the eight development rows by tenure and satisfaction. Describe two possible groupings. For example, group by shorter versus longer tenure, then try lower versus higher satisfaction. Which customers move together? Which separate?

A clustering algorithm performs a more formal version of this exercise, under chosen assumptions. Scaling matters: a variable measured over a large numerical range can dominate some distance calculations. The number of clusters, distance measure, and representation all influence the result. Different groups do not necessarily mean one analysis is wrong; they may answer different questions.

Give the groups descriptive names such as “shorter tenure, lower satisfaction” before using interpretive names such as “unhappy newcomers.” To earn the second name, inspect evidence beyond the coordinates. Do not put the renewal label into an ostensibly unsupervised analysis and then celebrate discovering the label.

4.6 Core: produce an analysis sheet

Time: 30 minutes. Inputs: Appendix G’s customer table and Rule A. A pencil or spreadsheet is enough.

1. State the target and list three questions the data cannot answer.
2. Reproduce the label counts by region, including denominators.
3. Apply Rule A to the development and test rows separately. Keep every error.
4. Write one association claim you can defend and one causal claim you must reject.
5. Propose the next data collection needed for an actual renewal decision.

Artefact: a one-page analysis sheet in the dossier. Success: distinguish label prevalence from prediction accuracy, report 2/4 test accuracy for Rule A, and refuse to describe a proxy as observed churn.

Optional build: import resources/lab-v1/customers.csv into a visual analysis tool. Set ID and split aside as metadata, identify the label as the target, and keep the test rows separate. A fitted tree may differ from the predefined rule; report its actual output. Do not tune it until it matches the printed answer. Eight development examples are not enough for a production claim.

4.7 What travels to the next chapter

The analysis sheet now contains a data dictionary, a target definition, a checkable result, and missing evidence. Next, privacy adds another question: what may you do with the fields and records, even when the analysis is useful? A sound pattern does not authorise a new use of personal information.

4.8 Further reading

• An Introduction to Statistical Learning: the authors’ accessible treatment of supervised learning, classification, resampling, and tree-based methods. Read the chapters on model assessment before treating a training result as performance.
• scikit-learn: common pitfalls: the project’s documentation on inconsistent preprocessing and data leakage. The implementation uses code, but the separation of development and evaluation data applies equally to visual tools.