8  What is it saying?

“Wonderful, another outage during my presentation.”

The positive word does not make this a positive experience. A score that loses the complaint can direct attention away from the customer who needs it. This chapter turns a collection of messages into something you can inspect without pretending a tidy number has captured every voice.

Use the six invented messages in Appendix G. They are a small teaching set, not a sample of Tessera’s customers. You can complete the analysis on paper before using any model.

8.1 Theme and tone answer different questions

A theme describes what a message concerns: billing, availability, access, or something else. Tone describes an interpretation of the expression: positive, negative, neutral, mixed, perhaps uncertain. Neither is the customer’s identity or a complete measure of urgency.

“The invoice was corrected. Thank you” concerns billing and reads positively. “Support was polite. My service is still down” contains praise and an unresolved outage. Reducing the second to a neutral average can hide the operational need.

Keep the source text attached to each label. A reviewer should be able to challenge the interpretation and identify a message that belongs to several groups. The useful output is a map back to the evidence, not just a coloured chart.

8.2 How text becomes a comparison

Text-analysis tools represent messages as numbers. A bag-of-words representation counts terms; TF-IDF adjusts weights according to their frequency within documents and across a collection. Embeddings provide another representation, often bringing some differently worded messages closer together.

Each representation loses something. Word counts do not preserve every relation between words. Removing stopwords can discard meaningful negation. Similar language does not guarantee the same intent, and different language can describe the same failure.

A pipeline might lowercase, tokenise, select terms, transform them into vectors, then compare distances. None of those is a universally correct setting. Preserve the original text so you can investigate the effect of each transformation. The author-maintained Introduction to Information Retrieval explains term weighting, similarity, and clustering in more detail.

8.3 Clustering proposes a grouping

A clustering method groups items according to the representation and objective you chose. K-means seeks a specified number of clusters under its distance-based objective; hierarchical methods organise nested groups. Neither determines the organisational importance of the resulting themes.

On six short messages, reading and arranging them yourself is more informative than celebrating a complex visualisation. Group S1 with S6 as availability concerns, then notice that S6 also praises support. Decide whether to allow multiple labels or preserve the mixed content in a note.

If a large cluster is named “customer dissatisfaction,” ask what decisions that label enables. It may be too broad to help. If every message gets its own cluster, the grouping may not reduce the workload. Choose the granularity for the decision and inspect borderline examples.

A second grouping method can challenge your view. Agreement between methods is corroboration only to the extent that they bring meaningfully different evidence or assumptions; it is not automatically cross-validation or independent ground truth.

8.4 A deliberately weak sentiment baseline

Appendix G defines a transparent word-counting scorer. Positive terms are wonderful, thank, clear, and polite. Negative terms are outage, fail, errors, and down. Count whole words without a negation rule; a tie is neutral.

This scorer returns neutral for S1: wonderful and outage cancel. It calls S5, “No more login errors after the patch,” negative because it counts errors and ignores the phrase that reverses the meaning. It also cannot return mixed, because mixed is not among its possible outputs.

Under the appendix’s reference key, it agrees on two of six messages. That is the result for this scorer, these inputs, and this key. It is not a measured error rate for every lexicon-based method. Some lexicon systems explicitly handle negation, punctuation, or intensity; VADER is an example. VADER, author-maintained implementation

A learned classifier or general-purpose language model may handle these examples better. It may also fail on unfamiliar language, irony, short context, or a different labelling convention. Do not ask it to imagine what another scorer would return and call that a comparison. Run the scorer, or label the estimate as untested.

8.5 Build a disagreement set

A disagreement set preserves examples on which two interpretations differ. It is the start of a review, not a verdict that one system is wrong.

For each disputed message, retain the original text, each label, the convention used, and a short reason for the final judgement. If two human reviewers reasonably disagree, record that too. Some tasks need an “uncertain” option or a way to represent multiple aspects.

Constructed exchange: an assistant calls S6 positive because support was polite. The reviewer points to the unresolved outage. The final annotation preserves both the praise and the service problem. The correction is supported by the words in the message, not by the model conceding the argument.

The same method scales to a larger sample. Select cases deliberately enough to inspect known risks, but distinguish that challenge set from a representative sample used to estimate an overall rate.

8.6 Aggregate scores can be wrong together

Averaging does not cure systematic errors. If a scorer misses negation in many messages, or handles one language poorly, its aggregate can retain that bias. Nor does a large overall sample make a tiny subgroup estimate reliable.

Before comparing groups, inspect sample sizes, missing responses, language, time period, product mix, and the suitability of the labels. A difference may be worth investigating without yet being attributable to the product or the model.

Do not use a sentiment score by itself to close a ticket, downgrade service, or decide a customer’s value. A theme summary can guide attention; a consequential action needs evidence and an accountable decision.

8.7 Core: annotate before automating

Time: 30 minutes. Inputs: Appendix G’s six messages, baseline dictionary, and reference key.

1. Cover the reference columns and assign a theme and tone yourself.
2. Underline the text supporting each label, preserving mixed cases.
3. Run the baseline by hand or in a spreadsheet.
4. Compare with the reference key and record all disagreements.
5. Write one operational conclusion the messages support and one they do not.

Artefact: an annotated disagreement table. Success: reproduce the baseline’s two agreements out of six, explain S5’s negation failure, and avoid claiming a population-wide error rate from this constructed set.

Optional tool comparison: import the bundled support CSV into a text tool or give the six messages to an approved model. Use the same label convention, record the actual output and version, and retain unresolved disagreements. The printed answer key checks your baseline; it does not prescribe what every model must say.

Next, Does it actually work? brings these habits into a formal evaluation: declared criteria, held-out inputs, visible counts, and a decision that survives an unfavourable result.