Data cleaning has always been the least glamorous and most expensive part of analytics — commonly cited at 60–80% of project effort. It is the phase AI has changed most in day-to-day feel, and the phase where AI introduces the most subtle risk.
The risk is not that the tool fails. A failure is visible and you fix it. The risk is that it succeeds at something slightly different from what you wanted, produces a tidy file, and never mentions the decision it made.
The rule that governs everything else
Ask for the change log, not the cleaned file.
If you take one thing from this article, that is it. A cleaned dataset with no record of what changed is a set of decisions you have inherited and cannot audit. When a number looks wrong three weeks later, you will have no way to trace it.
What a usable change log contains, per column: the rule applied, how many rows it touched, and one worked example. That is enough to spot a bad assumption in seconds.
The cleaning order that avoids rework
Safe to automate: profiling, validation, standardising
These three do not change the meaning of your data, only its presentation. Let AI do all of it.
Profiling — row counts, column types, null rates, unique counts, min and max. Historically a tedious first hour; now instant, and the results are reliable because they are arithmetic rather than judgement.
Validation against a schema — you state what the data should look like (a phone column contains phone numbers, amounts are positive, dates fall in a plausible range) and the tool reports every violation. The value is that it never gets bored on row 40,000.
Standardising formats — dates to ISO 8601, phone numbers to E.164, casing, whitespace, encoding. Mechanical, high-volume, and exactly what a machine should do.
One Malaysian-specific caution on dates: 3/1/25 is genuinely ambiguous. Read as DD/MM it is 3 January; as MM/DD it is 1 March. AI tools frequently default to the US reading. Always state the source convention explicitly, and check a few known dates afterwards.
Automate carefully: deduplication and imputation
These two change what your data means, so they need a human decision behind the rule.
Deduplication
Exact duplicates are trivial. Fuzzy duplicates are where judgement enters: are "Ahmad Bin Ali, 012-3456789" and "Ahmad B. Ali, 0123456789" the same person? Usually yes. Are "Syarikat Maju Sdn Bhd" and "Syarikat Maju Enterprise" the same company? Possibly not — they may be genuinely different legal entities with a shared name.
Set the matching rule yourself, run it, and review a sample of what it merged before accepting. An over-aggressive dedupe silently destroys records and is very hard to detect afterwards.
Sequence matters here: standardise before you deduplicate. If "KL" has not yet been mapped to "Kuala Lumpur", the matcher will never see those records as the same.
Imputation
Filling missing values is the single most consequential cleaning decision, because a filled value is indistinguishable from a real one downstream.
The discipline: impute only where you have an explicit, defensible rule — median for a numeric field where the distribution supports it, "Unknown" as a genuine category for text. Where you have no rule, leave the gap and flag it. An honest null is far more useful than a plausible invention, and analysts can decide how to handle it.
Be especially wary of a tool that fills gaps by inference from other columns. That is a model, not a cleaning step, and it belongs in your analysis with its assumptions visible — not silently inside your dataset.
Never automate: the review
Two things stay with a person.
Deciding whether the result is sane. Did the row count drop by a third? Did total revenue change after cleaning? It should not — and if it did, something removed records you needed. Always compare a few aggregate totals before and after.
Outliers. A tool can flag a value as anomalous. Only someone who knows the business can say whether RM 480,000 on one invoice is a data-entry error or your largest genuine order of the year. Deleting outliers because they look wrong is how real signal gets destroyed.
What AI genuinely cannot check
Of the six data quality dimensions, AI can assess five from the data alone. It cannot assess accuracy — whether the values correspond to reality.
A tool can confirm that an address is well-formed, in a valid postcode range, and consistently formatted. It cannot tell you the customer moved last year. That requires an external source of truth, and no amount of model capability substitutes for it.
This is why "the data is clean" and "the data is correct" are different claims, and why cleaning does not remove the need to know your data.
A workable prompt pattern
Rather than "clean this file", which invites silent decisions, specify the contract:
- State the source and its conventions — Malaysian system, dates DD/MM, amounts in RM.
- State the expected schema, column by column.
- Give explicit rules for missing values, and say what to do when no rule applies (flag, do not fill).
- Forbid row deletion. Ask for flagging instead.
- Require the change log: column, rule, rows affected, example.
- Ask for a "needs human" list of anything it was unsure about.
That last item is the one people omit and the one that catches the most. A tool asked to surface its uncertainty usually will.
Where this sits in the wider workflow
Cleaning is Phase 3 of the CRISP-DM lifecycle, and it remains a substantial share of effort even with AI — the change is that it is no longer the largest share, because modelling collapsed and deployment grew.
For the practical field guide to specific problems, see from messy export to analysis-ready.
Our AI Analytics programme covers data preparation, quality assessment and AI-assisted cleaning on real Malaysian business data — HRD Corp SBL-KHAS claimable for eligible employers.
