"Is our data clean?" is a question that cannot be answered as asked, because it bundles six independent properties. A dataset can be complete and inconsistent, valid and out of date, unique and wrong.
Separating them turns a vague worry into a checklist — and reveals which parts a tool can settle for you and which need a person.
The framing below follows the dimensions used in the DAMA-DMBOK, the standard data management reference, reduced to the six that carry most practical weight.
The six dimensions of data quality
1. Completeness — are values missing?
The easiest to measure: what percentage of each column is populated. AI answers this instantly.
The judgement is what counts as a problem. A 30% null rate in a "middle name" field is fine. A 3% null rate in "customer ID" is a serious problem, because those rows cannot be joined to anything.
Watch for disguised nulls — the string "N/A", a single dash, "0" in a field where zero is not a real value, or "1900-01-01" as a placeholder date. These count as populated and quietly corrupt averages.
2. Accuracy — do the values match reality?
The dimension that matters most and the only one that cannot be checked from the data alone.
An address can be perfectly formatted, in a valid postcode, consistently capitalised — and simply not where the customer lives. No model can tell you that. It requires an external source of truth: a verification call, a returned delivery, a check against SSM or a bank record.
Practical consequence: when someone says "the data is clean", ask which dimension they mean. Almost always they mean valid and consistent. Accuracy is assumed rather than tested, and that assumption is where the expensive errors live.
3. Validity — right format and range?
Does each value conform to its rules? A Malaysian NRIC has 12 digits. A phone number has a plausible prefix. An age is between 0 and 120. An invoice amount is not negative unless credit notes are permitted.
Fully checkable by machine, and the highest-yield automated check for the effort. Define the rules once and every future import is screened against them.
4. Consistency — same fact, same answer?
Does the same real-world thing get represented the same way, and do related values agree?
Two distinct failures. Representational: "KL", "Kuala Lumpur", "K.L." and "kuala lumpur" as four values for one city. Logical: a delivery date before the order date, or a customer's total spend not matching the sum of their transactions.
Both are machine-checkable, and the second is the one teams forget to test.
5. Uniqueness — is anything counted twice?
Duplicate records inflate customer counts, double-count revenue, and cause the same person to receive two of everything.
Exact duplicates are easy. Near-duplicates need a rule you set: which fields must match, and how closely. AI executes that rule well but should not choose it — the threshold is a business decision with real consequences either way.
6. Timeliness — current enough to be useful?
Correct data that is out of date is still the wrong answer. A price list from 2023, a contact who left the company, an address from before a move.
The useful question is not "how old is this?" but "how old is acceptable for this decision?" A monthly report tolerates week-old data. A stock check does not.
The practical failure is silent staleness: a feed stops updating and nothing alerts, so the dashboard keeps showing confident numbers from a snapshot that stopped moving.
A one-hour assessment
For any dataset you are about to rely on:
- Completeness — null rate per column, plus a hunt for disguised nulls.
- Validity — write the format rules for your five most important columns; count violations.
- Consistency — list distinct values in each categorical column. Duplicated concepts jump out immediately.
- Uniqueness — count exact duplicates on your key, then fuzzy-match on name plus phone.
- Timeliness — find the newest record. If it is older than expected, stop and find out why.
- Accuracy — sample 20 rows and verify against something external. Tedious, and the only way.
AI does steps 1–5 in minutes. Step 6 is an afternoon of human work, and it is the step that tells you whether the other five mattered.
Why this matters more with AI in the loop
When analysis was slow, bad data usually surfaced along the way — someone spent long enough with the numbers to notice something odd. Now an analysis can be produced in minutes, and poor-quality input flows straight through into a confident, well-formatted, wrong conclusion.
Quality assessment used to be implicit in the slowness. It now has to be deliberate. See data cleaning with AI for how to act on what the assessment finds.
Our AI Analytics programme covers data quality assessment and preparation for Malaysian business teams — HRD Corp SBL-KHAS claimable.
