Supplier Evaluation: Criteria, Matrix, and How to Automate the Process
In short
Supplier evaluation is the systematic process a company uses to measure the performance of those supplying it with goods or services, against defined and comparable criteria.
There are two distinct moments that often get conflated: the pre-award evaluation, which decides whether a supplier can come in, and the performance evaluation, which measures how it behaves once inside. They draw on different sources and carry different consequences.
The weighted matrix is the standard tool and the literature covers it well. Designing it is not the problem. Getting the data to fill it in, quarter after quarter, is.
What Is Supplier Evaluation?
Supplier evaluation is the process by which an organization measures, against defined criteria and a common scale, the performance of the companies supplying it with goods or services. Its purpose is to replace perception with comparable evidence, so that decisions such as renewing a contract, increasing volume or dropping a supplier can be backed by numbers.
It is not a one-off event. A supplier is evaluated before it comes in and keeps being evaluated while it operates, and those two exercises are not the same.
The Two Moments of Evaluation
Almost all the confusion on this topic comes from forcing both moments into a single matrix.
Pre-award evaluation. Happens before the first purchase order. It answers whether the supplier is allowed to work with us. It rests on documents and external checks: legal incorporation, tax standing, certifications, installed capacity, references, solvency. The data is either self-declared or comes from an outside source, and the decision is binary: in or out.
Performance evaluation. Happens afterwards, on a recurring basis. It answers how well the supplier is delivering. It rests on transactional data the operation itself generates: rejections at receiving, delivery compliance, price variation, incidents. The data is historical and the decision is graduated: more volume, a corrective action plan, or removal.
Pre-award evaluation | Performance evaluation | |
|---|---|---|
When | Before the first purchase | Quarterly or annually |
What it answers | Can it come in? | Is it delivering? |
Source of the data | Documents and external sources | Your own transactions |
Outcome | Approved or rejected | Comparable score |
Mixing them produces matrices where a supplier's ISO certification sits next to its OTIF for the last quarter, two things that neither refresh at the same pace nor mean the same thing.
Supplier Evaluation Criteria
Criteria should be objective, measurable and applied equally to every candidate. These five blocks cover most cases, and the important column in the table is not the criterion but the metric, because a criterion without a concrete metric is an opinion with a technical name.
Criterion | Concrete metric | What it reveals |
|---|---|---|
Quality | Rejection rate at receiving | Whether the product meets spec without rework |
Delivery | OTIF, compliance on date and quantity | Whether operations can plan around it |
Price | Variation against a reference price or index | Whether it stays competitive over time |
Service | Response time when something goes wrong | What happens when something breaks |
Risk | Current tax and legal standing, solvency | Whether it could disappear or create exposure |
One rule worth holding: criteria are agreed before offers come in. Adjusting them afterwards so the result favors a particular supplier destroys the traceability of the whole evaluation and leaves the process indefensible in an audit.
What Is Supplier Qualification?
Qualification is the formal part of the pre-award evaluation. It is the process of verifying that a supplier meets the minimum requirements to be contracted, and of leaving a record of that verification.
The difference from performance evaluation is one of nature, not degree. Qualification is an entry filter with a binary outcome and a supporting file. Performance evaluation is continuous measurement with a comparable outcome. A qualified supplier can perform badly, and a well-performing supplier can lose its qualification if a document expires or its tax standing changes.
That last point causes the most trouble in practice, and it is covered further down.
How to Build a Supplier Evaluation Matrix
The matrix is a weighted table that assigns a weight to each criterion and a score to each supplier, producing a comparable final grade. It turns a subjective decision into a defensible one.
Pick between four and six criteria. More than six and the matrix stops getting filled in. Fewer than four and it fails to discriminate.
Assign weights that add up to 100%. Weight reflects criticality, not habit. For a critical input, delivery weighs more; for a commodity, price does.
Define the scale and what each score means. From 1 to 5, with the threshold spelled out: a 5 in quality means under 1% rejections, not good quality.
Score everyone by the same yardstick. The scale is defined before looking at any supplier.
Calculate the weighted score and read the detail. Averages hide things. A supplier at 4.2 overall but 2 on risk is not a 4.2 supplier.
Worked example with weights:
Criterion | Weight | Supplier A | Supplier B |
|---|---|---|---|
Quality | 30% | 5 | 4 |
Delivery | 25% | 3 | 5 |
Price | 20% | 4 | 3 |
Service | 15% | 4 | 4 |
Risk | 10% | 2 | 5 |
Weighted score | 100% | 3.85 | 4.20 |
A looks better on quality and price, which is what tends to get looked at first. B wins because it delivers on time and carries no risk. That is exactly what the matrix is for: so the decision is not made by whichever criterion gets looked at first.
Where the Data for Each Criterion Comes From
This is where most guides stop and where the real problems begin. The matrix is designed, the weights are agreed, and then somebody has to fill in the cells.
Criterion | Where the data lives | Can it be pulled automatically? |
|---|---|---|
Quality | Quality system or the ERP receiving module | Yes, if rejections are logged by supplier |
Delivery | WMS, TMS or the ERP itself | Yes, comparing promised date against actual date |
Price | ERP, purchase order history | Yes |
Service | Email, calls, messages to the buyer | No |
Risk | External verification sources | Yes, if the lookup is integrated |
The service row is the one that breaks the exercise. In almost every organization, how responsive a supplier is lives in the inbox of whoever deals with it. There is no way to compute it. It gets estimated, and an estimate inside a weighted matrix contaminates the result with the appearance of precision.
There is a quieter problem: even when the data exists, it is almost never in one place. Rejections in one system, deliveries in another, prices in the ERP, verifications in a shared folder. Consolidating all of it for fifty suppliers is days of work, every quarter.
Deloitte documents this in its 2025 global chief procurement officer survey, covering more than 250 CPOs across 40 countries: 57% identify siloed working structures as the main obstacle to delivering value, and among the most effective risk mitigation strategies the second most cited, at 64%, is greater visibility into the supply chain. The obstacle is not a lack of criteria. It is that the information is scattered.
This is, at bottom, a question of data quality and which layer the controls sit in, applied to the procurement domain.
Why Evaluation Matrices Get Abandoned by the Third Quarter
The pattern repeats with surprising regularity. The matrix gets designed, filled in carefully the first time, with more effort the second, partially the third, and by the fourth nobody asks for it. It is not abandoned because of poor design. It is abandoned for three concrete reasons:
Criteria that depend on somebody remembering. Every cell that cannot be computed is a recurring manual task, and recurring manual tasks without a clear owner disappear.
Verifications that expire without warning. A supplier qualified in March can fall out of compliance by September without anyone touching it. Tax standing, certifications, insurance policies and registrations all have validity periods. A matrix filled in as a snapshot cannot see that: it sees the state on the day the verification was run.
Evaluations that trigger nothing. If a supplier scores 2.1 and still keeps receiving purchase orders, the next evaluation is paperwork. The effort of measuring only holds up if the measurement has a consequence.
The practical conclusion is uncomfortable but useful: a criterion that cannot be computed on its own is a criterion you will stop measuring. A matrix with four automatable criteria beats one with eight where half depend on memory.
What Happens After the Evaluation
The part almost no guide covers is what happens to the result. An operational evaluation has three outcomes defined in advance:
Approval, with or without conditions, and with a named approver for each case.
Corrective action plan, with a target metric and a review date, not a general conversation.
Block, which is the outcome that almost never gets resolved.
Actually blocking a supplier means the system will not let anyone raise a purchase order for it, not that an email goes out asking people to stop buying from it. That difference decides whether the evaluation carries authority or is a documentary exercise.
The same applies to verifications that expire. If a document's expiry does not trigger an automatic revalidation and, where warranted, a suspension, the qualification file describes a state that stopped being true.
Automation does not start at validation, it starts at capture. The documents backing the file, the tax certificate, the incorporation deed, the certifications, are processed to extract their fields automatically, so the supplier or the buyer confirms instead of transcribing. That removes the most common source of error and cuts the onboarding time, which is the part of the cycle where the weeks get lost.
This is no longer a matrix question. It is a question of how the supplier onboarding and maintenance process is built. When the supplier is governed as master data with centralized authoring, the criteria become rules that run on their own and the file stops aging.
Supplier Evaluation and ISO 9001
ISO 9001, in its clause on control of externally provided processes, products and services, requires organizations to determine and apply criteria for the evaluation, selection, performance monitoring and re-evaluation of external providers, and to retain documented information on those activities.
Two words in that requirement usually get skipped. Monitoring implies the evaluation is continuous, not an entry gate. And re-evaluation implies the result has an explicit expiry date. In other words, the standard already asks for exactly what gets abandoned by the third quarter in practice. Meeting it on paper is easy; sustaining it is what demands criteria that can be computed without manual intervention.
Case Study: 18,000 Monthly Verifications With Full Traceability
A global food company onboarded suppliers through email, paper and photocopies. The process took weeks and in some cases up to two months, but the more serious problem was not time: there was no systematic validation. A supplier on a denylist could be onboarded without passing any tax or anti-fraud screening. And because requests circulated by email, there was no record of who had verified what.
Arkon Data implemented the full process on its platform. Denylist screening became a blocking validation: if the result is positive, the record does not advance and never reaches the ERP. Approval criteria were split by function, with procurement, tax and compliance each seeing the exact status of every request. And the approved record is written directly into the client's Oracle Cloud environment, with no middleman.
Results to date:
More than 18,000 monthly executions with full traceability and zero lost requests
Onboarding time cut from months to minutes, subject to the approvals in the process
7 operating organizations across 4 languages, with 2,800 users
Bulk onboarding of 1,200 suppliers in Brazil against a regulatory deadline the previous global consultancy would not commit to
The relevant point here is not speed. It is that every one of those 18,000 executions left a record of what was verified, when, and who approved it. That is what turns an evaluation into auditable evidence.
Frequently Asked Questions About Supplier Evaluation
1. How often should a supplier be evaluated?
It depends on criticality. Critical suppliers or those with recent incidents, quarterly. Strategic ones, twice a year. The rest, annually. Evaluating everyone at the same frequency guarantees the evaluation becomes paperwork.
2. What is the difference between supplier qualification and supplier evaluation?
Qualification is the entry filter, with a binary outcome and a documentary file. Performance evaluation is the continuous measurement of whoever is already in, with a comparable outcome.
3. How many criteria should an evaluation matrix have?
Between four and six. The real limit is not conceptual but operational: how many cells the team can fill each period without the exercise being abandoned.
4. What is the weighted point method?
It means assigning a different weight to each criterion according to its importance, scoring every supplier on the same scale, and calculating a final grade. It is the most widely used method because it reflects that not all criteria carry equal weight.
5. Can you evaluate suppliers without specialized software?
Yes, with a spreadsheet, as long as the data for each criterion can be obtained without rebuilding it by hand. The constraint is not the evaluation tool, it is access to the information.
6. What happens if a supplier fails the evaluation?
There has to be a consequence defined before evaluating: a corrective action plan with a metric and a date, or an effective block in the system. Without a consequence, the evaluation loses its authority.
Automate Supplier Onboarding and Evaluation
A well-designed matrix is worth nothing if it depends on someone remembering to apply it. Arkon Data maps your supplier evaluation process and turns it into a flow that runs on rules, not reminders.
The criteria you defined become validations, keeping your master data intact. A supplier that does not comply does not advance, and the record does not reach the ERP until it has cleared every approval. Each verification is stamped with a date and an owner.
Our process engineers work with procurement, tax and quality to map how you evaluate today, including the criteria that live only in the head of whoever applies them. Out of that comes a standardized process running on clean data.
Let's set up a short conversation to review your current evaluation process and talk through how we can help you streamline it.
