What Evaluation Method Should I Use?

A checklist for practitioners

The range of evaluation methods available for you to use is fairly broad, and not all methods are well known. Here is a quick and dirty summary of the main evaluation methods that are available, together with some guidelines on when and how to use them.

Direct Appointment: Used frequently in the private sector, this method makes sense for low value or specialist projects where there is low risk; where you have prequalified suppliers who you know can do the job; where emergency works need doing or where it’s valuable to have a long-term relationship with a single supplier.
Most organisations have a dollar cap and some tight rules on the value of contracts that can be procured without competitive processes in place. This is especially important for public sector organisations, where Direct Appointment could lead to a perception of unfair practices. Be careful with this one!

Lowest Price Conforming (LPC): A common method that is, however, rapidly falling out of favour with best practice procurement professionals. Used correctly, this should involve:

1. A clear, ‘pass/ fail’, objective definition of what constitutes a ‘fail’ or non-conformance, included in the RFT. This should not be merely a score (e.g. under 35) without a tightly-defined description of what that score is attributable to.
2. Two envelopes [price and non-price], so that the evaluation of price information is obviously distinct from evaluation of attributes.
3. Best practice requires opening of the price envelopes first, then ranking of the prices from lowest to highest.
4. Due diligence on the lowest price bidder to check there are no tags that would have a financial effect, and no pricing errors.
5. Only the attributes of the lowest price bidder are checked for conformance to the pass/ fail standards defined in the RFT. If those attributes conform, the process is complete and the attributes of the other bidders do not need to be checked [this, in itself, is a significant saving in time].
6. The contract is awarded to the lowest price bidder provided it conforms. The attributes of the other bidders are discarded or returned.

The beauty of this method is that it does not allow value decisions on which bidder’s attributes are better than the others. The decision is cut and dried, with no opportunity for bias.

For this reason, LPC should only be used for contracts that are low in value, where scope is tightly defined, where risks are insignificant and could not have a material effect on the success of the project; and where there are no opportunities for whole-of-life cost benefits, innovations, or added value alternatives.
If any of these conditions don’t apply to your project (i.e. there are some risks, potential for innovation, or the project is somewhat complex), then LPC is not an appropriate method to find the best value for money supplier.

Weighted Attributes: Also a widely used method, involving assessment of attributes and price in varying proportions, to come up with a preferred bidder that optimises price and quality factors. While the weightings of the attributes and the price are entirely at the discretion of the procurement personnel acting for the client, there are some traps for the unwary. First up, here’s how it works:

The attributes and the price are all weighted. Responses should, as above, be received in two envelopes so that assessment of attributes and prices can be shown to be undertaken independently.

1. The suppliers’ attributes are evaluated and scored by the evaluation team.
    
2. A price score is calculated for each of the bidders. Naturally, the lowest price supplier should receive the maximum score, and those suppliers with higher prices should score less than that. Two price scoring formulae that are in common use are:

Price Score = 50 – {100 x (Tender Price/Estimate)}

or

Price Score = 100 x {(Lowest Tender Price)/(Tenderer X’s Price)}

The second version of the price score formulae above is a little easier to use and more intuitive, as the lowest price bidder always receives a score of 100 while other higher priced bidders are scored lower. It also doesn’t require an estimate, which can be an advantage where there is some uncertainty.

3. Once all the attributes are scored, and the Price of each bidder is scored, the scores are multiplied by the weights for the respective attributes and then added together. The highest weighted score is the winning bid.

The down side with using weighted attributes is that there is little transparency on the trade-offs between price and non-price components of the scoring. This may lead to some surprises when the answer pops out of the black box at the end of the process.

The Price Quality Method is a variation on Weighted Attributes, which provides for transparent trade-offs to assess the effects of the weights, before the RFT is finalised. Used correctly, this method achieves the same results as weighted attributes, but with a much higher level of transparency. It also provides the evaluators with the tools for a robust sensitivity analysis before RFTs go out, so they can tailor the process to deliver appropriate financial recognition for better quality solutions on a project.

For these reasons, Price/Quality is emerging as the most useful and effective method in many procurement situations. It’s most suitable where there is a degree of risk and complexity or innovation potential, and the potential effect of that warrants care in setting the weights for the attributes to deliver best value for money.

1. Like weighted attributes, the Price/Quality method starts by setting weights for the attributes and the price. At that point, before the RFT is released, it’s possible to review the Supplier Quality Premium (SQP) that is assigned to each attribute point. This is a dollar value that corresponds to the amount that the client is prepared to pay for each attribute point that one bidder scores higher than another. It’s a perfect ‘reality check’ for clients to consider, alongside the potential material impact of risks on the project and/ or the value that innovations might deliver to the project or the asset in future.
    
2. Effective procurement practitioners will test several scenarios for attribute weightings, until they are satisfied that the SQP is appropriate to the project.
    
3. At that point, the RFT goes to market and subsequently responses are received, as above, in two separate envelopes – one for attribute information, and a separate one for price information.
    
4. The attribute envelopes are opened and attributes are scored (ideally on the basis of an objective and anchored scoring system, but that’s another discussion!)
    
5. Scores are entered into a spreadsheet which calculates the SQP for each of the bidders. The lowest quality bidder receives a SQP of zero, as their attributes are the baseline and the client would not want to pay any extra for the quality of their solution.
    
6. The Price envelopes are then opened and the SQPs of each of the bidders are then deducted from the price.
    
7. The lowest adjusted price represents the winning bidder (the optimal weighted combination of highest quality attributes and lowest price).

Both Price/ Quality and Weighted Attributes methods rely heavily on selection of correct weightings for the attributes and the price. As a general rule, the greater the level of risk, complexity, uncertainty or potential for innovation, the higher the attribute weightings should be.

Red flags should be raised with these techniques if the balance between price and non-price factors is very uneven: At 70% Price and 30% attributes, the suppliers’ attribute information contributes very little to the decision-making process.

Is it really worth the bidders’ time and the evaluation team’s time to prepare and assess all of those attributes – or could you have saved huge amounts of time and cost by using Lowest Price Conforming evaluation?

If you’re not comfortable that LPC is the most appropriate method, given the level of risk and complexity of your project, then perhaps the 70/30 weightings are not appropriate and you should test a more even 50/50 or 60/40 combination.

Target Price [Purchaser Nominated Price]: This is a highly useful method when budgets are fixed, and the quality of the solution is the primary determinant in assessing the value you will receive for your spend. It’s especially useful where a Request for Proposal is sought (in other words, the client knows the outcome they need, but they’re relying on their suppliers to describe the best way to get there).

Here’s how it works:

1. The budget is set by the client and made clear in the RFP. The outcomes sought are also described in appropriate detail.
2. Suppliers are then asked to describe their proposed solution in relation to a list of weighted evaluation criteria.
3. The bids are then scored and the winner is assessed using the weighted attributes method (but with no need for a price score).

This is a simple and powerful method to achieve best value for a fixed budget. Within a wider context, this is probably the method you chose when you bought your last house, car or significant asset – you knew your budget cap, and went all-out to get the best possible value for that amount.

Quality-Based Selection, or Brook’s Law: This method applies when the scope of the project is unclear and/ or it is difficult, time-consuming or simply not possible to estimate the price of carrying out a contract. It is also useful when the relationship between the client and the supplier will need to be extremely cooperative to deliver the project successfully (for example, many Alliance or Public-Private Partnership projects are procured this way). In many cases, once the best supplier is selected, the client and the supplier(s) then work together to develop a ‘Target Outturn Cost [TOC]’, together with a pain share/ gain share mechanism that will incentivise outstanding performance as all parties work together to deliver the project.

No price is involved: simply a list of attributes that define the qualities that the client is looking for in a successful supplier (or group of suppliers). There are some more complex mechanisms that are commonly overlaid with this evaluation method, to quantify risks and undertake very robust cost/benefit analysis of the solutions offered.

Putting it All together:

So what should you consider, when choosing an evaluation method? Here’s a quick and dirty checklist to consider:

1. What pass/fail criteria should we include to eliminate unsuccessful suppliers at the start?
2. What level of risk/ complexity/ potential for innovation is there, and what material effect could that have on the outcomes? How do these relate to each of the attributes we should ask about?
3. How well defined is the scope of goods or activities required? Are we better to produce an outcome-related RFx document, rather than over-specifying (and possibly limiting innovation)?
4. Is there a set budget?
5. Is it reasonable or possible to cost this project accurately at tender stage?

The first step towards effective procurement processes is to understand the project and the supplier market, along with the risks and the opportunities to deliver added value. That analysis is scalable: it should be done, even on the smallest and simplest projects, to guide the right choice of evaluation method and the right tools to find the best value supplier.

With more attention and expertise invested into planning your procurement processes and choosing the right supplier selection method, you will achieve better results, and you’ll save time and money in the evaluation process.