How to use the review library

Who this is for

If you are evaluating a healthcare AI product for purchase, this guide is written for you. That includes CMIOs building a clinical AI strategy, CIOs running technology governance, IT governance committees scoring proposals, compliance officers validating vendor claims, and supply chain teams negotiating contracts.

You do not need a technical background to use the review library. The scoring method is designed to produce findings a non-technical board member can read and act on, because procurement decisions are made in rooms where most people are not engineers.

What a review tells you

Each review scores a product on six dimensions using the RUAIH Vendor Score method. Every dimension is scored from 0 to 3, and each score is backed by a named, dated, public source.

A review does not tell you whether the product works. Nobody outside the vendor can honestly answer that from the public record. What a review tells you is what evidence exists — and what does not — for the claims a vendor is making.

A high score means a buyer can defend the purchase. It means the evidence trail is strong enough that when your board, your regulator or your malpractice carrier asks "why did you buy this?", you have something to show them beyond a vendor deck.

A low score means the evidence is thin. It does not mean the product is bad. It means you are carrying more risk, because if something goes wrong, the public record will not support the decision.

That distinction matters. A strong product with no published evidence is a procurement risk. A weak product with excellent documentation is still a bad purchase. The score measures one thing — the defensibility of the buy — and it measures it well.

How to read the six dimensions

Each dimension answers a different question. Here is what to look for in each one, and what the score actually means for your process.

1. Independent validation

Look for the gap between level 1 and level 2. A vendor who asserts performance figures in a slide deck with no published method, no named population and no denominator is asking you to trust them. That is not evidence. It is marketing.

At level 2, the vendor has at least published a study or technical report with methods you can read. At level 3, someone with no commercial interest has confirmed the claim. The distance between "we say it works" and "someone else checked" is where most procurement risk sits.

2. Regulatory position

Check the actual clearance against the marketed indication. "FDA registered" is not a clearance — it means the company is listed with the FDA, which is an administrative step, not a regulatory finding. A product marketed for clinical decision support that holds a 510(k) for a different indication has a gap, and you need to know about it before the contract is signed.

If the product sits outside device regulation, the review checks whether the vendor explains correctly why. That explanation matters, because "we don't need clearance" and "we have clearance" are both assertions your compliance team needs to verify.

3. Transparency

A model card is the minimum. It tells you what the model was trained on, what it is intended for, and what its known limitations are. If a vendor cannot produce one, the product is a black box, and you are deploying a black box into clinical operations.

At level 3, look for versioning. If the vendor cannot tell you what changed between v2 and v3, they cannot tell your compliance team either. And your compliance team will need to know, because a model update mid-contract changes the thing you evaluated.

4. Security and data rights

SOC 2 is table stakes. A current third-party attestation and a BAA are the floor, not the ceiling. The real question — the one most buyers do not ask until too late — is data training rights. Can your patient data be used to train a shared model? Is that default-off, or do you have to find an opt-out buried in supplementary terms?

A vendor at level 3 on this dimension answers that question in plain language, with customer data excluded from training by default. If you have to negotiate your way out of it, the default tells you something about how the vendor thinks about your data.

5. Governance artefacts

This is the dimension nobody else scores, and it is what decides whether you survive an audit. Audit logging, human override capture, monitoring, drift detection — these are the artefacts your governance committee will need two years after go-live when someone asks whether the system operated as intended.

A product that scores well on every other dimension but has no governance artefacts will pass the procurement review and fail the post-market review. That is the worst possible outcome, because by then you are in production and your options are expensive.

6. Commercial terms

If you cannot leave, it is not a purchase. It is a dependency. Exit rights, data portability, capped price escalation and a service level with actual remedies — these are what separate a contract from a lock-in.

A vendor at level 0 on this dimension puts everything behind an NDA before you can evaluate it. That tells you something before the evaluation even starts: the terms are designed to be seen only after you are committed.

How to use a review in a procurement process

A review is most useful when it arrives at the right point in your process. Here is the workflow.

Step 1: Check the review before the vendor demo

Read the review before the first call. It takes ten minutes and it changes the questions you ask. A vendor who scores well on validation but poorly on governance artefacts is going to give you a strong demo and a weak answer on audit readiness — unless you ask.

Step 2: Use the due diligence checklist to track what you verify yourself

The review tells you what the public record shows. Your due diligence confirms what the vendor shows you directly. Use the due diligence checklist to track every claim you verify in person — the BAA you read, the SOC 2 report you inspected, the model card you were shown. The checklist is where the review meets your own process.

Step 3: Compare the vendor's claims against their published evidence

The most useful thing a review does is make gaps visible. If the vendor tells you in a demo that their product reduces documentation time by a specific percentage, check whether that figure appears in the review's validation dimension. If it does not, the claim is unsubstantiated on the public record, and you should ask for the study.

Step 4: Take the weakest dimension into contract negotiation

Every review leads with the weakest dimension. That is the dimension you take into the contract. If governance artefacts scored lowest, your contract should require audit logging, override capture, and a monitoring view — in writing, with a timeline. If commercial terms scored lowest, negotiate exit rights before you sign.

The weakest dimension is not a reason to walk away. It is the place where the contract needs to do the work the evidence does not.

Step 5: Re-check after six months

Every review is re-scored at least every six months. After you deploy, check the updated review. A vendor who was at level 1 on transparency at purchase and has since published a model card is a vendor who is moving in the right direction. A vendor whose security score dropped is a vendor you need to talk to.

What reviews do not cover

Stated here so you know the boundaries.

Fit. A review does not tell you whether the product fits your workflow, your EHR, your patient population or your staffing model. The right score for the wrong workflow is still the wrong purchase. That is what the readiness score and the toolkits are for.

Implementation quality. How a product deploys — the integration work, the training, the go-live support — is not visible on the public record and is not scored. It matters enormously, and it is something you can only evaluate through references and your own pilot.

Support. Post-sale support quality varies by account, by region and by contract tier. It is not reliably verifiable from public sources, and we do not score what we cannot verify.

The "will it work here" question. That is the question every buyer actually wants answered, and no external review can answer it. It depends on your data, your clinicians, your workflows, your IT infrastructure and your willingness to change how you work. What a review can do is tell you whether the evidence exists for the claim — the rest is yours to test.

Where to start

If you have not read the scoring method, start there. It takes fifteen minutes, and everything in the review library will make more sense once you understand what a level means and why there is no composite score.

• The RUAIH Vendor Score method — how we score, and why each dimension exists.
• Due diligence checklist — the working document you fill in alongside a review during procurement.
• Evidence landscapes — the research base behind each healthcare AI category, separate from any vendor.