6.6. Registration Metrics

In this section we cover the key metrics used for measuring and reporting registration error. This work was largely pioneered by a group at Vanderbilt and especially by the group led by Prof. Mike Fitzpatrick. A large part of CAS evaluation methodology and subsequent success in the OR has been born from the ability to understand the errors present in image-guidance system.

Registration Evaluation is nicely covered by Ziv Yaniv in Chapter 6, and section 6.4 of Peters and Cleary, “Image-Guided Interventions: Technology and Applications”.

For MPHY0026, none of these formula need memorizing.

In the text below, we’ve used notation from each paper. This is to encourage people to read it, alongside the paper. Or should we make the notation consistent on this page?

6.6.1. Evaluation Criteria

When preparing this section, I (Matt) naturally jumps to thoughts of FLE, FRE, TRE, covered below. But thanks to Ziv, it’s worth discussing these additional criteria first:

Fast - real-time. What is real-time really? Very context dependent, and often a marketing buzzword.
Accurate - [Maurer1998].
Robust - $N/2$ , meaning over half the data must be outliers to break the registration.
Automatic - No user interaction.
Reliable - Given clinical expectation, the registration is deemed a success.

Open Discussion.

6.6.2. Fiducial Localisation Error (FLE)

FLE is the error in determining the position of each fiducial / landmark [Maurer1998].
FLE “is essentially the root-mean-square distance between the exact and calculated fiducial positions” [Maurer1993].

$\sigma_{fl}^2 = E( \lVert {\bf e}_{fl} \rVert^2 )$

where $E$ means Expectation, i.e. average.

FLE is different for Image and Physical space

Questions

What are the assumptions here?

What errors exist in practice?

Measurements may have non-Gaussian or biased distribution
Note that optical tracker accuracy varies throughout physical measurement volume
Remember that FLE is UNKNOWN. We ESTIMATE it via repeated measurements.

6.6.3. Fiducial Registration Error (FRE)

For ordered, corresponding point sets ${\bf p}_j$ and ${\bf q}_j$ and tranform $T$ , point-based registration minimises:

$d(T)^2 = \frac{1}{N} \sum_{j=1}^{N} \lVert {\bf q}_j - T({\bf p}_j) \rVert^2$

So FRE is the minimum of this function. i.e. “the distance between corresponding fiducials after* registration and transformation” [Maurer1998].

Questions

Refer to [Maurer1998] directly

Look at design of fiducials for image space and physical space

What do you notice?

6.6.4. Target Registration Error (TRE)

TRE is “the distance between homologous points other than the centroids of fiducials” [Fitzpatrick1998].

Normally expressed as RMS error, like FRE
Points must not be a point used for Point Based Registration.
Registration accuracy not dependent on patient
So, validation can be done on phantoms - key for commercial adoption
So, in a lab validation, you normally use extra landmarks, which are used as targets with which to measure TRE
In the literature, we also see clinical targets, centroids of tumour etc.

6.6.5. Estimating TRE from FLE

The seminal paper by Fitzpatrick et al [Fitzpatrick1998] re-derived the following:

$\langle FRE^2 \rangle = (1 - 2/N) \langle FLE^2 \rangle$

where $N$ is the number of fiducials and $\langle \rangle$ means Expectation, i.e. average.

The main result of the paper was a formula to predict TRE from FLE:

$\langle TRE^2({\bf p}) \rangle \approx \frac{ \langle FLE^2 \rangle }{N} \left( 1 + \frac{1}{3} \sum_{k=1}^3 \frac{d_k^2}{f_k^2} \right)$

where ${\bf p}$ is a target point, $d_k^2$ the squared distance between the target and the $k$ th principal axis and $f_k^2$ is the root mean squared distance between the fiducial points and the same axis.

The last formula is often used in designing optically tracked tools [West2004], and for estimating the errors at a distance from a tool, e.g. endoscope [Shahidi2002].

6.6.6. FRE And TRE Are Uncorrelated

In [Fitzpatrick2009] Mike Fitzpatrick shows that FRE and TRE are uncorrelated
Proven mathematically, assuming independent Gaussian noise
Demonstrated in simulation

Strong advice: Do not use FRE as an indicator of accuracy

6.6.7. FRE And TRE Can Underestimate

The above papers assume independent, Gaussian noise on fiducials. There is a body of work analysing PBR when the noise is not so: [Batchelor2000], [Wiles2008], [Moghari2009], [Danilchenko2010] and also for tracking [Fitzpatrick2009] which is covered next week.

Nice illustration of clinical evaluation: [Shamir2009] from 2009.
Possibly underestimated due to non-Guassian effects
Illustrates how much work (15 years) done on PBR, and validation.

6.6.8. Do Not Claim FRE as TRE

Sometimes you cannot measure TRE. i.e. points on internal organs.
So, in practice you only have FLE and then FRE.
So you must report it as FRE. Not anything that sounds like TRE.
Don’t say “The accuracy of my system is X” where X is in fact FRE.

6.6.9. Fiducial Regsitration Educational Demo

For an interactive demo and game to study the effects of fiducial location on metrics such as FRE and TRE, please work through the Fiducial Registration Tutorial.