2. Make and Calibrate a Pointer
2.1. Introduction
This is the SciKit-Surgery tutorial on making a tracked pointer and the calibration of the pointer. It was developed as a two hour tutorial for online delivery during the 2020 Medical Image Computing Summer School hosted by UCL. Tracked pointers are an essential tool for image guided interventions, but are useful in a variety of applications.
The tutorial makes use of the Python application SciKit-SurgeryBARD
The tutorial is divided into four sections:
Introduction to tracked pointers (10 mins)
Introduction to ArUco markers and assembling your pointer (20 mins)
Calibration of your pointers (40 mins)
Estimating calibration and tracking accuracy (30 minutes)
Discussion and writing up results (20 minutes)
2.2. Learning Objectives
After completing the tutorial, students should be able to:
Describe how a tracked pointer can be used during image guided surgery.
Demonstrate the use of ArUco markers to track an object using SciKit-SurgeryBARD.
Perform a pivot calibration using SciKit-SurgeryBARD.
Estimate the accuracy of the calibrated tracked pointer.
2.3. Assumed Knowledge
SciKit-SurgeryBARD is Python software, it is assumed that pupils have a working Python installation and are able to install packages.
2.4. Installation
Follow the Python Setup instructions, specifically using Option 2 of the installation instructions.
2.5. Activate You Environment
If you’ve followed the Python Setup instructions, you should have ‘activated’ your python environment in the MPHY0026/.tox folder.
As a reminder, these are the commands that ‘activate’ the environment when your current working direction is the top-level folder of the MPHY0026 repository.
source .tox/test/bin/activate
or if you are a Windows user:
.tox\test\Scripts\activate
This means that your terminal is running the python environment with all the right libraries installed. So, make sure you have ‘activated’ the right environment before continuing.
2.7. Part 1 Introduction to Tracked Pointers
Tracked pointers enable the user to locate points and surfaces relative to the tracking system. Their main use for image guided surgery is to locate fiducial markers for use in point based registration or to digitise surfaces for surface based registration. More generally they can be used to make measurements and localise anatomy.
Tracked pointers consist of three parts.
The tip, this is the bit that makes contact with the patient or fiducial marker. The pointer must enable a unique physical point to be located, so the pointer is often pointed so the tip location is unambiguous. However it may also be spherical, so when inserted into a fiducial marker with a spherical divot the centre of the sphere is uniquely identifiable. For surgical applications the tip should be sterilisable.
The tracker marker. This is the part that is tracked by the tracking system, e.g. an electromagnetic coil or the reflective spheres used in optical tracking systems.
The frame and handle. In general it is not possible to place the tracking markers at the tip, so some sort of frame is needed to rigidly connect them. This frame can be designed to also act as a handle for the user.
Have a quick look at some of the videos on the linked (above) registration pages, and observe the types of pointers they use.
2.8. Part 2 Introduction to ArUco markers and assembling your pointer
This tutorial is designed to be carried out away from the operating theatre so we’ll make our own tracked pointer using (hopefully) readily available materials.
For the tip you’ll need something with a point, previously we have used pens, for today’s demo I found a metal skewer which has the benefit of not leaving pen marks.
For the tracking system we’ll use OpenCV’s implementation of the ArUco tracking
library which requires only a calibrated webcam or mobile phone camera and
the ability to print markers or show them on a screen. reg_pointerwithscale shows the
tags we will use for tracking the pointer. You can print them out from the printer ready pdf
and glue them to something rigid (cardboard etc),
or you can display them on your mobile phone screen using the
QR tag (reg_pointerqr).
The patten of six unique tags we will use for pointer tracking.
Scan this with your phone to open the tag image.
The ArUco tracking library relies on using computer vision to detect the corners of uniquely identifiable tags in a single frame of video. The position of the tag relative to the camera can then be calculated using the perspective n point algorithm, implemented in OpenCV. Solving this requires the tag corners in 3D space, which are defined in the file pointer.txt. The first column of pointer.txt defines the uniquely identifiable tag ID. The remaining 15 columns are the x,y,z coordinates of the tag centre and four corners, as shown here:
#tag id #centre (x,y,z) #corner0 (x,y,z) #corner1 (x,y,z) #corner2 (x,y,z) #corner3 (x,y,z)
208 -17.5 -8.75 0 -24.75 -16 0 -10.25 -16 0 -10.25 -1.5 0 -24.75 -1.5 0
295 0 -8.75 0 -7.25 -16 0 7.25 -16 0 7.25 -1.5 0 -7.25 -1.5 0
365 17.5 -8.75 0 10.25 -16 0 24.75 -16 0 24.75 -1.5 0 10.25 -1.5 0
31 -17.5 8.75 0 -24.75 1.5 0 -10.25 1.5 0 -10.25 16 0 -24.75 16 0
1 0 8.75 0 -7.25 1.5 0 7.25 1.5 0 7.25 16 0 -7.25 16 0
757 17.5 8.75 0 10.25 1.5 0 24.75 1.5 0 24.75 16 0 10.25 16 0
SciKit-SurgeryBARD uses a configuration file to set various parameters and the location of the pointer.txt file. You can download a suitable file here (config.json) or copy and paste from below.
{
"camera": {
"source": 0,
"window size": [640, 480],
"calibration directory": "data/example_camera_calib"
},
"tracker":{
"type" : "sksaruco",
"rigid bodies" : [
{
"name" : "modelreference",
"filename" : "data/reference.txt",
"aruco dictionary" : "DICT_ARUCO_ORIGINAL"
},
{
"name" : "pointerref",
"filename" : "data/pointer.txt",
"aruco dictionary" : "DICT_ARUCO_ORIGINAL",
"tag_width" : 32
}
]
},
"interaction": {
"keyboard": true
},
"out path" : "pointer_positions"
}
You will need to change the camera section, based on your results from the camera calibration tutorial. Check that the window size matches the images you used for calibration, which should have been saved as png images in the calibration directory. The pointer tag file is defined with the “pointer_tag_file” entry.
Underneath that is “tag_width”. If you printed your tags out
they should be 32 mm wide, however if you are using a screen to show your tags it may be harder to control the tag width. Looking at reg_pointerwithscale you’ll notice the horizontal line above the tags. You can measure the length of this line on your screen and enter the length into the configuration file under tag_width. This enables to scale your tags without having to change pointer.txt.
If you run SciKit-SurgeryBARD now with;
sksurgerybard -c config.json
you should be able to see that the tags are being tracked by the presence of silver spheres overlaid on the
tag centres, something like reg_pointer_tracking. Double check that you’ve set tag_width right, an incorrect value will make the next step (calibration) very difficult.
If the pointer tags are being tracked you should see silver spheres overlaid on the tags. Here the tag width was measured at 38 mm. If you set tag_width incorrectly in the configuration file the tags will still track, however if you set it too high (58 mm here) the tracker will locate the tags further away than they really are (small spheres at left), similarly if you set it too small (e.g. 18 mm) the spheres will appear closer (thus larger) than they should be.
2.9. Assembly of Your Tracker
Now you’re tracking your markers, assemble the pointer to your tracker markers. I’ve used gaffer tape to stick a skewer to the back of my phone ( reg_ass_pointer ). It is important that the assembly is rigid, you do not want the pointer tip to move relative to the markers.
The assembled tracked pointer.
2.10. Part 3 Calibration
The final stage in building your pointer is to determine the position of the pointer tip relative to the tracking markers. We refer to this as the pointer’s calibration. One method for finding the tip position is pivot calibration, where the tip of pointer is held stationary and the body of the pointer is pivoted about this fixed point.
2.11. Acquiring Data for Calibration
Watch the video below for a demonstration of how to use SciKit-SurgeryBARD to acquire a set of marker poses to use for calibration.
Acquiring marker poses for calibration amounts to pivoting the pointer through a cone of around 60 degrees while pressing ‘d’ in the SciKit-SurgeryBARD window. If it’s not already there you need to add the following to your configuration file to tell SciKit-SurgeryBARD to turn on keyboard interaction.
"interaction": {
"keyboard" : true
},
You should aim to capture at least 20 poses, around 100 would be ideal, but is important to try and spread them evenly around the imaginary cone coming up from the pivot point.
2.12. Performing a Pivot Calibration
Performing the pivot calibration involves finding the offset between the measured marker positions and the unknown tip position such that the pointer tip is stationary. SciKit-SurgeryBARD currently implements two algorithms to find the offset. These are “Algebraic one step”, “sphere fitting” which are described in Yanniv 2015. In addition the algebraic one step method can be used on conjunction with RANSAC to remove outliers from the optimisation. Try running:
bardPivotCalibration -i pointer_positions/bard_pointer_matrices/
You should see output like:
Pointer Offset = [[-180.34596358 -2.65290744 2.35962519]]
Pivot Location = [[-125.67858276 108.98254023 369.15847379]]
Residual Error = 9.914944136259292
The pointer offset is the position of the pointer tip relative to the marker pattern, this is what we’re trying to find. The pivot location is the location of the pivot relative to the tracking camera (the webcam). It is useful to compare this with your physical setup as it can give a quick indication of whether things have worked correctly. Looking at last value we see that the pivot location was about 37 cm from the camera. That seems about right given what we see in the video above. The last value is the residual error, which gives a measure of the spread of pointer tips around the estimated pivot location.
By default bardPivotCalibration.py uses the algebraic one step method. You can change this by supplying a configuration file with the -c command line flag. Create configuration file like this for sphere fitting
{
"method" : "sphere_fitting",
"init_parameters" : [-800, -90, -2000, 300]
}
and like this for RANSAC
{
"method": "ransac",
"number_iterations" : 10,
"error_threshold" : 4,
"consensus_threshold" : 0.25
}
then rerun pivot calibration with your configuration file.
bardPivotCalibration -i pointer_positions/bard_pointer_matrices/ -c your_config.json
What happens? It is likely you’ll need to change in the initial parameters for sphere fitting. The first three parameters are the estimated pivot location, and the last is the sphere radius. You could use the output from the algebraic one step method for this.
If time permits repeat this process (acquisition and calibration) several times and save your results in separate directories. How much do the results vary? What happens to the residual errors?
2.13. Part 4 Estimating Calibration Error
Knowing how accurately your pointer can locate things if very important for image guided surgery. This localisation accuracy forms the main part of Fiducial Localisation Error which will be covered in greater detail in the registration tutorial.
Start by picking one of your registrations and copying the pointer tip position into a pointer_tip.txt file like:
-180.34596358 -2.65290744 2.35962519
Then add “pointer_tag_to_tip” entry to the BARD configuration file like:
"pointer": {
"pointer_tag_to_tip": "data/pointer_tip.txt"
},
Now run SciKit-SurgeryBARD with;
sksurgerybard -c config.json
When your tags are visible you should now see an extra sphere, somewhere near the tip of the pointer.
Tracked pointer with tip. Note the additional sphere at the left hand side showing the estimated position of the pointer tip.
At this point the you may notice that the sphere marking the pointer tip is very jittery. This due to the small tracking errors at the markers being magnified by the lever arm of the pointer. A longer pointer should be more jittery. You can reduce the jitter by adding some tracking averaging with “smoothing buffer” entry in the tracker configuration like:
"tracker": {
"type" : "sksaruco",
"smoothing buffer" : 5,
etc ...
This will use a 5 frame rolling average to reduce the random tracking noise.
Now you can see where your calibration places the pointer tip and where it actually is you can make some estimates of the calibration accuracy. This will be easier with some sort of measuring device (a ruler for example, see reg_pointer_measure).
Use a ruler to measure the difference between the estimatated and apparent pointer tip positions in various orientations. Make sure you do it in a range of orientations.
Estimate the calibration error over a range of pointer poses. Make a note of it, then repeat the process for a different calibration from Part 3. Do this for as many calibrations as time allows. Try and answer the following questions.
How accurate is the calibration on average?
How much does calibration accuracy vary?
Was one method better than another?
Is there any link between the number of calibration data points and the calibration accuracy?
Is there any link between calibration residual errors and calibration accuracy?
If you have time, make a pointer with a shorter or longer shaft, what happens then?
Write up your results and share. That is the end of the tutorial, thank you.