3.1. Segmentation and Modelling
3.1.1. What is Segmentation?
“Segmentation is the process of partitioning an image into different meaningful segments.” (Wikipedia).
In the example below, of a T1 weighted MR scan of a brain, a meningioma has been identified.
So the image is partitioned into pixels that correspond to meningioma, and pixels that are not meningioma.
Fig. 3.1 Figure: Segmentation of a meningioma, visualised in 3D slicer, by RKikinis on Wikimedia, is licensed under CC BY-SA 3.0.
Also called:
Labelling: Each partition is assigned a separate label, either binary: background=0, meningioma=255.
Or multiple labels like Freesurfer:
Fig. 3.2 Figure: The software package FreeSurfer segments/labels many regions of the brain, by LarrabeeMGH on Wikimedia, licensed under CC BY-SA 4.0.
From statistics/machine learning literature, also called “Classification”, as you identify pixels as belonging to a certain class.
Representation:
Pixel mask
Contour definition (list of points)
Implicit surface (distance to zero-crossing)
3.1.2. Its a BIG field
There are:
many image types
many things being imaged
many artifacts like intensity inhomogeneity, beam hardening and artifacts caused by metal.
In addition, add in factors like, user dependent error, and hence:
There are many many research papers on segmentation.
Even review papers are split into sub-specialities.
3.1.3. Pixel-Based Methods
While many methods have arguably been superseded in the medical domain by methods using machine learning, it is still worth spending some time investigating these classical methods, as it gives you an appreciation of all the things that can go wrong. Most of the same issue can go wrong with machine learning methods too. While machine learning can learn more complicated functions than you or I could explicitly program, it would be still be foolish to assume that machine learning is a panacea for all segmentation issues.
3.1.3.1. Image Thresholding
Firsly, here is an introduction to Segmentation from a range of tutorials at Bioimage Analysis.
While this tutorial refers to biological images, rather than medical images, the concepts are the same.
How does this work in practice for some medical imaging? Here, Dr Clarkson gives a short overview:
Once you start thinking about individual pixels in the image, it’s important to appreciate the histogram of image intensities. This video from the Udacity YouTube channel, gives a useful introduction:
Following from simple thresholding then, and your understanding of the image histogram, a method that seeks to automatically pick nice thresholds is Otsu’s method. In the following video, Dr Clarkson gives an overview of how it might apply in medical imaging:
3.1.3.2. Region growing
One of the problems with image thresholding, is that it’s a global method, with no concept of connectivity. In other words, the algorithms do not consider that if adjacent pixels have the same intensity, they are likely to be part of the same object. Region growing algorithms aim to start from a seed point, and iteratively add pixels to the segmented volume, based on connectivity, and various heuristics, like whether or not adjacent pixels are within a close enough intensity range.
Here, is a demonstration of region growing algorithms:
3.1.3.3. K-Means
K-Means is a common method, not just in image-processing, but in data-clustering in general. Here is an introduction from the wonderful Computerphile channel:
Most general descriptions of K-means illustrate the algorithm on a 2D scatter plot. In a 2D scatter plot, there are two variables of interest, so each of the K-means is a vector of length 2. However, when used in image segmentation, of a single image, a single variable is used to denote the image intensity of each pixel. In the 1D case, you are essentially doing K-means, looking for K values representing peaks on the histogram of image intensities.
3.1.4. Atlas-Based methods
Another class of methods, fairly popular for a while in medical imaging, was that of atlas-based methods. An atlas is a reference image that has been accurately segmented, that you can consider to be a reference or template for all other segmentations. The atlas may be made from a single image, or from some process of averaging/combining a large number of scans and extracting an average image and segmenting that. The segmentation is often done manually, as this process is used to bootstrap the segmentation of a large database of other images.
For a single image atlas, the general process could be something like:
Identify a reference image. e.g. a healthy control subject of average age.
Manually segment the image, as accurately as possible.
For a new image that needs to be segmented, first register (align) it to the atlas
Copy the atlas labels onto the image that needs segmenting
In the following video, Prof. Paul Yushkevich who worked on ITK Snap, one of the most popular segmentation tools, gives an overview of Atlas Based Segmentation methods.
3.1.5. Model-Based Methods
The main categories of model-based methods are:
Statistical Shape Models (SSM) (next page and Statistical Shape Models).
3.1.6. Neural Network Based Methods
In recent years, since the 2012 ImageNet competition for example, machine learning and specifically deep learning have had great success at many computer vision tasks, including image segmentation, which in the deep learning literature is called semantic segmentation.
However, this simply cannot be a course on deep learning, or semantic segmentation. In this section of the notes, we just provide links to external resources that would give you an overview of how a neural network can be used to segment an image.
3.1.6.1. Convolutional Neural Networks
What to put in this section, as it could be a whole lifetime of study!
Here follows a list of resources that are useful to get up-to-speed with using Neural Networks.
A Keras tutorial from DataCamp illustrates what a neuron is.
A Keras tutorial on semantic segmentation as an introduction to segmentation.
The Deep Learning book.
Simon Prince’s new Understanding Deep Learning book.
Dr. Yipeng Hu’s tutorials from the course “Machine Learning in Medical Imaging (MPHY0041)” at UCL.
Dr. Yipeng Hu’s tutorials on segmentation from the course “Artificial Intelligence for Surgery and Intervention (MPHY0043)” at UCL.
To get started with your first network is not too hard. For example, (Even!) Prof. Clarkson coded up a UNet here, as part of his own learning process. But the real trick to “getting good”, is to practice on real problems. So, its not something you can just read. You need a project, and to be actively doing it.
3.1.6.2. Combining Neural Networks and Manual Annotation
Medical Images are still difficult to segment, even with the power of Artificial Intelligence and Deep Neural Networks. This is mainly due to a relative lack of data. In computer vision for example, algorithms can be trained on millions of images, but in the medical community, we might be dealing with only 100’s or at best 1000’s of images.
So, there is still a justification for training a neural network to partially segment things, and combining the network with manual input from the user for a manually-guided + AI hybrid.
3.1.7. Specific Challenges for CAS
Segmentation has additional problems that are specific to Computer Assisted Surgery (CAS):
Abnormal growths, so different shapes in training set compared to test set, or normal population
Post-op, metal artefacts, missing sections of anatomy
Low volume cases (one-by-one, each case different)
Class imbalance (lots of examples of good/healthy population, compared to few in diseased groups)
As a consequence, you still often end up with manual post-processing
3.1.8. What Tools Can I Use?
3.1.9. Commercial Services Exist
Think carefully about the cost-benefit of segmenting your own, and if training a Deep Learning model, will you outperform others with much larger datasets?
3.1.10. Segmentation of Pre-Op data
Once you have segmented/labelled/outlined, you can:
Use segmented models to measure size/volume/length pre-operatively.
Plan operation in terms of what to target, and what to avoid.
Intra-operatively, visualise where the target is. (More on visualisation later).
3.1.11. Segmentation of Intra-Op data
Segmentation can also be used intra-operatively, live, in real-time to aid things like computer vision based algorithms.
For example:
Video segmentation: e.g. in laparoscopic video, identify Liver/Not-Liver, use to track specific objects
Used segmentation to filter points: e.g. having identified Liver/Not-Liver, only do surface reconstruction on things that are Liver.
So, the use-cases pre-operatively, and intra-operatively are very different.
Different time constraints
Non-real time versus real time etc.