Researchers at AMBER, the SFI Research Centre for Advanced Materials and Bioengineering Research, the Trinity Advanced Microscopy Laboratory and the School of Physics at Trinity, have unveiled a project to maximise the imaging potential of electron microscopes. The Trinity team, with the support of the public, will create software to dramatically enhance our ability to examine, and visualise, fragile nanostructures.

The team are inviting everybody to visit their website where visitors can click through images of samples taken from an electron microscope and compare these against reconstructed images.

The idea is that humans ability for pattern recognition and visual comparison will generate information to teach a computer this very human task. The results of this citizen science project will enable the team to generate software that can recognise the most truthful reconstructed image of a real nanostructure.

Citizen science to help solve a real world challenge

As anyone who has tried to take a picture at night, close-up, or of a moving object, there are constraints to modern photography. You need enough light so that your camera can take an image, and you need to be able to focus your camera on the object you want to capture.

If you can’t, then you may end up with a blurry image that doesn’t accurately represent what you can see. The challenge of taking images of tiny particles with electron microscopes is similar, but with an additional issue: if you want to take a clean image that truly reflects the real nanoworld, you need a high-energy electron beam, which in turn can damage, or completely destroy the sample you want to image. Using lower energy can mean that the images created are not that useful for scientists.

Teaching a computer to recognise the best, most truthful representations of the nanoworld, in real time, so there is no time lag between taking an image, and being able to see it, is a real world challenge.

The science behind the citizen science

The science behind the machine learning tool created by the team is unique and involves a two-step automated process. Computer software is already used to deconstruct raw data images generated from electron microscopy into their component parts using principal component analysis.

The team want to create additional software that will automatically enable a computer to reconstruct the image thereby greatly speeding up the capacity for real time, lower energy, electron microscopy.

Professor Lewys Jones, AMBER investigator, SFI & Royal Society Research Fellow, and Ussher Assistant Professor in Ultramicroscopy at the School of Physics, said: “The combination of mathematical and computer tools used in this project is entirely new and could transform how we use these microscopes for research.

"At the Advanced Microscopy Laboratory we house the most powerful microscopes in Ireland, one of which can take images on the scale of individual atoms.

"But, as with all imaging techniques, there are limitations; one key limitation is the time between taking and processing an image, and then sending this on to the research scientist.

"With this new machine learning tool we hope to be able to support nanomaterials researchers with no time lag. This will enable the scientists we work with to give us immediate feedback on their image, so that if any adjustments need to be made, such as changing the orientation of sample, to get better results, and get better information, we can do it there and then.”

Machine learning techniques

Michael Mitchell, student at the School of Physics, says: “Applying machine learning techniques is not new in the area of image recognition, and is something that many data-scientists are familiar with.

"What makes our tool different is our two stage machine learning approach; the first phase being the purely mathematical, unsupervised image-component identification, which we then layer on with the second phase; the separation of useful and useless components, which in contrast to the first phase, is supervised and participant-trained.

"We hope that hundreds of people will take part in this project because this kind of human input will align closely with researcher preference where mathematical input might miss the mark. We can’t teach a computer without hundreds of human teachers!”

Michael Mitchell is an undergraduate in Theoretical Physics entering 3rd year, and studying as an intern at the Advanced Microscopy Laboratory in the summer of 2020 as a Laidlaw Scholar.