Researchers have developed a rationalized deep learning super-resolution microscopic imaging method

In recent years, computational super-resolution methods represented by deep learning can improve the resolution or signal-to-noise ratio of microscopic images without losing other imaging performance, showing broad application prospects
。 However, in response to the image requirements of biomedical research with high fidelity and quantitative analysis, deep learning microscopy imaging methods have three common problems: limited by the spectral frequency shift (spectral-bias) problem inherent in deep learning, the output image resolution cannot reach the ground truth level; Limited by super-resolution reconstruction, ill-posed problem of denoising problem and model-uncertainty of neural network models, the authenticity of reconstruction or prediction results cannot be guaranteed.
The training of deep neural networks requires a large amount of data, but the collection of high-quality training data is extremely difficult or even impossible to achieve
in many application scenarios.
At present, the research and development of deep learning microscopy methods are in full swing and show the potential to exceed the performance limits of traditional imaging, but the above problems hinder the use
of existing deep learning super-resolution or denoising methods in biological microscopy imaging experiments.

On October 6, the Li Dong research group of the Institute of Biophysics of the Chinese Academy of Sciences, together with the Department of Automation of Tsinghua University, the Institute of Brain and Cognitive Sciences of Tsinghua University, the Dai Qionghai Research Group of Tsinghua-IDG/McGovern Institute of Brain Science, and Jennifer Lippincott-Schwartz, Ph.
D.
of the Howard Hughes Medical Institute, in the form of an article in Nature Biotechnology, Published a paper
entitled Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes 。 This study proposes a set of rationalized deep learning (rDL) microscopy imaging technology framework, which integrates optical imaging models and physical priors with neural network structure design, rationalizes the network training and prediction process, so as to achieve high-performance, high-fidelity microscopic image denoising and super-resolution reconstruction, and combines the multi-modal structured light illumination microscope (Multi-SIM) and high-speed lattice light sheet microscope ( LLSM), which improves the imaging speed/time history of traditional TIRF/^GI-sim, 3D-SIM, LLS-SIM and LLSM by more than 30 times, achieves the current international fastest (684Hz), the longest imaging time course (up to 3 hours, more than 60,000 time points) live cell imaging performance, and the first time to the transporter (IFT) in high-speed swinging cilia (>30Hz) Rapid, multicolored, long-time, super-resolution observation
of multiple transport behaviors and liquid-liquid phase separation during intact cell division.
Nature Biotechnology also published a research briefing on this work
.

Specifically, the rationalized deep learning structured light super-resolution reconstruction architecture (rDL SIM) proposed by the Li Dong/Dai Qionghai research team is different from the end-to-end training mode of the existing super-resolution neural network model, but adopts a stepwise reconstruction strategy, which first uses the proposed neural network of fusion imaging physical model and structured light illumination a priori to denoise and high-frequency information enhancement of the original SIM image, and then performs SIM reconstruction through classical analysis algorithm to obtain the final super-resolution image
。 Compared with the super-resolution reconstruction neural network model DFCAN/DFGAN proposed by the team in Nature Methods last year, rDL SIM can reduce the uncertainty of super-resolution reconstruction results by 3~5 times, and achieve higher fidelity and reconstruction quality.
Compared to other denoising algorithms such as CARE, the rDL SIM recovers moiré fringes modulated in the original image and enhances high-frequency information by a factor of more than
10.

In addition, for widefield illumination or spot scanning imaging modalities such as lattice light sheet microscopy and confocal microscopy, the team proposed a learnable Fourier Domain Noise Suppression Module (FNSM).

The module can use OTF information to adaptively filter out noise in microscopic images
.
Based on this, the research team constructed a channel attention denoising neural network architecture embedded in FNSM, and based on the spatiotemporal continuity of the microscopy imaging data itself, a spatiotemporal interleaved sampling self-supervised training strategy (TiS/SiS-rDL)
was proposed.
This strategy can realize the training of denoising neural networks comparable to supervised learning without additional training data collection or time continuity of time series data, which solves the problem
that high-quality training data is difficult to obtain in actual biological imaging experiments.

The rationalized deep learning super-resolution microscopy imaging method can be applied to a variety of microscopic imaging modalities including 2D-SIM, 3D-SIM, LLSM, etc.
, providing high-resolution, high-fidelity microscopic image reconstruction performance, which can improve the imaging time history and imaging speed by up to 30 times and imaging speed
by up to 10 times compared with traditional methods 。 With the help of rDL imaging technology, the research team carried out many super-resolution in vivo imaging experiments that could not be carried out by previous imaging methods, and discussed its potential biological significance with Lippincott-Schwartz, Zhu Xueliang, researcher of the Center for Excellence and Innovation in Molecular Cell Science, Chinese Academy of Sciences, and He Kangmin, researcher of the Institute of Genetics and Developmental Biology of the Chinese Academy of Sciences, including: two-color, long-term (more than 1 hour), super-resolution ( 97nm resolution), which clearly and realistically records the dynamics of cell adhesion and migration without interfering with this long, fragile life process; Continuous super-resolution observation of high-speed oscillating cilia at the current fastest imaging rate of 684Hz for up to 60,000 time points without obvious photobleaching or cell activity damage in the process, and statistical analysis of cilia swing mode and frequency.
Ultrafast and super-resolution two-color imaging of swinging cilia and intracilia transporter (IFT) revealed a variety of new behaviors such as collision, recombination, and U-turn of IFT during travel.
Through the two-color, long-duration and super-resolution imaging of cCAS-DNA and ER, the directional movement, steering or diffusion of cGAS-DNA in the process of maintaining continuous contact with ER were observed, and the understanding of the interaction mechanism between membranous organelles and membraneless organelles was expanded.
The three-dimensional super-resolution in vivo imaging of nucleolar phosphoprotein (NPM1), RNA polymerase I subunit RPA49 and chromatin (H2B) during HeLa cell division was carried out over a long time course (12 seconds acquisition interval, more than 2.
5 hours), which realized the three-dimensional super-resolution in vivo continuous observation of the structural morphological changes of NPM1 and RPA49 during intact mitosis, and revealed the nucleolar formation and NPM1, Phase transition and interaction of RPA49 two membraneless subcellular structures; The Golgi apparatus was continuously photographed at a whole-cell body imaging frame rate of 10Hz for up to 10,000 time points, and the three-color, high-speed (on the order of seconds), and ultra-long time (on the order of hours, > 1000 time points) of subcellular structures such as endoplasmic reticulum, lysosomes, and mitochondria in the whole cell division process were realized, and the uniform distribution mechanism
of organelles in progeny cells during cell mitosis was explored.

Through the cross-innovation of artificial intelligence algorithm and optical microscopy imaging technology, the Li Dong/Dai Qionghai collaborative team proposed a rationalized deep learning super-resolution microscopy imaging framework, which solved the problems of resolution loss, prediction uncertainty, and difficult acquisition of training sets of existing deep learning imaging methods, which can provide more than 30 times the imaging speed and time course improvement for a variety of in vivo microscopy imaging modalities, and provide important research tools
for the development of cell biology, developmental biology, neuroscience and other fields 。 At the same time, the research idea of cross-innovation and combination of software and hardware of artificial intelligence algorithm and optical imaging principle adhered to and advocated by the research team has opened up a new technical path
for the development of modern optical microscopy.

His research work was supported
by the National Natural Science Foundation of China, the Ministry of Science and Technology, the Chinese Academy of Sciences, the China Postdoctoral Science Foundation, the Tencent Science Exploration Award, and the Shuimu Scholar Program of Tsinghua University.

Figure 1.
Rationalize the architecture of deep learning super-resolution microscopy imaging neural network

Figure 2.
Overview of the application of rationalized deep learning super-resolution microscopy imaging methods