CN108038874A - Towards the real-time registration apparatus of scanning electron microscope image and method of sequence section - Google Patents

Towards the real-time registration apparatus of scanning electron microscope image and method of sequence section Download PDF

Info

Publication number
CN108038874A
CN108038874A CN201711248908.4A CN201711248908A CN108038874A CN 108038874 A CN108038874 A CN 108038874A CN 201711248908 A CN201711248908 A CN 201711248908A CN 108038874 A CN108038874 A CN 108038874A
Authority
CN
China
Prior art keywords
mrow
image
corresponding points
msubsup
msub
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201711248908.4A
Other languages
Chinese (zh)
Other versions
CN108038874B (en
Inventor
陈曦
韩华
李国庆
谢启伟
沈丽君
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhongke Guanwei Beijing Technology Co Ltd
Original Assignee
Institute of Automation of Chinese Academy of Science
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institute of Automation of Chinese Academy of Science filed Critical Institute of Automation of Chinese Academy of Science
Priority to CN201711248908.4A priority Critical patent/CN108038874B/en
Publication of CN108038874A publication Critical patent/CN108038874A/en
Application granted granted Critical
Publication of CN108038874B publication Critical patent/CN108038874B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
    • G06F9/38—Concurrent instruction execution, e.g. pipeline, look ahead
    • G06F9/3818—Decoding for concurrent execution
    • G06F9/3822—Parallel decoding, e.g. parallel decode units
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30016—Brain

Abstract

The present invention relates to scanning electron microscope image registration field, and in particular to a kind of real-time registration apparatus of scanning electron microscope image towards sequence section and method, in order to improve the real-time of sem image registration.The real-time registration apparatus of the present invention includes FPGA and calculation server；Calculation server includes CPU and GPU.FPGA obtains sequence section view data in real time for being directly connected to Electronic Speculum, and calculates the corresponding points between contiguous slices image, finally sends the corresponding points information between the view data obtained from Electronic Speculum and contiguous slices to calculation server；CPU in calculation server, optimization is adjusted to the corresponding points position matched in sequence section；GPU in calculation server, image deformation is carried out according to the corresponding points position after adjustment.The present invention can form the high accuracy to Electronic Speculum system high throughput view data, the long sequence registration ability of low delay, and meet the imaging of high throughput Electronic Speculum sequence section matches somebody with somebody quasi need in real time.

Description

Towards the real-time registration apparatus of scanning electron microscope image and method of sequence section
Technical field
The present invention relates to scanning electron microscope image registration field, and in particular to a kind of scanning electron microscope image towards sequence section Real-time registration apparatus and method.
Background technology
Brain connection collection of illustrative plates research from macroscopic view, Jie's sight and micro-scale by building nervous system structures, and and physiological function Uniformity understand the operation principle of brain, wherein micro-scale connection collection of illustrative plates is directed to obtaining neuron and cynapse etc. fine The connection network of structure.The blur-free imaging of synaptic structure is necessarily dependent upon observation method-electron microscope of nanoscale, so as to In the nervous process tie line that most faint (20~30 nanometers) are followed the trail of in intensive neuropilem.
The neuromechanism connection network of micro-scale is generally obtained by the scanning electron microscope image three-dimensional reconstruction of sequence section Its three-dimensional appearance.Can not fast and stable obtain the high-resolution three-dimension sem image storehouse of nerve fiber, it is a wide range of prominent to become foundation One of limitation bottleneck of horizontal neutral net is touched, but lacks effective total solution both at home and abroad.
At present, existing sequence section sem image registration Algorithm is all processed offline sequence section image, i.e., all Sequence section image carries out image registration work again after all gathering.It is so not only no to utilize what is consumed during Image Acquisition The magnanimity time, causes to obtain the increase of nerve fiber three-dimensional data required calculating time, and can not in real time see and adopt Collection finishes the registration result of data, to instruct follow-up collection process, avoids insignificant sequence section micro-image Collecting work.Therefore, no matter from functional requirement or calculate in view of the time, for big scale of construction nerve fiber sequence section Electronic Speculum For image, development high throughput real time sequence section sem image registration technique, there is very important theory significance and practicality Value.
The content of the invention
In order to solve the above problem of the prior art, the present invention proposes a kind of scanning electron microscope (SEM) photograph towards sequence section , can be complete in real time while nerve fiber sequence section sem image registration accuracy is ensured as real-time registration apparatus and method Work into registration.
An aspect of of the present present invention, proposes a kind of real-time registration apparatus of scanning electron microscope image towards sequence section, including： FPGA (Field-Programmable Gate Array, field programmable gate array) and calculation server；
The FPGA, is configured to：Receive the slice image data from scanning electron microscope, and extract current slice image with it is upper Corresponding points between one sectioning image, described between the current slice view data and the image and a upper sectioning image The positional information of corresponding points is sent to the calculation server；
The calculation server, including：CPU (Central Processing Unit, central processing unit) and GPU (Graphics Processing Unit, image processor)；
The CPU, is configured to：Often receive the institute between a width slice image data and the image and a upper sectioning image Corresponding dot position information is stated, just all positions for having received the corresponding points on image are once adjusted, are optimized The position of the corresponding points afterwards；
The GPU, is configured to：After CPU often completes the once adjustment to the corresponding points, the GPU is all in accordance with excellent All images that received are carried out a deformation, so as to complete to have received image to all by the position of the corresponding points after change It is once registering.
Preferably, " slice image data from scanning electron microscope is received, and extracts current slice image and a upper slice map Corresponding points as between ", specifically include：
View data is received line by line from the scanning electron microscope and is cached, and SIFT feature is extracted according to the data of caching, until Receive the complete current slice image and complete SIFT feature extraction；
The SIFT feature of the current slice image is matched with the SIFT feature of a upper sectioning image, Obtain the corresponding points between the current slice image and a upper sectioning image；
Wherein, the line number of caching image data, determines according to the Size of Neighborhood for calculating SIFT feature.
Preferably, " often receive described corresponding between a width slice image data and the image and a upper sectioning image Dot position information, just once adjusts all positions for having received the corresponding points on image, the institute after being optimized State the position of corresponding points ", be specially：
Shown method according to the following formula, calculates corresponding when energy function E (w) is minimum valueValue：
And then by all corresponding points received on imagePosition adjustment for optimization after position
Wherein,
I is the sequence number of sequence section image；L is the sequence number of the current slice image；
c1=N (i-1, i) represents the quantity of the corresponding points between the i-th -1 sectioning image and i-th of sectioning image；It is right In the 1st section, there is c1=0；
c2=N (i, i+1) represents the quantity of the corresponding points between i-th of sectioning image and i+1 sectioning image；It is right Cut into slices in current l-th, since the L+1 section does not obtain also, there is c2=0；
K and l is the sequence number of corresponding points described in sequence section image, and k ≠ l；
For the position coordinates of k-th of corresponding points in i-th of sectioning image；
For the position coordinates of l-th of corresponding points in i-th of sectioning image；
Respectively position coordinatesMotion vector；
α and β is constant.
Preferably, rapid solving is carried out to min (E (w)) using multi-grid method.
Preferably,
Deformation is carried out to image using Moving Least Squares method on GPU.
Preferably, " after CPU often completes the once adjustment to the corresponding points, the GPU is all in accordance with the institute after optimization The position of corresponding points is stated, a deformation is carried out to all images that received, so as to complete to have received once matching somebody with somebody for image to all It is accurate ", be specially：
Different point v is chosen on the i-th width imagei, and corresponding rigid transformation matrix is calculated as follows
And then calculate the position coordinates after deformationUntil point all on the i-th width sectioning image completes deformation；
I=1,2 ..., L are taken, a deformation is carried out as stated above to all images that received, so as to complete to institute Have and received the once registering of image；
Wherein,
viFor the position at preceding any point of deformation in i-th of sectioning image；
For viRigid transformation matrix to be calculated on point；For viPosition of the point after deformation；
It is corresponding pointsPosition after adjustment；
γ is constant；
Represent corresponding pointsBy rigid transformationPosition coordinates afterwards.
Preferably, each different point v of parallel computation on the GPUiCorresponding rigid transformation matrixAfter deformation Position
Preferably, the FPGA is connected with the scanning electron microscope and the calculation server respectively by gigabit networking.
Another aspect of the present invention, proposes a kind of scanning electron microscope image Real-time Registration towards sequence section, is based on The real-time registration apparatus of scanning electron microscope image recited above towards sequence section, comprises the following steps：
Step S10, receives the slice image data from scanning electron microscope, and extracts current slice image and a upper slice map Corresponding points as between；
Step S20, according to all images having been received by and the corresponding points of extraction, has received on image to all The position of the corresponding points is adjusted, the position of the corresponding points after being optimized；
All images that received according to the position of the corresponding points after optimization, are carried out deformation, so that complete by step S30 Paired all registrations for having received image；
Step S40, judges whether to have received last width sectioning image, otherwise, goes to step S10.
Preferably, the corresponding points between current slice image and a upper sectioning image are extracted described in step S10, specifically For：
Calculate the SIFT feature of the current slice image, and the upper sectioning image with being buffered in FPGA SIFT feature is matched, and obtains the corresponding points between a upper sectioning image.
Preferably, after step slo, before step S20, further include：
Step S15, removes the SIFT feature for the upper sectioning image being buffered in FPGA, by the current slice The SIFT feature of image is cached, and is used for next sectioning image Feature Points Matching.
Beneficial effects of the present invention：
The real-time registration apparatus of scanning electron microscope image proposed by the present invention towards sequence section and method, based on FPGA+ The Heterogeneous Computing technology of CPU/GPU, and application sequence sectioning image registration Algorithm.The device and method takes full advantage of FPGA's The parallel processing capability of flowing structure and GPU, can while nerve fiber sequence section sem image registration accuracy is ensured Meet the imaging of high throughput Electronic Speculum sequence section match somebody with somebody quasi need in real time, realizes the registering in collection of sectioning image, solve from Line registration causes the problem that 3-dimensional image seriously lags.
Brief description of the drawings
Fig. 1 is that the embodiment of registration apparatus in real time of the invention forms schematic diagram；
Fig. 2 is the image acquisition mode schematic diagram of scanning electron microscope；
Fig. 3 is the flow diagram of Real-time Registration embodiment of the present invention.
Embodiment
The preferred embodiment of the present invention described with reference to the accompanying drawings.It will be apparent to a skilled person that this A little embodiments are only used for explaining the technical principle of the present invention, it is not intended that limit the scope of the invention.
The real-time registration apparatus of scanning electron microscope image proposed by the present invention towards sequence section, as shown in Figure 1, key is Start with from hardware-accelerated link, each link of sequence section registration Algorithm is realized on different hardware platforms, builds and is based on The real-time registration arrangement of Heterogeneous Computing of FPGA+CPU/GPU, and application sequence sectioning image registration Algorithm, realize that Electronic Speculum side gathers Side is registering in real time.Scanning electron microscope is first imaged sequence section, and during slice imaging, Electronic Speculum passes through kilomega network View data is sent to FPGA by network in real time, and is completed on FPGA the corresponding points between the section and a upper sectioning image and carried Take, corresponding points extraction result and original image are sent to calculation server by gigabit networking again.Calculation server stores institute There are the corresponding points between imaging slice and its contiguous slices, the corresponding points of current slice are added to all imaging slices In corresponding points, the optimization of corresponding points position is carried out on the CPU of calculation server.According to corresponding points position optimization as a result, Deformation is carried out to current all images for having received section on the GPU of calculation server.
A kind of embodiment of the real-time registration apparatus of scanning electron microscope image towards sequence section of the present invention, including one piece FPGA processing boards and a calculation server.FPGA is connected by gigabit networking with scanning electron microscope, calculation server.FPGA is received View data from scanning electron microscope, sends calculation server to again after processing.Calculation server includes CPU and GPU etc. and calculates Unit.
FPGA is configured to：Receive the slice image data from scanning electron microscope, and extract current slice image with it is upper all Corresponding points between picture, by the corresponding points between current slice view data and the image and a upper sectioning image Positional information is sent to calculation server；
CPU is configured to：It is described right between a width slice image data and the image and a upper sectioning image often to receive Dot position information is answered, is just once adjusted all positions for having received the corresponding points on image, the correspondence after being optimized The position of point；
GPU is configured to：After CPU often completes the once adjustment to the corresponding points, after the GPU is all in accordance with optimization All images that received are carried out a deformation by the position of corresponding points, so as to complete to have received the once registering of image to all.
In the present embodiment, " receive the slice image data from scanning electron microscope, and extract current slice image with it is upper all Corresponding points between picture ", specifically include：
View data is received line by line from the scanning electron microscope and is cached, and SIFT feature is extracted according to the data of caching, until Receive the complete current slice image and complete SIFT feature extraction；
The SIFT feature of current slice image is matched with the SIFT feature of a upper sectioning image, is obtained current Corresponding points between sectioning image and a upper sectioning image；
Wherein, the line number of caching image data, determines according to the Size of Neighborhood for calculating SIFT feature.
Electronic Speculum often completes piece image the imaging of data line, is just sent to the row data in imaging process FPGA processing, without being retransmited when entire image imaging finishes.Since the imaging mode of scanning electron microscope is electron beam Electronic signal is inspired in sample surfaces, and the signal strength in electron beam scanning region is obtained using detector, finally line by line View data is obtained, as shown in Figure 2.This mode is very suitable for being handled using the flowing structure of FPGA.Using which Electron microscopic data is handled, without waiting for complete finishing image scanning, when can be substantially improved that registration is full-range in real time Between utilization rate.
The calculating of SIFT feature needs the neighborhood information of current position, thus only a line view data when, it is impossible to count SIFT feature is calculated, it is necessary to cache some row data in FPGA, then start to calculate, the line number of caching image data is by calculating SIFT The Size of Neighborhood of feature determines.
In the present embodiment, " often receive described between a width slice image data and the image and a upper sectioning image Corresponding dot position information, just once adjusts all positions for having received the corresponding points on image, after obtaining optimization The corresponding points position ", be specially：
According to the method shown in formula (1), calculate corresponding when energy function E (w) is minimum valueValue：
And then by all corresponding points received on sectioning imagePosition adjustment for optimization after position
Wherein,
I is the sequence number of sequence section image；L is the sequence number of current slice image；
c1=N (i-1, i) represents the quantity of corresponding points between the i-th -1 sectioning image and i-th of sectioning image；For 1 section, there is c1=0；
c2=N (i, i+1) represents the quantity of corresponding points between i-th of sectioning image and i+1 sectioning image；For working as Preceding l-th section, since the L+1 section does not obtain also, there is c2=0；
K and l is the sequence number of corresponding points in sequence section image, and k ≠ l；
For the position coordinates of k-th of corresponding points in i-th of sectioning image；
For the position coordinates of l-th of corresponding points in i-th of sectioning image；
Respectively position coordinatesMotion vector；
E (w) is the energy function of motion vector w；
α and β is constant.
Rapid solving is carried out to min (E (w)) using multi-grid method.
In the present embodiment, deformation is carried out to image using Moving Least Squares method on GPU.
Specifically, different point v is chosen on the i-th width imagei, and calculate corresponding rigid transformation matrix by formula (2)
And then calculate the position coordinates after deformationUntil point all on the i-th width sectioning image completes deformation；
I=1,2 ..., L are taken, a deformation is carried out as stated above to all images that received, so as to complete to institute Have and received the once registering of image；
Wherein,
viFor the position at preceding any point of deformation in i-th of sectioning image；For viRigid transformation square to be calculated on point Battle array；For viPosition of the point after deformation；It is corresponding pointsPosition after adjustment；γ For constant；Represent corresponding pointsBy rigid transformationPosition coordinates afterwards.
Due to every bit v in imageiRigid transformationCalculating all independently carry out, therefore can utilize GPU it is parallel Calculate, reduce total calculating time.
According to described above, cannot be corresponded to when the registration apparatus rigid connection of the present embodiment receives the 1st width image Point extraction, corresponding points adjustment, and image deformation；But since the 2nd width image, the new sectioning image of a width is often received, Extraction with regard to carrying out corresponding points for the width new images, and all images having been received by are subjected to a corresponding points adjustment respectively With an image deformation, this way, ensure that the image received from Electronic Speculum can carry out registration in real time, be shown as three-dimensional Image.
The embodiment of a kind of scanning electron microscope image Real-time Registration towards sequence section of the present invention, based on institute above The real-time registration apparatus of the scanning electron microscope image towards sequence section stated, as shown in figure 3, comprising the following steps：
Step S10, receives the slice image data from scanning electron microscope, and extracts current slice image and a upper slice map Corresponding points as between；
Step S20, according to all images having been received by and the corresponding points of extraction, has received on image to all The position of the corresponding points is adjusted, the position of the corresponding points after being optimized；
All images that received according to the position of the corresponding points after optimization, are carried out deformation, so that complete by step S30 Paired all registrations for having received image；
Step S40, judges whether to have received last width sectioning image, if so, then terminating registration task；Otherwise, turn To step S10.
The corresponding points between current slice image and a upper sectioning image, tool are extracted in the present embodiment, described in step S10 Body is：
Calculate the SIFT feature of the current slice image, and the upper sectioning image with being buffered in FPGA SIFT feature is matched, and obtains the corresponding points between a upper sectioning image.
In the present embodiment, after step slo, before step S20, further include：
Step S15, removes the SIFT feature for the upper sectioning image being buffered in FPGA, by the current slice The SIFT feature of image is cached, and is used for next sectioning image Feature Points Matching.
Those skilled in the art should be able to recognize that, each exemplary dress described with reference to the embodiments described herein Put and method and step, can be realized with electronic hardware, computer software or the combination of the two, in order to clearly demonstrate electronics The interchangeability of hardware and software, generally describes each exemplary composition and step according to function in the above description Suddenly.These functions are performed with electronic hardware or software mode actually, and the application-specific and design depending on technical solution are about Beam condition.Those skilled in the art can realize described function to each specific application using distinct methods, but It is this realization it is not considered that beyond the scope of this invention.
So far, the preferred embodiment shown in the drawings technical solution that the invention has been described, still, this area are had been combined Technical staff is it is easily understood that protection scope of the present invention is expressly not limited to these embodiments.Without departing from this On the premise of the principle of invention, those skilled in the art can make correlation technique feature equivalent change or replacement, these Technical solution after changing or replacing it is fallen within protection scope of the present invention.

Claims (11)

  1. A kind of 1. real-time registration apparatus of scanning electron microscope image towards sequence section, it is characterised in that including：FPGA and calculating take Business device；
    The FPGA, is configured to：Receive the slice image data from scanning electron microscope, and extract current slice image with it is upper all Corresponding points between picture, will be described corresponding between the current slice view data and the image and a upper sectioning image The positional information of point is sent to the calculation server；
    The calculation server, including：CPU and GPU；
    The CPU, is configured to：It is described right between a width slice image data and the image and a upper sectioning image often to receive Dot position information is answered, just all positions for having received the corresponding points on image are once adjusted, after being optimized The position of the corresponding points；
    The GPU, is configured to：After CPU often completes the once adjustment to the corresponding points, after the GPU is all in accordance with optimization The corresponding points position, to it is all received images carry out a deformation, so as to complete to have received the one of image to all Secondary registration.
  2. 2. real-time registration apparatus according to claim 1, it is characterised in that " receive the sectioning image from scanning electron microscope Data, and extract the corresponding points between current slice image and a upper sectioning image ", specifically include：
    View data is received line by line from the scanning electron microscope and is cached, SIFT feature is extracted according to the data of caching, until receiving To the complete current slice image and complete SIFT feature extraction；
    The SIFT feature of the current slice image is matched with the SIFT feature of a upper sectioning image, is obtained Corresponding points between the current slice image and a upper sectioning image；
    Wherein, the line number of caching image data, determines according to the Size of Neighborhood for calculating SIFT feature.
  3. 3. real-time registration apparatus according to claim 1, it is characterised in that " often receive a width slice image data with As soon as and the corresponding dot position information between the image and upper sectioning image, to all corresponding points received on image Position once adjusted, the position of the corresponding points after being optimized ", be specially：
    Shown method according to the following formula, calculates corresponding when energy function E (w) is minimum valueValue：
    <mfenced open = "" close = ""> <mtable> <mtr> <mtd> <mrow> <mi>min</mi> <mrow> <mo>(</mo> <mrow> <mi>E</mi> <mrow> <mo>(</mo> <mi>w</mi> <mo>)</mo> </mrow> </mrow> <mo>)</mo> </mrow> <mo>=</mo> <mi>min</mi> <mo>(</mo> <munderover> <mi>&amp;Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </munderover> <munderover> <mi>&amp;Sigma;</mi> <mrow> <mi>k</mi> <mo>=</mo> <mn>1</mn> </mrow> <msub> <mi>c</mi> <mn>1</mn> </msub> </munderover> <mo>|</mo> <mo>|</mo> <msubsup> <mi>p</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>+</mo> <msubsup> <mi>w</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>-</mo> <msubsup> <mi>p</mi> <mi>k</mi> <mrow> <mi>i</mi> <mo>+</mo> <mn>1</mn> </mrow> </msubsup> <mo>-</mo> <msubsup> <mi>w</mi> <mi>k</mi> <mrow> <mi>i</mi> <mo>+</mo> <mn>1</mn> </mrow> </msubsup> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <mo>+</mo> <mi>&amp;alpha;</mi> <munderover> <mi>&amp;Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>L</mi> </munderover> <munderover> <mi>&amp;Sigma;</mi> <mrow> <mi>k</mi> <mo>=</mo> <mn>1</mn> </mrow> <mrow> <msub> <mi>c</mi> <mn>1</mn> </msub> <mo>+</mo> <msub> <mi>c</mi> <mn>2</mn> </msub> </mrow> </munderover> <munderover> <munder> <mi>&amp;Sigma;</mi> <mrow> <mi>l</mi> <mo>=</mo> <mn>1</mn> </mrow> </munder> <mrow> <mi>l</mi> <mo>&amp;NotEqual;</mo> <mi>k</mi> </mrow> <mrow> <msub> <mi>c</mi> <mn>1</mn> </msub> <mo>+</mo> <msub> <mi>c</mi> <mn>2</mn> </msub> </mrow> </munderover> <mfrac> <mrow> <mo>|</mo> <mo>|</mo> <msubsup> <mi>w</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>-</mo> <msubsup> <mi>w</mi> <mi>l</mi> <mi>i</mi> </msubsup> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> </mrow> <mrow> <mo>|</mo> <mo>|</mo> <msubsup> <mi>p</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>-</mo> <msubsup> <mi>p</mi> <mi>l</mi> <mi>i</mi> </msubsup> <mo>|</mo> <msub> <mo>|</mo> <mn>2</mn> </msub> </mrow> </mfrac> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <mo>+</mo> <mi>&amp;beta;</mi> <munderover> <mi>&amp;Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>L</mi> </munderover> <munderover> <mi>&amp;Sigma;</mi> <mrow> <mi>k</mi> <mo>=</mo> <mn>1</mn> </mrow> <mrow> <msub> <mi>c</mi> <mn>1</mn> </msub> <mo>+</mo> <msub> <mi>c</mi> <mn>2</mn> </msub> </mrow> </munderover> <mo>|</mo> <mo>|</mo> <msubsup> <mi>w</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> <mo>)</mo> </mrow> </mtd> </mtr> </mtable> </mfenced>
    And then by all corresponding points received on sectioning imagePosition adjustment for optimization after positionI=1,2 ..., L, k=1,2 ..., c1+c2；
    Wherein,
    I is the sequence number of sequence section image；L is the sequence number of the current slice image；
    c1=N (i-1, i) represents the quantity of the corresponding points between the i-th -1 sectioning image and i-th of sectioning image；For the 1st A section, there is c1=0；
    c2=N (i, i+1) represents the quantity of the corresponding points between i-th of sectioning image and i+1 sectioning image；For working as Preceding l-th section, since the L+1 section does not obtain also, there is c2=0；
    K and l is the sequence number of corresponding points described in sequence section image, and k ≠ l；
    For the position coordinates of k-th of corresponding points in i-th of sectioning image；
    For the position coordinates of l-th of corresponding points in i-th of sectioning image；
    Respectively position coordinatesMotion vector；
    α and β is constant.
  4. 4. real-time registration apparatus according to claim 3, it is characterised in that using multi-grid method to min (E (w)) into Row rapid solving.
  5. 5. real-time registration apparatus according to claim 3, it is characterised in that Moving Least Squares are utilized on the GPU Method to carry out deformation to image.
  6. 6. real-time registration apparatus according to claim 5, it is characterised in that " often completed once to the corresponding points in CPU Adjustment after, the GPU all in accordance with the corresponding points after optimization position, to it is all received images carry out a shape Become, so as to complete to have received the once registering of image to all ", be specially：
    Different point v is chosen on the i-th width sectioning imagei, and corresponding rigid transformation matrix is calculated as follows
    <mrow> <mi>m</mi> <mi>i</mi> <mi>n</mi> <mrow> <mo>(</mo> <msub> <mi>l</mi> <msup> <mi>v</mi> <mi>i</mi> </msup> </msub> <mo>)</mo> </mrow> <mo>=</mo> <mi>m</mi> <mi>i</mi> <mi>n</mi> <mrow> <mo>(</mo> <munderover> <mi>&amp;Sigma;</mi> <mrow> <mi>k</mi> <mo>=</mo> <mn>1</mn> </mrow> <mrow> <msub> <mi>c</mi> <mn>1</mn> </msub> <mo>+</mo> <msub> <mi>c</mi> <mn>2</mn> </msub> </mrow> </munderover> <msubsup> <mi>w</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>|</mo> <mo>|</mo> <msub> <mi>l</mi> <msup> <mi>v</mi> <mi>i</mi> </msup> </msub> <mo>(</mo> <msubsup> <mi>p</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>)</mo> <mo>-</mo> <msubsup> <mi>q</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> <mo>)</mo> </mrow> </mrow>
    And then calculate the position coordinates after deformationUntil point all on the i-th width sectioning image completes deformation；
    Take i=1,2 ..., L, to it is all received images as stated above carry out a deformation so that complete to it is all Receive the once registering of image；
    Wherein,
    viFor the position at preceding any point of deformation in i-th of sectioning image；
    For viRigid transformation matrix to be calculated on point；For viPosition of the point after deformation；
    It is corresponding pointsPosition after adjustment；
    γ is constant；
    Represent corresponding pointsBy rigid transformationPosition coordinates afterwards.
  7. 7. real-time registration apparatus according to claim 6, it is characterised in that each difference of parallel computation on the GPU Point viCorresponding rigid transformation matrixWith the position after deformation
  8. 8. the real-time registration apparatus according to any one of claim 1-7, it is characterised in that the FPGA passes through kilomega network Network is connected with the scanning electron microscope and the calculation server respectively.
  9. 9. a kind of scanning electron microscope image Real-time Registration towards sequence section, it is characterised in that based in claim 1-8 Any one of them comprises the following steps towards the real-time registration apparatus of scanning electron microscope image of sequence section：
    Step S10, receives the slice image data from scanning electron microscope, and extract current slice image and a upper sectioning image it Between corresponding points；
    Step S20, according to all images having been received by and the corresponding points of extraction, to described in all received on image The position of corresponding points is adjusted, the position of the corresponding points after being optimized；
    All images that received according to the position of the corresponding points after optimization, are carried out deformation by step S30, so as to complete pair All registrations for having received image；
    Step S40, judges whether to have received last width sectioning image, otherwise, goes to step S10.
  10. 10. Real-time Registration according to claim 9, it is characterised in that current slice figure is extracted described in step S10 Picture and the corresponding points between a upper sectioning image, are specially：
    Calculate the SIFT feature of the current slice image, and the SIFT of the upper sectioning image with being buffered in FPGA Characteristic point is matched, and obtains the corresponding points between a upper sectioning image.
  11. 11. Real-time Registration according to claim 10, it is characterised in that after step slo, before step S20, Further include：
    Step S15, removes the SIFT feature for the upper sectioning image being buffered in FPGA, by the current slice image SIFT feature cached, for next sectioning image Feature Points Matching use.
CN201711248908.4A 2017-12-01 2017-12-01 Scanning electron microscope image real-time registration device and method for sequence slices Active CN108038874B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711248908.4A CN108038874B (en) 2017-12-01 2017-12-01 Scanning electron microscope image real-time registration device and method for sequence slices

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711248908.4A CN108038874B (en) 2017-12-01 2017-12-01 Scanning electron microscope image real-time registration device and method for sequence slices

Publications (2)

Publication Number Publication Date
CN108038874A true CN108038874A (en) 2018-05-15
CN108038874B CN108038874B (en) 2020-07-24

Family

ID=62095075

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711248908.4A Active CN108038874B (en) 2017-12-01 2017-12-01 Scanning electron microscope image real-time registration device and method for sequence slices

Country Status (1)

Country Link
CN (1) CN108038874B (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109242894A (en) * 2018-08-06 2019-01-18 广州视源电子科技股份有限公司 A kind of image alignment method and system based on Moving Least
CN110517242A (en) * 2019-08-23 2019-11-29 强联智创（北京）科技有限公司 A kind of aneurysmal analysis method and device
CN112184638A (en) * 2020-09-14 2021-01-05 南京市儿童医院 Automatic kidney biopsy electron microscope picture identification method based on deep learning-comprehensive model
CN112396608A (en) * 2020-11-30 2021-02-23 中国科学院自动化研究所 Biological tissue electron microscope image correction method, system and device based on X-ray image
CN112988395A (en) * 2021-04-20 2021-06-18 宁波兰茜生物科技有限公司 Pathological analysis method and device of extensible heterogeneous edge computing framework
CN114049252A (en) * 2021-09-27 2022-02-15 中国科学院自动化研究所 Scanning electron microscope three-dimensional image acquisition system and method for sequence slicing

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4053729A4 (en) 2020-09-23 2023-06-07 Changxin Memory Technologies, Inc. Chip product comparison method and apparatus, chip product modeling method and apparatus, and storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102920470A (en) * 2012-10-18 2013-02-13 苏州生物医学工程技术研究所 Medical imaging system and method with double-mode fusion
US20130147789A1 (en) * 2011-12-08 2013-06-13 Electronics & Telecommunications Research Institute Real-time three-dimensional real environment reconstruction apparatus and method
CN103856727A (en) * 2014-03-24 2014-06-11 北京工业大学 Multichannel real-time video splicing processing system
CN104616304A (en) * 2015-02-11 2015-05-13 南京理工大学 Self-adapting support weight stereo matching method based on field programmable gate array (FPGA)
CN105279762A (en) * 2015-11-20 2016-01-27 北京航空航天大学 An oral cavity soft and hard tissue CT sequence and three-dimensional grid model registration method
CN106708777A (en) * 2017-01-23 2017-05-24 张军 Multi-core heterogeneous CPU - CPU - FPGA architecture
CN107247681A (en) * 2017-05-27 2017-10-13 上海德衡数据科技有限公司 A kind of big data engine prototype based on multinuclear isomery CPU GPU FPGA
EP3249535A1 (en) * 2016-05-25 2017-11-29 Imagination Technologies Limited Assessing performance of a hardware design using formal verification

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130147789A1 (en) * 2011-12-08 2013-06-13 Electronics & Telecommunications Research Institute Real-time three-dimensional real environment reconstruction apparatus and method
CN102920470A (en) * 2012-10-18 2013-02-13 苏州生物医学工程技术研究所 Medical imaging system and method with double-mode fusion
CN103856727A (en) * 2014-03-24 2014-06-11 北京工业大学 Multichannel real-time video splicing processing system
CN104616304A (en) * 2015-02-11 2015-05-13 南京理工大学 Self-adapting support weight stereo matching method based on field programmable gate array (FPGA)
CN105279762A (en) * 2015-11-20 2016-01-27 北京航空航天大学 An oral cavity soft and hard tissue CT sequence and three-dimensional grid model registration method
EP3249535A1 (en) * 2016-05-25 2017-11-29 Imagination Technologies Limited Assessing performance of a hardware design using formal verification
CN106708777A (en) * 2017-01-23 2017-05-24 张军 Multi-core heterogeneous CPU - CPU - FPGA architecture
CN107247681A (en) * 2017-05-27 2017-10-13 上海德衡数据科技有限公司 A kind of big data engine prototype based on multinuclear isomery CPU GPU FPGA

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109242894A (en) * 2018-08-06 2019-01-18 广州视源电子科技股份有限公司 A kind of image alignment method and system based on Moving Least
CN109242894B (en) * 2018-08-06 2021-04-09 广州视源电子科技股份有限公司 Image alignment method and system based on mobile least square method
CN110517242A (en) * 2019-08-23 2019-11-29 强联智创（北京）科技有限公司 A kind of aneurysmal analysis method and device
CN112184638A (en) * 2020-09-14 2021-01-05 南京市儿童医院 Automatic kidney biopsy electron microscope picture identification method based on deep learning-comprehensive model
CN112184638B (en) * 2020-09-14 2024-02-06 南京市儿童医院 Automatic identification method for kidney biopsy electron microscope picture based on deep learning-comprehensive model
CN112396608A (en) * 2020-11-30 2021-02-23 中国科学院自动化研究所 Biological tissue electron microscope image correction method, system and device based on X-ray image
CN112396608B (en) * 2020-11-30 2021-05-04 中国科学院自动化研究所 Biological tissue electron microscope image correction method, system and device based on X-ray image
CN112988395A (en) * 2021-04-20 2021-06-18 宁波兰茜生物科技有限公司 Pathological analysis method and device of extensible heterogeneous edge computing framework
CN112988395B (en) * 2021-04-20 2021-08-24 宁波兰茜生物科技有限公司 Pathological analysis method and device of extensible heterogeneous edge computing framework
CN114049252A (en) * 2021-09-27 2022-02-15 中国科学院自动化研究所 Scanning electron microscope three-dimensional image acquisition system and method for sequence slicing

Also Published As

Publication number Publication date
CN108038874B (en) 2020-07-24

Similar Documents

Publication Publication Date Title
CN108038874A (en) Towards the real-time registration apparatus of scanning electron microscope image and method of sequence section
CN105447888B (en) A kind of UAV Maneuver object detection method judged based on effective target
WO2021000423A1 (en) Pig weight measurement method and apparatus
CN103778636B (en) A kind of feature construction method for non-reference picture quality appraisement
CN105806765A (en) Refined characterization method of micro CT scanning soil body space pore structure
WO2011155150A1 (en) Image processing apparatus, image processing method, and program
CN108491757A (en) Remote sensing image object detection method based on Analysis On Multi-scale Features study
CN109889326A (en) A kind of Internet of Things framework based on block chain and have verified that data prove common recognition method
CN105744256A (en) Three-dimensional image quality objective evaluation method based on graph-based visual saliency
CN104680538B (en) SAR image CFAR object detection methods based on super-pixel
CN106204447A (en) The super resolution ratio reconstruction method with convolutional neural networks is divided based on total variance
CN102075786A (en) Method for objectively evaluating image quality
CN109000557A (en) A kind of nuclear fuel rod pose automatic identifying method
CN106225681A (en) A kind of Longspan Bridge health status monitoring device
CN107909588A (en) Partition system under MRI cortex based on three-dimensional full convolutional neural networks
CN104463104B (en) A kind of stationary vehicle target rapid detection method and device
CN107315994A (en) Clustering algorithm based on Spectral Clustering space trackings
CN106683084B (en) It is a kind of based in the ranks as the TDI image deformation degree method for objectively evaluating of bias estimation
CN104144339B (en) A kind of matter based on Human Perception is fallen with reference to objective evaluation method for quality of stereo images
CN106291553B (en) A kind of discontinuous scattering center coordinate restoration methods based on ISAR image sequence
WO2022188030A1 (en) Crowd density estimation method, electronic device and storage medium
CN104182931B (en) Super resolution method and device
Fu et al. Screen content image quality assessment using Euclidean distance
CN104820972B (en) A kind of infrared image ME noise remove methods based on in-orbit statistic of classification
CN103886333B (en) Method for active spectral clustering of remote sensing images

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20211110

Address after: 101407 room 413, floor 4, building 2, yard 11, Xingke East Street, Yanqi Economic Development Zone, Huairou District, Beijing

Patentee after: Zhongke Guanwei (Beijing) Technology Co., Ltd

Address before: 100190 No. 95, Zhongguancun East Road, Haidian District, Beijing

Patentee before: Institute of automation, Chinese Academy of Sciences