CN108765283B - Device and method for processing super-resolution positioning microscopic imaging mixed density data - Google Patents
Device and method for processing super-resolution positioning microscopic imaging mixed density data Download PDFInfo
- Publication number
- CN108765283B CN108765283B CN201810410930.2A CN201810410930A CN108765283B CN 108765283 B CN108765283 B CN 108765283B CN 201810410930 A CN201810410930 A CN 201810410930A CN 108765283 B CN108765283 B CN 108765283B
- Authority
- CN
- China
- Prior art keywords
- sparse
- density
- board card
- original image
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 28
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000003384 imaging method Methods 0.000 title claims abstract description 18
- 239000000284 extract Substances 0.000 claims abstract description 7
- 230000008569 process Effects 0.000 claims abstract description 7
- 238000004891 communication Methods 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 8
- 238000002073 fluorescence micrograph Methods 0.000 claims description 7
- 230000009467 reduction Effects 0.000 claims description 5
- 238000003672 processing method Methods 0.000 claims description 3
- 238000000386 microscopy Methods 0.000 claims description 2
- 230000004807 localization Effects 0.000 abstract description 5
- 238000012800 visualization Methods 0.000 abstract description 3
- 238000013480 data collection Methods 0.000 abstract 1
- 239000012472 biological sample Substances 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000004020 luminiscence type Methods 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000007476 Maximum Likelihood Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10056—Microscopic image
- G06T2207/10061—Microscopic image from scanning electron microscope
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
- Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)
Abstract
The invention discloses a device and a method for processing super-resolution positioning microscopic imaging mixed density data, wherein the device comprises: the system comprises a data acquisition card, a PC, an FPGA board card and a GPU board card. The data collection card is used for collecting original image data detected by the camera and is directly connected with the PC through a PCIE slot of the mainboard; the method comprises the following steps that a PC extracts original image data in a data acquisition card through control software, transmits the original image data to an FPGA board card, and the FPGA board card judges and classifies sparse and high-density fluorescent molecules in the original image data and transmits sparse and high-density molecular subregions to the PC; the GPU board card in the PC is used for processing sparse molecules and high-density molecules by different positioning algorithms respectively. The biological fluorescent sample has large sparse molecular area, high positioning speed and large image quantity. While the high density molecular domains have relatively small mass and slow localization speed. And in the reconstruction process, the real-time processing and visualization of sparse/high-density mixed molecule super-resolution image reconstruction are realized.
Description
Technical Field
The invention belongs to the technical field of super-resolution positioning microscopic imaging, and particularly relates to a device and a method for processing mixed density data of super-resolution positioning microscopic imaging.
Background
The super-resolution positioning imaging technology can realize the spatial resolution of 20nm, can research the complex work mechanism in the cell from the molecular level, and becomes an indispensable research tool in the field of life science research. The data output speed of a fast-developing weak light detector (sCMOS) reaches nearly 1GB/s, and various sparse molecular positioning algorithms are available at present, the speed is relatively high, and the precision is different. The high-density molecular positioning algorithm has slow calculation speed due to its complex molecular model, and is a problem of much concern in the industry. For a common biological sample, in an original image of super-resolution positioning imaging, the situation that most signals are sparse molecular luminescence and a small part of signals are high-density molecular luminescence almost universally exists. Thus, if only a simple sparse molecular localization algorithm is used, it is inevitably necessary to discard the image of the high-density molecular luminescence portion, so that partial image deletion exists as a result. However, if the high-density molecular localization algorithm is used to process data, the data processing speed is reduced, and real-time data processing and visualization of the super-resolution image cannot be realized. Mixed density molecular localization algorithms need to be developed.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a device and a method for processing super-resolution positioning micro-imaging mixed density data. The method aims to solve the problem that the existing data processing method cannot give consideration to the ubiquitous uneven molecular density distribution and single molecular positioning algorithm in the biological sample.
As one aspect of the present invention, the present invention provides an apparatus for super-resolution positioning microscopy imaging mixed density data processing, comprising:
the system comprises a data acquisition card, a PC, an FPGA board card and a GPU board card; the data acquisition card is used for acquiring original image data detected by the camera and is directly connected with the PC through a PCIE slot of the mainboard; the method comprises the steps that a PC extracts original image data in a data acquisition card through control software, the original image data are transmitted to an FPGA board card, the FPGA board card judges and classifies sparse molecules and high-density fluorescent molecules in the original image data, the sparse and high-density molecular sub-regions are transmitted to the PC, and a GPU board card in the PC processes the sparse and high-density molecular sub-regions through different positioning algorithms.
Preferably, direct communication is performed between the storage area in the PC and the FPGA board card.
Preferably, the PC is provided with two memory areas, the first memory area is used for storing the sparse molecular sub-area, the second memory area is used for the high-density molecular sub-area, and the GPU board card respectively extracts and processes data in the first memory area and the second memory area.
Preferably, the PC includes a plurality of GPU boards, a part of the GPU boards are used for performing positioning processing on the sparse molecular sub-region, and the rest of the GPU boards are used for performing positioning processing on the high-density molecular sub-region.
Preferably, the PC and the FPGA board are connected through a PCIE interface.
As another aspect of the present invention, the present invention provides a method based on the above apparatus, where the FPGA board extracts sparse fluorescent molecules and high-density fluorescent molecules from the original image data according to the following steps:
step 1: carrying out noise reduction and background removal processing on original image data;
step 2: judging fluorescence points of the processed original image data to obtain a fluorescence image;
and step 3: and judging the fluorescence map by using a sparse fluorescence point judgment method, if the image area to be judged meets the sparse condition, extracting sparse subregions, and otherwise, extracting high-density subregions.
Preferably, the sparse subregion extracted subregion is smaller than the high-density subregion extracted subregion.
In general, compared with the prior art, the above technical solution contemplated by the present invention can achieve the following beneficial effects:
1. the FPGA receives an original fluorescence image transmitted by software developed based on a data acquisition card SDK in a PCIE interface at a computer end, and performs the work of noise reduction, background removal, fluorescence point judgment, sparse/high-density molecular screening, subregion extraction and the like of super-resolution positioning imaging. The sparse molecular subregion and the high-density molecular subregion are accessed separately, the GPU board card reads different storage units, knows whether the memory units are sparse or high-density molecular subregions according to head file signals of the different storage units, performs sparse/high-density hybrid positioning by the GPU board card, performs accelerated positioning imaging by adopting multi-core GPU board card pipeline operation, and thus adopts different molecular positioning algorithms.
Since the sparse molecular region has a relatively large distribution, even if the positioning speed is high, the number of molecules is large, and much time is still needed. And if the distribution quantity is not very large, real-time image processing can be realized by adopting PALMER or other high-density molecular positioning algorithms. Therefore, real-time processing and visualization of super-resolution images can be achieved with sparse/high density hybrid molecular localization.
2. Sending the original image to FPGA by PCIE interface
The invention ensures that the data processing of super-resolution positioning imaging is not limited by a camera interface any more, and no matter the camera is transmitted into the interface by USB, Cameralink or other data, the original image data of the acquisition card can be read by software developed based on the data acquisition card SDK without exchanging an FPGA interface mode for the whole system, and the original image data is transmitted to the FPGA through the transmitting end of a standard PCIE interface to perform data preprocessing work. The technical problem that different systems are equipped with different systems due to the fact that the camera interfaces are not uniform is solved.
3. DMA direct communication between FPGA board card and GPU board card
According to the invention, the FPGA board card end and the GPU board card of the PC are in direct DMA communication through the PCIE interface, and the CPU does not participate in instruction judgment and communication, so that the time is saved. And the sparse/high-density molecular subregions sent out by the FPGA board card are respectively transmitted to the storage units corresponding to the PC through the PCIE interfaces.
Drawings
FIG. 1 is a schematic structural diagram of an apparatus for super-resolution positioning micro-imaging mixed density data processing according to the present invention;
FIG. 2 is a schematic diagram of an FPGA provided by the present invention;
fig. 3 is a schematic diagram of reading and writing of direct communication among the FPGA board card, the DMA, and the FPGA board card provided by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
The invention provides a device and a method for processing super-resolution positioning micro-imaging mixed density data. The device aims to solve the problem that in the preparation process of a biological sample, when the distribution of a fluorophore is not uniform, the single algorithm is adopted to cause inaccurate judgment of fluorescence points or serious lag of calculation speed. And the whole set of data processing is independent of the camera interface, and can be Cameralink, USB or any other common interface type.
As shown in fig. 1, the apparatus provided by the present invention includes a data acquisition card, a PC, an FPGA board card, and a GPU board card; the PC comprises a memory, a storage, a mainboard and a CPU. The FPGA board card comprises an FPGA and FPGA peripheral circuits, the FPGA board card and the GPU board card are both inserted into a computer mainboard through PCIE interfaces, and a CPU gives instructions to control the use condition of a bus. The FPGA board card provides a bus needing to be used for the PC, the CPU responds, and an idle bus is provided for Direct communication between the FPGA board card and the GPU board card, namely DMA (Direct Memory Access), so that the efficiency is high and the speed is high. The camera is connected with the PC through the data acquisition card, the data acquisition card is directly connected with the PC through the PCIE slot of the mainboard, and the PC reads original image data information in the data acquisition card through software developed based on the data acquisition card SDK.
The method comprises the steps that original image data are transmitted to an FPGA through a PCIE interface, data preprocessing work of a super-resolution positioning imaging algorithm is carried out on the FPGA, and the data preprocessing work comprises noise reduction, background removal, sparse and high-density fluorescent point judgment and sparse and high-density fluorescent molecule sub-region extraction of the original image data.
And transmitting the sparse and high-density molecular subregion data to the PC through the PCIE interface, and respectively storing the sparse and high-density molecular subregions into a first storage region and a second storage region in the PC memory. And reading the data of different storage areas by the GPU card, and adopting different subsequent processing methods. Sparse molecular subregion data accelerates the data processing process through the GPU integrated circuit board, and real-time imaging can be performed even if the maximum likelihood estimation algorithm with high recognized precision at present is adopted. And the high-density molecular subregion data can be subjected to subsequent fluorescent spot positioning by adopting a high-density positioning algorithm, such as a PALMER algorithm. Although the high-density positioning algorithm is complex in calculation model and low in calculation speed, a balance exists for the situation that most data are sparse fluorescence points and few are high-density fluorescence points in a biological sample, so that the whole super-resolution positioning imaging can be completed in almost the same time regardless of sparse or high-density molecules.
As shown in fig. 2, the FPGA operates as follows:
step 1: carrying out noise reduction and background removal processing on original image data;
step 2: and extracting a sparse molecular subregion and a high-density molecular subregion in the processed original image data. The step 2 specifically comprises the following steps:
step 21: judging original image data by adopting a fluorescence point judgment method, if fluorescence points exist in a data area to be judged, extracting the data area to be judged, and if not, discarding the data area to be judged; further, a fluorescence image was obtained.
Step 22: the whole fluorescence image is judged by using a sparse fluorescence point judgment method, in the embodiment, the sparse fluorescence points are judged by adopting the following method, and other standards can be adopted to judge the sparse fluorescence points and the high-density fluorescence points according to the user requirements:
extracting a 3x3 sub-area corresponding to the current processing pixel position in the fluorescence image as a judged fluorescence signal, extracting the current pixel value of the background fluctuation intensity image as a threshold value, and then judging as follows:
(1) in the 3x3 sub-region, the centermost pixel is the maximum.
(2) In the 3x3 sub-region, the centermost pixel is greater than 2 times the threshold.
(3) In the 3x3 sub-region, the sum of the most central pixel and its 8 domain pixels is greater than 11 times the threshold value; the centermost pixel and its 4 domain pixel sum is greater than 9 times the threshold.
And when the region to be judged meets the 3 sparse conditions, the region is a sparse fluorescent spot, sparse subregion extraction is carried out, if the region to be judged does not meet the 3 sparse conditions, the region to be judged is a high-density fluorescent spot, high-density subregion extraction is carried out, the extracted subregions of the sparse subregion and the extracted subregions of the high-density subregion are different in size, and the subregions of different sizes can improve the data extraction precision.
Step 23: sub-region extraction
And (4) taking a sparse subregion extraction method as a judgment standard, and entering a sparse subregion storage unit when the fluorescent points are judged to be sparse. Otherwise, the high density is judged and sent to the high density sub-area memory unit.
As shown in fig. 3, a schematic diagram of reading and writing of direct communication among the FPGA board, the DMA, and the FPGA board. The direct communication includes the steps of:
step 1: and loading the driver in the DMA, establishing a PC device and an FPGA device, and initializing and recording hardware resources. And the PC detects whether the firmware in the FPGA board card needs to be updated, and if so, the firmware of the FPGA board card is updated through the DMA.
Step 2: the PC sets image parameters, and the DMA sets a related parameter register, wherein the parameters comprise parameters such as pixel size, resolution ratio and the like.
And step 3: the PC sets file reading interruption or inquires whether a file reading instruction exists or not by a CPU program, and starts file reading operation and DMA after the file reading interruption or the file reading instruction exists. Initiating DMA includes setting a DMA address and size.
And 4, step 4: and the PC starts to read the file, and image processing data transmission is carried out between the FPGA board card and the PC memory.
And 5: and after the data transmission of the FPGA board card is finished, triggering interruption and informing the DMA. After receiving the message, the DMA informs the PC to start the next file reading request.
Step 6: and (5) repeating the steps 3 to 5 to realize multiple file reading operations between the PC memory and the FPGA.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (5)
1. An apparatus for super-resolution positioning microscopy imaging mixed density data processing, comprising: the system comprises a data acquisition card, a PC, an FPGA board card and a GPU board card, wherein the data acquisition card is used for acquiring original image data detected by a camera and is directly connected with the PC through a PCIE slot of a mainboard; the PC extracts original image data in the data acquisition card through control software and transmits the original image data to the FPGA board card through PCIE reading; the FPGA board card judges and classifies sparse molecules and high-density fluorescent molecules in original image data, the sparse and high-density molecular sub-regions are transmitted to a PC through PCIE writing, the GPU board card in the PC processes the sparse and high-density molecular sub-regions through different positioning algorithms, and the PC and the FPGA board card are connected through a PCIE interface.
2. The apparatus of claim 1, wherein direct communication is between a memory area in the PC and the FPGA card.
3. The device as claimed in claim 1 or 2, wherein two memory areas are provided in the PC, the first memory area is used for storing sparse sub-areas, the second memory area is used for high-density sub-areas, and the GPU card respectively extracts and processes data in the first memory area and the second memory area.
4. The device of claim 1 or 2, wherein the PC comprises a plurality of GPU cards, a part of the GPU cards are used for performing fluorescent molecule positioning processing on the sparse subarea by using a sparse single-molecule positioning algorithm, and the rest of the GPU cards are used for performing fluorescent molecule positioning processing on the high-density subarea by using a high-density positioning algorithm.
5. The data processing method based on the device of claim 1, characterized in that the FPGA board card extracts sparse fluorescent molecules and high-density fluorescent molecules in the original image data according to the following steps:
step 1: carrying out noise reduction and background removal processing on original image data;
step 2: judging fluorescence points of the processed original image data to obtain a fluorescence image;
and step 3: judging the fluorescence map by using a sparse fluorescence point judgment method, if the image area to be judged meets the sparse condition, extracting sparse subregions, otherwise, extracting high-density subregions;
the step 3 comprises the following steps:
extracting a 3x3 sub-area corresponding to the current processing pixel position in the fluorescence image as a judged fluorescence signal, extracting the current pixel value of the background fluctuation intensity image as a threshold value, and then judging as follows:
(1) in the 3x3 sub-region, the centermost pixel is the maximum;
(2) in the 3x3 sub-region, the centermost pixel is greater than 2 times the threshold;
(3) in the 3x3 sub-region, the sum of the most central pixel and its 8 domain pixels is greater than 11 times the threshold value; the sum of the most central pixel and its 4 domain pixels is greater than 9 times the threshold value;
and when the area to be judged meets the 3 sparse conditions, the area is a sparse fluorescent spot, sparse subregion extraction is carried out, if the area does not meet the 3 sparse conditions, the area is judged to be a high-density fluorescent spot in advance, high-density subregion extraction is carried out, and the extracted subregion of the sparse subregion is smaller than the extracted subregion of the high-density subregion.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810410930.2A CN108765283B (en) | 2018-05-02 | 2018-05-02 | Device and method for processing super-resolution positioning microscopic imaging mixed density data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810410930.2A CN108765283B (en) | 2018-05-02 | 2018-05-02 | Device and method for processing super-resolution positioning microscopic imaging mixed density data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108765283A CN108765283A (en) | 2018-11-06 |
| CN108765283B true CN108765283B (en) | 2020-11-24 |
Family
ID=64008808
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810410930.2A Active CN108765283B (en) | 2018-05-02 | 2018-05-02 | Device and method for processing super-resolution positioning microscopic imaging mixed density data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108765283B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102288589A (en) * | 2011-06-23 | 2011-12-21 | 深圳大学 | Method and device for positioning single fluorescent probe, and super-resolution imaging method and system |
| CN103592278A (en) * | 2013-11-21 | 2014-02-19 | 中国计量学院 | Random positioning super-resolution microscopy method and device based on fluorescence-emission kill mechanism |
| CN106897969A (en) * | 2017-02-20 | 2017-06-27 | 华中科技大学 | A kind of data processing equipment and method for super-resolution positioning micro-imaging |
-
2018
- 2018-05-02 CN CN201810410930.2A patent/CN108765283B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102288589A (en) * | 2011-06-23 | 2011-12-21 | 深圳大学 | Method and device for positioning single fluorescent probe, and super-resolution imaging method and system |
| CN103592278A (en) * | 2013-11-21 | 2014-02-19 | 中国计量学院 | Random positioning super-resolution microscopy method and device based on fluorescence-emission kill mechanism |
| CN106897969A (en) * | 2017-02-20 | 2017-06-27 | 华中科技大学 | A kind of data processing equipment and method for super-resolution positioning micro-imaging |
Non-Patent Citations (1)
| Title |
|---|
| 基于sCOMS相机的超分辨定位成像技术;桂丹 等;《中国激光》;20180228;第45卷(第2期);第2-4节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108765283A (en) | 2018-11-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3839754A1 (en) | Acceleration control system chip based on thresholding method, and robot | |
| CN107527317B (en) | Data transmission system based on image processing | |
| WO2012169088A1 (en) | Image processing apparatus, image processing method and image processing system | |
| CN110582071B (en) | In-vehicle device, information processing method, and recording medium | |
| CN108121982B (en) | Method and device for acquiring facial single image | |
| CN111027343B (en) | Bar code region positioning method and device | |
| CN109525836B (en) | Embedded multimode is survey device simultaneously based on FPGA | |
| CN110599453A (en) | Panel defect detection method and device based on image fusion and equipment terminal | |
| WO2018207261A1 (en) | Image analysis device | |
| CN110795955A (en) | System and method for rapidly identifying multiple bar codes | |
| JP5480149B2 (en) | System and method for supporting defect detection of inspection object | |
| CN108765283B (en) | Device and method for processing super-resolution positioning microscopic imaging mixed density data | |
| CN103838694B (en) | FPGA high-speed USB interface data reading method | |
| CN115495498B (en) | Data association method, system, electronic equipment and storage medium | |
| CN111050027A (en) | Lens distortion compensation method, device, equipment and storage medium | |
| CN111124959B (en) | Hardware unloading method for processing NVMe protocol management command | |
| CN109086231B (en) | Access method and device of input and output equipment | |
| EP3764238A1 (en) | Processing circuit, information processing apparatus, and information processing method | |
| CN111753722B (en) | Fingerprint identification method and device based on feature point type | |
| CN110264488B (en) | Binary image edge extraction device | |
| Fan et al. | Xilinx Zynq 7000 SoC based character feature extraction system for OCR inspection of integrated circuit chips | |
| CN114973394A (en) | Gesture motion recognition method and device, electronic equipment and computer storage medium | |
| CN111611843A (en) | Face detection preprocessing method, device, equipment and storage medium | |
| KR100615694B1 (en) | Control system for controlling plural function block | |
| CN115396634B (en) | Camera link collection system based on FPGA |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |