CN107545567A - The method for registering and device of biological tissue's sequence section micro-image - Google Patents
The method for registering and device of biological tissue's sequence section micro-image Download PDFInfo
- Publication number
- CN107545567A CN107545567A CN201710638426.3A CN201710638426A CN107545567A CN 107545567 A CN107545567 A CN 107545567A CN 201710638426 A CN201710638426 A CN 201710638426A CN 107545567 A CN107545567 A CN 107545567A
- Authority
- CN
- China
- Prior art keywords
- image
- msubsup
- micro
- mrow
- corresponding points
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 238000004422 calculation algorithm Methods 0.000 claims description 17
- 238000003860 storage Methods 0.000 claims description 11
- 230000009466 transformation Effects 0.000 claims description 5
- 230000008520 organization Effects 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 claims description 3
- 238000001000 micrograph Methods 0.000 claims 1
- 238000003384 imaging method Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 10
- 230000008569 process Effects 0.000 description 10
- 239000000523 sample Substances 0.000 description 10
- 230000008859 change Effects 0.000 description 6
- 239000012472 biological sample Substances 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000009825 accumulation Methods 0.000 description 2
- 238000005520 cutting process Methods 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000001010 compromised effect Effects 0.000 description 1
- 229910003460 diamond Inorganic materials 0.000 description 1
- 239000010432 diamond Substances 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000001035 drying Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000010884 ion-beam technique Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005381 potential energy Methods 0.000 description 1
- 238000012876 topography Methods 0.000 description 1
Landscapes
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The present invention relates to image procossing and neuroscience field, and in particular to a kind of method for registering and device of biological tissue's sequence section micro-image.Aim to solve the problem that the problem of true and reliable corresponding points information can not be searched out between biological tissue's contiguous slices micro-image.For this purpose, a kind of method for registering of biological tissue's sequence section micro-image is proposed in the present invention, including：Obtain the corresponding points of each contiguous slices micro-image in biological tissue's sequence section micro-image；The position of the acquired each corresponding points of adjustment；According to the position of each corresponding points after adjustment, image deformation is carried out to biological tissue's sequence section micro-image.True and reliable corresponding points information between contiguous slices micro-image can be found by the present invention, so as to ensure that the precision of biological tissue's sequence section micro-image registration.
Description
Technical field
The present invention relates to image procossing and neuroscience field, and in particular to a kind of biological tissue's sequence section micro-image
Method for registering and device.
Background technology
At present, biological tissue's sequence section micro-image three-dimensional rebuilding method based on Electronic Speculum imaging technique mainly includes sequence
Row cross-sectional imaging method and sequence section imaging method.Wherein, the step of sequence cross-sectional imaging method mainly includes：By biological group
Tissue samples are fixed in electron microscopic sample chamber, cut or use the side of ion beam bombardment by diamond cutter after imaging is shown it
Formula removes the sample surfaces of ten nano thickness, and treated sample is imaged again, repeats said process until obtaining institute
There is the image of sample.The image sequence obtained by above-mentioned imaging method can be easy to be configured, but in imaging process
In can not keeping sample, and then can not to sample carry out secondary imaging.Sequence section imaging method is will by ultramicrotome
Biological tissue samples are cut into the section of tens nanometer thickness, and collect on conductive carrier, then are put into Electronic Speculum and are imaged.
This method, which can retain, cuts into slices and repeatedly utilizes, and by way of more Electronic Speculum parallel imagings image can be accelerated to obtain, phase
Than in sequence cross-sectional imaging method, sequence section imaging method is more applicable for the high accuracy reconstruction of large sample block.But sequence is cut
Piece imaging method due to occurring different degrees of tissue deformation in sample preparation, section, collection and imaging process, and fold,
The problems such as tear, it is therefore desirable to which these section micro-images are carried out three-dimensional reconstruction by increasingly complex registration Algorithm.
The three dimensional non-linear collocation method of sequence section imaging method mainly includes Scale invariant Operator Method and Block- matching
Method.Wherein, Scale invariant Operator Method is to select piece image as base layer image, forwardly and rearwardly registering adjacent successively
The image of layer, up to all images, registration finishes two-by-two.Although this method reduces the difficulty of 3-D view registration, but wherein
The final registration result of strong influence is understood in the accumulation and propagation of error, especially for the image away from datum layer, to meet
The continuity of image sequence registration result, its deformation are significantly larger than truth.
Another kind is block matching method, and its key step includes：First, it is assumed that the shape of biological sample is in contiguous slices figure
The smooth change as in, and the non-linear deformation suffered by each sectioning image be it is separate, it is unrelated with contiguous slices.So
Afterwards, each sectioning image is expressed as a triangle spring grid, grid vertex is found by Block- matching two-by-two and cut at other
Correspondence position in piece, the interior position relationship between section of section embody elasticity about by elastic potential energy virtual between grid vertex
Beam, global elastic restraint is added in registration process to minimize non-linear deformation.In order to ensure the validity of Block- matching result,
Too big change can not occur for picture material between this method requires contiguous slices, i.e., propose rigors to slice thickness.For cutting
The violent abnormal conditions of the biological sample change in shape such as fold, tear on piece be present, this method can not be also effectively treated.
Simultaneously as the microstructure of biological tissue samples does not possess the shape of rule, i.e., biological tissue samples are in difference
The content of image between section should differ, and simply have necessarily similar in the case where slice thickness is smaller
Property, therefore so either use Scale invariant operator or Block- matching, can not all be searched out between contiguous slices micro-image
True and reliable corresponding points information, so as to cause the registration accuracy of biological tissue's sequence section micro-image relatively low.
The content of the invention
It has been to solve not searching out between contiguous slices micro-image to solve above mentioned problem of the prior art
The technical problem of true and reliable corresponding points information, an aspect of of the present present invention, it is proposed that a kind of biological tissue's sequence section is micro-
The method for registering of image, including：
Obtain the corresponding points of each contiguous slices micro-image in biological tissue's sequence section micro-image；
The position of the acquired each corresponding points of adjustment；
According to the position of each corresponding points after the adjustment, image is carried out to biological tissue's sequence section micro-image
Deformation.
Preferably, " corresponding points for obtaining each contiguous slices micro-image in biological tissue's sequence section micro-image "
Corresponding points including obtaining each adjacent layer biological tissue sequence section micro-image using SIFT flow algorithms.
Preferably, " corresponding points that each adjacent tomographic image is obtained using SIFT flow algorithms " are specifically included：
Image change is carried out to the first section micro-image in two contiguous slices micro-images using SIFT flow algorithms
Shape, and by the second section micro-image in the first section micro-image and described two contiguous slices micro-images after deformation
Matched；
Grid is set respectively on the first section micro-image and the second section micro-image after the matching, and it is described
The position coordinates that grid is corresponded in first section micro-image and the second section micro-image is identical；
Choose corresponding with grid vertex in the first section micro-image after the deformation in the first section micro-image
Location point is as its corresponding points；And choose location point corresponding to grid vertex in the second section micro-image and corresponded to as it
Point.
Preferably, " positions of the acquired each corresponding points of adjustment " are included acquired in method adjustment shown according to the following formula
The position of each corresponding points：
Wherein, the i is the sequence number of section micro-image, and the k and l are respectively the sequence of corresponding points in section micro-image
Number, and l ≠ k, it is describedIt is described for the position coordinates of k-th of corresponding points in i-th of section micro-imageFor institute's rheme
Put coordinateMotion vector, the E (w) be motion vector w energy function, the α and β are respectively constant.
Preferably, " according to the position of each corresponding points after the adjustment, to biological tissue's sequence section micro-image
Carry out image deformation " include：Image deformation, tool are carried out to biological tissue's sequence section micro-image using Moving Least
Body includes：
Shown method obtains position of the specific trellis summit after image deformation described in section micro-image according to the following formula
Put coordinate：
All corresponding points after removing specific trellis summit in the section micro-image are obtained using bilinear interpolation to exist
Position coordinates after image deformation；
Wherein, cut into slices before the v is image deformation the position coordinates on specific trellis summit in micro-image, the k is
The sequence number of corresponding points, the p in section micro-imagekFor the position of k-th of corresponding points in micro-image of being cut into slices before image deformation
Put coordinate, the qkFor the position coordinates of k-th of corresponding points in micro-image of being cut into slices after image deformation, qk=pk+wk, the wk
For the motion vector of k-th of corresponding points in section micro-image, the λkFor weight, andThe α
For constant, the lvIt is the rigid transformation matrix at position coordinates v correspondence position points.
The second aspect of the present invention, it is proposed that a kind of registration apparatus of biological tissue's sequence section micro-image, it is described to match somebody with somebody
Standard apparatus includes：
Acquisition module, it is configured to obtain the correspondence of each contiguous slices micro-image in biological tissue's sequence section micro-image
Point；
Adjusting module, it is configured to adjust the position of acquired each corresponding points；
Deformation module, the position of each corresponding points after being configured to according to the adjustment, to biological tissue's sequence section
Micro-image carries out image deformation.
Preferably, the acquisition module also includes：
First module, it is configured to aobvious to the first section in two contiguous slices micro-images using SIFT flow algorithms
Micro- image carries out anamorphose, and by the first section micro-image and described two contiguous slices micro-images after deformation
Second section micro-image is matched；
Second unit, it is configured on the first section micro-image after the matching and the second section micro-image respectively
Grid is set, and the position coordinates that grid is corresponded in the first section micro-image and the second section micro-image is identical；
Third unit, it is configured to choose in the first section micro-image in micro-image of being cut into slices with first after the deformation
Location point corresponding to grid vertex is as its corresponding points；And choose position corresponding to grid vertex in the second section micro-image
Put a conduct its corresponding points.
Preferably, adjusting module includes adjustment unit, and the adjustment unit is configured to method adjustment shown according to the following formula
The position for each corresponding points that the acquisition module obtains：
Wherein, the i is the sequence number of section micro-image, and the k and l are respectively the sequence of corresponding points in section micro-image
Number, and l ≠ k, it is describedIt is described for the position coordinates of k-th of corresponding points in i-th of section micro-imageFor institute's rheme
Put coordinateMotion vector, the E (w) be motion vector w energy function, the α and β are respectively constant.
The third aspect of the present invention, it is proposed that a kind of storage device, wherein a plurality of program is stored with, suitable for by processor
Load and perform to realize the method for registering of above-mentioned biological tissue's sequence section micro-image.
The fourth aspect of the present invention, it is proposed that a kind of processing unit, including
Processor, it is adapted for carrying out each bar program；And
Storage device, suitable for storing a plurality of program；
Described program is suitable to be loaded by processor and performed to realize above-mentioned biological tissue's sequence section micro-image
Method for registering.
Compared with immediate prior art, above-mentioned technical proposal at least has the advantages that：
1st, the method for registering of biological tissue's sequence section micro-image includes in the present invention：Obtain biological tissue's sequence section
The corresponding points of each contiguous slices micro-image in micro-image；The position of the acquired each corresponding points of adjustment；According to the adjustment
The position of each corresponding points afterwards, image deformation is carried out to biological tissue's sequence section micro-image.This method is aobvious in section
When tissue deformation, fold and tear problem occurs in micro- image, it can also find between each contiguous slices micro-image and truly may be used
The corresponding points information leaned on, avoid the accumulation of error in registration process and propagate the influence to registration result, so as to improve biology
The registering precision of organization order's section micro-image.
2nd, compared to more traditional Scale invariant operator, each adjacent layer biological tissue sequence is obtained using SIFT flow algorithms
The corresponding points of section micro-image, make use of characteristic point dense in image, avoid due to many phases in biological tissue be present
The error hiding like caused by structure, also cause the more robust when handling the larger three-dimensional scenic matching of deformation.
3rd, the corresponding points of each adjacent layer biological tissue sequence section micro-image are obtained by using SIFT flow algorithms,
And two grids with same position of setting in the section micro-image matched, the summit of grid is only chosen as corresponding
Point, on the one hand reducing amount of calculation, still further aspect can be evenly distributed in image using the corresponding points of this method extraction,
Regional area is avoided to lack corresponding points and cause registration error larger.
4th, adjustment corresponding points position during, using minimize energy function come calculate the displacement of corresponding points to
Amount, can make the motion vector of corresponding points small as far as possible and similar, so as to reduce the non-linear deformation of section micro-image.
5th, rigid transformation carried out by least square method during carrying out image deformation to section micro-image so that
The rigidity and yardstick of topography are maintained, so that biological sample can keep its local structure as far as possible
Degree of rigidity.
Brief description of the drawings
Fig. 1 is that the key step flow of the method for registering of biological tissue's sequence section micro-image in the embodiment of the present invention is shown
It is intended to；
Fig. 2 is the primary structure signal of the registration apparatus of biological tissue's sequence section micro-image in the embodiment of the present invention
Figure.
Embodiment
The preferred embodiment of the present invention described with reference to the accompanying drawings.It will be apparent to a skilled person that this
A little embodiments are used only for explaining the technical principle of the present invention, it is not intended that limit the scope of the invention.
Below in conjunction with the accompanying drawings, the method for registering images of biological tissue's sequence section micro-image in the embodiment of the present invention is entered
Row explanation.
As shown in figure 1, in the present embodiment biological tissue's sequence section micro-image method for registering include step S100,
S200 and S300.
Step S100, obtain the corresponding points of each contiguous slices micro-image in biological tissue's sequence section micro-image.Tool
Body, SIFT flow algorithms can be used to obtain pair of each adjacent layer biological tissue sequence section micro-image in the present embodiment
Ying Dian, specifically include：
Step S101, using SIFT flow algorithms to the first section micro-image in two contiguous slices micro-images
Anamorphose is carried out, and the first section micro-image after deformation is cut with second in described two contiguous slices micro-images
Piece micro-image is matched；
Step S102, after the matching first section micro-image and second section micro-image on net is set respectively
Lattice, and the position coordinates that grid is corresponded in the first section micro-image and the second section micro-image is identical；
Step S103, choose first section micro-image in after the deformation first section micro-image in grid top
Location point is as its corresponding points corresponding to point；And choose location point corresponding to grid vertex in the second section micro-image and make
For its corresponding points.
For biological tissue's sequence section micro-image, reliable corresponding points are found on contiguous slices micro-image
It is extremely difficult.This is due to that the structure of biological sample is continually changing, and different patterns is embodied in different sections
Feature.When the thinner thickness of section, certain similitude between contiguous slices be present, and similarity degree depends on the part of sample
Structure change feature and the thickness of section.When slice thickness is relatively thin, the similarity degree increase between contiguous slices, be advantageous to image
Registration, but higher requirement is proposed to sample making course simultaneously, fold, damage etc. are also easily produced during section is collected and is asked
Topic.And when cutting into slices thicker, find that reliable corresponding points are relatively difficult between contiguous slices, because traditional Scale invariant
Operator or block matching algorithm are had a great influence by the similarity degree of matching area.In addition, many similar knots in biological tissue be present
Structure, if merely with characteristic point sparse in image, it is easy to cause error hiding.
Compared to more traditional Scale invariant operator, SIFT-flow methods are being located as a kind of dense Feature Correspondence Algorithm
More robust when managing the larger three-dimensional scenic matching of deformation.Therefore, we are extracted using SIFT-flow methods on contiguous slices
Corresponding points.
Although all points can serve as corresponding points on the image matched, consider that the calculating of subsequent algorithm is born
Load, only corresponding points of the selected part point as contiguous slices micro-image.Setting two on the sectioning image matched has
The grid of same position, corresponding points are taken as the summit of grid, and the position of grid vertex can utilize SIFT-flow to calculate before deformation
Obtained Deformation Field is back-calculated to obtain.The density of grid has a great impact to amount of calculation, and density is bigger, and involved corresponding points are got over
More, registration accuracy is higher, but amount of calculation is bigger.The corresponding points extracted using this method can be evenly distributed in image, kept away
Exempt from regional area causes registration error larger due to lacking corresponding points.
Step S200, adjust the position of acquired each corresponding points.
Specifically, each correspondence that can according to the following formula acquired in the method set-up procedure S100 shown in (1) in the present embodiment
The position of point：
Each meaning of parameters is in formula (1)：I is the sequence number of section micro-image, and k and l are respectively in section micro-image
The sequence number of corresponding points, and l ≠ k,For i-th section micro-image in k-th of corresponding points position coordinates,To be described
Position coordinatesMotion vector, the E (w) be motion vector w energy function, α and β are respectively constant.
Wherein, Section 1It is data item, limits contiguous slices micro-image
Corresponding points have consistent coordinate, i.e. corresponding points have identical x-y after the adjustment；Section 2It is smooth item, limiting adjacent corresponding points has similar motion vector；Section 3Limit
Determine motion vectorSmall as far as possible, α is used for controlling the proportion of smooth item in energy function E (w), and β is used for controlling energy function E
(w) proportion of motion vector item is limited in.
The corresponding points position that adjacent section micro-image extracts to obtain two-by-two needs to be adjusted so that these corresponding points
Position is consistent.Although various deformation in a slice be present, it is true that most sectioning image still reflects biological tissue
Real structure.It can make the motion vector of corresponding points as far as possible by minimizing energy function to calculate the motion vector of corresponding points
It is small and similar, with reduce section non-linear deformation.
Step S300, according to the position of each corresponding points after adjustment, figure is carried out to biological tissue's sequence section micro-image
As deformation.
Specifically, Moving Least can be used to carry out biological tissue's sequence section micro-image in the present embodiment
Image deformation, specifically include：
The method shown in (2) obtains specific trellis summit described in section micro-image after image deformation according to the following formula
Position coordinates：
All corresponding points after removing specific trellis summit in the section micro-image are obtained using bilinear interpolation to exist
Position coordinates after image deformation；
Each meaning of parameters is in formula (2)：V be image deformation before cut into slices micro-image in specific trellis summit position
Coordinate, k be section micro-image in corresponding points sequence number, pkFor k-th of corresponding points in micro-image of being cut into slices before image deformation
Position coordinates, qkFor the position coordinates of k-th of corresponding points in micro-image of being cut into slices after image deformation, qk=pk+wk, wkTo cut
The motion vector of k-th of corresponding points, λ in piece micro-imagekFor weight, andα is constant, lvIt is position
Put the rigid transformation matrix at coordinate v correspondence position points.
It is global smooth by the deformation results obtained by Moving Least Squares method, and passes through rigid transformation, office
The rigidity and yardstick of portion's image are maintained, so that biological sample can keep the firm of its partial structurtes as far as possible
Property.The method based on Moving Least progress anamorphose applied in this step is not to act on deformation function often
One pixel, but image gridding deformation function will be applied to grid vertex afterwards, two-wire is used to other pixels
The method of property difference enters line translation, is compromised between registration accuracy and amount of calculation.
The embodiment of method for registering based on above-mentioned biological tissue's sequence section micro-image, present invention also offers one kind
The registration apparatus of biological tissue's sequence section micro-image.Below in conjunction with the accompanying drawings, to biological tissue's sequence in the embodiment of the present invention
The registration apparatus of section micro-image illustrates.
As shown in Fig. 2 the registration apparatus of biological tissue's sequence section micro-image includes acquisition module, adjusted in the present embodiment
Mould preparation block and deformation module.Wherein, acquisition module is configurable to each adjacent in acquisition biological tissue sequence section micro-image
The corresponding points of section micro-image.Adjusting module is configurable to adjust the position of acquired each corresponding points.Deformation module can
To be configured to the position according to each corresponding points after adjustment, image deformation is carried out to biological tissue's sequence section micro-image.
Further, acquisition module can also include first module, second unit and third unit in the present embodiment.Its
In, first module is configurable to micro- to the first section in two contiguous slices micro-images using SIFT flow algorithms
Image carries out anamorphose, and by the in the first section micro-image and described two contiguous slices micro-images after deformation
Two section micro-images are matched.Second unit is configurable to the first section micro-image after matching and the second section
Grid is set respectively on micro-image, and the position that grid is corresponded in the first section micro-image and the second section micro-image is sat
Mark identical.Third unit is configurable to choose in the first section micro-image and net in the first section micro-image after deformation
The location point of lattice vertex correspondence is as its corresponding points；And choose position corresponding to grid vertex in the second section micro-image
Point is used as its corresponding points.
Further, adjusting module can also include adjustment unit in the present embodiment, and the adjustment unit can be according to public affairs
The position for each corresponding points that method adjustment shown in formula (1) obtains.
The embodiment of the method for registering of above-mentioned biological tissue's sequence section micro-image, its technical principle, the skill solved
Art problem and caused technique effect are similar, and person of ordinary skill in the field can be understood that, for the side of description
Just and succinctly, the specific work process of the registration apparatus of biological tissue's sequence section micro-image of foregoing description and speak on somebody's behalf
It is bright, the method for registering of aforementioned biological organization order section micro-image is may be referred to, will not be repeated here.
It will be understood by those skilled in the art that the method for registering of above-mentioned biological tissue's sequence section micro-image also includes one
Other a little known features, such as processor, controller, memory etc., wherein, memory include but is not limited to random access memory,
It is flash memory, read-only storage, programmable read only memory, volatile memory, nonvolatile memory, serial storage, parallel
Memory or register etc., processor include but is not limited to CPLD/FPGA, DSP, arm processor, MIPS processors etc., in order to
Embodiment of the disclosure is unnecessarily obscured, these known structures are not shown.
It will be understood by those skilled in the art that the module in the device in embodiment can adaptively be changed
And they are arranged in one or more devices different from the embodiment.Can the module in embodiment or unit or
Component is combined into a module or unit or component, and can be divided into multiple submodule or subelement or subgroup in addition
Part.In addition at least some in such feature and/or process or unit exclude each other, any combinations can be used
To all features disclosed in this specification (including adjoint claim, summary and accompanying drawing) and such disclosed any side
All processes or unit of method or equipment are combined.Unless expressly stated otherwise, this specification (including adjoint right will
Ask, make a summary and accompanying drawing) disclosed in each feature can be replaced by the alternative features for providing identical, equivalent or similar purpose.
The embodiment of method for registering based on above-mentioned biological tissue's sequence section micro-image, present invention also offers one kind
Storage device.A plurality of program is stored with the present embodiment in storage device, the program be applied to loaded by processor and performed with
Realize the method for registering of above-mentioned biological tissue's sequence section micro-image.
The embodiment of method for registering based on above-mentioned biological tissue's sequence section micro-image, present invention also offers one kind
Processing unit.Processing unit can include processor and storage device in the present embodiment.Wherein, processor is adapted for carrying out each bar journey
Sequence, storage device is suitable to store a plurality of program, and these programs are suitable to be loaded by processor and performed to realize above-mentioned biology
The method for registering of organization order's section micro-image.
Person of ordinary skill in the field can be understood that, for convenience and simplicity of description, foregoing description
Storage device, the specific work process of processing unit and relevant explanation, may be referred to the corresponding process in aforementioned system embodiment,
It will not be repeated here.
The all parts embodiment of the present invention can be realized with hardware, or to be run on one or more processor
Software module realize, or realized with combinations thereof.It will be understood by those of skill in the art that it can use in practice
Microprocessor or digital signal processor (DSP) realize some in server according to embodiments of the present invention, client
Or some or all functions of whole parts.The present invention be also implemented as perform method as described herein one
Partly or completely equipment or program of device (for example, PC programs and PC program products).Such journey for realizing the present invention
Sequence can be stored on PC computer-readable recording mediums, or can have the form of one or more signal.Such signal can be from
Download and obtain on internet website, either provide on carrier signal or provided in the form of any other.
In addition, it will be appreciated by those of skill in the art that although some embodiments described herein include other embodiments
In included some features rather than further feature, but the combination of the feature of different embodiments means in of the invention
Within the scope of and form different embodiments.For example, in claims of the present invention, embodiment claimed
It is one of any mode to use in any combination.
It should be noted that the present invention will be described rather than limits the invention for above-described embodiment, and ability
Field technique personnel can design alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference symbol between bracket should not be configured to limitations on claims.Word "comprising" does not exclude the presence of not
Element or step listed in the claims.Word "a" or "an" before element does not exclude the presence of multiple such
Element.The present invention can be realized by means of including the hardware of some different elements and by means of properly programmed PC.
If in the unit claim for listing equipment for drying, several in these devices can be come specific by same hardware branch
Embody.The use of word first, second, and third does not indicate that any order.These words can be construed to title.
So far, combined preferred embodiment shown in the drawings describes technical scheme, still, this area
Technical staff is it is easily understood that protection scope of the present invention is expressly not limited to these embodiments.Without departing from this
On the premise of the principle of invention, those skilled in the art can make equivalent change or replacement to correlation technique feature, these
Technical scheme after changing or replacing it is fallen within protection scope of the present invention.
Claims (10)
1. a kind of method for registering of biological tissue's sequence section micro-image, it is characterised in that the method for registering includes：
Obtain the corresponding points of each contiguous slices micro-image in biological tissue's sequence section micro-image；
The position of the acquired each corresponding points of adjustment；
According to the position of each corresponding points after the adjustment, image deformation is carried out to the section micro-image.
2. method for registering according to claim 1, it is characterised in that
" corresponding points for obtaining each contiguous slices micro-image in biological tissue's sequence section micro-image " include using
SIFT flow algorithms obtain the corresponding points of each adjacent layer biological tissue sequence section micro-image.
3. method for registering according to claim 2, it is characterised in that
" corresponding points that each adjacent tomographic image is obtained using SIFT flow algorithms " are specifically included：
Anamorphose is carried out to the first section micro-image in two contiguous slices micro-images using SIFT flow algorithms,
And the first section micro-image after deformation and the second section micro-image in described two contiguous slices micro-images are entered
Row matching；
Grid, and described first are set respectively on the first section micro-image and the second section micro-image after the matching
The position coordinates that grid is corresponded in section micro-image and the second section micro-image is identical；
Choose position corresponding with grid vertex in the first section micro-image after the deformation in the first section micro-image
Point is used as its corresponding points；And choose second section micro-image in location point corresponding to grid vertex as its corresponding points.
4. according to the method for registering described in any one of claims 1 to 3, it is characterised in that
" positions of the acquired each corresponding points of adjustment " include the acquired each corresponding points of method adjustment shown according to the following formula
Position：
<mrow>
<mi>m</mi>
<mi>i</mi>
<mi>n</mi>
<mrow>
<mo>(</mo>
<mi>E</mi>
<mo>(</mo>
<mi>w</mi>
<mo>)</mo>
<mo>)</mo>
</mrow>
<mo>=</mo>
<mfenced open = "(" close = ")">
<mtable>
<mtr>
<mtd>
<mstyle>
<munder>
<mo>&Sigma;</mo>
<mi>i</mi>
</munder>
</mstyle>
<mstyle>
<munder>
<mo>&Sigma;</mo>
<mi>k</mi>
</munder>
</mstyle>
<mo>|</mo>
<mo>|</mo>
<msubsup>
<mi>p</mi>
<mi>k</mi>
<mi>i</mi>
</msubsup>
<mo>+</mo>
<msubsup>
<mi>w</mi>
<mi>k</mi>
<mi>i</mi>
</msubsup>
<mo>-</mo>
<msubsup>
<mi>p</mi>
<mi>k</mi>
<mrow>
<mi>i</mi>
<mo>+</mo>
<mn>1</mn>
</mrow>
</msubsup>
<mo>-</mo>
<msubsup>
<mi>w</mi>
<mi>k</mi>
<mrow>
<mi>i</mi>
<mo>+</mo>
<mn>1</mn>
</mrow>
</msubsup>
<mo>|</mo>
<msubsup>
<mo>|</mo>
<mn>2</mn>
<mn>2</mn>
</msubsup>
</mtd>
</mtr>
<mtr>
<mtd>
<mrow>
<mo>+</mo>
<mi>&alpha;</mi>
<munder>
<mo>&Sigma;</mo>
<mi>i</mi>
</munder>
<munder>
<mo>&Sigma;</mo>
<mi>k</mi>
</munder>
<munderover>
<mo>&Sigma;</mo>
<mi>l</mi>
<mrow>
<mi>l</mi>
<mo>&NotEqual;</mo>
<mi>k</mi>
</mrow>
</munderover>
<mfrac>
<mrow>
<mo>|</mo>
<mo>|</mo>
<msubsup>
<mi>w</mi>
<mi>k</mi>
<mi>i</mi>
</msubsup>
<mo>-</mo>
<msubsup>
<mi>w</mi>
<mi>l</mi>
<mi>i</mi>
</msubsup>
<mo>|</mo>
<msubsup>
<mo>|</mo>
<mn>2</mn>
<mn>2</mn>
</msubsup>
</mrow>
<mrow>
<mo>|</mo>
<mo>|</mo>
<msubsup>
<mi>p</mi>
<mi>k</mi>
<mi>i</mi>
</msubsup>
<mo>-</mo>
<msubsup>
<mi>p</mi>
<mi>l</mi>
<mi>i</mi>
</msubsup>
<mo>|</mo>
<msub>
<mo>|</mo>
<mn>2</mn>
</msub>
</mrow>
</mfrac>
<mo>+</mo>
<mi>&beta;</mi>
<munder>
<mo>&Sigma;</mo>
<mi>i</mi>
</munder>
<munder>
<mo>&Sigma;</mo>
<mi>k</mi>
</munder>
<mo>|</mo>
<mo>|</mo>
<msubsup>
<mi>w</mi>
<mi>k</mi>
<mi>i</mi>
</msubsup>
<mo>|</mo>
<msubsup>
<mo>|</mo>
<mn>2</mn>
<mn>2</mn>
</msubsup>
</mrow>
</mtd>
</mtr>
</mtable>
</mfenced>
</mrow>
Wherein, the i is the sequence number of section micro-image, and the k and l are respectively the sequence number of corresponding points in section micro-image,
And l ≠ k, it is describedIt is described for the position coordinates of k-th of corresponding points in i-th of section micro-imageFor the position
CoordinateMotion vector, the E (w) be motion vector w energy function, the α and β are respectively constant.
5. method for registering according to claim 4, it is characterised in that
" according to the position of each corresponding points after the adjustment, image shape is carried out to biological tissue's sequence section micro-image
Become " include：Image deformation is carried out to biological tissue's sequence section micro-image using Moving Least, specifically included：
Shown method obtains position of the specific trellis summit after image deformation described in section micro-image and sat according to the following formula
Mark：
<mrow>
<mi>m</mi>
<mi>i</mi>
<mi>n</mi>
<mrow>
<mo>(</mo>
<msub>
<mi>l</mi>
<mi>v</mi>
</msub>
<mo>)</mo>
</mrow>
<mo>=</mo>
<mi>m</mi>
<mi>i</mi>
<mi>n</mi>
<mrow>
<mo>(</mo>
<munder>
<mo>&Sigma;</mo>
<mi>k</mi>
</munder>
<msub>
<mi>&lambda;</mi>
<mi>k</mi>
</msub>
<mo>|</mo>
<mo>|</mo>
<msub>
<mi>l</mi>
<mi>v</mi>
</msub>
<mrow>
<mo>(</mo>
<msub>
<mi>p</mi>
<mi>k</mi>
</msub>
<mo>)</mo>
</mrow>
<mo>-</mo>
<msub>
<mi>q</mi>
<mi>k</mi>
</msub>
<mo>|</mo>
<msubsup>
<mo>|</mo>
<mn>2</mn>
<mn>2</mn>
</msubsup>
<mo>)</mo>
</mrow>
</mrow>
Using all corresponding points after removing specific trellis summit in the bilinear interpolation acquisition section micro-image in image
Position coordinates after property deformation；
Wherein, the v is the position coordinates on specific trellis summit in micro-image of being cut into slices before image deformation, and the k is section
The sequence number of corresponding points in micro-image, the pkPosition for k-th of corresponding points in micro-image of being cut into slices before image deformation is sat
Mark, the qkFor the position coordinates of k-th of corresponding points in micro-image of being cut into slices after image deformation, qk=pk+wk, the wkTo cut
The motion vector of k-th of corresponding points in piece micro-image, the λkFor weight, andThe α is normal
Number, the lvIt is the rigid transformation matrix at position coordinates v correspondence position points.
6. a kind of registration apparatus of biological tissue's sequence section micro-image, it is characterised in that the registration apparatus includes：
Acquisition module, it is configured to obtain the corresponding points of each contiguous slices micro-image in biological tissue's sequence section micro-image；
Adjusting module, it is configured to adjust the position of acquired each corresponding points；
Deformation module, the position of each corresponding points after being configured to according to the adjustment are micro- to biological tissue's sequence section
Image carries out image deformation.
7. the registration apparatus of biological tissue's sequence section micro-image according to claim 6, it is characterised in that described to obtain
Modulus block also includes：
First module, it is configured to using SIFT flow algorithms to the first section micrograph in two contiguous slices micro-images
As carrying out anamorphose, and by second in the first section micro-image and described two contiguous slices micro-images after deformation
Section micro-image is matched；
Second unit, it is configured to set respectively on the after the matching first section micro-image and the second section micro-image
Grid, and the position coordinates that grid is corresponded in the first section micro-image and the second section micro-image is identical；
Third unit, it is configured to choose in the first section micro-image and grid in the first section micro-image after the deformation
The location point of vertex correspondence is as its corresponding points；And choose location point corresponding to grid vertex in the second section micro-image
As its corresponding points.
8. the registration apparatus of biological tissue's sequence section micro-image according to claim 6 or 7, it is characterised in that institute
Stating adjusting module includes adjustment unit, and the adjustment unit is configured to the method adjustment acquisition module shown according to the following formula and obtained
The position of each corresponding points taken：
<mrow>
<mi>m</mi>
<mi>i</mi>
<mi>n</mi>
<mrow>
<mo>(</mo>
<mi>E</mi>
<mo>(</mo>
<mi>w</mi>
<mo>)</mo>
<mo>)</mo>
</mrow>
<mo>=</mo>
<mfenced open = "(" close = ")">
<mtable>
<mtr>
<mtd>
<mstyle>
<munder>
<mo>&Sigma;</mo>
<mi>i</mi>
</munder>
</mstyle>
<mstyle>
<munder>
<mo>&Sigma;</mo>
<mi>k</mi>
</munder>
</mstyle>
<mo>|</mo>
<mo>|</mo>
<msubsup>
<mi>p</mi>
<mi>k</mi>
<mi>i</mi>
</msubsup>
<mo>+</mo>
<msubsup>
<mi>w</mi>
<mi>k</mi>
<mi>i</mi>
</msubsup>
<mo>-</mo>
<msubsup>
<mi>p</mi>
<mi>k</mi>
<mrow>
<mi>i</mi>
<mo>+</mo>
<mn>1</mn>
</mrow>
</msubsup>
<mo>-</mo>
<msubsup>
<mi>w</mi>
<mi>k</mi>
<mrow>
<mi>i</mi>
<mo>+</mo>
<mn>1</mn>
</mrow>
</msubsup>
<mo>|</mo>
<msubsup>
<mo>|</mo>
<mn>2</mn>
<mn>2</mn>
</msubsup>
</mtd>
</mtr>
<mtr>
<mtd>
<mrow>
<mo>+</mo>
<mi>&alpha;</mi>
<munder>
<mo>&Sigma;</mo>
<mi>i</mi>
</munder>
<munder>
<mo>&Sigma;</mo>
<mi>k</mi>
</munder>
<munderover>
<mo>&Sigma;</mo>
<mi>l</mi>
<mrow>
<mi>l</mi>
<mo>&NotEqual;</mo>
<mi>k</mi>
</mrow>
</munderover>
<mfrac>
<mrow>
<mo>|</mo>
<mo>|</mo>
<msubsup>
<mi>w</mi>
<mi>k</mi>
<mi>i</mi>
</msubsup>
<mo>-</mo>
<msubsup>
<mi>w</mi>
<mi>l</mi>
<mi>i</mi>
</msubsup>
<mo>|</mo>
<msubsup>
<mo>|</mo>
<mn>2</mn>
<mn>2</mn>
</msubsup>
</mrow>
<mrow>
<mo>|</mo>
<mo>|</mo>
<msubsup>
<mi>p</mi>
<mi>k</mi>
<mi>i</mi>
</msubsup>
<mo>-</mo>
<msubsup>
<mi>p</mi>
<mi>l</mi>
<mi>i</mi>
</msubsup>
<mo>|</mo>
<msub>
<mo>|</mo>
<mn>2</mn>
</msub>
</mrow>
</mfrac>
<mo>+</mo>
<mi>&beta;</mi>
<munder>
<mo>&Sigma;</mo>
<mi>i</mi>
</munder>
<munder>
<mo>&Sigma;</mo>
<mi>k</mi>
</munder>
<mo>|</mo>
<mo>|</mo>
<msubsup>
<mi>w</mi>
<mi>k</mi>
<mi>i</mi>
</msubsup>
<mo>|</mo>
<msubsup>
<mo>|</mo>
<mn>2</mn>
<mn>2</mn>
</msubsup>
</mrow>
</mtd>
</mtr>
</mtable>
</mfenced>
</mrow>
Wherein, the i is the sequence number of section micro-image, and the k and l are respectively the sequence number of corresponding points in section micro-image,
And l ≠ k, it is describedIt is described for the position coordinates of k-th of corresponding points in i-th of section micro-imageFor the position
CoordinateMotion vector, the E (w) be motion vector w energy function, the α and β are respectively constant.
9. a kind of storage device, wherein being stored with a plurality of program, it is characterised in that described program is applied to by processor loading simultaneously
Perform to realize the method for registering of biological tissue's sequence section micro-image described in claim any one of 1-5.
10. a kind of processing unit, including
Processor, it is adapted for carrying out each bar program；And
Storage device, suitable for storing a plurality of program；
Characterized in that, described program is suitable to be loaded by processor and performed to realize：Life described in claim any one of 1-5
The method for registering of thing organization order section micro-image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710638426.3A CN107545567B (en) | 2017-07-31 | 2017-07-31 | Registration method and device for biological tissue sequence section microscopic image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710638426.3A CN107545567B (en) | 2017-07-31 | 2017-07-31 | Registration method and device for biological tissue sequence section microscopic image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107545567A true CN107545567A (en) | 2018-01-05 |
| CN107545567B CN107545567B (en) | 2020-05-19 |
Family
ID=60970436
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710638426.3A Active CN107545567B (en) | 2017-07-31 | 2017-07-31 | Registration method and device for biological tissue sequence section microscopic image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107545567B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109242894A (en) * | 2018-08-06 | 2019-01-18 | 广州视源电子科技股份有限公司 | Image alignment method and system based on mobile least square method |
| CN109584368A (en) * | 2018-10-18 | 2019-04-05 | 中国科学院自动化研究所 | The construction method and device of biological sample three-dimensional structure |
| CN110596068A (en) * | 2019-10-11 | 2019-12-20 | 中国科学院自动化研究所 | Strip sequence slice optical imaging device and imaging method based on line scanning |
| CN110751680A (en) * | 2019-10-17 | 2020-02-04 | 中国科学院云南天文台 | Image processing method with fast alignment algorithm |
| CN111751371A (en) * | 2020-06-24 | 2020-10-09 | 武汉中纪生物科技有限公司 | Immunohistochemical digital slide reading system and method |
| CN112381718A (en) * | 2020-11-24 | 2021-02-19 | 中国科学院自动化研究所 | Automatic stitching method, system and device for microscopic sequence images based on affine transformation |
| CN112396608A (en) * | 2020-11-30 | 2021-02-23 | 中国科学院自动化研究所 | Biological tissue electron microscope image correction method, system and device based on X-ray image |
| CN114049252A (en) * | 2021-09-27 | 2022-02-15 | 中国科学院自动化研究所 | Scanning electron microscope three-dimensional image acquisition system and method for sequence slicing |
| CN114113084A (en) * | 2021-11-11 | 2022-03-01 | 福州迈新生物技术开发有限公司 | Real-time accompanying film reading method based on digital pathological image and storage device |
| CN116246019A (en) * | 2023-02-27 | 2023-06-09 | 上海迪派生物科技有限公司 | 3D reconstruction method, device, equipment and medium for pathological section |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103543168A (en) * | 2013-10-12 | 2014-01-29 | 华南理工大学 | Method and system for X ray detection on multilayer package substrate defects |
| CN104751455A (en) * | 2015-03-13 | 2015-07-01 | 华南农业大学 | Crop image dense matching method and system |
| CN105427308A (en) * | 2015-11-20 | 2016-03-23 | 中国地质大学（武汉） | Sparse and dense characteristic matching combined image registration method |
| CN106203269A (en) * | 2016-06-29 | 2016-12-07 | 武汉大学 | A kind of based on can the human face super-resolution processing method of deformation localized mass and system |
| CN106407988A (en) * | 2016-09-05 | 2017-02-15 | 中国科学院自动化研究所 | Electron microscopy serial slice image-based synapse detection method |
-
2017
- 2017-07-31 CN CN201710638426.3A patent/CN107545567B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103543168A (en) * | 2013-10-12 | 2014-01-29 | 华南理工大学 | Method and system for X ray detection on multilayer package substrate defects |
| CN104751455A (en) * | 2015-03-13 | 2015-07-01 | 华南农业大学 | Crop image dense matching method and system |
| CN105427308A (en) * | 2015-11-20 | 2016-03-23 | 中国地质大学（武汉） | Sparse and dense characteristic matching combined image registration method |
| CN106203269A (en) * | 2016-06-29 | 2016-12-07 | 武汉大学 | A kind of based on can the human face super-resolution processing method of deformation localized mass and system |
| CN106407988A (en) * | 2016-09-05 | 2017-02-15 | 中国科学院自动化研究所 | Electron microscopy serial slice image-based synapse detection method |
Non-Patent Citations (5)
| Title |
|---|
| CE LIU 等: "SIFT Flow: Dense Correspondence across Scenes and its Applications", 《IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE》 * |
| HUIWEN CAO 等: "Microscopic neural image registration based on the structure of mitochondria", 《SPIE MEDICAL IMAGING》 * |
| XI CHEN 等: "Wrinkle Image Registration for Serial Microscopy Sections", 《2015 11TH INTERNATIONAL CONFERENCE ON SIGNAL-IMAGE TECHNOLOGY & INTERNET-BASED SYSTEMS》 * |
| 汤敏 等: "生物组织连续切片的两步配准及快速重建", 《中国图象图形学报》 * |
| 苏昭: "图像配准算法研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109242894A (en) * | 2018-08-06 | 2019-01-18 | 广州视源电子科技股份有限公司 | Image alignment method and system based on mobile least square method |
| CN109584368B (en) * | 2018-10-18 | 2021-05-28 | 中国科学院自动化研究所 | Method and device for constructing three-dimensional structure of biological sample |
| CN109584368A (en) * | 2018-10-18 | 2019-04-05 | 中国科学院自动化研究所 | The construction method and device of biological sample three-dimensional structure |
| CN110596068A (en) * | 2019-10-11 | 2019-12-20 | 中国科学院自动化研究所 | Strip sequence slice optical imaging device and imaging method based on line scanning |
| CN110751680A (en) * | 2019-10-17 | 2020-02-04 | 中国科学院云南天文台 | Image processing method with fast alignment algorithm |
| CN111751371B (en) * | 2020-06-24 | 2021-01-08 | 武汉中纪生物科技有限公司 | Immunohistochemical digital slide reading system and method |
| CN111751371A (en) * | 2020-06-24 | 2020-10-09 | 武汉中纪生物科技有限公司 | Immunohistochemical digital slide reading system and method |
| CN112381718A (en) * | 2020-11-24 | 2021-02-19 | 中国科学院自动化研究所 | Automatic stitching method, system and device for microscopic sequence images based on affine transformation |
| CN112396608A (en) * | 2020-11-30 | 2021-02-23 | 中国科学院自动化研究所 | Biological tissue electron microscope image correction method, system and device based on X-ray image |
| CN112396608B (en) * | 2020-11-30 | 2021-05-04 | 中国科学院自动化研究所 | Biological tissue electron microscope image correction method, system and device based on X-ray image |
| CN114049252A (en) * | 2021-09-27 | 2022-02-15 | 中国科学院自动化研究所 | Scanning electron microscope three-dimensional image acquisition system and method for sequence slicing |
| CN114113084A (en) * | 2021-11-11 | 2022-03-01 | 福州迈新生物技术开发有限公司 | Real-time accompanying film reading method based on digital pathological image and storage device |
| WO2023082650A1 (en) * | 2021-11-11 | 2023-05-19 | 福州迈新生物技术开发有限公司 | Real-time concomitant slice reading method based on digital pathological image, and storage device |
| CN116246019A (en) * | 2023-02-27 | 2023-06-09 | 上海迪派生物科技有限公司 | 3D reconstruction method, device, equipment and medium for pathological section |
| CN116246019B (en) * | 2023-02-27 | 2024-01-05 | 上海迪派生物科技有限公司 | 3D reconstruction method, device, equipment and medium for pathological section |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107545567B (en) | 2020-05-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107545567A (en) | The method for registering and device of biological tissue's sequence section micro-image | |
| Lanzavecchia et al. | Conical tomography of freeze-fracture replicas: a method for the study of integral membrane proteins inserted in phospholipid bilayers | |
| CN111553858B (en) | Image restoration method and system based on generation countermeasure network and application thereof | |
| AU2011206927A1 (en) | A computed tomography imaging process and system | |
| CN102750704A (en) | Step-by-step video camera self-calibration method | |
| JP7506987B2 (en) | AI-enabled volume reconstruction | |
| CN105654483A (en) | Three-dimensional point cloud full-automatic registration method | |
| JP2010154920A (en) | Three-dimensional shape data processor, three-dimensional shape data processing method, and program | |
| CN109584223A (en) | Pulmonary vascular dividing method in CT image | |
| CN107481224A (en) | Method for registering images and device, storage medium and equipment based on structure of mitochondria | |
| CN104952068B (en) | A kind of vertebra characteristic point automatic identifying method | |
| US7912271B2 (en) | Method for determining final projection matrices | |
| Škorvánková et al. | Automatic estimation of anthropometric human body measurements | |
| DE10254907A1 (en) | Surface contouring method, especially for use with medical image data from computer tomography, whereby planes are defined through a 3D image volume and surface contours determined on the contours of each plane | |
| CN112396608B (en) | Biological tissue electron microscope image correction method, system and device based on X-ray image | |
| CN102599887B (en) | Optical projection tomography method based on helical scanning track | |
| TW201835800A (en) | Method for determining/correcting defects and associated devices | |
| JP2011161104A (en) | Image generator, image generation method, and program thereof | |
| CN104793160B (en) | A kind of method of the frequency aliasing effect for reducing lack sampling magnetic resonance imaging | |
| Ding et al. | Automatic 3D reconstruction of SEM images based on Nano-robotic manipulation and epipolar plane images | |
| CN107194998B (en) | A kind of method of multi-layer three-dimension point cloud single-layered | |
| CN110428392A (en) | A kind of Method of Medical Image Fusion based on dictionary learning and low-rank representation | |
| Baba et al. | A novel “ghost”-free tomographic image reconstruction method applicable to rotary-shadowed replica specimens | |
| CN114049252A (en) | Scanning electron microscope three-dimensional image acquisition system and method for sequence slicing | |
| Chen et al. | Hybrid‐attention densely connected U‐Net with GAP for extracting livers from CT volumes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |