CN110279429B - Four-dimensional ultrasonic reconstruction method and device - Google Patents
Four-dimensional ultrasonic reconstruction method and device Download PDFInfo
- Publication number
- CN110279429B CN110279429B CN201910510059.8A CN201910510059A CN110279429B CN 110279429 B CN110279429 B CN 110279429B CN 201910510059 A CN201910510059 A CN 201910510059A CN 110279429 B CN110279429 B CN 110279429B
- Authority
- CN
- China
- Prior art keywords
- dimensional
- respiratory
- dimensional image
- images
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/48—Diagnostic techniques
- A61B8/483—Diagnostic techniques involving the acquisition of a 3D volume of data
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/5215—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data
- A61B8/5238—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for combining image data of patient, e.g. merging several images from different acquisition modes into one image
- A61B8/5246—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for combining image data of patient, e.g. merging several images from different acquisition modes into one image combining images from the same or different imaging techniques, e.g. color Doppler and B-mode
Abstract
The embodiment of the invention provides a four-dimensional ultrasonic reconstruction method and a device, wherein the method comprises the following steps: acquiring first two-dimensional image sequences at different cross section positions of a target three-dimensional organ and respiratory signals respectively corresponding to the first two-dimensional image sequences, and dividing all the respiratory signals into a plurality of groups corresponding to a plurality of respiratory states; extracting two-dimensional images corresponding to each breathing state in each first two-dimensional image sequence based on the breathing signals corresponding to each two-dimensional image respectively so as to form a corresponding second two-dimensional image sequence aiming at each breathing state respectively; and for each respiratory state, reconstructing the three-dimensional volume data image in the respiratory state based on the second two-dimensional image sequence corresponding to the respiratory state, and displaying the four-dimensional ultrasonic image based on all the three-dimensional volume data images by arranging all the respiratory states. The embodiment of the invention can effectively improve the success rate of four-dimensional imaging and the imaging image quality and effectively reduce the cost.
Description
Technical Field
The invention relates to the technical field of ultrasonic image processing, in particular to a four-dimensional ultrasonic reconstruction method and device.
Background
The development of four-dimensional (4D) imaging provides researchers and the medical community with respiratory motion of three-dimensional organs, i.e., 4D imaging can form a 4D (3D + t) image sequence with three-dimensional motion according to three-dimensional images of organs in different respiratory states, such as 4D Magnetic Resonance Imaging (MRI), 4D Computed Tomography (CT), and the like.
In 4D imaging, the MR/CT has the capability of high-resolution imaging so that the tissue structure becomes clear, and by visualizing the three-dimensional respiratory motion inside a human body, a researcher can be better assisted to master the rule of organ motion. However, in addition to the long data acquisition process, the two imaging methods generate a large amount of radiation during the CT imaging process, which causes radiation pollution, while the MR imaging is relatively expensive and costly.
In recent years, ultrasound has been widely used in 4D imaging technology due to its features of high real-time performance, high safety, and low cost. Currently, 4D imaging processing is mainly performed according to 3D US images, and a 3D US probe can extract 3D US images with good real-time performance. However, the acquired 3D US images have a limited field of view, low resolution, and structures with high acoustic impedance may cause occlusion, resulting in incomplete image acquisition, which may seriously affect the success rate and image quality of 4D imaging. In addition, 3D US probes are expensive and costly.
Disclosure of Invention
In order to overcome the above problems or at least partially solve the above problems, embodiments of the present invention provide a four-dimensional ultrasonic reconstruction method and apparatus, so as to effectively improve the success rate of four-dimensional imaging and the quality of an imaged image, and effectively reduce the cost.
In a first aspect, an embodiment of the present invention provides a four-dimensional ultrasonic reconstruction method, including:
acquiring first two-dimensional image sequences at different cross section positions of a target three-dimensional organ and respiratory signals corresponding to two-dimensional images in the first two-dimensional image sequences respectively, and dividing all the respiratory signals into a plurality of groups according to the signal values, wherein each group corresponds to a respiratory state;
extracting two-dimensional images corresponding to the breathing states in each first two-dimensional image sequence based on the breathing signals corresponding to the two-dimensional images in each first two-dimensional image sequence respectively so as to form corresponding second two-dimensional image sequences aiming at the breathing states respectively;
and for each respiratory state, reconstructing a three-dimensional volume data image in the respiratory state based on a second two-dimensional image sequence corresponding to the respiratory state, and reconstructing a four-dimensional ultrasonic image of the target three-dimensional organ for display based on all the three-dimensional volume data images by arranging all the respiratory states.
In a second aspect, an embodiment of the present invention provides a four-dimensional ultrasound reconstruction apparatus, including:
the data acquisition module is used for acquiring first two-dimensional image sequences at different cross section positions of a target three-dimensional organ and respiratory signals corresponding to two-dimensional images in the first two-dimensional image sequences respectively, and dividing all the respiratory signals into a plurality of groups according to the size of a signal value, wherein each group corresponds to a respiratory state;
the image extraction module is used for extracting two-dimensional images corresponding to the respiratory states in each first two-dimensional image sequence based on respiratory signals corresponding to the two-dimensional images in each first two-dimensional image sequence respectively so as to form corresponding second two-dimensional image sequences aiming at the respiratory states respectively;
and the reconstruction display module is used for reconstructing a three-dimensional volume data image in the respiratory state based on the second two-dimensional image sequence corresponding to the respiratory state for each respiratory state, and reconstructing a four-dimensional ultrasonic image of the target three-dimensional organ for display based on all the three-dimensional volume data images by arranging all the respiratory states.
In a third aspect, an embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the processor implements the steps of the four-dimensional ultrasound reconstruction method according to the first aspect.
In a fourth aspect, embodiments of the present invention provide a non-transitory computer-readable storage medium having stored thereon computer instructions which, when executed by a computer, implement the steps of the four-dimensional ultrasound reconstruction method according to the first aspect.
According to the four-dimensional ultrasonic reconstruction method and device provided by the embodiment of the invention, the two-dimensional images at different cross section positions of the target three-dimensional organ and the breathing signals corresponding to the two-dimensional images are acquired, the two-dimensional images are divided again according to the breathing signals to form a new two-dimensional image sequence, and finally, the accurate reconstruction and display of the four-dimensional ultrasonic image of the target three-dimensional organ are realized by performing three-dimensional reconstruction on all the two-dimensional images in each breathing state and arranging all the breathing states, so that the success rate of four-dimensional imaging and the imaging image quality can be effectively improved, and the cost can be effectively reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a four-dimensional ultrasonic reconstruction method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a four-dimensional ultrasonic reconstruction method according to another embodiment of the present invention;
FIG. 3 is a schematic flow chart of data processing and image classification in the flow chart of the four-dimensional ultrasonic reconstruction method shown in FIG. 2;
fig. 4 is a schematic structural diagram of a four-dimensional ultrasonic reconstruction apparatus according to an embodiment of the present invention;
fig. 5 is a schematic physical structure diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention without any creative efforts belong to the protection scope of the embodiments of the present invention.
Aiming at the problems of low success rate and image quality and high cost of 4D imaging in the prior art, the embodiment of the invention acquires two-dimensional images at different cross section positions of a target three-dimensional organ and respiratory signals corresponding to the two-dimensional images, re-divides the two-dimensional images into new two-dimensional image sequences according to the respiratory signals, and finally realizes accurate reconstruction and display of four-dimensional ultrasonic images of the target three-dimensional organ by performing three-dimensional reconstruction on all the two-dimensional images in each respiratory state and arranging all the respiratory states, thereby effectively improving the success rate of four-dimensional imaging and the quality of the imaged images and effectively reducing the cost. Embodiments of the present invention will be described and illustrated with reference to various embodiments.
Fig. 1 is a schematic flow chart of a four-dimensional ultrasonic reconstruction method according to an embodiment of the present invention, as shown in fig. 1, the method includes:
s101, acquiring first two-dimensional image sequences at different cross section positions of a target three-dimensional organ and respiratory signals corresponding to two-dimensional images in the first two-dimensional image sequences respectively, and dividing all the respiratory signals into a plurality of groups according to the signal values, wherein each group corresponds to a respiratory state.
Specifically, the embodiment of the invention respectively acquires and acquires data at a plurality of different cross section positions of a target three-dimensional organ, wherein the acquired data comprises two-dimensional images at the cross sections and corresponding breathing signals when the two-dimensional images are acquired. For the same cross section position, a plurality of two-dimensional images can be continuously collected for a plurality of times, and the two-dimensional images form a two-dimensional image sequence based on the same cross section, namely the first two-dimensional image sequence. Then, for a plurality of different cross-sectional positions, a plurality of different first two-dimensional image sequences and respiratory signals corresponding to each two-dimensional image in the first two-dimensional image sequence can be correspondingly acquired.
Then, the respiratory signals corresponding to all the two-dimensional images in all the first two-dimensional image sequences are grouped according to the size of the signal value, each group actually corresponds to one respiratory phase, and each respiratory phase is set to be in a specific respiratory state. For example, all the respiratory signals may be arranged in order from small to large to obtain the maximum and minimum values, and then all the respiratory signals may be divided into a plurality of respiratory phases according to the equal respiratory signal intervals or the equal respiratory signal number, where each respiratory phase corresponds to a specific respiratory state.
S102, extracting the two-dimensional images corresponding to the breathing states in each first two-dimensional image sequence based on the breathing signals corresponding to the two-dimensional images in each first two-dimensional image sequence, so as to form corresponding second two-dimensional image sequences aiming at the breathing states.
It can be understood that, on the basis of grouping all the respiratory signals to obtain a plurality of respiratory states, the embodiment of the present invention extracts the two-dimensional images in each first two-dimensional image sequence corresponding to each respiratory state according to the respiratory phase to which the respiratory signal of the two-dimensional image in each first two-dimensional image sequence belongs, and then includes a plurality of two-dimensional images in each respiratory state. Thus, on a per-respiratory-state basis, a new image sequence, i.e., a second two-dimensional image sequence, can be formed from the plurality of two-dimensional images in each respiratory state. It can be understood that through the conversion of the first two-dimensional image sequence to the second two-dimensional image sequence, the image sequence is converted from being divided based on different positions to being divided based on different breathing states, that is, clustering of two-dimensional images at different cross section positions with the same breathing state is realized.
S103, for each respiratory state, reconstructing a three-dimensional volume data image in the respiratory state based on the second two-dimensional image sequence corresponding to the respiratory state, and reconstructing a four-dimensional ultrasonic image of the target three-dimensional organ for display based on all three-dimensional volume data images by arranging all respiratory states.
It can be understood that, in this step, based on the above conversion, the three-dimensional ultrasound image reconstruction in the corresponding respiratory state is performed by using all the two-dimensional images in each respiratory state, and the three-dimensional ultrasound image obtained by reconstruction is the three-dimensional volume data image in the corresponding respiratory state. For example, for the respiratory state i, a second two-dimensional image sequence is corresponding to the respiratory state i, where the second two-dimensional image sequence includes a plurality of two-dimensional images at different cross-sectional positions of the target three-dimensional organ acquired in the respiratory state i, the three-dimensional ultrasound image can be reconstructed by using the two-dimensional images, and the reconstructed three-dimensional ultrasound image is a three-dimensional volume data image in the respiratory state i.
Thus, for each respiratory state, a corresponding three-dimensional volume data image will be obtained. And finally, arranging all the respiratory states according to the sizes to obtain an ordered arrangement, arranging the three-dimensional data images corresponding to all the respiratory states according to a corresponding order, finally obtaining a four-dimensional ultrasonic image containing respiratory state information, namely the four-dimensional ultrasonic image of the target three-dimensional organ obtained by reconstruction, and then carrying out four-dimensional display on the four-dimensional ultrasonic image by using related display equipment.
According to the four-dimensional ultrasonic reconstruction method provided by the embodiment of the invention, the two-dimensional images at different cross section positions of the target three-dimensional organ and the breathing signals corresponding to the two-dimensional images are acquired, the two-dimensional images are divided again according to the breathing signals to form a new two-dimensional image sequence, and finally, the accurate reconstruction and display of the four-dimensional ultrasonic image of the target three-dimensional organ are realized by performing three-dimensional reconstruction on all the two-dimensional images in each breathing state and arranging all the breathing states, so that the success rate of four-dimensional imaging and the imaging image quality can be effectively improved, and the cost can be effectively reduced.
Optionally, according to the above embodiments, the step of extracting the two-dimensional image corresponding to each respiratory state in each first two-dimensional image sequence specifically includes: for each first two-dimensional image sequence, determining the respiratory state of each two-dimensional image based on the respiratory signal corresponding to each two-dimensional image in the first two-dimensional image sequence; for any respiratory state, the similarity calculation between the two-dimensional image at the first cross section position in the respiratory state and all the two-dimensional images at the adjacent next cross section position in the respiratory state is carried out, and the two-dimensional image with the maximum similarity at the first cross section position is extracted and used as the two-dimensional image corresponding to the respiratory state.
Specifically, the embodiment of the present invention processes the first two-dimensional image sequence at each cross-sectional position, and extracts two-dimensional images corresponding to different respiratory states. Starting from the first cross section position, defining the currently processed cross section position as the first cross section position, and determining the corresponding relation between the corresponding two-dimensional image and the corresponding breathing phase by judging the breathing phase to which the breathing signal corresponding to the two-dimensional image belongs, namely determining the corresponding relation between the corresponding two-dimensional image and the breathing phase (corresponding breathing state), for the first two-dimensional image sequence at the first cross section position.
Therefore, for each respiratory state, each cross-sectional position corresponding to each respiratory state comprises a plurality of two-dimensional images, taking one respiratory state i as an example, after the two-dimensional image of each respiratory state is taken out at the first cross-sectional position, the 2D US image of the respiratory state i at the first cross-sectional position and all the two-dimensional images of the respiratory state i at the next adjacent position are used for calculating the similarity measure, and the two-dimensional image with the highest similarity calculation result in all the two-dimensional images at the first cross-sectional position is extracted and used as the image for reconstructing the four-dimensional ultrasound image in the respiratory state i at the position.
Optionally, specifically, an L2 norm calculation method may be adopted to perform similarity calculation, where a specific calculation formula is as follows:
in the formula (I), the compound is shown in the specification,
to represent
And
the L2 norm of (a),
representing a two-dimensional image at a first cross-sectional position in a respiratory state i, i ═ 1,2,. multidot.n, n representing the total number of packets,
represents the two-dimensional image at the next cross-sectional position under the respiratory state i, j ═ 1,2The total number of two-dimensional images belonging to the respiratory state i in the first sequence of two-dimensional images at the cross-sectional position.
According to the method and the device, based on the concept that the images at the adjacent positions have similarity, on the premise that the respiratory phase of the US image is identified by using the respiratory signal, the similarity between the adjacent images is calculated, and the two-dimensional image at the current position is accurately screened, so that the phenomenon that the tissue boundary in the volume data is discontinuous is obviously improved.
Optionally, the following processing procedure may also be adopted to extract the two-dimensional images corresponding to the respiratory states in each first two-dimensional image sequence: firstly, aiming at the respiratory state of each group, based on the signal values of all respiratory signals of the group, calculating the average signal value corresponding to the respiratory state of the group; secondly, for any first two-dimensional image sequence, determining the respiratory state of each two-dimensional image based on the respiratory signal corresponding to each two-dimensional image in the first two-dimensional image sequence; finally, for any respiratory state, the two-dimensional image with the respiratory signal value closest to the average signal value is selected as the two-dimensional image corresponding to the respiratory state by calculating the closeness degree of the respiratory signal corresponding to each two-dimensional image in the respiratory state and the average signal value.
That is, in the embodiment of the present invention, the statistical average is used as a reference to select the corresponding two-dimensional image in each first two-dimensional image sequence to perform the three-dimensional volume data reconstruction in each respiratory state. Specifically, for each respiratory state, i.e. respiratory phase, the average value of all respiratory signals with a signal value in that phase is determined, and the average value is used as the average signal value of that respiratory phase. Then, for each first two-dimensional image sequence, determining a corresponding relationship between a corresponding two-dimensional image and a respiratory phase by judging the respiratory phase to which the respiratory signal of the two-dimensional image belongs, that is, determining the corresponding relationship between the corresponding two-dimensional image and the respiratory phase. Thus, for each respiratory state, a plurality of two-dimensional images is included for each cross-sectional location below it. Finally, for each respiratory state, the respiratory signals of the plurality of two-dimensional images corresponding to each cross section position under the respiratory state are respectively compared with the obtained average signal value of the respiratory state, and the two-dimensional image with the respiratory signal value closest to the average signal value of the respiratory stage is extracted and used for reconstructing the three-dimensional volume data of the respiratory stage.
Further, before the step of dividing all the respiratory signals into a plurality of groups according to the size of the signal value, the four-dimensional ultrasound reconstruction method according to the embodiment of the present invention may further include: performing histogram statistics on all respiratory signals to obtain effective upper boundaries and effective lower boundaries of all respiratory signals, and rejecting abnormal respiratory signals and two-dimensional images corresponding to the abnormal respiratory signals by comparing the values of all respiratory signals with the effective upper boundaries and the effective lower boundaries; correspondingly, all the breathing signals after the abnormal breathing signals are removed are divided into a plurality of groups according to the size of the signal value, and each group corresponds to one breathing state.
It can be understood that, since the phenomenon of the movement of the acquired object inevitably occurs during the process of acquiring the two-dimensional image, and the range of the acquired respiratory signal value exceeds the range of the movement of normal respiratory motion, the embodiment of the present invention performs histogram statistics on all acquired respiratory signals in advance, and removes abnormal respiratory signal values with abnormal movement according to the statistical result.
Optionally, the step of obtaining the first two-dimensional image sequences at different cross-sectional positions of the target three-dimensional organ and the respiratory signals corresponding to the two-dimensional images in the first two-dimensional image sequences respectively specifically includes: for any cross section position, controlling a two-dimensional ultrasonic probe to collect a group of target three-dimensional organ cross section ultrasonic sequence images with 2 or 3 respiratory motion cycles at the cross section position as a first two-dimensional image sequence, and acquiring respiratory signals corresponding to all two-dimensional images in the first two-dimensional image sequence by tracking a mark point preset on a target three-dimensional organ by using an electromagnetic tracking method.
It can be understood that, in the embodiment of the present invention, the mechanical arm is used to acquire 2D US sequences of different cross-section portions of the target organ, and specifically, the mechanical arm is used to control the common two-dimensional ultrasonic probe to acquire the required two-dimensional image data. If the target three-dimensional organ traverses each cross section position along a given direction, the mechanical arm controls the two-dimensional ultrasonic probe to move from top to bottom, and the two-dimensional ultrasonic probe moves at intervals of 1mm each time. Particularly, when the two-dimensional ultrasonic probe is moved each time, the parallelism between the ultrasonic sequence image planes in the acquisition process can be controlled, and the two-dimensional ultrasonic probe is kept to move on a straight line vertical to the ultrasonic planes.
Before the two-dimensional ultrasonic probe is moved each time, the ultrasonic probe acquires a group of target organ cross section ultrasonic sequence images with 2 or 3 respiratory motion cycles at the current position, meanwhile, the mark points at the set position are tracked by an electromagnetic tracking method to acquire the respiratory signals of the image sequence, and after the acquisition is finished, the mechanical arm controls the ultrasonic probe to move downwards to acquire the next group of image sequence.
And after the two-dimensional ultrasonic probe is moved to a new cross section position, the data acquisition step is repeatedly executed until the data acquisition traverses the whole target three-dimensional organ, and all the first two-dimensional image sequences and the respiratory signals corresponding to the two-dimensional images in the first two-dimensional image sequences are obtained.
To further illustrate the technical solutions of the embodiments of the present invention, the embodiments of the present invention provide the following specific processing flows according to the above embodiments, but do not limit the scope of the embodiments of the present invention.
Figure 2 is a flow chart of a four-dimensional ultrasonic reconstruction method according to another embodiment of the present invention,
fig. 3 is a schematic flow chart of data processing and image classification in the flow chart of the four-dimensional ultrasonic reconstruction method shown in fig. 2, as shown in fig. 2 and 3, the method includes:
first, a 2D US sequence { I ] of each part is obtained by data acquisition1，...,INAnd its corresponding set of respiratory signals { S }1,...,SNAnd performing histogram statistics on all respiratory signals to obtain upper and lower boundaries of respiratory signal values so as to eliminate abnormal values.
Secondly, from the set of respiratory signals { S }1,...,SNObtaining upper and lower boundaries of the respiration signal as maximum and minimum values of the segments, and dividing each cross section according to the obtained maximum and minimum valuesThe breathing signal of the face position in this interval is divided into 15 breathing phases, the image representation in each breathing phase having the same breathing state.
Again, for the first set of two-dimensional image sequences I1And extracting the image of which the respiratory signal value is closest to the average value of the respiratory phase, and using the image for reconstructing the three-dimensional volume data of the respiratory phase.
Specifically, at the first cross-sectional position, after the two-dimensional image of each respiratory phase is taken out, the 2D US image I of the respiratory phase I (i.e. respiratory state I) at the cross-sectional position 1 is used1 iThe similarity measure is calculated for all images of this breathing phase i at the next adjacent cross-sectional position, using the L2 norm as follows:
in the formula (I), the compound is shown in the specification,
to represent
And
the L2 norm of (a),
represents a two-dimensional image at a first cross-sectional position in a respiratory state i, i 1, 2., 15,
represents the two-dimensional image at the next cross-sectional position under the respiratory state i, j ═ 1, 2., k, k represents the total number of two-dimensional images belonging to the respiratory state i in the first sequence of two-dimensional images at said next cross-sectional position.
And then, extracting an image with the minimum L2 norm value in all two-dimensional images in the respiratory state i at the first cross section position as an image used for reconstruction of the siwei ultrasonic image in the respiratory stage i at the position.
And repeating the processing process, and extracting the same breathing node image at each cross section position until the images at all the cross section positions are extracted.
And finally, respectively generating 3D US images by the extracted image sequences of all respiratory phases, and obtaining 4D US images in a half period for displaying according to respiratory signal value arrangement.
The embodiment of the invention combines the image information and the respiratory signal to carry out 4D US reconstruction, so that the phenomenon of discontinuous tissue boundary in the volume data is obviously improved. Meanwhile, 4D ultrasonic data are obtained by using a common 2D ultrasonic probe, and the 4D ultrasonic data are reconstructed by using a 2D ultrasonic sequence, so that the 4D ultrasonic image has a large visual angle and high quality, and the defects of the 3D US probe are effectively compensated.
Based on the same concept, the embodiments of the present invention provide a four-dimensional ultrasonic reconstruction apparatus according to the above embodiments, which is used for implementing four-dimensional ultrasonic reconstruction in the above embodiments. Therefore, the description and definition in the four-dimensional ultrasonic reconstruction method in each of the above embodiments may be used for understanding each execution module in the embodiments of the present invention, and reference may be made to the above embodiments specifically, and details are not described here.
According to an embodiment of the present invention, a four-dimensional ultrasound reconstruction apparatus is shown in fig. 4, which is a schematic structural diagram of the four-dimensional ultrasound reconstruction apparatus provided in the embodiment of the present invention, and the apparatus may be used to implement four-dimensional ultrasound reconstruction in the foregoing method embodiments, and the apparatus includes: a data acquisition module 401, an image extraction module 402 and a reconstruction presentation module 403. Wherein:
the data acquisition module 401 is configured to acquire first two-dimensional image sequences at different cross-sectional positions of a target three-dimensional organ and respiratory signals corresponding to two-dimensional images in the first two-dimensional image sequences, and divide all the respiratory signals into a plurality of groups according to signal values, where each group corresponds to a respiratory state; the image extraction module 402 is configured to extract two-dimensional images corresponding to each respiratory state in each first two-dimensional image sequence based on the respiratory signal corresponding to each two-dimensional image in each first two-dimensional image sequence, so as to form a corresponding second two-dimensional image sequence for each respiratory state; the reconstruction display module 403 is configured to, for each respiratory state, reconstruct a three-dimensional volume data image in the respiratory state based on the second two-dimensional image sequence corresponding to the respiratory state, and display a four-dimensional ultrasound image of the target three-dimensional organ based on all three-dimensional volume data images by arranging all respiratory states.
Specifically, the data acquisition module 401 acquires and acquires data at a plurality of different cross-sectional positions of the target three-dimensional organ, where the acquired data includes two-dimensional images at the cross-sectional positions and respiratory signals corresponding to the acquired two-dimensional images. For the same cross section position, the data acquisition module 401 may continuously acquire a plurality of two-dimensional images for a plurality of times, and the two-dimensional images form a two-dimensional image sequence based on the same cross section, that is, the first two-dimensional image sequence. Then, for a plurality of different cross-sectional positions, a plurality of different first two-dimensional image sequences and respiratory signals corresponding to each two-dimensional image in the first two-dimensional image sequence can be correspondingly acquired.
Then, the data obtaining module 401 groups the respiratory signals corresponding to all the two-dimensional images in all the first two-dimensional image sequences according to the magnitude of the signal value, and each group actually corresponds to one respiratory phase, so that each respiratory phase is set to a specific respiratory state. For example, the data obtaining module 401 may arrange all the respiratory signals in order from small to large, obtain the maximum and minimum values, and then divide all the respiratory signals into a plurality of respiratory phases according to an equal respiratory signal interval or an equal respiratory signal quantity, where each respiratory phase corresponds to a specific respiratory state.
Then, the image extraction module 402 extracts the two-dimensional images in the first two-dimensional image sequences respectively corresponding to the respiratory states according to the respiratory phases to which the respiratory signals of the two-dimensional images in the first two-dimensional image sequences belong, and then the two-dimensional images in each respiratory state include a plurality of two-dimensional images. Thus, on the basis of each respiratory state, the image extraction module 402 may respectively form a new image sequence, i.e., a second two-dimensional image sequence, from the plurality of two-dimensional images in each respiratory state. It can be understood that through the conversion of the first two-dimensional image sequence to the second two-dimensional image sequence, the image sequence is converted from being divided based on different positions to being divided based on different breathing states, that is, clustering of two-dimensional images at different cross section positions with the same breathing state is realized.
Finally, the reconstruction display module 403 reconstructs the three-dimensional ultrasound image in each respiratory state by using all the two-dimensional images in each respiratory state, and the three-dimensional ultrasound image obtained by reconstruction is the three-dimensional volume data image in the corresponding respiratory state. Thus, a corresponding three-dimensional volume data image is obtained for each breathing state. Then, the reconstruction display module 403 obtains an ordered arrangement by arranging all the respiratory states, and at the same time, arranges the three-dimensional volume data images corresponding to each respiratory state according to the corresponding order, and finally obtains a four-dimensional ultrasound image containing respiratory state information, that is, a four-dimensional ultrasound image of the target three-dimensional organ obtained by reconstruction, and performs four-dimensional display on the four-dimensional ultrasound image by using a related display device.
According to the four-dimensional ultrasonic reconstruction device provided by the embodiment of the invention, the two-dimensional images at different cross section positions of the target three-dimensional organ and the breathing signals corresponding to the two-dimensional images are acquired by arranging the corresponding execution modules, the two-dimensional images are divided again according to the breathing signals to form a new two-dimensional image sequence, and finally, the accurate reconstruction and display of the four-dimensional ultrasonic image of the target three-dimensional organ are realized by performing three-dimensional reconstruction on all the two-dimensional images in each breathing state and arranging all the breathing states, so that the success rate of four-dimensional imaging and the imaging image quality can be effectively improved, and the cost can be effectively reduced.
It is understood that, in the embodiment of the present invention, each relevant program module in the apparatus of each of the above embodiments may be implemented by a hardware processor (hardware processor). Moreover, the four-dimensional ultrasonic reconstruction apparatus according to the embodiment of the present invention can implement the four-dimensional ultrasonic reconstruction process of each method embodiment by using the program modules, and when the apparatus is used to implement the four-dimensional ultrasonic reconstruction in each method embodiment, the beneficial effects produced by the apparatus according to the embodiment of the present invention are the same as those of the corresponding method embodiment, and reference may be made to the method embodiments, which are not described herein again.
As a further aspect of the embodiments of the present invention, the present embodiment provides an electronic device according to the above embodiments, the electronic device includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the four-dimensional ultrasound reconstruction method according to the above embodiments.
Further, the electronic device of the embodiment of the present invention may further include a communication interface and a bus. Referring to fig. 5, an entity structure diagram of an electronic device provided in an embodiment of the present invention includes: at least one memory 501, at least one processor 502, a communication interface 503, and a bus 504.
The memory 501, the processor 502 and the communication interface 503 complete mutual communication through the bus 504, and the communication interface 503 is used for information transmission between the electronic device and the two-dimensional ultrasound image device; the memory 501 stores a computer program that can be executed on the processor 502, and when the processor 502 executes the computer program, the steps of the four-dimensional ultrasound reconstruction method according to the embodiments described above are implemented.
It is understood that the electronic device at least includes a memory 501, a processor 502, a communication interface 503 and a bus 504, and the memory 501, the processor 502 and the communication interface 503 are connected in communication with each other through the bus 504, and can complete communication with each other, for example, the processor 502 reads program instructions of the four-dimensional ultrasound reconstruction method from the memory 501. In addition, the communication interface 503 can also implement communication connection between the electronic device and the two-dimensional ultrasound image device, and can complete mutual information transmission, such as four-dimensional ultrasound reconstruction and the like through the communication interface 503.
When the electronic device is running, the processor 502 calls the program instructions in the memory 501 to perform the methods provided by the above-described method embodiments, including for example: acquiring first two-dimensional image sequences at different cross section positions of a target three-dimensional organ and respiratory signals corresponding to two-dimensional images in the first two-dimensional image sequences respectively, and dividing all the respiratory signals into a plurality of groups according to the signal values, wherein each group corresponds to a respiratory state; extracting two-dimensional images corresponding to each breathing state in each first two-dimensional image sequence based on the breathing signals corresponding to each two-dimensional image in each first two-dimensional image sequence to form corresponding second two-dimensional image sequences respectively aiming at each breathing state; and for each respiratory state, reconstructing a three-dimensional volume data image in the respiratory state based on the second two-dimensional image sequence corresponding to the respiratory state, and reconstructing a four-dimensional ultrasonic image of the target three-dimensional organ for display and the like based on all three-dimensional volume data images by arranging all respiratory states.
The program instructions in the memory 501 may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand-alone product. Alternatively, all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, where the program may be stored in a computer-readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
Embodiments of the present invention further provide a non-transitory computer-readable storage medium according to the above embodiments, on which computer instructions are stored, and when the computer instructions are executed by a computer, the method for four-dimensional ultrasound reconstruction according to the above embodiments is implemented, for example, the method includes: acquiring first two-dimensional image sequences at different cross section positions of a target three-dimensional organ and respiratory signals corresponding to two-dimensional images in the first two-dimensional image sequences respectively, and dividing all the respiratory signals into a plurality of groups according to the signal values, wherein each group corresponds to a respiratory state; extracting two-dimensional images corresponding to each breathing state in each first two-dimensional image sequence based on the breathing signals corresponding to each two-dimensional image in each first two-dimensional image sequence to form corresponding second two-dimensional image sequences respectively aiming at each breathing state; and for each respiratory state, reconstructing a three-dimensional volume data image in the respiratory state based on the second two-dimensional image sequence corresponding to the respiratory state, and reconstructing a four-dimensional ultrasonic image of the target three-dimensional organ for display and the like based on all three-dimensional volume data images by arranging all respiratory states.
According to the electronic device and the non-transitory computer readable storage medium provided by the embodiments of the present invention, by executing the four-dimensional ultrasonic reconstruction method described in each of the embodiments, two-dimensional images at different cross-sectional positions of a target three-dimensional organ and respiratory signals corresponding to the two-dimensional images are acquired, the two-dimensional images are subdivided into new two-dimensional image sequences for the respiratory signals, and finally, all the two-dimensional images in each respiratory state are three-dimensionally reconstructed and arranged, so that accurate reconstruction and display of four-dimensional ultrasonic images of the target three-dimensional organ are achieved, the success rate of four-dimensional imaging and the quality of the imaged images can be effectively improved, and the cost can be effectively reduced.
It is to be understood that the above-described embodiments of the apparatus, the electronic device and the storage medium are merely illustrative, and that elements described as separate components may or may not be physically separate, may be located in one place, or may be distributed on different network elements. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on such understanding, the technical solutions mentioned above may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a usb disk, a removable hard disk, a ROM, a RAM, a magnetic or optical disk, etc., and includes several instructions for causing a computer device (such as a personal computer, a server, or a network device, etc.) to execute the methods described in the method embodiments or some parts of the method embodiments.
In addition, it should be understood by those skilled in the art that in the specification of the embodiments of the present invention, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
In the description of the embodiments of the invention, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description. Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects.
However, the disclosed method should not be interpreted as reflecting an intention that: that is, the claimed embodiments of the invention require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of an embodiment of this invention.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and not to limit the same; although embodiments of the present invention have been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. A four-dimensional ultrasound reconstruction method, comprising:
acquiring first two-dimensional image sequences at different cross section positions of a target three-dimensional organ and respiratory signals corresponding to two-dimensional images in the first two-dimensional image sequences respectively, and dividing all the respiratory signals into a plurality of groups according to the signal values, wherein each group corresponds to a respiratory state;
extracting two-dimensional images corresponding to the breathing states in each first two-dimensional image sequence based on the breathing signals corresponding to the two-dimensional images in each first two-dimensional image sequence respectively so as to form corresponding second two-dimensional image sequences aiming at the breathing states respectively;
for each respiratory state, reconstructing a three-dimensional volume data image in the respiratory state based on a second two-dimensional image sequence corresponding to the respiratory state, and reconstructing a four-dimensional ultrasonic image of the target three-dimensional organ for display based on all the three-dimensional volume data images by arranging all the respiratory states; wherein the step of extracting the two-dimensional image corresponding to each respiratory state in each first two-dimensional image sequence specifically includes:
for each first two-dimensional image sequence, determining the respiratory state of each two-dimensional image based on the respiratory signal corresponding to each two-dimensional image in the first two-dimensional image sequence;
for any one respiratory state, extracting a two-dimensional image with the maximum similarity at the first cross section position as a two-dimensional image corresponding to the respiratory state by performing similarity calculation between the two-dimensional image at the first cross section position in the respiratory state and all two-dimensional images at the adjacent next cross section position in the respiratory state;
or, the step of extracting the two-dimensional image corresponding to each respiratory state in each first two-dimensional image sequence specifically includes:
aiming at the respiratory state of each group, calculating an average signal value corresponding to the respiratory state of the group based on the signal values of all respiratory signals of the group;
for any first two-dimensional image sequence, determining the respiratory state of each two-dimensional image based on the respiratory signal corresponding to each two-dimensional image in the first two-dimensional image sequence;
for any one of the respiratory states, the two-dimensional image with the respiratory signal value closest to the average signal value is selected as the two-dimensional image corresponding to the respiratory state by calculating the closeness degree of the respiratory signal corresponding to each two-dimensional image in the respiratory state and the average signal value.
2. The four-dimensional ultrasound reconstruction method according to claim 1, wherein the similarity calculation is performed specifically by using an L2 norm calculation method, and a specific calculation formula is as follows:
in the formula (I), the compound is shown in the specification,
to represent
And
the L2 norm of (a),
representing a two-dimensional image at a first cross-sectional position in a respiratory state i, i ═ 1, 2., n, representing the total number of said groups,
represents the two-dimensional image at the next cross-sectional position under the respiratory state i, j ═ 1, 2., k, k represents the total number of two-dimensional images belonging to the respiratory state i in the first sequence of two-dimensional images at said next cross-sectional position.
3. The four-dimensional ultrasound reconstruction method according to any one of claims 1-2, further comprising, prior to the step of dividing all the respiratory signals into a plurality of groups according to the magnitude of signal values:
performing histogram statistics on all the respiratory signals to obtain effective upper boundaries and effective lower boundaries of all the respiratory signals, and eliminating abnormal respiratory signals and two-dimensional images corresponding to the abnormal respiratory signals by comparing the values of all the respiratory signals with the effective upper boundaries and the effective lower boundaries;
correspondingly, all the breathing signals after the abnormal breathing signals are removed are divided into a plurality of groups according to the size of the signal value, and each group corresponds to one breathing state.
4. The four-dimensional ultrasound reconstruction method according to claim 1, wherein the step of obtaining the first two-dimensional image sequence at different cross-sectional positions of the target three-dimensional organ and the respiratory signal corresponding to each two-dimensional image in each first two-dimensional image sequence specifically includes:
and for any cross section position, controlling a two-dimensional ultrasonic probe to collect a group of cross section ultrasonic sequence images of the target three-dimensional organ with 2 or 3 respiratory motion cycles at the cross section position to serve as the first two-dimensional image sequence, and acquiring respiratory signals corresponding to all two-dimensional images in the first two-dimensional image sequence by tracking a mark point preset on the target three-dimensional organ by using an electromagnetic tracking method.
5. The four-dimensional ultrasound reconstruction method according to claim 4, wherein the step of controlling the two-dimensional ultrasound probe comprises in particular:
and controlling the planes of the images of the ultrasonic sequence to be parallel in the acquisition process, and keeping the two-dimensional ultrasonic probe to move on a straight line vertical to the ultrasonic plane.
6. A four-dimensional ultrasound reconstruction apparatus, comprising:
the data acquisition module is used for acquiring first two-dimensional image sequences at different cross section positions of a target three-dimensional organ and respiratory signals corresponding to two-dimensional images in the first two-dimensional image sequences respectively, and dividing all the respiratory signals into a plurality of groups according to the size of a signal value, wherein each group corresponds to a respiratory state;
the image extraction module is used for extracting two-dimensional images corresponding to the respiratory states in each first two-dimensional image sequence based on respiratory signals corresponding to the two-dimensional images in each first two-dimensional image sequence respectively so as to form corresponding second two-dimensional image sequences aiming at the respiratory states respectively;
the reconstruction display module is used for reconstructing a three-dimensional volume data image in each respiratory state based on the second two-dimensional image sequence corresponding to the respiratory state, and reconstructing a four-dimensional ultrasonic image of the target three-dimensional organ for display based on all the three-dimensional volume data images by arranging all the respiratory states;
wherein the extracting the two-dimensional image corresponding to each respiratory state in each first two-dimensional image sequence specifically includes:
for each first two-dimensional image sequence, determining the respiratory state of each two-dimensional image based on the respiratory signal corresponding to each two-dimensional image in the first two-dimensional image sequence;
for any one respiratory state, the similarity calculation between the two-dimensional image at the first cross section position in the respiratory state and all the two-dimensional images at the adjacent next cross section position in the respiratory state is carried out, and the two-dimensional image with the maximum similarity at the first cross section position is extracted and used as the two-dimensional image corresponding to the respiratory state.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the four-dimensional ultrasound reconstruction method according to any of claims 1 to 5.
8. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein the computer instructions, when executed by a computer, implement the steps of the four-dimensional ultrasound reconstruction method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910510059.8A CN110279429B (en) | 2019-06-13 | 2019-06-13 | Four-dimensional ultrasonic reconstruction method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910510059.8A CN110279429B (en) | 2019-06-13 | 2019-06-13 | Four-dimensional ultrasonic reconstruction method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110279429A CN110279429A (en) | 2019-09-27 |
| CN110279429B true CN110279429B (en) | 2021-02-09 |
Family
ID=68004740
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910510059.8A Active CN110279429B (en) | 2019-06-13 | 2019-06-13 | Four-dimensional ultrasonic reconstruction method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110279429B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112089438B (en) * | 2020-08-31 | 2021-07-30 | 北京理工大学 | Four-dimensional reconstruction method and device based on two-dimensional ultrasonic image |
| CN116740219B (en) * | 2023-08-14 | 2024-01-09 | 之江实验室 | Three-dimensional photoacoustic tomography method, device, equipment and readable storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107822660A (en) * | 2016-07-26 | 2018-03-23 | 美国西门子医疗解决公司 | Generate method, ultrasonic system and the storage medium of ultrasonoscopy |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6966878B2 (en) * | 2003-08-28 | 2005-11-22 | Ge Medical Systems Global Technology Company, Llc | Method and apparatus for obtaining a volumetric scan of a periodically moving object |
| CN101529471B (en) * | 2006-10-17 | 2013-01-23 | 皇家飞利浦电子股份有限公司 | Four-dimensional reconstruction method and system of regions exhibiting multiple phases of periodic motion |
| DE102007009017B3 (en) * | 2007-02-23 | 2008-09-25 | Siemens Ag | Arrangement for supporting a percutaneous procedure |
| US8121669B2 (en) * | 2008-04-07 | 2012-02-21 | Mitsubishi Electric Research Laboratories, Inc. | Method for tracking soft tissue masses in images using directed graphs |
| US8852106B2 (en) * | 2009-04-13 | 2014-10-07 | Hitachi Aloka Medical, Ltd. | Ultrasound diagnostic apparatus |
-
2019
- 2019-06-13 CN CN201910510059.8A patent/CN110279429B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107822660A (en) * | 2016-07-26 | 2018-03-23 | 美国西门子医疗解决公司 | Generate method, ultrasonic system and the storage medium of ultrasonoscopy |
Non-Patent Citations (1)
| Title |
|---|
| Towards Real-time Registration of 4D Ultrasound Images;Pezhman Foroughi等;《Proceedings of the 28th IEEE EMBS Annual International Conference》;20060903;第404-407页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110279429A (en) | 2019-09-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9697639B2 (en) | Three-dimensional model data generation device, method and program | |
| Huang et al. | Bezier interpolation for 3-D freehand ultrasound | |
| CN109754394B (en) | Three-dimensional medical image processing device and method | |
| CN104520897B (en) | Picture noise reduces and/or image resolution ratio improves | |
| CN101084528B (en) | A method and a system for integration of medical diagnostic information and a geometric model of a movable body | |
| WO2007061931A2 (en) | Fully automatic vessel tree segmentation | |
| CN109523584A (en) | Image processing method, device, multi-mode imaging system, storage medium and equipment | |
| CA2619308A1 (en) | Method and apparatus for automatic 4d coronary modeling and motion vector field estimation | |
| WO2008091583A2 (en) | Image-based extraction for vascular trees | |
| CN110279429B (en) | Four-dimensional ultrasonic reconstruction method and device | |
| Raza et al. | Brain image representation and rendering: A survey | |
| CN110251231A (en) | The method and device that ultrasonic three-dimensional is rebuild | |
| CN114241077B (en) | CT image resolution optimization method and device | |
| CN109118554A (en) | The electrical impedance images method for reconstructing decomposed based on low-rank sparse | |
| CN114399594A (en) | Automatic curved surface reconstruction method for vessel wall image based on center line extraction | |
| KR100680232B1 (en) | Method for analyzing hippocampus for aiding diagnosis of brain diseases and the recording media therein readable by computer | |
| JPH09198490A (en) | Three-dimensional discrete data projector | |
| JP2021186430A (en) | Image processing device and image processing method | |
| Jin et al. | A new approach of arc skeletonization for tree-like objects using minimum cost path | |
| CN107170009B (en) | Medical image-based goggle base curve data measurement method | |
| Moraes et al. | 3D Lanczos interpolation for medical volumes | |
| EP1447772A1 (en) | A method of lung lobe segmentation and computer system | |
| US20140032180A1 (en) | Method and apparatus for computing deformation of an object | |
| CN110927644A (en) | Method for measuring mechanical asynchrony in left ventricle based on cardiac nuclear magnetic resonance imaging | |
| Cheng et al. | Sdct-gan: reconstructing CT from biplanar x-rays with self-driven generative adversarial networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |