CN116193231B - Method and system for handling minimally invasive surgical field anomalies - Google Patents
Method and system for handling minimally invasive surgical field anomalies Download PDFInfo
- Publication number
- CN116193231B CN116193231B CN202211298711.2A CN202211298711A CN116193231B CN 116193231 B CN116193231 B CN 116193231B CN 202211298711 A CN202211298711 A CN 202211298711A CN 116193231 B CN116193231 B CN 116193231B
- Authority
- CN
- China
- Prior art keywords
- image
- module
- minimally invasive
- video
- haze
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The invention discloses a method and a system for processing visual field abnormality of minimally invasive surgery, wherein the method comprises the steps of acquiring video images in real time; removing jitter of the video image; removing haze of the video image; correcting the brightness of the video image; outputting the repaired image and displaying the repaired image to a doctor; the system comprises an acquisition module, a shake removal module, a haze removal module, a correction module and a display module; video jitter, haze and brightness are corrected by accessing video data captured by the surgical lens, so that the definition and stability of a surgical field picture are improved; the problems of picture shaking, long-time lens fogging and picture darkness which cannot be processed by the current physical method can be solved, anxiety caused by visual fatigue of operators can be relieved, and the safety of the operation can be improved; and the long-time stable output of the operation field picture ensures that an operator does not need to repeatedly wipe and process the lens, thereby greatly reducing the times of the lens entering and exiting the body cavity and the inside and outside of the body cavity, obviously shortening the operation time and improving the operation fluency.
Description
Technical Field
The invention relates to the field of artificial intelligence, in particular to a method and a system for processing visual field abnormality of minimally invasive surgery.
Background
Currently, research in the field of minimally invasive endoscopic surgery is mostly focused on analyzing objects in surgical videos, such as anatomical structures, surgical instruments, and the like. However, there is little concern about the presentation of the surgical field during surgery. In the minimally invasive surgery process, the surgical field of a doctor of a main knife almost completely depends on the picture presented on a display through a lens controlled by a handrail, however, due to long surgical process, the shaking of the handrail under fatigue, improper light adjustment and inaccurate focusing, and the fog under the surgical field caused by long-term surgical operation, the above factors cause poor surgical field exposure, visual fatigue of operators, serious restriction on smooth operation, prolonged surgical time, and even increased surgical collateral damage and relay-open surgery rate.
The clear exposure of the surgical field is one of the necessary conditions to ensure the successful operation and reduce the surgical trauma. The existing lens defogging technology comprises preheating a lens, smearing an antifogging agent on the lens, wiping the lens with iodophor, cleaning the lens with carbon dioxide, sucking negative pressure for defogging and the like, however, only by various physical methods, the lens can be prevented from fogging in a short time, secondary or repeated treatment is required in the follow-up process, the operation steps are complicated, anxiety is brought to operators, and the operation safety is reduced.
Disclosure of Invention
The invention aims to solve the problems and designs a method and a system for processing visual field abnormality of minimally invasive surgery.
The invention realizes the above purpose through the following technical scheme:
a method for treating a minimally invasive surgical field abnormality, comprising:
s1, acquiring a video image of a minimally invasive surgery to be processed in real time;
s2, removing jitter of the video image by adopting a visual field stable model;
s3, removing haze of the video image by using a visual field defogging model;
s4, correcting the brightness of the video image by using an automatic brightness contrast adjustment algorithm;
s5, outputting the repaired image and displaying the repaired image to a doctor.
A system for handling minimally invasive surgical field abnormalities, implementing the method for handling minimally invasive surgical field abnormalities described above, comprising:
an acquisition module; the acquisition module is used for acquiring video data in the abdominal cavity when the minimally invasive surgery is performed in real time;
a jitter removal module for removing jitter of video data; the data signal output end of the acquisition module is connected with the data signal input end of the jitter removal module;
the haze removing module is used for removing haze of the video data; the data signal output end of the shake removing module is connected with the data signal input end of the haze removing module;
the correction module is used for correcting the brightness of the video data; the data signal output end of the haze removing module is connected with the data signal input end of the correcting module;
the display module is used for displaying the repaired video image; the data signal output end of the correction module is connected with the data signal input end of the display module.
An apparatus for treating a minimally invasive surgical field abnormality, comprising:
a reservoir; the memory is used for storing a computer program;
a processor; the processor is configured to execute the computer program, and when the computer program in the storage is executed by the processor, the method for handling minimally invasive surgery visual field anomalies as described above is implemented.
The invention has the beneficial effects that: by accessing video data captured by the surgical lens and correcting video jitter, haze and brightness, the definition and stability of a surgical field picture are improved; the method can solve the problems of shaking of pictures, long-time lens fogging and dark pictures which cannot be processed by the current physical method, is beneficial to relieving anxiety caused by visual fatigue of operators and improves operation safety. In addition, the long-time stable output of the operation field picture ensures that an operator does not need to repeatedly wipe and process the lens, thereby greatly reducing the times of the lens entering and exiting the body cavity and the inside and outside of the body cavity, obviously shortening the operation time and improving the operation fluency.
Drawings
FIG. 1 is a schematic flow chart for handling minimally invasive surgical field anomalies in accordance with the present invention;
FIG. 2 is a schematic diagram of the present invention for removing video image jitter in minimally invasive surgical field anomalies;
FIG. 3 is a schematic flow chart of the method for removing haze from video images in the minimally invasive surgery visual field abnormality;
fig. 4 is a cumulative gaussian distribution diagram used in the present invention to treat minimally invasive surgical field abnormalities.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It will be apparent that the described embodiments are some, but not all, embodiments of the invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
In the description of the present invention, it should be understood that the directions or positional relationships indicated by the terms "upper", "lower", "inner", "outer", "left", "right", etc. are based on the directions or positional relationships shown in the drawings, or the directions or positional relationships conventionally put in place when the inventive product is used, or the directions or positional relationships conventionally understood by those skilled in the art are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the apparatus or elements referred to must have a specific direction, be configured and operated in a specific direction, and therefore should not be construed as limiting the present invention.
Furthermore, the terms "first," "second," and the like, are used merely to distinguish between descriptions and should not be construed as indicating or implying relative importance.
In the description of the present invention, it should also be noted that, unless explicitly specified and limited otherwise, terms such as "disposed," "connected," and the like are to be construed broadly, and for example, "connected" may be either fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.
The following describes specific embodiments of the present invention in detail with reference to the drawings.
A method for treating a minimally invasive surgical field abnormality, as shown in fig. 1, comprising:
s1, acquiring a video image of the minimally invasive surgery to be processed in real time.
S2, removing jitter of the video image by adopting a visual field stable model; the method specifically comprises the following steps:
s21, extracting each source frame of the video imageAnd encoded as its characteristic->；
S22, using a given flow fieldDistortion characteristics->；
S23, utilizing a fusion method based on CNN neural network to characterize the distortionFusion to fusion characteristics->The fusion modes include mean fusion, gaussian weighted fusion, maximum value sorting fusion and stream error weighted fusion, wherein the mean fusion is as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,for the warped feature map ++>K is a warped mask t For features and masks to be warped and aligned with the target frame, n s For the nth frame, τ is a very small constant.
S24, for each frame, its distortion characteristicsFusion characteristics->Cascading is used as an input of a frame generator;
s25, the frame generator obtains the presented target frameAnd its corresponding confidence map->Wherein the frame generator we do this using a common UNet;
s26, calculating residual errors and adding back generated images because high-frequency details may be lost by using a decoder to draw (warp) image features, calculating residual errors and transmitting the residual errors to a target frame to recover the lost detailsAs shown in fig. 2, specifically:
1) Features to be extractedReconstructing the frame as an input to a frame generator;
2) Reconstructing a frame from an input imageSubtracting to obtain the rest->；
3) Using flow fieldsWarping the residual to obtain a warped residual +.>；
4) Residual error to be distortedAdd back the generated image->；
S27, end useFor->Weighted presentation output stable frame +.>The weighting method is as follows:
wherein the method comprises the steps ofFor inputting an image +.>For pre-distortion feature map->For a given flow field +.>For the warped feature map ++>For feature map after fusing CNN feature, < +.>For rendered image +.>For the confidence map corresponding to the rendered image, < ->For reconstructing the difference between the image and the input image, +.>For flow distortion->And (5) a subsequent image.
S3, removing haze of the video image by using a visual field defogging model; as shown in fig. 3, the method specifically includes:
s31, acquiring a fog graph modeling model, wherein the fog graph modeling model is expressed as I (x) =J (x) t (x) +A (1-t (x)), wherein I (x) is an atomized image, J (x) is a restored image, t (x) is transmissivity, A is global atmospheric light, J (x) t (x) is called direct attenuation, and A (1-t (x)) represents atmospheric light;
s32, transforming the fog pattern into a modelWhere c denotes a certain channel of the color image, Ω denotes a window area in the image, I c For atomizing the value of a channel in RGB of the figure, J c To restore the value of a channel in RGB of the graph;
s33, combining the prior of the dark channel to obtain；
S34, estimating the transmittance, which is expressed as；
S35, calculating a restored image, which is expressed as。
S4, correcting the brightness of the video image by using an automatic brightness contrast adjustment algorithm; the method specifically comprises the following steps:
s41, converting an RGB color space image of the video image with haze removed into an HSV color space image;
s42, extracting brightness channel image information;
s43, counting a Gaussian distribution diagram of the brightness channel values, and modifying the brightness channel values according to the accumulated Gaussian distribution, as shown in FIG. 4;
s44, original image brightness channel data are modified, and the original image brightness channel data are converted into RGB color space images.
S5, outputting the repaired image and displaying the repaired image to a doctor.
The intense shaking of the lens can cause dizziness to the person watching the video to different degrees, and the influence of the dizziness on the doctor of the doctor is huge during the operation. Electronic equipment often chooses to use electronic anti-shake (EIS) to achieve video stabilization, but such techniques compress our field of view size because it cuts the boundaries of the video to achieve video stabilization.
The method uses a frame synthesis algorithm to realize the full-frame video stabilization. A dense warped field is first estimated from neighboring frames and then stable frames are synthesized by fusing the warped content. The core technical innovation is hybrid space fusion based on learning, and the method relieves the artifacts caused by inaccurate optical flow and rapid moving objects.
Haze, fog and smog are all phenomena generated due to atmospheric absorption and scattering, and the algorithm estimates the transmission rate and restores the scene radiance through an atmospheric scattering model by means of the depth map of a fog image, so that the fog is effectively removed from a single image; the defogging can obviously improve the visibility of the scene and correct the light color shift caused by air. Moreover, the haze-free image is visually more pleasing to the doctor of the main knife to handle; the method can realize clear visual field without any physical method in the operation process, so that the visual field in the operation is continuously clear and the frequency of repeatedly taking out the endoscope for wiping is reduced.
Too much exposure, severely excessive exposure; or too little exposure, the serious shortage of exposure can lead to the degradation of the image definition. The method uses histogram equalization to automatically adjust the problem that the exposure of an image is too high or too low, and as color distortion can be caused by respectively equalizing three channels of RGB, the system selects the brightness (V) in HSV to realize the effect of automatically adjusting the brightness of the image.
By accessing video data captured by the surgical lens and correcting video jitter, haze and brightness, the definition and stability of a surgical field picture are improved; the method can solve the problems of shaking of pictures, long-time lens fogging and dark pictures which cannot be processed by the current physical method, is beneficial to relieving anxiety caused by visual fatigue of operators and improves operation safety. In addition, the long-time stable output of the operation field picture ensures that an operator does not need to repeatedly wipe and process the lens, thereby greatly reducing the times of the lens entering and exiting the body cavity and the inside and outside of the body cavity, obviously shortening the operation time and improving the operation fluency.
A system for handling minimally invasive surgical field abnormalities, implementing a method for handling minimally invasive surgical field abnormalities as described above, comprising:
an acquisition module; the acquisition module is used for acquiring video data in the abdominal cavity when the minimally invasive surgery is performed in real time;
a jitter removal module for removing jitter of video data; the data signal output end of the acquisition module is connected with the data signal input end of the jitter removal module;
the haze removing module is used for removing haze of the video data; the data signal output end of the shake removing module is connected with the data signal input end of the haze removing module;
the correction module is used for correcting the brightness of the video data; the data signal output end of the haze removing module is connected with the data signal input end of the correcting module;
the display module is used for displaying the repaired video image; the data signal output end of the correction module is connected with the data signal input end of the display module.
An apparatus for treating a minimally invasive surgical field abnormality, comprising:
a reservoir; the memory is used for storing a computer program;
a processor; the processor is configured to execute the computer program, and when the computer program in the storage is executed by the processor, the method for handling minimally invasive surgery visual field anomalies as described above is implemented.
The technical scheme of the invention is not limited to the specific embodiment, and all technical modifications made according to the technical scheme of the invention fall within the protection scope of the invention.
Claims (5)
1. A method for treating a minimally invasive surgical field abnormality, comprising:
s1, acquiring a video image of a minimally invasive surgery to be processed in real time;
s2, removing jitter of the video image by adopting a visual field stable model; the method specifically comprises the following steps:
s21, extracting each source frame of the video imageAnd encoded into its characteristics/>；
S22, using a given flow fieldDistortion characteristics->；
S23, utilizing a fusion method based on CNN neural network to characterize the distortionFusion to fusion characteristics->；
S24, for each frame, its distortion characteristicsFusion characteristics->Cascading is used as an input of a frame generator;
s25, the frame generator obtains the presented target frameAnd its corresponding confidence map->，/>A confidence map corresponding to the rendered image;
s26, calculating residual errors and adding back generated imagesThe method comprises the steps of carrying out a first treatment on the surface of the The method comprises the following steps:
1) Features to be extractedReconstructing the frame as an input to a frame generator;
2) Reconstructing a frame from an input imageSubtracting to obtain the rest +.>，/>For reconstructing a difference between the image and the input image;
3) Using flow fieldsWarping the residual error to obtain a warped residual error fatin>；
4) Residual error to be distortedAdd back the generated image->，/>Is a rendered image;
s27, useFor the addition of the residual error->Output stable frame for weighted presentation>，/>A confidence map corresponding to the rendered image;
s3, removing haze of the video image by using a visual field defogging model;
s4, correcting the brightness of the video image by using an automatic brightness contrast adjustment algorithm;
s5, outputting the repaired image and displaying the repaired image to a doctor.
2. The method for treating visual field abnormalities for minimally invasive surgery according to claim 1, characterized in that it comprises, in S3:
s31, acquiring a fog graph modeling model, wherein the fog graph modeling model is expressed as I (x) =J (x) t (x) +A (1-t (x)), wherein I (x) is an atomized image, J (x) is a restored image, t (x) is transmissivity, A is global atmospheric light, J (x) t (x) is called direct attenuation, and A (1-t (x)) represents atmospheric light;
s32, transforming the fog pattern into a modelWhere c denotes a certain channel of the color image, Ω denotes a window area in the image, I c For atomizing the value of a channel in RGB of the figure, J c To restore the value of a channel in RGB of the graph;
s33, combining the prior of the dark channel to obtain；
S34, estimating the transmittance, which is expressed as；
S35, calculating a restored image, which is expressed as。
3. The method for treating visual field abnormalities for minimally invasive surgery according to claim 1, characterized in that it comprises, in S4:
s41, converting an RGB color space image of the video image with haze removed into an HSV color space image;
s42, extracting brightness channel image information;
s43, counting a Gaussian distribution diagram of the brightness channel value, and modifying the brightness channel value according to the accumulated Gaussian distribution;
s44, original image brightness channel data are modified, and the original image brightness channel data are converted into RGB color space images.
4. A system for handling minimally invasive surgical field abnormalities, implementing a method for handling minimally invasive surgical field abnormalities as claimed in any of claims 1-3, comprising:
an acquisition module; the acquisition module is used for acquiring video data in the abdominal cavity when the minimally invasive surgery is performed in real time;
a jitter removal module for removing jitter of video data; the data signal output end of the acquisition module is connected with the data signal input end of the jitter removal module;
the haze removing module is used for removing haze of the video data; the data signal output end of the shake removing module is connected with the data signal input end of the haze removing module;
the correction module is used for correcting the brightness of the video data; the data signal output end of the haze removing module is connected with the data signal input end of the correcting module;
the display module is used for displaying the repaired video image; the data signal output end of the correction module is connected with the data signal input end of the display module.
5. A device for treating visual field abnormalities of minimally invasive surgery, comprising:
a reservoir; the memory is used for storing a computer program;
a processor; a processor for executing a computer program, the computer program in a memory being executed by the processor for implementing a method for handling minimally invasive surgical field abnormalities according to any of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211298711.2A CN116193231B (en) | 2022-10-24 | 2022-10-24 | Method and system for handling minimally invasive surgical field anomalies |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211298711.2A CN116193231B (en) | 2022-10-24 | 2022-10-24 | Method and system for handling minimally invasive surgical field anomalies |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116193231A CN116193231A (en) | 2023-05-30 |
| CN116193231B true CN116193231B (en) | 2023-07-18 |
Family
ID=86447688
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211298711.2A Active CN116193231B (en) | 2022-10-24 | 2022-10-24 | Method and system for handling minimally invasive surgical field anomalies |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116193231B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110191320A (en) * | 2019-05-29 | 2019-08-30 | 合肥学院 | Video jitter based on pixel timing motion analysis and freeze detection method and device |
| WO2021093718A1 (en) * | 2019-11-15 | 2021-05-20 | 北京金山云网络技术有限公司 | Video processing method, video repair method, apparatus and device |
| CN113163120A (en) * | 2021-04-21 | 2021-07-23 | 安徽清新互联信息科技有限公司 | Transformer-based video anti-shake method |
Family Cites Families (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012041492A1 (en) * | 2010-09-28 | 2012-04-05 | MAX-PLANCK-Gesellschaft zur Förderung der Wissenschaften e.V. | Method and device for recovering a digital image from a sequence of observed digital images |
| IN2015DN03877A (en) * | 2012-11-12 | 2015-10-02 | Behavioral Recognition Sys Inc | |
| CN103810725B (en) * | 2014-03-12 | 2016-06-08 | 北京理工大学 | A kind of video stabilizing method based on global optimization |
| EP3166468A4 (en) * | 2014-07-10 | 2018-03-07 | M.S.T. Medical Surgery Technologies Ltd. | Improved interface for laparoscopic surgeries - movement gestures |
| US20180068451A1 (en) * | 2016-09-08 | 2018-03-08 | Qualcomm Incorporated | Systems and methods for creating a cinemagraph |
| US10609284B2 (en) * | 2016-10-22 | 2020-03-31 | Microsoft Technology Licensing, Llc | Controlling generation of hyperlapse from wide-angled, panoramic videos |
| WO2018225346A1 (en) * | 2017-06-05 | 2018-12-13 | ソニー株式会社 | Medical system and control unit |
| CN110602487B (en) * | 2019-09-06 | 2021-04-20 | 高新兴科技集团股份有限公司 | Video image jitter detection method based on TSN (time delay network) |
| CN111524166B (en) * | 2020-04-22 | 2023-06-30 | 北京百度网讯科技有限公司 | Video frame processing method and device |
| US11367165B2 (en) * | 2020-05-19 | 2022-06-21 | Facebook Technologies, Llc. | Neural super-sampling for real-time rendering |
| CN113742527A (en) * | 2021-11-08 | 2021-12-03 | 成都与睿创新科技有限公司 | Method and system for retrieving and extracting operation video clips based on artificial intelligence |
| CN217365787U (en) * | 2021-11-25 | 2022-09-06 | 华中科技大学协和深圳医院 | Video real-time defogging enhancement device of endoscope |
| CN114881896B (en) * | 2022-07-12 | 2022-10-04 | 广东欧谱曼迪科技有限公司 | Endoscope image real-time defogging method and device, electronic equipment and storage medium |
-
2022
- 2022-10-24 CN CN202211298711.2A patent/CN116193231B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110191320A (en) * | 2019-05-29 | 2019-08-30 | 合肥学院 | Video jitter based on pixel timing motion analysis and freeze detection method and device |
| WO2021093718A1 (en) * | 2019-11-15 | 2021-05-20 | 北京金山云网络技术有限公司 | Video processing method, video repair method, apparatus and device |
| CN113163120A (en) * | 2021-04-21 | 2021-07-23 | 安徽清新互联信息科技有限公司 | Transformer-based video anti-shake method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116193231A (en) | 2023-05-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110148095B (en) | Underwater image enhancement method and enhancement device | |
| CN112419162B (en) | Image defogging method, device, electronic equipment and readable storage medium | |
| CN111292258B (en) | Image defogging method based on dark channel prior and bright channel prior | |
| Bolkar et al. | Deep smoke removal from minimally invasive surgery videos | |
| CN104899836B (en) | A kind of Misty Image intensifier and method based near infrared multispectral imaging | |
| US20160057343A1 (en) | Medical imaging system and method for obtaining medical image | |
| CN110910319B (en) | Operation video real-time defogging enhancement method based on atmospheric scattering model | |
| CN108717686A (en) | A kind of real-time video defogging method based on dark channel prior | |
| CN114066780B (en) | 4k endoscope image defogging method and device, electronic equipment and storage medium | |
| CN106846259A (en) | A kind of the real-time of laparoscopic surgery video frame rate self adaptation goes smog method | |
| CN110827221A (en) | Single image defogging method based on double-channel prior and side window guide filtering | |
| WO2023217046A1 (en) | Image processing method and apparatus, nonvolatile readable storage medium and electronic device | |
| CN116193231B (en) | Method and system for handling minimally invasive surgical field anomalies | |
| CN112465720B (en) | Image defogging method and device based on image sky segmentation and storage medium | |
| CN108093175A (en) | A kind of adaptive defogging method of real-time high-definition video and device | |
| CN115170417A (en) | Defogging method and system for neuroendoscope | |
| CN114584675B (en) | Self-adaptive video enhancement method and device | |
| CN113432723A (en) | Image processing method, system and computer system for weakening stray radiation | |
| CN112508797A (en) | System and method for real-time defogging in images | |
| Choi et al. | Self-supervised real-time video stabilization | |
| CN112465728B (en) | Video image processing method, system, electronic device and storage medium | |
| CN115205731A (en) | Video noise reduction device and method and terminal | |
| CN111738928B (en) | Endoscope defogging method based on probability optimization and neural network | |
| CN107194894A (en) | A kind of video defogging method and its system | |
| Yoo et al. | 7‐5: Synthetic Dataset for Improving UDC Video Restoration Network Performance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |