CN110336963A - A kind of dynamic image pro cess system and image processing method - Google Patents
A kind of dynamic image pro cess system and image processing method Download PDFInfo
- Publication number
- CN110336963A CN110336963A CN201910491736.6A CN201910491736A CN110336963A CN 110336963 A CN110336963 A CN 110336963A CN 201910491736 A CN201910491736 A CN 201910491736A CN 110336963 A CN110336963 A CN 110336963A
- Authority
- CN
- China
- Prior art keywords
- module
- register
- configuration
- image
- clock
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/40—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
- H04N25/42—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by switching between different modes of operation using different resolutions or aspect ratios, e.g. switching between interlaced and non-interlaced mode
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/50—Control of the SSIS exposure
- H04N25/57—Control of the dynamic range
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/70—SSIS architectures; Circuits associated therewith
- H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
- H04N25/77—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
- H04N25/772—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising A/D, V/T, V/F, I/T or I/F converters
Abstract
A kind of dynamic image pro cess system disclosed by the invention and image processing method, including imaging sensor interconnected and FPGA, imaging sensor includes register module and image generating module, and image generating module stores M kind register mode in register module for generating image；FPGA includes clock module, configuration module and receiving processing module, and receiving processing module is used to receive the image of image generating module transmission and handles it, and M kind clock module is stored in clock module, stores M kind configuration mode in configuration module；Wherein, M kind register mode, M kind clock module and M kind configuration mode correspond.A kind of dynamic image pro cess system provided by the invention and image processing method, by carrying out dynamic configuration to FPGA and imaging sensor, successively change the transmission frame per second of imaging sensor and the processing speed of FPGA in a very short period of time by the effect of configuration module in FPGA, to promote FPGA to the processing capacity of imaging sensor.
Description
Technical field
The present invention relates to field of image processings, and in particular to a kind of dynamic image pro cess system and image processing method.
Background technique
With the development of image processing techniques, imaging sensor using more and more, especially in industrial circle, machine view
The step of feel, is instead of pure artificial most of work；And the generally industrial camera that machine vision uses in industrial circle,
Industrial camera includes imaging sensor and image processing module, and in the prior art, image processing module generally uses FPGA
Processing is acquired to image, the image after being handled by FPGA again pass through interface (such as USB interface, CameraLink,
Gigabit ethernet interface etc.) output.
In existing image processing system, imaging sensor is kept fixed the output frame rate of FPGA, i.e. image sensing
Device is kept fixed frame per second and transmits image to FPGA, and FPGA is kept fixed rate and handles the image received；If transmitting
It needs to change transmission frame per second in the process, generallys use the following two kinds mode and handled:
(1) DDR buffer is configured in FPGA, when performing image processing, client sends frame request signal control figure
It is buffered in DDR buffer as the transmission frame per second of sensor change image, then by the image data under new transmission frame per second, then
FPGA handles the image data in DDR buffer.In above-mentioned treatment process, due to the transmission frame per second of imaging sensor
Changed, but FPGA is received and the rate of processing image is there is no changing, even if imaging sensor transmits image
Frame per second is lower, and the processing speed of FPGA is still high, the problems such as so as to cause FPGA power consumption is high, system stability is poor.
(2) the brilliant battle array for increasing adjustment clock outside imaging sensor, when which is used to adjust the work of imaging sensor
Clock, then brilliant battle array and FPGA are connected to control centre, adjusted respectively by control centre imaging sensor work clock and
The processing speed of FPGA is consistent the transmission rate of the two and processing speed.It is needed using the system of this processing mode
It is additional to increase brilliant battle array and control centre, the integrated level of entire processing system is reduced, and control mode is cumbersome, reduces image
The efficiency of processing.
Summary of the invention
The object of the present invention is to provide a kind of dynamic image pro cess system and image processing methods, by FPGA and image
Sensor carries out dynamic configuration, successively changes the biography of imaging sensor in a very short period of time by the effect of configuration module in FPGA
The processing speed of defeated frame per second and FPGA, to promote FPGA to the processing capacity of imaging sensor.
To achieve the goals above, the present invention adopts the following technical scheme: a kind of dynamic image pro cess system, including it is mutual
The imaging sensor and FPGA of connection, described image sensor include register module and image generating module, and described image produces
Raw module stores M kind register mode in the register module for generating image；
The FPGA includes clock module, configuration module and receiving processing module, and the receiving processing module is for receiving
The image of described image generation module transmission is simultaneously handled it, and the register module and the clock module are all connected with institute
Image generating module is stated, the register module and the clock module are all connected with configuration module, store M in the clock module
Clock module is planted, stores M kind configuration mode in the configuration module；Wherein, M kind register mode, M kind clock module and M
Kind configuration mode corresponds；Wherein, M is the integer more than or equal to 2；
Configuration module selects one of configuration mode in M kind configuration mode in the FPGA, and by the configuration mode
Successively be transmitted to the register module and the clock module so that register module and clock module successively select it is corresponding
Register mode and clock module；The register module and the clock module are by corresponding register mode and clock module
It is transmitted to described image generation module, described image generation module is made to carry out image transmitting according to corresponding clock module.
Further, the system also includes processing end, the processing end connects the configuration module, described for triggering
Configuration module selects one of configuration mode in M kind configuration mode.
Further, the configuration module connects the register module by SPI interface.
Further, described image generation module is connected with the receiving processing module by coffret, and the biography
Defeated interface includes data transmission interface, row selection interface and column selection interface.
A kind of image processing method provided by the invention, includes the following steps:
Configuration module selects one of configuration mode in M kind configuration mode in S01:FPGA, and by the configuration mode
It is successively transmitted to register module and clock module, so that register module and clock module successively select corresponding register mould
Formula and clock module；Wherein, the FPGA includes clock module, configuration module and receiving processing module, described image sensor
Including register module and image generating module；
S02: corresponding register mode and clock module are transmitted to by the register module and the clock module
Described image generation module, so that image generating module carries out image transmitting according to the clock module after selection.
Further, the configuration module connects processing end, and the configuration mould is triggered in processing end described in the step S01
Block selects one of configuration mode in M kind configuration mode.
Further, the processing end is the end PC.
Further, configuration module described in the step S01 sends SPI signal to the register module, controls it
Select corresponding register mode.
Further, image generating module described in the step S02 passes through coffret to the receiving processing module
Send image, wherein the coffret includes data transmission interface, row selection interface and column selection interface.
The invention has the benefit that configuration module and clock module is arranged in the present invention in FPGA, in imaging sensor
Middle setting register module, and the configuration mode stored in configuration module, the clock module that stores in clock module and deposit
The register mode stored in device module corresponds, and controls register module by configuration module and clock module successively selects
Corresponding mode, so that the transmission frame per second of imaging sensor and the processing speed of FPGA successively change in a very short period of time
Become, to promote FPGA to the processing capacity of imaging sensor；Since image processing system of the present invention can change image in real time
The transmission frame per second of sensor and the processing speed of FPGA, and keep the two rate consistent, FPGA only includes clock mould in the present invention
Block, configuration module and receiving processing module avoid the problem of needing to increase DDR buffer or brilliant battle array in the prior art, from
And entire processing system structure is simplified, moreover it is possible to ensure the real-time effectiveness of data processing.
Detailed description of the invention
Attached drawing 1 is a kind of structural schematic diagram of dynamic image pro cess system of the present invention.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, with reference to the accompanying drawing to specific reality of the invention
The mode of applying is described in further detail.
As shown in Fig. 1, a kind of dynamic image pro cess system provided by the invention, including imaging sensor interconnected
And FPGA, imaging sensor include register module and image generating module, image generating module is for generating image, register
M kind register mode is stored in module.Wherein, image sensing implement body can use ISA8201 chip.
FPGA includes clock module, configuration module and receiving processing module, and receiving processing module is for receiving image generation
The image of module transmission is simultaneously handled it, and register module and clock module are all connected with image generating module, register mould
Block and clock module are all connected with configuration module, M kind clock module are stored in clock module, storage M kind configures mould in configuration module
Formula；Wherein, M kind register mode, M kind clock module and M kind configuration mode correspond；Wherein, M is more than or equal to 2
Integer.For example, if when clock module configures tetra- kinds of clock modules of 10M, 20M, 30M and 40M, configuration module and register module
It include four kinds of configuration modes corresponding with above-mentioned four kinds of clock modules and four kinds of register modes, wherein 10M clock module table
Show 10,000,000 transmission rates per second.
As shown in Fig. 1, specific connection relationship are as follows: the register mould in configuration module connection imaging sensor in FPGA
Clock module in block and FPGA, register module and clock module are all connected with the image generating module in imaging sensor,
Image generating module connects receiving processing module in FPGA；Preferably, configuration module connects register module by SPI interface；
Preferably, image generating module is connected with receiving processing module by coffret, and coffret includes data transmission interface,
Row selection interface and column selection interface.System further includes processing end in the present invention, and processing end connects the configuration module in FPGA, is used
In the configuration mode of triggering configuration module, thus cause the change of the transmission frame per second of imaging sensor and the processing speed of FPGA,
Specific processing end can be PC etc..
When system starts to carry out image procossing in the present invention, configuration module selects it in M kind configuration mode in FPGA
A kind of middle configuration mode, and the configuration mode after selection is successively transmitted to register module and clock module, so that register
Module and clock module successively select corresponding register mode and clock module；Register module is by the register after selection
Mode is transmitted to image generating module, and the clock module after selection is transmitted to image generating module, made by subsequent clock module
It obtains image generating module and carries out image transmitting according to the clock module after selection.
When needing to change transmission rate of the imaging sensor to FPGA, configuration module is in M in configuration mode in FPGA
Another configuration mode is switched to, and the configuration mode after switching is successively transmitted to register module and clock module, make
It obtains register module and clock module successively switches corresponding register mode and clock module；Register module and clock module
Be connected to image generating module, and by after switching register mode and clock module be transmitted to image generating module, make
It obtains image generating module and carries out image transmitting according to the clock module after switching.
It is worth noting that in the present invention in FPGA after configuration module handover configurations mode, since it connects clock mould
Block and register module, but it acts on the time on clock module and register module with sequencing, the present invention is true
Configuration module after protecting selection or switching first acts on register module, that is, ensures that configuration module first configures image sensing
Device, then FPGA is configured；This is because: the processing speed that the transmission rate for first adjusting imaging sensor adjusts FPGA again can
To ensure that adjustment process steadily carries out, it is if the processing speed for first adjusting FPGA adjusts imaging sensor transmission rate again and will cause
System disorder.Specifically, configuration module can complete the successive effect of register module and clock module in the following way: matching
It sets module and first chooses a kind of configuration mode in M kind configuration mode, first the configuration mode is transmitted in register module, so that
Register module selects corresponding register mode, then the configuration mode is transmitted in clock module, so that clock module selects
Select corresponding clock module.
A kind of image processing method provided by the invention, includes the following steps:
S01: when system starts to carry out image procossing in the present invention, configuration module is matched in M kind in processing end triggering FPGA
It sets and selects one of configuration mode (the first configuration mode) in mode, and the configuration mode after selection is successively transmitted to deposit
Device module and clock module, so that register module and clock module successively select corresponding register mode (the first register
Mode) and clock module (the first clock module)；Specifically, configuration module sends SPI signal to register module, its choosing is controlled
Select corresponding register mode.Wherein, processing end can be the end PC.
S02: register module and clock module are connected to image generating module, and by the register mode after selection
(the first register mode) and clock module (the first clock module) are transmitted to image generating module, so that image generating module is pressed
Image transmitting is carried out according to the clock module (the first clock module) of selection.Specifically, image generating module by coffret to
Receiving processing module sends image, wherein coffret includes data transmission interface, row selection interface and column selection interface.
S03: when transmission rate of the imaging sensor to FPGA needs to change, configuration module exists in processing end triggering FPGA
Switch another configuration mode (the second configuration mode) in M kind configuration mode, and the configuration mode after switching is successively transmitted
To register module and clock module, so that register module and clock module successively switch corresponding register mode (second
Register mode) and clock module (second clock mode)；Specifically, configuration module sends SPI signal, control to register module
It makes it and switches corresponding register mode.
S04: register module and clock module are connected to image generating module, and by the register mode after switching
(the second register mode) and clock module (second clock mode) are transmitted to image generating module, so that image generating module is pressed
Image transmitting is carried out according to the clock module (second clock mode) after switching.Specifically, image generating module passes through coffret
Image is sent to receiving processing module, wherein coffret includes data transmission interface, row selection interface and column selection interface.
Above-mentioned first clock module, the first register mode and the first configuration mode are corresponding mode, second clock
Mode, the second register mode and the second configuration mode are corresponding mode, and only a kind of naming method, this field skill
Art personnel can carry out any name to above-mentioned corresponding mode.
Configuration module and clock module is arranged in the present invention in FPGA, and register module is arranged in the image sensor, and
And the register stored in the configuration mode stored in configuration module, the clock module and register module that store in clock module
Mode corresponds, and controls register module by configuration module and clock module successively selects corresponding mode, so that
The transmission frame per second of imaging sensor and the processing speed of FPGA successively change in a very short period of time, to promote FPGA to figure
As the processing capacity of sensor；Due to image processing system of the present invention can change in real time imaging sensor transmission frame per second and
The processing speed of FPGA, and keep the two rate consistent, FPGA only includes clock module, configuration module and receiving area in the present invention
Module is managed, the problem of needing to increase DDR buffer or brilliant battle array in the prior art is avoided, so that entire processing system knot
Structure is simplified, moreover it is possible to ensure the real-time effectiveness of data processing.
The above description is only a preferred embodiment of the present invention, and the embodiment is not intended to limit protection model of the invention
It encloses, therefore all with the variation of equivalent structure made by specification and accompanying drawing content of the invention, similarly should be included in this hair
In the protection scope of bright appended claims.
Claims (9)
1. a kind of dynamic image pro cess system, which is characterized in that including imaging sensor interconnected and FPGA, described image
Sensor includes register module and image generating module, and described image generation module is for generating image, the register mould
M kind register mode is stored in block；
The FPGA includes clock module, configuration module and receiving processing module, and the receiving processing module connects described image
Generation module, for receiving the image of described image generation module transmission and handling it, the register module and institute
It states clock module and is all connected with described image generation module, the register module and the clock module are all connected with configuration module,
M kind clock module is stored in the clock module, stores M kind configuration mode in the configuration module；Wherein, M kind register mould
Formula, M kind clock module and M kind configuration mode correspond；Wherein, M is the integer more than or equal to 2；
Configuration module selects one of configuration mode in M kind configuration mode in the FPGA, and successively by the configuration mode
It is transmitted to the register module and clock module, so that register module and clock module successively select corresponding register mould
Formula and clock module；Corresponding register mode and clock module are transmitted to the figure by the register module and clock module
As generation module, so that image generating module carries out image transmitting according to corresponding clock module.
2. a kind of dynamic image pro cess system according to claim 1, which is characterized in that the system also includes processing
End, the processing end connect the configuration module, select one of which in M kind configuration mode for triggering the configuration module
Configuration mode.
3. a kind of dynamic image pro cess system according to claim 1, which is characterized in that the configuration module passes through SPI
Interface connects the register module.
4. a kind of dynamic image pro cess system according to claim 1, which is characterized in that described image generation module and connect
It receives processing module to connect by coffret, and the coffret includes data transmission interface, row selection interface and column selection
Interface.
5. a kind of image processing method, which comprises the steps of:
Configuration module selects one of configuration mode in M kind configuration mode in S01:FPGA, and successively by the configuration mode
It is transmitted to register module and clock module, so that register module and the clock module successively select corresponding register mould
Formula and clock module；Wherein, the FPGA includes clock module, configuration module and receiving processing module, described image sensor
Including register module and image generating module；
S02: corresponding register mode and clock module are transmitted to described image and produced by the register module and clock module
Raw module, so that image generating module carries out image transmitting according to the clock module after selection.
6. a kind of image processing method according to claim 5, which is characterized in that the configuration module connects processing end,
The configuration module is triggered in the step S01 and selects one of configuration mode in M kind configuration mode in the processing end.
7. a kind of image processing method according to claim 6, which is characterized in that the processing end is the end PC.
8. a kind of image processing method according to claim 5, which is characterized in that the configuration module is in the step
SPI signal is sent to the register module in S01, it is controlled and selects corresponding register mode.
9. a kind of image processing method according to claim 5, which is characterized in that described image generation module is in the step
Image is sent to the receiving processing module by coffret in rapid S02, wherein the coffret includes that data transmission connects
Mouthful, row selection interface and column selection interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910491736.6A CN110336963B (en) | 2019-06-06 | 2019-06-06 | Dynamic image processing system and image processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910491736.6A CN110336963B (en) | 2019-06-06 | 2019-06-06 | Dynamic image processing system and image processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110336963A true CN110336963A (en) | 2019-10-15 |
| CN110336963B CN110336963B (en) | 2022-03-29 |
Family
ID=68140921
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910491736.6A Active CN110336963B (en) | 2019-06-06 | 2019-06-06 | Dynamic image processing system and image processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110336963B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113163113A (en) * | 2021-03-29 | 2021-07-23 | 广州安凯微电子股份有限公司 | Image sensor frame rate and exposure control system and method |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6947070B2 (en) * | 2001-05-21 | 2005-09-20 | Pentax Corporation | Video scope utilized in electronic endoscope system |
| CN102055894A (en) * | 2010-11-23 | 2011-05-11 | 无锡市博阳精密机械制造有限公司 | Modularized CCD industrial camera |
| CN102082901A (en) * | 2009-12-01 | 2011-06-01 | 株式会社理光 | Imaging apparatus |
| CN102665049A (en) * | 2012-03-29 | 2012-09-12 | 中国科学院半导体研究所 | Programmable visual chip-based visual image processing system |
| CN103647901A (en) * | 2013-12-24 | 2014-03-19 | 中国科学院半导体研究所 | Low-cost image data collection transmission system free of external storage and based on field programmable gate array (FPGA) |
| CN104427232A (en) * | 2013-08-30 | 2015-03-18 | 安凯(广州)微电子技术有限公司 | Method, device and system for realizing image acquisition |
| WO2016018653A1 (en) * | 2014-07-31 | 2016-02-04 | Apple Inc. | Sensor data rescaler for image signal processing |
| CN206195934U (en) * | 2016-11-08 | 2017-05-24 | 中国科学院西安光学精密机械研究所 | High -speed CMOS camera imaging system |
-
2019
- 2019-06-06 CN CN201910491736.6A patent/CN110336963B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6947070B2 (en) * | 2001-05-21 | 2005-09-20 | Pentax Corporation | Video scope utilized in electronic endoscope system |
| CN102082901A (en) * | 2009-12-01 | 2011-06-01 | 株式会社理光 | Imaging apparatus |
| CN102055894A (en) * | 2010-11-23 | 2011-05-11 | 无锡市博阳精密机械制造有限公司 | Modularized CCD industrial camera |
| CN102665049A (en) * | 2012-03-29 | 2012-09-12 | 中国科学院半导体研究所 | Programmable visual chip-based visual image processing system |
| CN104427232A (en) * | 2013-08-30 | 2015-03-18 | 安凯(广州)微电子技术有限公司 | Method, device and system for realizing image acquisition |
| CN103647901A (en) * | 2013-12-24 | 2014-03-19 | 中国科学院半导体研究所 | Low-cost image data collection transmission system free of external storage and based on field programmable gate array (FPGA) |
| WO2016018653A1 (en) * | 2014-07-31 | 2016-02-04 | Apple Inc. | Sensor data rescaler for image signal processing |
| CN206195934U (en) * | 2016-11-08 | 2017-05-24 | 中国科学院西安光学精密机械研究所 | High -speed CMOS camera imaging system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113163113A (en) * | 2021-03-29 | 2021-07-23 | 广州安凯微电子股份有限公司 | Image sensor frame rate and exposure control system and method |
| CN113163113B (en) * | 2021-03-29 | 2022-06-07 | 广州安凯微电子股份有限公司 | Image sensor frame rate and exposure control system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110336963B (en) | 2022-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100586403C (en) | Visual sense prosthesis image processing device and method | |
| Conradt et al. | An embedded AER dynamic vision sensor for low-latency pole balancing | |
| CN106412403A (en) | 3D camera module and 3D camera device | |
| DE10041085A1 (en) | Portable orientation system for blind person has sensor unit coupled to signal processing device converting visual information into output signals for different sense | |
| CN112311989B (en) | High-speed imaging and transmission system | |
| WO2003036565A3 (en) | System and method for obtaining video of multiple moving fixation points within a dynamic scene | |
| CN110336963A (en) | A kind of dynamic image pro cess system and image processing method | |
| CN106375642B (en) | Image acquisition and processing device and object of which movement image capturing system | |
| CN104820418A (en) | Embedded vision system for mechanical arm and method of use | |
| CN105165255B (en) | A kind of intelligent cotton picker system based on computer vision technique | |
| CN106412557A (en) | 3D camera control method and 3D camera control device | |
| CN102528821A (en) | Bionic multi-vision physical platform based on multiple independent holders | |
| CN106921857A (en) | Three-dimensional display system and stereo display method | |
| CN102547237B (en) | Dynamic monitoring system based on multiple image acquisition devices | |
| CN109106567A (en) | A kind of more sighting distance display systems for myopia | |
| CN105117690A (en) | Finger vein identification control system based on infrared detection circuit | |
| CN101159126B (en) | Method for stabilizing handhold equipment image and device | |
| CN202078749U (en) | Remote controller and system capable of controlling toy | |
| CN109603011A (en) | A kind of photon surveys skin skin beautifying apparatus and its processing method | |
| CN109286749A (en) | High bandwidth Image Acquisition, pretreatment and dissemination system and image processing system | |
| CN103209287A (en) | Photographing method of regulation and control camera lens, special camera and special glasses frame | |
| KR102290328B1 (en) | Video output device | |
| CN107485840A (en) | A kind of ten thousand frame high-speed camera electric chronographs and its method of work | |
| CN207168999U (en) | A kind of ten thousand frame high-speed camera electric chronographs | |
| CN110781871A (en) | Iris recognition device and recognition method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |