CN109194923B - Video image processing device, system and method based on local non-uniform resolution - Google Patents
Video image processing device, system and method based on local non-uniform resolution Download PDFInfo
- Publication number
- CN109194923B CN109194923B CN201811212816.5A CN201811212816A CN109194923B CN 109194923 B CN109194923 B CN 109194923B CN 201811212816 A CN201811212816 A CN 201811212816A CN 109194923 B CN109194923 B CN 109194923B
- Authority
- CN
- China
- Prior art keywords
- resolution
- region
- video image
- interest
- mapping
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 59
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000013507 mapping Methods 0.000 claims abstract description 92
- 238000012544 monitoring process Methods 0.000 claims abstract description 47
- 230000005540 biological transmission Effects 0.000 claims abstract description 19
- 230000006835 compression Effects 0.000 claims abstract description 12
- 238000007906 compression Methods 0.000 claims abstract description 12
- 230000007704 transition Effects 0.000 claims description 74
- 230000009466 transformation Effects 0.000 claims description 19
- 230000006872 improvement Effects 0.000 claims description 16
- 230000009467 reduction Effects 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 11
- 238000011426 transformation method Methods 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 9
- 238000003672 processing method Methods 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 238000004458 analytical method Methods 0.000 abstract description 4
- 230000000007 visual effect Effects 0.000 abstract description 4
- 238000001514 detection method Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000033001 locomotion Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 241000699670 Mus sp. Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000009365 direct transmission Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005549 size reduction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440245—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/47205—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/4728—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Image Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention discloses video image processing equipment, system and method based on local non-uniform resolution, and relates to the technical field of image processing. The video image processing apparatus includes: a memory to store data and instructions; a processor capable of determining at least one region of interest in the surveillance video image; and performing non-uniform resolution mapping processing on the monitoring video image to obtain a small-size video image before video encoding or transmission, wherein more resolution is allocated to the region of interest, and the rectangular shape and size of each frame of video image after mapping are maintained for encoding compression. The invention can ensure the monitoring visual field range and improve the local video resolution of the warning area during video analysis, playback and preview under the limit of the existing bandwidth, hardware capability and video storage cost.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to an image processing technology based on local non-uniform resolution.
Background
In the use of video monitoring systems, users often want to have a camera to monitor a sufficiently wide field of view, and a monitor screen to display a large number of areas, and also want to see local details clearly, which requires that the monitoring system can support as large an image resolution as possible. However, the image resolution actually used by the video monitoring system is limited by the transmission bandwidth, the hardware processing capability, the video storage cost and the like of the video system, and the maximum resolution of the monitoring camera cannot be used, but only a smaller resolution can be used. Thus, if video resolution is adaptively controlled according to user requirements and system performance limitations, it will help to improve the performance of the monitoring system.
Various resolution self-adaptive control methods are proposed in the existing monitoring technology research. Such as: in order to adapt to different transmission bandwidths and hardware capabilities, some surveillance video systems adopt a method of simultaneously encoding and storing multiple resolution code streams (a main code stream and a secondary code stream), and select a code stream with a certain resolution for playback according to the hardware playback capability. The method is suitable for the situation that the hardware processing capacity and the video transmission bandwidth of the video recording system have a certain margin compared with the processing of the independent main code stream. In some monitoring systems, in order to improve the definition of areas such as faces, areas of interest such as faces are analyzed and identified by adopting an artificial intelligence algorithm, and a video encoder is controlled to allocate more code rates to the areas so as to reduce quality loss of the areas caused by video compression. But this method has limited effect if the resolution of the original input image is not high. Some monitoring applications compress and transmit uncorrected fisheye lens pictures to maintain high resolution in the picture center, but because of picture distortion, video compression efficiency is affected, direct display of distorted fisheye video also affects the relationship between object recognition and distance judgment of users, and a monitoring sensitive area is not necessarily located in the picture center, so that application limitation is more.
Currently, in monitoring applications, perimeter protection (setting a virtual guard line or a virtual guard zone on a video background screen by a user) is a common application scenario. In one application, the user needs a wide visual field range of the picture, can contain the whole range of one or more areas needing to be alerted, and leaves enough observation areas outside the alerted areas so as to observe the process of people or objects entering or exiting the alerted areas; on the other hand, the user has different requirements on the definition of different areas of the monitoring picture, and the interest of the local warning area is far higher than that of other areas. Particularly, when an event occurs in the warning area, the user has a strong need to preview or replay the picture of the warning area after amplifying, but if the original resolution of the monitoring video is low, the details cannot be seen even if the image is amplified.
Disclosure of Invention
The invention aims at: the defects of the prior art are overcome, and video image processing equipment, system and method based on local non-uniform resolution are provided. The monitoring video image processing equipment with the local non-uniform resolution provided by the invention can ensure the monitoring visual field range and promote the local video resolution of the guard zone during video analysis, playback and preview under the limit of the existing bandwidth, hardware capability and video storage cost.
In order to achieve the above object, the present invention provides the following technical solutions:
the present invention provides a video image processing apparatus based on local non-uniform resolution, comprising:
a memory to store data and instructions;
a processor, when executing the instructions, for,
acquiring a monitoring video image;
determining at least one region of interest in the surveillance video image;
and performing non-uniform resolution mapping processing on the monitoring video image to obtain a small-size video image before video encoding or transmission, wherein more resolution is allocated to the region of interest, and the rectangular shape and size of each frame of video image after mapping are maintained for encoding compression.
Further, the processor is configured to, upon video playback or preview,
and performing resolution de-mapping processing to recover the image proportion relation when the small-size video image is reversely mapped back to the large-size display image, and outputting a high-resolution image when the region of interest is locally enlarged.
Further, the processor includes a region of interest setting module for,
generating at least one region of interest on the video image based on the manual input;
and/or automatically generating at least one region of interest on the video image based on the captured scene features;
and/or editing the region of interest to reconfigure the region of interest of the video image, the editing including adjusting a range of the region of interest, adding new regions of interest, and/or deleting existing regions of interest.
Further, the processor is configured, when determining at least one region of interest,
setting a local resolution improvement multiple m of a region of interest;
setting a resolution transition region around the region of interest, wherein the resolutions of pixels outside the region of interest and the transition region keep the low resolution set by the original system to form a low resolution region, and the low resolution region is not influenced by resolution mapping;
the local resolution improvement multiple m is smaller than the minimum scale ratio of the transition region to the corresponding region of interest in each direction.
Further, the processor is configured, in generating the small-size video image,
different resolution mapping modes are used for reducing different areas of the video image, the concerned area is scaled by using a reduction scale k as a high resolution area, the low resolution area is scaled by using a reduction scale L, and k=l/m.
Further, the mapping mode of the transition region adopts a transmission transformation method or a polynomial projection transformation method.
The invention also provides a video image processing system based on the local non-uniform resolution, which comprises:
the monitoring camera is used for acquiring an original video image with high resolution;
an image processing apparatus for performing a local non-uniform resolution mapping process on the aforementioned original video image, the image processing apparatus being the apparatus of any one of claims 1 to 6;
and the monitoring display is used for displaying the video image processed by the image processing equipment.
The invention also provides a video image processing method based on the local non-uniform resolution, which comprises the following steps:
acquiring a monitoring video image;
determining at least one region of interest in the surveillance video image;
before video coding or transmission, carrying out non-uniform resolution mapping processing on an original monitoring video image to obtain a small-size video image, wherein more resolution is allocated to the concerned region, and the rectangular shape and size of each frame of mapped video image are maintained;
and carrying out coding compression on the video image after the non-uniform resolution mapping processing.
Further, the step of performing the non-uniform resolution mapping process on the original surveillance video image to obtain a small-sized video image includes,
setting a local resolution improvement multiple m of a region of interest, setting a resolution transition region around the region of interest, wherein the resolutions of the pixels outside the region of interest and the transition region keep the low resolution set by an original system to form a low resolution region, and the local resolution improvement multiple m is smaller than the minimum value of the scale ratio of the transition region and the corresponding region of interest in all directions;
the method comprises the steps of reducing different areas of a video image by using different resolution mapping modes, wherein the concerned area is scaled by using a reduction scale k as a high resolution area, and the low resolution area is scaled by using a reduction scale L, wherein k=L/m; the mapping mode of the transition region adopts a transmission transformation method or a polynomial projection transformation method.
The invention also provides a non-uniform resolution mapping method based on perspective transformation, which comprises the following steps:
step 1, setting a high resolution area and a transition area boundary on an original image; the high-resolution area and the resolution transition area are in the shape of any polygon, the two polygons have the same vertex number n, and the corresponding relation between the vertex coordinates of the two n-sided polygons and the end points of each side on the image is recorded;
step 2, reordering the polygon vertexes of the regional boundary clockwise; firstly, selecting a vertex A with the highest vertical position from vertexes of a boundary polygon of a transition area, selecting a vertex with the horizontal position at the leftmost position when a plurality of vertexes with the same height exist, and sequencing the vertexes of the boundary polygon of the transition area and the boundary polygon of a high resolution area in a clockwise direction according to the connection relation of edges from the point A;
step 3, pairing the vertexes of the transition area and the high-resolution area in pairs;
a. calculating whether each vertex of the boundary of the high-resolution area meets the matable condition for the vertex A of the boundary of the transition area;
b. selecting a high-resolution area vertex and the point A from the pairing candidate point set of the point A, and detecting whether the pairing points corresponding to the vertexes of the transition area meet the pairing-available condition;
when the trial matching vertex which does not meet the matching condition exists, the trial matching fails, and the vertex of the high-resolution area is removed from the matching candidate point set of the point A; otherwise, the trial matching is successful, and the distance sum of each matching point of the two polygons is used as the matching cost value of the point;
c. b, circulating until the trial matching of all candidate points in the pairing candidate point set of the point A and the point A is completed;
d. comparing the matching cost value of each point successfully matched with the point A, selecting the candidate point with the minimum cost as the final matching point, and completing the matching of the vertex of the high-resolution area and each vertex of the transition area in the clockwise direction;
step 4, calculating the coordinates of each vertex on the target mapping chart;
step 5, dividing the transition area into a plurality of mapping quadrilaterals;
step 6, solving quadrilateral mapping and reflection matrix; and solving a unique mapping matrix and a reflection matrix for each group of corresponding mapping quadrangles according to the perspective transformation formula.
Further, the method also comprises the following steps:
mapping and shrinking the high-resolution image to obtain a small-size video image, and/or performing reflection amplifying treatment on the small-size video image during video playback or previewing;
in the processing process, when the target pixel point is positioned in a high resolution area or a low resolution area, mapping processing is carried out according to the corresponding scaling factor; when the target pixel point is positioned in the resolution transition area, judging which perspective transformation mapping quadrangle the target pixel point is positioned in, and then mapping by applying the corresponding perspective transformation relation.
Compared with the prior art, the invention has the following advantages and positive effects by taking the technical scheme as an example:
1) According to the invention, the non-uniform resolution information can be transmitted on the reduced video image according to different requirements of users on different areas of the picture. Even if the original camera image is transmitted and encoded in a reduced size due to bandwidth, hardware performance, storage cost and the like, the high-resolution information of the warning area set by the user can be maintained, the allowable view angle range of the monitoring picture can be enlarged, the accuracy of algorithms such as out-of-range detection/area intrusion detection and the like can be improved, and the user can conveniently observe the object details in an amplified mode during playback.
2) According to the method for dividing the monitoring video image into the three areas of the high resolution area, the low resolution area and the resolution transition area according to the local characteristics, the video coding compression rate reduction caused by frequent horizontal movement area image distortion can be reduced because the high resolution area image and the low resolution area image are not distorted, and the storage cost of the monitoring video is reduced. This is more suitable for encoding compression than the existing direct transmission of fisheye lens images to ensure high resolution in the center of the image.
3) The mapping method of the resolution transition region based on perspective transformation, disclosed by the invention, has the characteristics of adaptation to various transition region shapes, gentle resolution change, low cost and easiness in realization, is beneficial to reducing the distortion degree of a mapping image and reducing the quality loss caused by image distortion and recovery.
Drawings
Fig. 1 is a schematic block diagram of a surveillance video system commonly used in the prior art.
Fig. 2 is a schematic block diagram of a video image processing system based on local non-uniform resolution according to an embodiment of the present invention.
Fig. 3 is a diagram showing an example of performing local non-uniform mapping on a high-resolution large image to obtain a small-size image according to an embodiment of the present invention.
Fig. 4 is a diagram illustrating pairing candidate points of a vertex a according to an embodiment of the present invention.
Fig. 5 is an exemplary diagram of dividing a transition region between an original image and a mapped image into a plurality of corresponding perspective transformation quadrilaterals according to an embodiment of the present invention.
Fig. 6 is a flowchart of a video image processing method based on local non-uniform resolution according to an embodiment of the present invention.
Fig. 7 is a flowchart of a perspective transformation-based non-uniform resolution mapping method according to an embodiment of the present invention.
Reference numerals illustrate:
the monitoring camera 110, the image processing device 120, and the monitoring display 130.
Description of the embodiments
The video image processing apparatus, system and method based on local non-uniform resolution disclosed in the present invention are described in further detail below with reference to the accompanying drawings and detailed description. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be regarded as being isolated, and they may be combined with each other to achieve a better technical effect. In the drawings of the embodiments described below, like reference numerals appearing in the various drawings represent like features or components and are applicable to the various embodiments. Thus, once an item is defined in one drawing, no further discussion thereof is required in subsequent drawings.
It should be noted that the structures, proportions, sizes, etc. shown in the drawings are merely used in conjunction with the disclosure of the present specification, and are not intended to limit the applicable scope of the present invention, but rather to limit the scope of the present invention. The scope of the preferred embodiments of the present invention includes additional implementations in which functions may be performed out of the order described or discussed, including in a substantially simultaneous manner or in an order that is reverse, depending on the function involved, as would be understood by those of skill in the art to which embodiments of the present invention pertain.
Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but should be considered part of the specification where appropriate. In all examples shown and discussed herein, any specific values should be construed as merely illustrative, and not a limitation. Thus, other examples of the exemplary embodiments may have different values.
Examples
Video surveillance systems may generally consist of several modules, video camera shooting, video analysis and processing, video transmission, video recording, video playback and display. For an analog video camera, the monitoring system transmits an analog video, and the video is analyzed and processed, stored in a coding mode and played back and displayed at a receiving end; for a digital video camera, the monitoring system performs analysis processing and coding compression at a transmitting end, and only needs to be stored and played back for display at a receiving end.
In surveillance video applications, perimeter protection (a user sets a virtual guard line or a virtual guard zone on a video background screen) is a common application scenario. In such applications, the user needs to have a wide visual field range on one hand, can contain the whole range of one or more areas needing to be alerted, and leaves enough observation areas outside the alerted areas so as to observe the process of people or objects entering or exiting the alerted areas; on the other hand, the user has different requirements on the definition of different areas of the monitoring picture, and the interest of the local warning area is far higher than that of other areas. Especially, when an event occurs in the warning area, the user has a strong demand for previewing or replaying the enlarged picture of the warning area, and needs to see details such as faces, articles and the like.
That is, the virtual guard zone is a region of interest of the user, which is desired to be configured with high resolution. But at a fixed resolution, the resolution of each local area decreases as the field of view of the picture increases.
Referring to fig. 1, in a surveillance video system commonly used in the prior art, a high-resolution large image obtained by a surveillance camera is reduced to obtain a low-resolution small image, and then the small image is compressed and stored by video coding; when the video is played back or previewed, the video is decoded, a low-resolution small image is output to a user, the user can enlarge the small image according to the needs, and at the moment, the monitor display outputs a low-resolution large image, so that the definition of the image is difficult to guarantee.
I.e. only the resolution of the image can be sacrificed in order to ensure a field of view. When the local resolution is low, the accuracy of algorithms such as out-of-range detection/area intrusion detection is affected, and the details cannot be seen even if the user performs an amplifying operation on the image when playing back the monitoring video. In addition, when the resolution of the sensitive area is low, the accuracy of the algorithms such as out-of-range detection/area intrusion detection is also affected.
Referring to fig. 2, a video image processing system based on local non-uniform resolution is provided in the present invention.
The video image processing system includes a monitor camera 110, an image processing device 120 and a monitor display 130 in communication.
The monitoring camera 110 is used for acquiring an original video image with high resolution. The monitoring camera 110 may be a digital video camera or an analog video camera.
The image processing device 120 is configured to perform a local non-uniform resolution mapping process on the original video image.
The monitor display 130 is configured to display the video image processed by the image processing device. By way of example and not limitation, the monitor display may be a liquid crystal display.
The network for data transmission of the various parts of the system may be an intranet, the internet, or any type or combination of one or more wired or wireless networks.
The image processing device 120 is a video image processing device based on a local non-uniform resolution, which may include a memory and a processor in particular.
And the memory is used for storing data and instructions.
A processor, when executing the instructions, for,
acquiring a monitoring video image;
determining at least one region of interest in the surveillance video image;
and performing non-uniform resolution mapping processing on the monitoring video image to obtain a small-size video image before video encoding or transmission, wherein more resolution is allocated to the region of interest, and the rectangular shape and size of each frame of video image after mapping are maintained for encoding compression.
In this embodiment, the image processing device 120 may include a memory storing one or more instructions and a processor for executing the one or more instructions, which when executed, may configure the processor to provide the functionality described herein. Of course, the image processing device may also include other components commonly found in computing devices, such as one or more input/output components, such as cameras, keyboards, mice, network adapters, etc., for inputting information to or outputting information from the processing device.
With continued reference to fig. 2, the processor is also operative, upon video playback or preview: and performing resolution de-mapping processing to recover the image proportion relation when the decoded small-size video image is reversely mapped back to the large-size display image, and outputting a high-resolution image when the concerned area is locally enlarged.
In this embodiment, the processor may include a region of interest setting module. The region of interest setting module is configured to: generating at least one region of interest on the video image based on the manual input; and/or automatically generating at least one region of interest on the video image based on the captured scene features; and/or editing the region of interest to reconfigure the region of interest of the video image, the editing including adjusting a range of the region of interest, adding new regions of interest, and/or deleting existing regions of interest.
As such, the region of interest of the present invention (the region with higher resolution desired for the user) may be determined in a variety of ways. For example, one or more regions of interest are set based on manual input by a user. In another embodiment, the region of interest may be automatically determined, preferably based on known or determined characteristics of the scene being captured (e.g., a fence on a road, a parting line, etc.).
Preferably, the region of interest setting module supports a user to set a fixed region of interest (high resolution area) and a local resolution improvement factor m on the monitor screen as needed, and to set an affected resolution transition region around the region of interest.
The local resolution improvement multiple m is required to be smaller than the minimum scale ratio of the transition region to the corresponding region of interest in each direction. The resolution of the pixels outside each concerned region and the transition region keeps the low resolution set by the original system, namely, the pixels are used as the low resolution region, the pixels are not influenced by resolution mapping, the region outside the concerned region where the horizontal motion of the object frequently occurs is reserved as the low resolution region, and the coding compression efficiency of the mapped video can be improved.
Referring to fig. 3, an example of setting a region of interest is shown, in which a high-resolution region as a region of interest is located at a middle position of a resolution transition region. Of course, the above-mentioned division manner is taken as an example and not by way of limitation, and the high-resolution area serving as the region of interest may be located in the middle of the resolution transition area, may be located at the edge of the resolution transition area, or may be located at any other position of the screen. The setting of the region of interest of the present invention is not limited with respect to the prior art.
When the processor processes, the resolution mapping module can use different resolution mapping modes to reduce different partitioned areas on the original video large graph:
a lower image reduction scale k is applied to the high resolution region and an image reduction scale L is used to scale the low resolution region, the k=l/m.
And then, mapping the resolution transition region of the original image to the blank region left on the target small image in a designed mapping mode. The mapping mode of the transition region can be performed by adopting methods such as perspective transformation, polynomial projection transformation and the like.
Referring to the lower graph of fig. 3, a non-uniform resolution small graph is obtained after mapping the high resolution large graph. The shape and size of the mapped small picture are not changed, so that the traditional video coding and decoding module can be adopted for compression coding and decoding.
The picture content of the small picture subjected to mapping is distorted and deformed, and in order to facilitate the observation of a user, the processor is provided with a reflection module, and the reflection module is used for amplifying the small picture and recovering the image proportional relation according to different resolution area division and a corresponding resolution mapping method so as to eliminate the picture deformation.
Therefore, the high resolution area set by the original image can still keep high resolution when displayed through the system processing, and further, the detail of the image can be conveniently observed when the user enlarges.
Referring to fig. 4, the exemplary steps of the method for image processing using the video image processing system described above are preferably as follows:
s110, acquiring a monitoring video image.
S120, determining at least one region of interest in the monitoring video image.
And S130, before video coding or transmission, carrying out non-uniform resolution mapping processing on the original monitoring video image to obtain a small-size video image, wherein more resolution is allocated to the region of interest, and the rectangular shape and size of each frame of mapped video image are maintained.
And S140, encoding and compressing the video image after the non-uniform resolution mapping processing.
In the step S130, the step of performing the non-uniform resolution mapping process on the original monitoring video image to obtain the small-size video image may specifically be as follows:
setting a local resolution improvement multiple m of the concerned region, setting a resolution transition region around the concerned region, wherein the resolution of the concerned region and the pixels outside the transition region keeps the low resolution set by the original system to form a low resolution region, and the local resolution improvement multiple m is smaller than the minimum value of the scale ratio of the transition region and the corresponding concerned region in each direction;
the method comprises the steps of reducing different areas of a video image by using different resolution mapping modes, wherein the concerned area is scaled by using a reduction scale k as a high resolution area, and the low resolution area is scaled by using a reduction scale L, wherein k=L/m; the mapping mode of the transition region adopts a transmission transformation method or a polynomial projection transformation method.
For mapping of resolution transition regions, a non-uniform resolution mapping method based on perspective transformation is proposed herein, see fig. 5, the method comprising the steps of:
1) A high resolution region and a transition region boundary are set on the original image.
Preferably, the high resolution area and the resolution transition area defined by the user on the original image are defined as arbitrary polygons, the two polygons must have the same number of vertices n, and the n polygons may be convex polygons or concave polygons. And respectively recording the corresponding relation between the vertex coordinates of the two n-polygons and the end points of each side on the original image.
2) The region boundary polygon vertices are reordered clockwise.
First, selecting the vertex A with the highest vertical position from the vertices of the boundary polygon of the transition area, and selecting the vertex with the leftmost horizontal position if a plurality of vertices with the same height exist. Then, according to the recorded correspondence between the polygon edges and the end points of the region, two edges connected with the vertex A and corresponding vertices can be obtained. The first side which is rotated and contacted with the forward horizontal axis from the point A in the clockwise direction is selected, and the direction of traversing each vertex of the polygon along the side from the point A is clockwise. Starting from the point A, according to the connection relation of the sides, sequencing the vertexes of the boundary polygon of the transition area in the clockwise direction. Respectively designated a, B, C, D, etc.
The same is done for the high resolution region boundary polygons.
3) The transition region and the high resolution region are paired in pairs.
a. And (3) calculating whether each vertex of the high-resolution area boundary meets the matable condition for the vertex A of the transition area boundary.
Referring to fig. 6, in this embodiment, the pairing condition of the vertex a is: the line connecting the vertex of the high resolution area to A for pairing must lie within the angle of the two edges from A and not intersect the edges of the high resolution area boundary.
And the high-resolution area vertexes meeting the conditions form a point pairing candidate point set. If the pairing candidate point set is empty, the region setting fails, and the step 1) is returned to prompt the user to reset the shapes of the transition region and the high-resolution region.
b. Selecting a high-resolution area vertex and A point trial matching pair from the matching candidate point set of the A point: and pairing the remaining vertexes of the two n-polygons in a pair-wise manner sequentially according to the clockwise direction, and detecting whether the pairing conditions are met by the corresponding trial pairing points of the vertexes of the transition region.
If the trial matching vertex which does not meet the matching condition exists, the trial matching fails, and the vertex of the high resolution area is removed from the matching candidate point set of the point A; otherwise, the trial matching is successful, and the distance between each matching point of the two polygons is summed to be used as the matching cost value of the point.
c. And b, circulating the step b until the trial matching of all candidate points in the pairing candidate point set of the point A and the point A is completed. If the candidate points which meet the success of trial matching do not exist in the candidate point set, the region setting fails, and the step 1) is returned to prompt the user to reset the shapes of the transition region and the high-resolution region.
d. And comparing the matching cost values of the points successfully matched with the point A, selecting the candidate point with the minimum cost value as the final matching point, and completing the matching of the vertexes of the high-resolution area and the vertexes of the transition area in the clockwise direction. The pairing vertices may be labeled (A0, A0) (B0, B0) (C0, C0) (D0, D0), etc., as shown in fig. 7.
4) And calculating the coordinates of each vertex on the target mapping graph.
And calculating the coordinates of the vertex pairs of each area on the mapping small graph according to the image size reduction ratio L and the local resolution improvement multiple m. Can be labeled (A1, A1) (B1, B1) (C1, C1) (D1, D1), etc. If the high resolution area vertices (a 1, b1, c1, d1, etc.) on the mapping small graph are located outside the transition area boundary, the local resolution improvement factor m set by the user is too large, the local resolution improvement factor m value needs to be reduced, and the step 4) is repeated until the local resolution improvement factor m value reaches a reasonable range.
5) The transition region is divided into a number of mapping quadrilaterals.
Judging whether two endpoints of any side of the transition area of the original image and the corresponding pairing points of the endpoints are on a straight line or not. If the four points are not in a straight line, they form an artwork transition area quadrilateral, such as A0B0B0A0. The four points form a corresponding target map transition area quadrangle at the corresponding points on the target map, as shown in fig. 7, for example, A1B1 A1. In this way, the original image transition area and the target image transition area can be respectively decomposed into 2-n mapping quadrilaterals and are in one-to-one correspondence.
6) Solving the quadrilateral mapping and the reflection matrix.
According to the perspective transformation formula, a unique mapping matrix and a reflection matrix can be solved for each group of mapping quadrilaterals corresponding to each group.
Further, the method comprises the following steps:
7) And mapping (shrinking) the original high-resolution image to obtain a small-size video image.
Firstly judging which region the target pixel point is in, and if the pixel is in a high resolution region or a low resolution region, performing scaling according to the corresponding scaling coefficient; if the target pixel is located in the resolution transition region, determining which perspective transformation mapping quadrilateral it is located in, and then applying the corresponding perspective transformation relation to perform pixel mapping.
8) During video playback or preview, the small-size video image is subjected to inverse mapping (enlargement) processing in the same way as in step 7).
In the processing process, when the target pixel point is positioned in a high resolution area or a low resolution area, mapping processing is carried out according to the corresponding scaling factor; when the target pixel point is positioned in the resolution transition area, judging which perspective transformation mapping quadrangle the target pixel point is positioned in, and then mapping by applying the corresponding perspective transformation relation.
The non-uniform resolution monitoring system equipment provided by the invention can be used for mapping the high-resolution large-size video image of the camera to the small-size video image for transmission, encoding and storage through non-uniform resolution mapping according to the warning area setting of the user. Higher resolution is preserved for guard areas in small-size video. When the video is browsed or played back, the small-size video image is reversely mapped back to the large-size display image, and high resolution is obtained in the warning area of the display image. Further, the monitoring video image is divided into three areas of high resolution, low resolution and resolution transition area, different resolution mapping methods are adopted for mapping, and a small-size rectangular target mapping chart is synthesized, so that coding transmission is facilitated. And the mapping method of the resolution transition region is optimized.
Those skilled in the art will appreciate that other conventional methods of determining the region of interest may also be used, such as determining an overlapping coverage area based on views of multiple cameras capturing the scene and determining the overlapping area as the region of interest. Although the above example shows only 1 region, it should be understood that there may be more than 2 regions of interest.
The aforementioned processor may be any type of processor, such as a general purpose central processing unit ("CPU") or a special purpose microprocessor such as an embedded microcontroller or a digital signal processor ("DSP").
The memory may be any type of memory suitable for storing and accessing electronic information, or a combination thereof, such as temporary Random Access Memory (RAM) or non-temporary memory, such as Read Only Memory (ROM), hard drive memory, database memory, optical drive memory, optical memory, etc.
The memory may include data and instructions that, when executed by the processor, may configure or cause the device to perform or implement the functions or aspects described above (e.g., process one or more steps). In addition, the device may also include other components typically found in computing systems, such as an operating system, a queue manager, device drivers, database drivers, or one or more network protocols, etc., stored in memory and executed by the processor.
In the above description, the disclosure of the present invention is not intended to limit itself to these aspects. Rather, the components may be selectively and operatively combined in any number within the scope of the present disclosure. In addition, terms like "comprising," "including," and "having" should be construed by default as inclusive or open-ended, rather than exclusive or closed-ended, unless expressly defined to the contrary. All technical, scientific, or other terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Common terms found in dictionaries should not be too idealized or too unrealistically interpreted in the context of the relevant technical document unless the present disclosure explicitly defines them as such. Any alterations and modifications of the present invention, which are made by those of ordinary skill in the art based on the above disclosure, are intended to be within the scope of the appended claims.
Claims (9)
1. A video image processing apparatus based on local non-uniform resolution, characterized by comprising:
a memory to store data and instructions;
a processor, when executing the instructions, for,
acquiring a monitoring video image;
determining at least one region of interest in the surveillance video image;
and prior to video encoding or transmission, performing non-uniform resolution mapping processing on the surveillance video image to obtain a small-size video image, wherein more resolution is allocated to the region of interest while preserving the rectangular shape and size of each frame of video image after mapping for encoding compression;
wherein the processor, when determining at least one region of interest, is further configured to: setting a local resolution improvement multiple m of a region of interest, wherein the region of interest is a high resolution region; setting an affected resolution transition region around the region of interest, wherein the resolutions of pixels outside the region of interest and the transition region keep the low resolution set by the original system to form a low resolution region, and the low resolution region is not affected by resolution mapping; the local resolution improvement multiple m is smaller than the minimum scale ratio of the transition region to the corresponding region of interest in each direction.
2. The apparatus according to claim 1, wherein: the processor is used during video playback or preview,
and performing resolution de-mapping processing to recover the image proportion relation when the small-size video image is reversely mapped back to the large-size display image, and outputting a high-resolution image when the region of interest is locally enlarged.
3. The apparatus according to claim 1 or 2, characterized in that: the processor includes a region of interest setting module for,
generating at least one region of interest on the video image based on the manual input;
and/or automatically generating at least one region of interest on the video image based on the captured scene features;
and/or editing the region of interest to reconfigure the region of interest of the video image, the editing including adjusting a range of the region of interest, adding new regions of interest, and/or deleting existing regions of interest.
4. The apparatus according to claim 1, wherein: the processor is configured to, when generating a small-size video image,
different resolution mapping modes are used for reducing different areas of the video image, the concerned area is scaled by using a reduction scale k as a high resolution area, the low resolution area is scaled by using a reduction scale L, and k=l/m.
5. The apparatus according to claim 4, wherein: and the mapping mode of the transition region adopts a transmission transformation method or a polynomial projection transformation method.
6. A video image processing system based on local non-uniform resolution, comprising:
the monitoring camera is used for acquiring an original video image with high resolution;
an image processing apparatus for performing a local non-uniform resolution mapping process on the aforementioned original video image, the image processing apparatus being the apparatus of any one of claims 1 to 5;
and the monitoring display is used for displaying the video image processed by the image processing equipment.
7. The video image processing method based on the local non-uniform resolution is characterized by comprising the following steps:
acquiring a monitoring video image;
determining at least one region of interest in the surveillance video image;
before video coding or transmission, carrying out non-uniform resolution mapping processing on an original monitoring video image to obtain a small-size video image, wherein more resolution is allocated to the concerned region, and the rectangular shape and size of each frame of mapped video image are maintained;
encoding and compressing the video image after the non-uniform resolution mapping treatment;
the step of performing non-uniform resolution mapping processing on the original monitoring video image to obtain a small-size video image comprises the following steps:
setting a local resolution improvement multiple m of a region of interest, wherein the region of interest is a high resolution region; setting an affected resolution transition region around the region of interest, wherein the resolutions of pixels outside the region of interest and the transition region keep the low resolution set by the original system to form a low resolution region, and the low resolution region is not affected by resolution mapping; the local resolution improvement multiple m is smaller than the minimum scale ratio of the transition region to the corresponding region of interest in each direction.
8. The method according to claim 7, wherein:
the method comprises the steps of reducing different areas of a video image by using different resolution mapping modes, wherein the concerned area is scaled by using a reduction scale k as a high resolution area, and the low resolution area is scaled by using a reduction scale L, wherein k=L/m;
the mapping mode of the transition region adopts a transmission transformation method or a polynomial projection transformation method.
9. A non-uniform resolution mapping method based on perspective transformation is characterized by comprising the following steps:
step 1, setting a high resolution area and a transition area boundary on an original image; the high-resolution area and the resolution transition area are in the shape of any polygon, the two polygons have the same vertex number n, and the corresponding relation between the vertex coordinates of the two n-sided polygons and the end points of each side on the image is recorded;
step 2, reordering the polygon vertexes of the regional boundary clockwise; firstly, selecting a vertex A with the highest vertical position from vertexes of a boundary polygon of a transition area, selecting a vertex with the horizontal position at the leftmost position when a plurality of vertexes with the same height exist, and sequencing the vertexes of the boundary polygon of the transition area and the boundary polygon of a high resolution area in a clockwise direction according to the connection relation of edges from the point A;
step 3, pairing the vertexes of the transition area and the high-resolution area in pairs;
a. calculating whether each vertex of the boundary of the high-resolution area meets the matable condition for the vertex A of the boundary of the transition area;
b. selecting a high-resolution area vertex and the point A from the pairing candidate point set of the point A, and detecting whether the pairing points corresponding to the vertexes of the transition area meet the pairing-available condition;
when the trial matching vertex which does not meet the matching condition exists, the trial matching fails, and the vertex of the high-resolution area is removed from the matching candidate point set of the point A; otherwise, the trial matching is successful, and the distance sum of each matching point of the two polygons is used as the matching cost value of the point;
c. b, circulating until the trial matching of all candidate points in the pairing candidate point set of the point A and the point A is completed;
d. comparing the matching cost value of each point successfully matched with the point A, selecting the candidate point with the minimum cost as the final matching point, and completing the matching of the vertex of the high-resolution area and each vertex of the transition area in the clockwise direction;
step 4, calculating the coordinates of each vertex on the target mapping chart;
step 5, dividing the transition area into a plurality of mapping quadrilaterals;
step 6, solving quadrilateral mapping and reflection matrix; solving a unique mapping matrix and a reflection matrix for each group of corresponding mapping quadrangles according to a perspective transformation formula;
and, further comprising the steps of:
mapping and shrinking the high-resolution image to obtain a small-size video image, and performing reflection amplifying treatment on the small-size video image during video playback or previewing; when the target pixel point is positioned in a high resolution area or a low resolution area, mapping is carried out according to the corresponding scaling coefficient; when the target pixel point is positioned in the resolution transition region, judging which perspective transformation mapping quadrangle the target pixel point is positioned in, and then mapping by applying the corresponding perspective transformation relation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811212816.5A CN109194923B (en) | 2018-10-18 | 2018-10-18 | Video image processing device, system and method based on local non-uniform resolution |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811212816.5A CN109194923B (en) | 2018-10-18 | 2018-10-18 | Video image processing device, system and method based on local non-uniform resolution |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109194923A CN109194923A (en) | 2019-01-11 |
| CN109194923B true CN109194923B (en) | 2023-08-29 |
Family
ID=64945442
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811212816.5A Active CN109194923B (en) | 2018-10-18 | 2018-10-18 | Video image processing device, system and method based on local non-uniform resolution |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109194923B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020173414A1 (en) * | 2019-02-25 | 2020-09-03 | 昀光微电子（上海）有限公司 | Human vision characteristic-based near-eye display method and device |
| CN111698453B (en) * | 2019-03-11 | 2022-02-08 | 杭州海康威视系统技术有限公司 | Video processing method and device |
| EP3742738B1 (en) * | 2019-05-24 | 2021-09-08 | Mirriad Advertising PLC | Incorporating visual objects into video material |
| CN110572656B (en) * | 2019-09-19 | 2021-11-19 | 江苏视博云信息技术有限公司 | Encoding method, image processing method, device, system, storage medium and equipment |
| CN110650357B (en) | 2019-09-27 | 2023-02-10 | 腾讯科技（深圳）有限公司 | Video decoding method and device |
| WO2021073336A1 (en) * | 2019-10-18 | 2021-04-22 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | A system and method for creating real-time video |
| CN110796997B (en) * | 2019-11-14 | 2021-12-21 | 京东方科技集团股份有限公司 | Method and device for realizing non-uniform resolution display |
| CN113518249B (en) * | 2020-04-10 | 2023-03-10 | 华为技术有限公司 | Far-end image processing method and device |
| CN112884803B (en) * | 2020-08-18 | 2023-11-24 | 眸芯科技（上海）有限公司 | Real-time intelligent monitoring target detection method and device based on DSP |
| CN113630622B (en) * | 2021-06-18 | 2024-04-26 | 中图云创智能科技(北京)有限公司 | Panoramic video image processing method, server, target equipment, device and system |
| CN113592712A (en) * | 2021-07-30 | 2021-11-02 | 北京百度网讯科技有限公司 | Image processing method, device, equipment, storage medium and cloud VR system |
| CN114274877B (en) * | 2021-11-29 | 2023-06-23 | 惠州华阳通用智慧车载系统开发有限公司 | Rearview mirror image display control method |
| CN114520873A (en) * | 2021-12-31 | 2022-05-20 | 广州文远知行科技有限公司 | Sweeper video acceleration method and system |
| CN116761019A (en) * | 2023-08-24 | 2023-09-15 | 瀚博半导体(上海)有限公司 | Video processing method, system, computer device and computer readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105635624A (en) * | 2014-10-27 | 2016-06-01 | 华为技术有限公司 | Video image processing method, device and system |
| CN106791854A (en) * | 2016-11-22 | 2017-05-31 | 北京疯景科技有限公司 | Image Coding, coding/decoding method and device |
| CN108475420A (en) * | 2015-11-16 | 2018-08-31 | 阿尔卡特朗讯美国公司 | Multiresolution compressed sensing image procossing |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5492139B2 (en) * | 2011-04-27 | 2014-05-14 | 富士フイルム株式会社 | Image compression apparatus, image expansion apparatus, method, and program |
-
2018
- 2018-10-18 CN CN201811212816.5A patent/CN109194923B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105635624A (en) * | 2014-10-27 | 2016-06-01 | 华为技术有限公司 | Video image processing method, device and system |
| CN108475420A (en) * | 2015-11-16 | 2018-08-31 | 阿尔卡特朗讯美国公司 | Multiresolution compressed sensing image procossing |
| CN106791854A (en) * | 2016-11-22 | 2017-05-31 | 北京疯景科技有限公司 | Image Coding, coding/decoding method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109194923A (en) | 2019-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109194923B (en) | Video image processing device, system and method based on local non-uniform resolution | |
| US8855441B2 (en) | Method and apparatus for transforming a non-linear lens-distorted image | |
| CN112204993B (en) | Adaptive panoramic video streaming using overlapping partitioned segments | |
| TWI535285B (en) | Conference system, surveillance system, image processing device, image processing method and image processing program, etc. | |
| US10038841B1 (en) | System for streaming multiple regions deriving from a wide-angle camera | |
| CN111161660B (en) | Data processing system | |
| CN112365407B (en) | Panoramic stitching method for camera with configurable visual angle | |
| CN108564551B (en) | Fisheye image processing method and fisheye image processing device | |
| JP2007067917A (en) | Image data processing apparatus | |
| KR20100105351A (en) | Banana codec | |
| CN102598651B (en) | Motion video processing unit and method, the camera head of motion video processing unit is installed | |
| WO2003019471A2 (en) | Method and apparatus for encoding photogrraphic images | |
| US7433544B2 (en) | Apparatus and method for producing thumbnail images and for improving image quality of re-sized images | |
| KR102308604B1 (en) | Method, apparatus and stream for formatting immersive video for legacy and immersive rendering devices | |
| WO2018068680A1 (en) | Image processing method and device | |
| US11528412B2 (en) | Apparatus and method for stitching together multiple images | |
| JP5473903B2 (en) | Method and system for processing images | |
| WO2023280266A1 (en) | Fisheye image compression method, fisheye video stream compression method and panoramic video generation method | |
| WO2021083150A1 (en) | Zoom method and device, aircraft, flight system and storage medium | |
| EP3091742A1 (en) | Device and method for encoding a first image of a scene using a second image having a lower resolution and captured at the same instant | |
| KR102498870B1 (en) | Camera system and method for processing image thereof | |
| JPH0562798B2 (en) | ||
| CN115134604A (en) | Panoramic video compression method and device, computer equipment and storage medium | |
| CN117456145A (en) | Video processing method, device, electronic equipment and readable storage medium | |
| CN117853980A (en) | Video processing method, apparatus, electronic device, and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 201210 floor 8, building 1, No. 298, Xiangke Road, Pudong New Area, Shanghai Applicant after: MOUXIN TECHNOLOGY (SHANGHAI) Co.,Ltd. Address before: 201210 Room 502, building 1, No. 800, Naxian Road, China (Shanghai) pilot Free Trade Zone, Pudong New Area, Shanghai Applicant before: MOUXIN TECHNOLOGY (SHANGHAI) Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |