CN111523533B - Method and device for determining area of object from image - Google Patents
Method and device for determining area of object from image Download PDFInfo
- Publication number
- CN111523533B CN111523533B CN201910106122.1A CN201910106122A CN111523533B CN 111523533 B CN111523533 B CN 111523533B CN 201910106122 A CN201910106122 A CN 201910106122A CN 111523533 B CN111523533 B CN 111523533B
- Authority
- CN
- China
- Prior art keywords
- convolution
- map data
- feature map
- candidate
- convolution kernel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The application provides a method and a device for determining an area where an object is located from an image. The method for determining the area where the object is located in the image comprises the following steps: acquiring an image to be identified; acquiring feature map data of the image to be identified; obtaining object parts in the feature map data; and determining the area where the object in the image to be identified is located according to the object part. The method provided by the application is adopted to improve the calculation efficiency of object identification.
Description
Technical Field
The present invention relates to the field of object recognition, and in particular, to a method and an apparatus for determining an area where an object is located from an image.
Background
Currently, area-based object recognition methods have been the dominant method in the field of object recognition. Currently, most advanced zone-based object detectors are implemented using a zone of interest pooling layer in combination with a zone candidate network.
By learning embedded spatial information from higher-order convolutional neural network (Convolutional Neural Network, CNN) features, area-based object detectors have made tremendous progress over traditional non-CNN approaches such as variable component models (Deformable Parts Model, DPM). Recently, region-based full convolution networks (Region-based Fully Convolutional Network, R-FCNs) have further optimized the performance and computational efficiency of object recognition by reducing the overhead of duplicate predictions through the Region-of-interest pooling layer on the final FCN (Fully Convolutional Network) signature.
However, these current region-based object recognition methods must use a large amount of frame labeling training data to perform model training to obtain object components, which results in low calculation efficiency of object recognition. Especially in advanced regional candidate networks (RPNs), when the size of the convolution kernel of the CNN used increases, the calculation amount of training parameters is very large, further reducing the calculation efficiency of object recognition
Disclosure of Invention
The application provides a method and a device for determining an area where an object is located from an image, so as to improve the calculation efficiency of object identification.
The application provides a method for determining an area where an object is located from an image, which comprises the following steps:
acquiring an image to be identified;
acquiring feature map data of the image to be identified;
obtaining object parts in the feature map data;
and determining the area where the object in the image to be identified is located according to the object part.
Optionally, the acquiring the feature map data of the image to be identified includes:
and extracting features of the image to be identified by using a convolutional neural network, and acquiring feature map data of the image to be identified.
Optionally, obtaining the object part in the feature map data includes:
Performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result;
and obtaining the object parts in the characteristic map data according to the convolution result.
Optionally, the method for determining the area where the object is located in the image further includes:
determining an initial height of a candidate convolution kernel and an initial width of the candidate convolution kernel;
taking the initial height as a reference, performing multiplication operation of a first multiple to obtain a candidate height of the candidate convolution kernel;
taking the initial width as a reference, performing multiplication operation of a second multiple to obtain a candidate width of the candidate convolution kernel;
obtaining size information of the candidate convolution kernel according to the initial height, the initial width, the candidate height and the candidate width;
and obtaining the convolution kernel according to the size information of the candidate convolution kernel.
Optionally, the initial height of the candidate convolution kernel is 1, the initial width of the candidate convolution kernel is 1, the first multiple is 2, and the second multiple is 2.
Optionally, the obtaining the convolution kernel according to the size information of the candidate convolution kernel includes:
determining the size of the candidate convolution kernel according to the size information of the candidate convolution kernel;
Amplifying the size of the candidate convolution kernel to obtain amplified size information of the candidate convolution kernel;
and obtaining the convolution kernel according to the amplified size information of the candidate convolution kernel.
Optionally, the size information of the candidate convolution kernel includes height information of the candidate convolution kernel and width information of the candidate convolution kernel, and the amplified size information of the candidate convolution kernel includes the amplified height information of the candidate convolution kernel and the amplified width information of the candidate convolution kernel;
the determining the size of the candidate convolution kernel according to the size information of the candidate convolution kernel includes: determining the height of the candidate convolution kernel according to the height information of the candidate convolution kernel, and determining the width of the candidate convolution kernel according to the width information of the candidate convolution kernel;
amplifying the size of the candidate convolution kernel to obtain amplified size information of the candidate convolution kernel, wherein the amplifying comprises the following steps: amplifying the height of the candidate convolution kernel to obtain amplified height information of the candidate convolution kernel, and amplifying the width of the candidate convolution kernel to obtain amplified width information of the candidate convolution kernel.
Optionally, the performing convolution operation on the feature map data and the convolution kernel to obtain a convolution result includes:
judging whether filling processing is needed for the feature map data according to the attribute of the feature map data and the convolution kernel;
if yes, filling the feature map data, and carrying out convolution operation on the filled feature map data and the convolution kernel to obtain a convolution result.
Optionally, the performing convolution operation on the feature map data and the convolution kernel to obtain a convolution result includes:
performing convolution operation on the feature map data and the convolution kernels with the first number and different sizes respectively to obtain convolution results corresponding to the convolution kernels with the first number and different sizes;
the step of obtaining the object parts in the feature map data according to the convolution result comprises the following steps: and obtaining object parts in the feature map data according to convolution results corresponding to the convolution kernels with different sizes of the first number.
Optionally, the method for determining the area where the object is located in the image further includes:
and obtaining the first number of convolution kernels with different sizes according to the attribute of the feature map data.
Optionally, the obtaining the object part in the feature map data according to the convolution results corresponding to the convolution kernels with the first number and different sizes includes:
and synthesizing convolution results corresponding to the convolution kernels with different sizes of the first number to obtain object parts in the feature map data.
Optionally, the synthesizing the convolution results corresponding to the convolution kernels with different sizes of the first number to obtain object parts in the feature map data includes:
splicing convolution results corresponding to the convolution kernels with different sizes to obtain spliced feature vectors of the feature map data;
and obtaining object parts of an interested region in the feature map data according to the spliced feature vector, wherein the interested region is a region possibly containing the object in the feature map data.
Optionally, the region of interest is determined in the feature map data according to a size of a convolution kernel corresponding to the feature map data.
Optionally, the determining, according to the object portion, an area where the object in the image to be identified is located includes:
determining a target area containing the object in the feature map data from the region of interest according to the object score;
And determining the area where the object in the image to be identified is located according to the target area.
The application provides a device for determining an area where an object is located from an image, which comprises the following steps:
the image acquisition unit is used for acquiring an image to be identified;
the feature acquisition unit is used for acquiring feature map data of the image to be identified;
an object division obtaining unit, configured to obtain an object division in the feature map data;
and the determining unit is used for determining the area where the object in the image to be identified is located according to the object parts.
Optionally, the feature acquisition unit is specifically configured to:
and extracting features of the image to be identified by using a convolutional neural network, and acquiring feature map data of the image to be identified.
Optionally, the object obtaining unit is specifically configured to:
performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result;
and obtaining the object parts in the characteristic map data according to the convolution result.
Optionally, the device for determining the area where the object is located from the image further includes a convolution kernel obtaining unit, configured to:
determining an initial height of a candidate convolution kernel and an initial width of the candidate convolution kernel;
Taking the initial height as a reference, performing multiplication operation of a first multiple to obtain a candidate height of the candidate convolution kernel;
taking the initial width as a reference, performing multiplication operation of a second multiple to obtain a candidate width of the candidate convolution kernel;
based on the initial height, the initial width, the candidate height and the candidate width, obtaining size information of the candidate convolution kernel;
and obtaining the convolution kernel according to the size information of the candidate convolution kernel.
Optionally, in the convolution kernel obtaining unit, an initial height of the candidate convolution kernel is 1, an initial width of the candidate convolution kernel is 1, the first multiple is 2, and the second multiple is 2.
Optionally, the convolution kernel obtaining unit is further configured to:
determining the size of the candidate convolution kernel according to the size information of the candidate convolution kernel;
amplifying the size of the candidate convolution kernel to obtain amplified size information of the candidate convolution kernel:
and obtaining the convolution kernel according to the amplified size information of the candidate convolution kernel.
Optionally, in the convolution kernel obtaining unit, the size information of the candidate convolution kernel includes height information of the candidate convolution kernel and width information of the candidate convolution kernel, and the enlarged size information of the candidate convolution kernel includes the enlarged height information of the candidate convolution kernel and the enlarged width information of the candidate convolution kernel;
The determining the size of the candidate convolution kernel according to the size information of the candidate convolution kernel includes: determining the height of the candidate convolution kernel according to the height information of the candidate convolution kernel, and determining the width of the candidate convolution kernel according to the width information of the candidate convolution kernel;
the step of amplifying the size of the candidate convolution kernel to obtain amplified size information of the candidate convolution kernel includes: amplifying the height of the candidate convolution kernel to obtain amplified height information of the candidate convolution kernel, and amplifying the width of the candidate convolution kernel to obtain amplified width information of the candidate convolution kernel.
Optionally, the convolution kernel obtaining unit is further configured to:
judging whether filling processing is needed for the feature map data according to the attribute of the feature map data and the convolution kernel;
if yes, filling the feature map data, and carrying out convolution operation on the filled feature map data and the convolution kernel to obtain a convolution result.
Optionally, the object obtaining unit is specifically configured to:
performing convolution operation on the feature map data and the convolution kernels with the first number and different sizes respectively to obtain convolution results corresponding to the convolution kernels with the first number and different sizes;
The step of obtaining the object parts in the feature map data according to the convolution result comprises the following steps: and obtaining object parts in the feature map data according to convolution results corresponding to the convolution kernels with different sizes of the first number.
Optionally, the object division obtaining unit further includes a convolution kernel obtaining unit with different sizes, where the convolution kernel obtaining unit is configured to:
and obtaining the first number of convolution kernels with different sizes according to the attribute of the feature map data.
Optionally, the object obtaining unit is specifically configured to:
and synthesizing convolution results corresponding to the convolution kernels with different sizes of the first number to obtain object parts in the feature map data.
Optionally, the object obtaining unit is further configured to:
splicing convolution results corresponding to the convolution kernels with different sizes to obtain spliced feature vectors of the feature map data;
and obtaining object parts of an interested region in the feature map data according to the spliced feature vector, wherein the interested region is a region possibly containing the object in the feature map data.
Optionally, the object obtaining unit is further configured to:
The region of interest is determined in the feature map data according to the size of the convolution kernel corresponding to the feature map data.
Optionally, the determining unit is specifically configured to:
determining a target area containing the object in the feature map data from the region of interest according to the object score;
and determining the area where the object in the image to be identified is located according to the target area.
The application provides an electronic device, the electronic device includes:
a processor;
a memory for storing a linked list of processing programs which, when read and executed by the processor, perform the following operations:
acquiring an image to be identified;
acquiring feature map data of the image to be identified;
obtaining object parts in the feature map data;
and determining the area where the object in the image to be identified is located according to the object part.
The present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
acquiring an image to be identified;
acquiring feature map data of the image to be identified;
obtaining object parts in the feature map data;
and determining the area where the object in the image to be identified is located according to the object part.
The application provides a method for determining an area where an object is located from an object to be identified, which comprises the following steps:
acquiring an object to be identified;
acquiring feature map data of the object to be identified;
obtaining object parts in the feature map data;
and determining the area where the object in the object to be identified is located according to the object part.
Optionally, the obtaining the object part in the feature map data includes:
performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result;
and obtaining the object parts in the characteristic map data according to the convolution result.
Optionally, the method for determining the area where the object is located from the object to be identified further includes:
determining an initial height of a candidate convolution kernel and an initial width of the candidate convolution kernel;
taking the initial height as a reference, performing multiplication operation of a first multiple to obtain a candidate height of the candidate convolution kernel;
taking the initial width as a reference, performing multiplication operation of a second multiple to obtain a candidate width of the candidate convolution kernel;
obtaining size information of the candidate convolution kernel according to the initial height, the initial width, the candidate height and the candidate width;
And obtaining the convolution kernel according to the size information of the candidate convolution kernel.
Optionally, the initial height of the candidate convolution kernel is 1, the initial width of the candidate convolution kernel is 1, the first multiple is 2, and the second multiple is 2.
Optionally, the performing convolution operation on the feature map data and the convolution kernel to obtain a convolution result includes:
performing convolution operation on the feature map data and the convolution kernels with the first number and different sizes respectively to obtain convolution results corresponding to the convolution kernels with the first number and different sizes;
the step of obtaining the object parts in the feature map data according to the convolution result comprises the following steps: and obtaining object parts in the feature map data according to convolution results corresponding to the convolution kernels with different sizes of the first number.
Compared with the prior art, the application has the following advantages:
by adopting the method for determining the area of the object from the image, the object part in the feature map data is directly obtained, so that a large amount of frame marking training data is not needed to be used for model training in the object recognition process, and the calculation efficiency of object recognition is improved.
Drawings
FIG. 1 is a flow chart of a method of a first embodiment of the present application;
FIG. 2 is a schematic illustration of determining an area of an object from an image according to a first embodiment of the present application;
fig. 3 is a network architecture diagram of a BING (Binarized Normed Gradients) -RPN (Region Proposal Network, area candidate network) according to a first embodiment of the present application;
FIG. 4 is a schematic diagram of an object identification branch of a BING-RPN network according to a first embodiment of the present application;
FIG. 5 is a schematic diagram of convolution computation in a BING-RPN network according to a first embodiment of the present application;
fig. 6 is a schematic view of an apparatus according to a second embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is, however, susceptible of embodiment in many other ways than those herein described and similar generalizations can be made by those skilled in the art without departing from the spirit of the application and the application is therefore not limited to the specific embodiments disclosed below.
The first embodiment of the application provides a method for determining an area where an object is located from an image. Referring to fig. 1, a flowchart of a method according to a first embodiment of the present application is shown. A method for determining an area where an object is located from an image according to a first embodiment of the present application will be described in detail with reference to fig. 1. The method comprises the following steps:
Step S101: and acquiring an image to be identified.
The method is used for acquiring the image to be identified.
Fig. 2 is a schematic diagram of a process of determining an area of an object from an image according to the present embodiment. The first row in fig. 2 provides an image comprising one bird, the background of the image comprising green grass. The object to be identified is an image containing birds. In general, birds in an image are taken as objects to be identified. By adopting the method provided by the embodiment, the target area comprising the object to be identified is obtained from the image, and the object is identified.
Referring to fig. 3, a network architecture diagram of a BING-RPN employing the method provided in this embodiment is shown. In fig. 3, the image to be identified is the leftmost image of the figure that contains the dog.
Unlike conventional BING, this embodiment provides a modified BING method, please refer to FIGS. 4 and 5. Fig. 4 is a schematic diagram of an object recognition branch of the BING-RPN network provided in this embodiment. FIG. 5 is a schematic diagram of convolution computation in a BING-RPN network. The following step description will be described in more detail with reference to fig. 4 and 5.
Step S102: and acquiring the feature map data of the image to be identified.
The step is used for acquiring the feature map data of the image to be identified.
The obtaining the feature map data of the image to be identified comprises the following steps:
and extracting features of the image to be identified by using a convolutional neural network, and acquiring feature map data of the image to be identified.
The convolutional neural network in this embodiment may be a commonly used ResNet, acceptance V3, acceptance V4, etc. Referring to fig. 3, the backbone CNN network is the convolutional neural network.
Step S103: and obtaining the object parts in the characteristic diagram data.
The step is used for obtaining the object parts in the characteristic map data by utilizing the negative Laplace operator.
The laplace operator is a tool commonly used in image processing for sharpening of images. The method provided by the embodiment uses the negative Laplace operator in the calculation of the object parts, and obtains higher object recognition efficiency.
An object score (object score) is used in an RPN network, and refers to a probability that a candidate region in feature map data corresponding to an image to be identified contains an object to be identified. The greater the value of the object score, which is between 0 and 1, the greater the probability that the candidate region contains the object to be identified.
The obtaining the object part in the feature map data by using the negative Laplacian comprises the following steps:
Performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result, wherein the weight of the convolution kernel is a negative Laplacian;
and obtaining the object parts in the characteristic map data according to the convolution result.
The feature map data x in the present embodiment can be expressed using the following formula.
x∈R h×w×C ，
Where h represents the height of the feature map data, w represents the width of the feature map data, and C represents the number of channels of the feature map data. For example, the feature map data x may be an 8×8×1536 data block employed in acceptance-V4.
The BING-RPN provided in this embodiment performs multiple parallel deep convolution operations on the feature map data x, where the convolution kernel of the deep convolution operation is obtained using the following formula:
wherein, the liquid crystal display device comprises a liquid crystal display device,
and->
Referring to fig. 4, the BING-RPN provided in this embodiment outputs N area candidates in the C dimension. Wherein N is obtained using the following formula:
where h is the height of the input feature map data, w is the width of the input feature map data, h θ ，w θ Is the height and width of the convolution kernel of different sizes, multiplied by 1 x 1 as described above.
The method for determining the area of the object from the image further comprises the following steps:
And obtaining the first number of convolution kernels with different sizes according to the attribute of the feature map data.
By the above formula of N, it can be seen that the first number N can be obtained by the height and width data of the feature map data.
The method for determining the area of the object from the image further comprises the following steps:
determining an initial height of a candidate convolution kernel and an initial width of the candidate convolution kernel;
taking the initial height as a reference, performing multiplication operation of a first multiple to obtain a candidate height of the candidate convolution kernel;
taking the initial width as a reference, performing multiplication operation of a second multiple to obtain a candidate width of the candidate convolution kernel;
obtaining size information of the candidate convolution kernel according to the initial height, the initial width, the candidate height and the candidate width;
and obtaining the convolution kernel according to the size information of the candidate convolution kernel.
The initial height of the candidate convolution kernel is 1, the initial width of the candidate convolution kernel is 1, the first multiple is 2, and the second multiple is 2.
For example, according to the above method, convolution kernel sizes of 1×1,2×1,4×1,8×1,1×2,2×2,4×2,8×2,1×4,2×4,4×4,8×4,1×8,2×8,4×8,8×8 can be obtained. By adopting the searching strategy of the convolution kernel which is multiplied by 1, the acquisition efficiency of the convolution kernel is improved. Please refer to fig. 5 (a) and (b). In fig. 5, the bold frame represents the convolution kernel, and the right (0, 1) represents (x, y, convolution kernel width, convolution kernel height), respectively, x, y being the coordinates of the upper left-most corner of the frame.
The obtaining the convolution kernel according to the size information of the candidate convolution kernel includes:
determining the size of the candidate convolution kernel according to the size information of the candidate convolution kernel;
amplifying the size of the candidate convolution kernel according to the size of the negative Laplacian, and obtaining the amplified size information of the candidate convolution kernel;
and obtaining the convolution kernel according to the amplified size information of the candidate convolution kernel.
Because objects are generally considered to have good closed boundaries, the visual field (recovery field) obtained from the region of interest (Region of interest) is insufficient for object recognition. The method for determining the area where the object is located in the image provided by the embodiment enlarges the size of the convolution kernel.
The size information of the candidate convolution kernel comprises the height information of the candidate convolution kernel and the width information of the candidate convolution kernel, and the amplified size information of the candidate convolution kernel comprises the amplified height information of the candidate convolution kernel and the amplified width information of the candidate convolution kernel;
the determining the size of the candidate convolution kernel according to the size information of the candidate convolution kernel includes: determining the height of the candidate convolution kernel according to the height information of the candidate convolution kernel, and determining the width of the candidate convolution kernel according to the width information of the candidate convolution kernel;
The step of amplifying the size of the candidate convolution kernel according to the size of the negative Laplacian to obtain amplified size information of the candidate convolution kernel, including: amplifying the height of the candidate convolution kernel according to the height of the negative Laplacian to obtain amplified height information of the candidate convolution kernel, and amplifying the width of the candidate convolution kernel according to the width of the negative Laplacian to obtain amplified width information of the candidate convolution kernel.
The step of carrying out convolution operation on the feature map data and the convolution kernel to obtain a convolution result comprises the following steps:
judging whether filling processing is needed for the feature map data according to the attribute of the feature map data and the convolution kernel;
if yes, filling the feature map data, and carrying out convolution operation on the filled feature map data and the convolution kernel to obtain a convolution result.
Since the enlargement of the convolution kernel causes non-uniformity of the convolution calculation, a 0-filling operation is required for the feature map data. Please refer to fig. 5 (c) and (d). Taking (d) in fig. 5 as an example, pad (2, 4) represents that 2 lines are newly expanded in each of the upward and downward directions in the height direction on the basis of 8×8 feature map data. In the width direction, 4 columns are newly expanded in both the left and right directions. The expanded rows and columns are filled with 0 complements.
And performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result, wherein the weight of the convolution kernel is a negative Laplacian, and the method comprises the following steps:
performing convolution operation on the feature map data and the convolution kernels with the first number and different sizes respectively to obtain convolution results corresponding to the convolution kernels with the first number and different sizes, wherein the weights of the convolution kernels with the first number and different sizes are negative Laplacian operators;
the step of obtaining the object parts in the feature map data according to the convolution result comprises the following steps: and obtaining object parts in the feature map data according to convolution results corresponding to the convolution kernels with different sizes of the first number.
Given a givenThe negative Laplace operator has a height and width of k, and if a field of view of k×k times is to be obtained, a size of (k·h can be used θ )×(k·w θ ) And subjecting the convolution kernel to a fill process of (k-1)/2 h, (k-1)/2 w complement of 0. Based on the analysis, the BING-RPN provided by this embodiment can be expressed using the following formula:
BING(x；Θ，k)＝{Conv depthwise (x'; θ) |θ ε Θ (k) } equation 1
Where x' is the filling data after the filling process by (k-1)/2 h, (k-1)/2 w of 0 complement to the input feature map data x, k is the enlargement ratio of the visual field, and Θ (k) is the set of convolution kernels that follow the change of k. When k=1, the BING algorithm is equivalent to a BING algorithm that is not filled with 0. Please refer to fig. 4. In fig. 4, the feature map data of 8×8×c at the lowest, and the penultimate line is the size of the filling, for example, the filling size (2, 1), representing expansion of two lines each in the upward and downward directions, and expansion of one column each in the left and right directions. And supplementing 0 on the expanded data. The third last row is a convolution kernel of 16 sizes obtained by enlarging k=3 for the previously obtained 1×1,2×1,4×1,8×1,1×2,2×2,4×2,8×2,1×4,2×4,4×4,1×8,2×8,4×8,8×8 convolution kernel, and k=3. The weight of the convolution kernel adopts a manually given negative Laplacian.
In existing BING implementations, a training model is used to obtain object scores. The method provided by this example, found that it was feasible to calculate object scores using the negative laplace operator, and demonstrated by the ImageNet 2012 experimental dataset commonly used in the industry. Based on the above analysis, the convolution kernel may be selected as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,
is a two-dimensional Laplacian filter, and the variable sigma x ，σ y Proportional to the size of the laplace filter. Can set [ sigma ] x ，σ y ]＝[1.4，1.4]The corresponding convolution kernel has a size of 9 x 9.Θ in the equation 2 obj With' formula 1, an N x C-dimensional vector is generated, where each element corresponds to an object component in a channel. The minimum size of the laplace filter is 3×3, so k=3 is used to maintain the resolution of the output. Object recognition can be achieved by BING (x; Θ) obj ', 3). FIG. 4 shows BING (x; Θ) obj ' 3) calculation process. As can be seen from fig. 4, the calculation process includes: firstly, feature map data are acquired, then filling processing is carried out, then convolution operation is carried out with a plurality of convolution kernels with different convolution kernel sizes, and each obtained convolution operation result is synthesized to obtain object components.
The obtaining the object part in the feature map data according to the convolution results corresponding to the convolution kernels with the first number and different sizes comprises the following steps:
and synthesizing convolution results corresponding to the convolution kernels with different sizes of the first number to obtain object parts in the feature map data.
The step of synthesizing the convolution results corresponding to the convolution kernels with different sizes of the first number to obtain object parts in the feature map data, includes:
splicing convolution results corresponding to the convolution kernels with different sizes to obtain spliced feature vectors of the feature map data;
and obtaining object parts of the region of interest in the feature map data according to the spliced feature vector, wherein the region of interest is a region possibly containing the object in the feature map data.
The region of interest is determined in the feature map data according to the size of the convolution kernel corresponding to the feature map data.
Step S104: and determining the area where the object in the image to be identified is located according to the object part.
The method comprises the step of determining the area where the object in the image to be identified is located according to the object part.
The determining the area where the object in the image to be identified is located according to the object part comprises the following steps:
determining a target area containing the object in the feature map data from the region of interest according to the object score;
and determining the area where the object in the image to be identified is located according to the target area.
Referring to fig. 3, the target area is a frame including a dog.
In the foregoing embodiment, a method for determining an area where an object is located in an image is provided, and correspondingly, the application also provides a device for determining an area where an object is located in an image. Referring to fig. 6, a flowchart of an embodiment of an apparatus for determining an area of an object from an image is shown. Since this embodiment, i.e. the second embodiment, is substantially similar to the method embodiment, the description is relatively simple, and reference should be made to the description of the method embodiment for relevant points. The device embodiments described below are merely illustrative.
The device for determining the area of the object from the image comprises:
an image acquisition unit 601, configured to acquire an image to be identified;
a feature acquisition unit 602, configured to acquire feature map data of the image to be identified;
An object division obtaining unit 603, configured to obtain an object division in the feature map data by using a negative laplacian operator;
and the determining unit 604 is used for determining the area where the object in the image to be identified is located according to the object part.
In this embodiment, the feature acquiring unit is specifically configured to:
and extracting features of the object to be identified by using a convolutional neural network, and acquiring feature map data of the object to be identified.
In this embodiment, the object obtaining unit is specifically configured to:
performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result, wherein the weight of the convolution kernel is a negative Laplacian;
and obtaining the object parts in the characteristic map data according to the convolution result.
In this embodiment, the apparatus for determining an area where an object is located from an image further includes a convolution kernel obtaining unit, configured to:
determining an initial height of a candidate convolution kernel and an initial width of the candidate convolution kernel;
taking the initial height as a reference, performing multiplication operation of a first multiple to obtain a candidate height of the candidate convolution kernel;
taking the initial width as a reference, performing multiplication operation of a second multiple to obtain a candidate width of the candidate convolution kernel;
Obtaining size information of the candidate convolution kernel according to the initial height, the initial width, the candidate height and the candidate width;
and obtaining the convolution kernel according to the size information of the candidate convolution kernel.
In this embodiment, in the convolution kernel obtaining unit, an initial height of the candidate convolution kernel is 1, an initial width of the candidate convolution kernel is 1, the first multiple is 2, and the second multiple is 2.
In this embodiment, the convolution kernel obtaining unit is further configured to:
determining the size of the candidate convolution kernel according to the size information of the candidate convolution kernel;
amplifying the size of the candidate convolution kernel according to the size of the negative Laplacian, and obtaining the amplified size information of the candidate convolution kernel;
and obtaining the convolution kernel according to the amplified size information of the candidate convolution kernel.
In this embodiment, in the convolution kernel obtaining unit, the size information of the candidate convolution kernel includes height information of the candidate convolution kernel and width information of the candidate convolution kernel, and the enlarged size information of the candidate convolution kernel includes the enlarged height information of the candidate convolution kernel and the enlarged width information of the candidate convolution kernel;
The determining the size of the candidate convolution kernel according to the size information of the candidate convolution kernel includes: determining the height of the candidate convolution kernel according to the height information of the candidate convolution kernel, and determining the width of the candidate convolution kernel according to the width information of the candidate convolution kernel;
the step of amplifying the size of the candidate convolution kernel according to the size of the negative Laplacian to obtain amplified size information of the candidate convolution kernel, including: amplifying the height of the candidate convolution kernel according to the height of the negative Laplacian to obtain amplified height information of the candidate convolution kernel, and amplifying the width of the candidate convolution kernel according to the width of the negative Laplacian to obtain amplified width information of the candidate convolution kernel.
In this embodiment, the convolution kernel obtaining unit is further configured to:
judging whether filling processing is needed for the feature map data according to the attribute of the feature map data and the convolution kernel;
if yes, filling the feature map data, and carrying out convolution operation on the filled feature map data and the convolution kernel to obtain a convolution result.
In this embodiment, the object obtaining unit is specifically configured to:
performing convolution operation on the feature map data and the convolution kernels with the first number and different sizes respectively to obtain convolution results corresponding to the convolution kernels with the first number and different sizes, wherein the weights of the convolution kernels with the first number and different sizes are negative Laplacian operators;
the step of obtaining the object parts in the feature map data according to the convolution result comprises the following steps: and obtaining object parts in the feature map data according to convolution results corresponding to the convolution kernels with different sizes of the first number.
In this embodiment, the object division obtaining unit further includes convolution kernel obtaining units with different sizes, where the convolution kernel obtaining units with different sizes are configured to:
and obtaining the first number of convolution kernels with different sizes according to the attribute of the feature map data.
Optionally, the object obtaining unit is specifically configured to:
and synthesizing convolution results corresponding to the convolution kernels with different sizes of the first number to obtain object parts in the feature map data.
In this embodiment, the object obtaining unit is further configured to:
Splicing convolution results corresponding to the convolution kernels with different sizes to obtain spliced feature vectors of the feature map data;
and obtaining object parts of an interested region in the feature map data according to the spliced feature vector, wherein the interested region is a region possibly containing the object in the feature map data.
In this embodiment, the object obtaining unit is further configured to:
the region of interest is determined in the feature map data according to the size of the convolution kernel corresponding to the feature map data.
In this embodiment, the determining unit is specifically configured to:
determining a target area containing the object in the feature map data from the region of interest according to the object score;
and identifying the object in the object to be identified according to the target area.
A third embodiment of the present application provides an electronic device, including:
a processor;
a memory for storing a linked list of processing programs which, when read and executed by the processor, perform the following operations:
acquiring an image to be identified;
acquiring feature map data of the image to be identified;
obtaining object parts in the feature map data;
And determining the area where the object in the image to be identified is located according to the object part.
A fourth embodiment of the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
acquiring an image to be identified;
acquiring feature map data of the image to be identified;
obtaining object parts in the feature map data;
and determining the area where the object in the image to be identified is located according to the object part.
The fifth embodiment Shen Di provides a method for determining an area where an object is located from an object to be identified, and is substantially similar to the first embodiment, so that the description is relatively simple, and reference is made to relevant parts of the first embodiment for details.
The method for determining the area where the object is located from the image comprises the following steps:
acquiring an object to be identified;
acquiring feature map data of the object to be identified;
obtaining object parts in the feature map data by using a negative Laplacian operator;
and identifying the object in the object to be identified according to the object part. The object to be identified can be a static picture or a video acquired by video acquisition equipment such as a camera.
In this embodiment, the obtaining the object portion in the feature map data by using the negative laplacian operator includes:
performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result, wherein the weight of the convolution kernel is a negative Laplacian;
and obtaining the object parts in the characteristic map data according to the convolution result.
In this embodiment, the method for determining an area where an object is located from an image further includes:
determining an initial height of a candidate convolution kernel and an initial width of the candidate convolution kernel;
taking the initial height as a reference, performing multiplication operation of a first multiple to obtain a candidate height of the candidate convolution kernel;
taking the initial width as a reference, performing multiplication operation of a second multiple to obtain a candidate width of the candidate convolution kernel;
obtaining size information of the candidate convolution kernel according to the initial height, the initial width, the candidate height and the candidate width;
and obtaining the convolution kernel according to the size information of the candidate convolution kernel.
While the preferred embodiment has been described, it is not intended to limit the invention thereto, and any person skilled in the art may make variations and modifications without departing from the spirit and scope of the present invention, so that the scope of the present invention shall be defined by the claims of the present application.
In one typical configuration, a computing device includes one or more operators (CPUs), an input/output interface, a network interface, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
1. Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer readable media, as defined herein, does not include non-transitory computer readable media (transmission media), such as modulated data signals and carrier waves.
2. It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Claims (17)
1. A method for determining an area of an object from an image, comprising:
acquiring an image to be identified;
acquiring feature map data of the image to be identified;
obtaining object parts in the feature map data;
determining the region where the object in the image to be identified is located according to the object part;
wherein, the obtaining the object part in the feature map data includes: performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result, and obtaining object parts in the feature map data according to the convolution result, wherein the object parts specifically comprise: and respectively carrying out convolution operation on the feature map data and the convolution kernels with the first number and different sizes to obtain convolution results corresponding to the convolution kernels with the first number and different sizes, and obtaining object parts in the feature map data according to the convolution results corresponding to the convolution kernels with the first number and different sizes.
2. The method for determining an area where an object is located from an image according to claim 1, wherein the acquiring feature map data of the image to be identified includes:
and extracting features of the image to be identified by using a convolutional neural network, and acquiring feature map data of the image to be identified.
3. The method of determining an area of an object from an image of claim 1, further comprising:
determining an initial height of a candidate convolution kernel and an initial width of the candidate convolution kernel;
taking the initial height as a reference, performing multiplication operation of a first multiple to obtain a candidate height of the candidate convolution kernel;
taking the initial width as a reference, performing multiplication operation of a second multiple to obtain a candidate width of the candidate convolution kernel;
obtaining size information of the candidate convolution kernel according to the initial height, the initial width, the candidate height and the candidate width;
and obtaining the convolution kernel according to the size information of the candidate convolution kernel.
4. A method of determining an area of an object from an image according to claim 3, wherein the initial height of the candidate convolution kernel is 1, the initial width of the candidate convolution kernel is 1, the first multiple is 2, and the second multiple is 2.
5. A method for determining a region of an object from an image according to claim 3, wherein the obtaining the convolution kernel according to the size information of the candidate convolution kernel comprises:
determining the size of the candidate convolution kernel according to the size information of the candidate convolution kernel;
amplifying the size of the candidate convolution kernel to obtain amplified size information of the candidate convolution kernel;
and obtaining the convolution kernel according to the amplified size information of the candidate convolution kernel.
6. The method for determining an area where an object is located in an image according to claim 5, wherein the size information of the candidate convolution kernel includes height information of the candidate convolution kernel and width information of the candidate convolution kernel, and the size information of the enlarged candidate convolution kernel includes the height information of the enlarged candidate convolution kernel and the width information of the enlarged candidate convolution kernel;
the determining the size of the candidate convolution kernel according to the size information of the candidate convolution kernel includes: determining the height of the candidate convolution kernel according to the height information of the candidate convolution kernel, and determining the width of the candidate convolution kernel according to the width information of the candidate convolution kernel;
The step of amplifying the size of the candidate convolution kernel to obtain amplified size information of the candidate convolution kernel includes: amplifying the height of the candidate convolution kernel to obtain amplified height information of the candidate convolution kernel, and amplifying the width of the candidate convolution kernel to obtain amplified width information of the candidate convolution kernel.
7. The method for determining an area where an object is located in an image according to claim 3, wherein the performing a convolution operation on the feature map data and a convolution kernel to obtain a convolution result includes:
judging whether filling processing is needed for the feature map data according to the attribute of the feature map data and the convolution kernel;
if yes, filling the feature map data, and carrying out convolution operation on the filled feature map data and the convolution kernel to obtain a convolution result.
8. The method of determining an area of an object from an image of claim 1, further comprising:
and obtaining the first number of convolution kernels with different sizes according to the attribute of the feature map data.
9. The method for determining an area where an object is located in an image according to claim 1, wherein the obtaining the object portion in the feature map data according to the convolution results corresponding to the convolution kernels of the first number and the different sizes includes:
And synthesizing convolution results corresponding to the convolution kernels with different sizes of the first number to obtain object parts in the feature map data.
10. The method for determining an area where an object is located in an image according to claim 9, wherein the synthesizing the convolution results corresponding to the convolution kernels with different sizes of the first number to obtain the object portion in the feature map data includes:
splicing convolution results corresponding to the convolution kernels with different sizes to obtain spliced feature vectors of the feature map data;
and obtaining object parts of an interested region in the feature map data according to the spliced feature vector, wherein the interested region is a region possibly containing the object in the feature map data.
11. The method of determining a region of an object from an image according to claim 10, wherein the region of interest is determined in the feature map data according to a size of a convolution kernel corresponding to the feature map data.
12. The method for determining a region where an object is located in an image according to claim 10, wherein determining the region where the object is located in the image to be identified according to the object score comprises:
Determining a target area containing the object in the feature map data from the region of interest according to the object score;
and determining the area where the object in the image to be identified is located according to the target area.
13. An apparatus for determining an area of an object from an image, comprising:
the image acquisition unit is used for acquiring an image to be identified;
the feature acquisition unit is used for acquiring feature map data of the image to be identified;
an object division obtaining unit, configured to obtain an object division in the feature map data;
the determining unit is used for determining the area where the object in the image to be identified is located according to the object parts;
wherein, the obtaining the object part in the feature map data includes: performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result, and obtaining object parts in the feature map data according to the convolution result, wherein the object parts specifically comprise: and respectively carrying out convolution operation on the feature map data and the convolution kernels with the first number and different sizes to obtain convolution results corresponding to the convolution kernels with the first number and different sizes, and obtaining object parts in the feature map data according to the convolution results corresponding to the convolution kernels with the first number and different sizes.
14. An electronic device, the electronic device comprising:
a processor;
a memory for storing a linked list of processing programs which, when read and executed by the processor, perform the following operations:
acquiring an image to be identified;
acquiring feature map data of the image to be identified;
obtaining object parts in the feature map data;
determining the region where the object in the image to be identified is located according to the object part;
wherein, the obtaining the object part in the feature map data includes: performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result, and obtaining object parts in the feature map data according to the convolution result, wherein the object parts specifically comprise: and respectively carrying out convolution operation on the feature map data and the convolution kernels with the first number and different sizes to obtain convolution results corresponding to the convolution kernels with the first number and different sizes, and obtaining object parts in the feature map data according to the convolution results corresponding to the convolution kernels with the first number and different sizes.
15. A computer readable storage medium having stored thereon a computer program, characterized in that the program, when executed by a processor, realizes the steps of:
Acquiring an image to be identified;
acquiring feature map data of the image to be identified;
obtaining object parts in the feature map data;
determining the region where the object in the image to be identified is located according to the object part;
wherein, the obtaining the object part in the feature map data includes: performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result, and obtaining object parts in the feature map data according to the convolution result, wherein the object parts specifically comprise: and respectively carrying out convolution operation on the feature map data and the convolution kernels with the first number and different sizes to obtain convolution results corresponding to the convolution kernels with the first number and different sizes, and obtaining object parts in the feature map data according to the convolution results corresponding to the convolution kernels with the first number and different sizes.
16. A method for determining an area of an object from an object to be identified, comprising:
acquiring an object to be identified;
acquiring feature map data of the object to be identified;
obtaining object parts in the feature map data;
determining the region where the object in the object to be identified is located according to the object score;
wherein, the obtaining the object part in the feature map data includes: performing convolution operation on the feature map data and a convolution kernel to obtain a convolution result, and obtaining object parts in the feature map data according to the convolution result, wherein the object parts specifically comprise: and respectively carrying out convolution operation on the feature map data and the convolution kernels with the first number and different sizes to obtain convolution results corresponding to the convolution kernels with the first number and different sizes, and obtaining object parts in the feature map data according to the convolution results corresponding to the convolution kernels with the first number and different sizes.
17. The method of determining an area of an object from an object to be identified according to claim 16, further comprising:
determining an initial height of a candidate convolution kernel and an initial width of the candidate convolution kernel;
taking the initial height as a reference, performing multiplication operation of a first multiple to obtain a candidate height of the candidate convolution kernel;
taking the initial width as a reference, performing multiplication operation of a second multiple to obtain a candidate width of the candidate convolution kernel;
obtaining size information of the candidate convolution kernel according to the initial height, the initial width, the candidate height and the candidate width;
and obtaining the convolution kernel according to the size information of the candidate convolution kernel.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910106122.1A CN111523533B (en) | 2019-02-01 | 2019-02-01 | Method and device for determining area of object from image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910106122.1A CN111523533B (en) | 2019-02-01 | 2019-02-01 | Method and device for determining area of object from image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111523533A CN111523533A (en) | 2020-08-11 |
| CN111523533B true CN111523533B (en) | 2023-07-07 |
Family
ID=71900036
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910106122.1A Active CN111523533B (en) | 2019-02-01 | 2019-02-01 | Method and device for determining area of object from image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111523533B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113378873A (en) * | 2021-01-13 | 2021-09-10 | 杭州小创科技有限公司 | Algorithm for determining attribution or classification of target object |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104063719A (en) * | 2014-06-27 | 2014-09-24 | 深圳市赛为智能股份有限公司 | Method and device for pedestrian detection based on depth convolutional network |
| CN104933722A (en) * | 2015-06-29 | 2015-09-23 | 电子科技大学 | Image edge detection method based on Spiking-convolution network model |
| CN107315995A (en) * | 2017-05-18 | 2017-11-03 | 中国科学院上海微系统与信息技术研究所 | A kind of face identification method based on Laplce's logarithm face and convolutional neural networks |
| CN109086656A (en) * | 2018-06-06 | 2018-12-25 | 平安科技（深圳）有限公司 | Airport foreign matter detecting method, device, computer equipment and storage medium |
-
2019
- 2019-02-01 CN CN201910106122.1A patent/CN111523533B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104063719A (en) * | 2014-06-27 | 2014-09-24 | 深圳市赛为智能股份有限公司 | Method and device for pedestrian detection based on depth convolutional network |
| CN104933722A (en) * | 2015-06-29 | 2015-09-23 | 电子科技大学 | Image edge detection method based on Spiking-convolution network model |
| CN107315995A (en) * | 2017-05-18 | 2017-11-03 | 中国科学院上海微系统与信息技术研究所 | A kind of face identification method based on Laplce's logarithm face and convolutional neural networks |
| CN109086656A (en) * | 2018-06-06 | 2018-12-25 | 平安科技（深圳）有限公司 | Airport foreign matter detecting method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111523533A (en) | 2020-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10803554B2 (en) | Image processing method and device | |
| US9418458B2 (en) | Graph image representation from convolutional neural networks | |
| CN111161349B (en) | Object posture estimation method, device and equipment | |
| CN110852349B (en) | Image processing method, detection method, related equipment and storage medium | |
| JP2019087252A (en) | Apparatus and method for performing deconvolution operation in neural network | |
| US20130287250A1 (en) | Method and apparatus for tracking object in image data, and storage medium storing the same | |
| CN111091123A (en) | Text region detection method and equipment | |
| US20120163704A1 (en) | Apparatus and method for stereo matching | |
| CN108961180B (en) | Infrared image enhancement method and system | |
| CN111696110B (en) | Scene segmentation method and system | |
| CN110176024B (en) | Method, device, equipment and storage medium for detecting target in video | |
| CN111814905A (en) | Target detection method, target detection device, computer equipment and storage medium | |
| CN108022244B (en) | Hypergraph optimization method for significant target detection based on foreground and background seeds | |
| WO2017050083A1 (en) | Element identification method and device | |
| CN111738036B (en) | Image processing method, device, equipment and storage medium | |
| EP3942462B1 (en) | Convolution neural network based landmark tracker | |
| CN109255382B (en) | Neural network system, method and device for picture matching positioning | |
| CN110910445B (en) | Object size detection method, device, detection equipment and storage medium | |
| CN109902588B (en) | Gesture recognition method and device and computer readable storage medium | |
| CN114241388A (en) | Video instance segmentation method and segmentation device based on space-time memory information | |
| CN111523533B (en) | Method and device for determining area of object from image | |
| CN111914596A (en) | Lane line detection method, device, system and storage medium | |
| CN111027551B (en) | Image processing method, apparatus and medium | |
| US20140023260A1 (en) | Biological unit segmentation with ranking based on similarity applying a geometric shape and scale model | |
| CN116797830A (en) | Image risk classification method and device based on YOLOv7 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |