CN111339344A - Indoor image retrieval method and device and electronic equipment - Google Patents
Indoor image retrieval method and device and electronic equipment Download PDFInfo
- Publication number
- CN111339344A CN111339344A CN202010116190.9A CN202010116190A CN111339344A CN 111339344 A CN111339344 A CN 111339344A CN 202010116190 A CN202010116190 A CN 202010116190A CN 111339344 A CN111339344 A CN 111339344A
- Authority
- CN
- China
- Prior art keywords
- image
- line
- currently acquired
- clustering center
- structural
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5854—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using shape and object relationship
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Library & Information Science (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Databases & Information Systems (AREA)
- Image Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides an indoor image retrieval method and device and electronic equipment, and belongs to the technical field of image processing. Wherein, the method comprises the following steps: extracting the characteristics of the currently acquired image to acquire the line characteristics of each structural line in the image; determining a clustering center corresponding to each structural line according to the similarity between the line characteristic of each structural line and each preset clustering center; determining a difference vector corresponding to each clustering center according to the line characteristics of each clustering center and each corresponding structure line; performing fusion processing on the difference vector corresponding to each clustering center to generate structural line characteristics corresponding to the currently acquired image; and determining the images similar to the currently acquired images in the image library according to the structural line characteristics corresponding to the currently acquired images and the similarity between the structural line characteristics corresponding to the images in the image library. Therefore, by the indoor image retrieval method, the complexity of feature matching is reduced, and the accuracy of indoor positioning is improved.
Description
Technical Field
The application relates to the technical field of image processing, in particular to the technical field of computer vision, and provides an indoor image retrieval method and device and electronic equipment.
Background
The indoor image retrieval positioning refers to retrieving an existing image similar to a currently acquired image according to an image in a preset image library and preprocessed image information, and further performing positioning according to the retrieved similar image and information stored in the image library.
In the related art, when indoor positioning is performed, a similar image matched with a currently acquired image is determined according to the matching degree of the point features of the currently acquired image and the point features of images in a preset image library, and then indoor positioning is performed according to position information of the similar image.
However, the point feature data of the image is too much and has no significant features, so that the matching error phenomenon is easy to occur, and the accuracy of indoor image retrieval and positioning is reduced.
Disclosure of Invention
The method, the device and the electronic equipment for searching the indoor image are used for solving the problems that in the related technology, due to the fact that the point feature data volume of the image is too much and the point feature data volume does not have the significant feature, the matching error phenomenon is easy to occur in a mode of searching and positioning the indoor image through the point feature, and the accuracy of searching and positioning the indoor image is reduced.
An embodiment of an aspect of the present application provides an indoor image retrieval method, including: extracting the characteristics of the currently acquired image to acquire the line characteristics of each structural line in the image; determining a clustering center corresponding to each structural line according to the similarity between the line characteristic of each structural line and each preset clustering center; determining a difference vector corresponding to each clustering center according to the line characteristics of each clustering center and each corresponding structure line; fusing the difference vector corresponding to each clustering center to generate a structural line characteristic corresponding to the currently acquired image; and determining the images similar to the currently acquired images in the image library according to the structural line characteristics corresponding to the currently acquired images and the similarity between the structural line characteristics corresponding to the images in the image library.
An embodiment of another aspect of the present application provides an indoor image retrieval device, including: the first acquisition module is used for extracting the characteristics of the currently acquired image and acquiring the line characteristics of each structural line in the image; the first determining module is used for determining the clustering center corresponding to each structural line according to the similarity between the line characteristic of each structural line and each preset clustering center; the second determining module is used for determining a difference vector corresponding to each clustering center according to the line characteristics of each clustering center and each corresponding structural line; the generating module is used for carrying out fusion processing on the difference vector corresponding to each clustering center to generate the structural line characteristics corresponding to the currently acquired image; and the third determining module is used for determining the images similar to the currently acquired images in the image library according to the similarity between the structural line features corresponding to the currently acquired images and the structural line features corresponding to the images in the image library.
An embodiment of another aspect of the present application provides an electronic device, which includes: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the indoor image retrieval method as described above.
In another aspect, the present application provides a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are configured to cause the computer to execute the indoor image retrieval method as described above.
Any of the embodiments of the above applications has the following advantages or benefits: the structural line characteristics of the currently acquired image are determined according to the line characteristics of the structural lines included in the currently acquired image and the preset clustering center, and then the images similar to the currently acquired image in the image library are determined by utilizing the structural line characteristics of the currently acquired image. Because the technical means of extracting the characteristics of the currently acquired image, acquiring the line characteristics of each structural line in the image, determining the clustering center corresponding to each structural line according to the similarity between the line characteristics of each structural line and each preset clustering center, determining the difference vector corresponding to each clustering center according to the line characteristics of each clustering center and each corresponding structural line, fusing the difference vectors corresponding to each clustering center to generate the structural line characteristics corresponding to the currently acquired image, and determining the image similar to the currently acquired image in the image library according to the similarity between the structural line characteristics corresponding to the currently acquired image and the structural line characteristics corresponding to each image in the image library are adopted, the method overcomes the problem that the indoor image retrieval and positioning are carried out through the point characteristics due to the fact that the point characteristic data volume of the image is too much and does not have the significant characteristics, the matching error phenomenon is easy to occur, the problem of accuracy of indoor image retrieval and positioning is lowered, and therefore the technical effects that the structural characteristics of the indoor image are fully reflected through line characteristics, the complexity of characteristic matching is reduced, and the accuracy of indoor positioning is improved are achieved.
Other effects of the above-described alternative will be described below with reference to specific embodiments.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
fig. 1 is a schematic flowchart of an indoor image retrieval method according to an embodiment of the present disclosure;
fig. 2 is a schematic flowchart of another indoor image retrieval method according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of an indoor image retrieving device according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
The embodiment of the application aims at the problems that in the related art, due to the fact that the point feature data volume of an image is too much and the point feature data volume does not have a significant feature, matching errors are prone to occurring in a mode of carrying out indoor image retrieval positioning through point features, and accuracy of indoor image retrieval positioning is reduced, and the indoor image retrieval method is provided.
Hereinafter, a method, an apparatus, an electronic device, and a storage medium for retrieving an indoor image according to the present invention will be described in detail with reference to the accompanying drawings.
Fig. 1 is a schematic flowchart of an indoor image retrieval method according to an embodiment of the present disclosure.
As shown in fig. 1, the indoor image retrieval method includes the following steps:
It should be noted that, when the indoor positioning is realized through the indoor image retrieval, the image of the current position may be acquired through an image acquisition device such as a camera, and the currently acquired image is matched with the image in the image library to determine an image similar to the currently acquired image in the image library, and then the current position is positioned according to the position information of the similar image.
The line feature of the structure line refers to a feature vector representing the structure line. It should be noted that the dimension of the line feature may be preset according to actual needs, and this is not limited in the embodiment of the present application. For example, the dimension of the line feature may be 128 dimensions.
In this embodiment of the present application, an arbitrary structural line extraction algorithm may be adopted to perform feature extraction on the currently acquired image to obtain a structural line included in the currently acquired image, and perform feature representation on the obtained structural line to generate a line feature of each structural line.
As a possible implementation manner, manhattan structural line extraction may be performed on a currently acquired image, and line features of each structural line are determined according to the obtained gray values of the pixel points around each structural line.
The structural features of the indoor image are obvious, and the structural lines in the image can well represent the structural features of the image, so that the indoor image is characterized by the structural lines, and the accuracy of indoor image retrieval can be improved.
And 102, determining the clustering center corresponding to each structural line according to the similarity between the line characteristics of each structural line and each preset clustering center.
The similarity between the line features and the clustering centers can be measured by using the distances between the line features and the clustering centers. Specifically, the similarity between the linear feature and the clustering center is in positive correlation with the distance between the linear feature and the clustering center.
It should be noted that, in actual use, a parameter for measuring the similarity between the line feature of the structural line and the preset clustering center may be selected according to actual needs, which is not limited in the embodiment of the present application.
In the embodiment of the present application, the structural lines in the currently acquired image may be classified in a clustering manner first. Optionally, the distance between the line feature of each structural line and each preset clustering center may be determined as the similarity between the line feature of each structural line and each preset clustering center, and then the clustering center corresponding to each structural line is determined according to the relationship between the similarity between the line feature of each structural line and each preset clustering center and the first threshold.
Specifically, if the similarity between the line feature of the structural line and the preset clustering center a is greater than or equal to the first threshold, it may be determined that the clustering center corresponding to the structural line is the clustering center a.
Furthermore, image acquisition can be performed on the indoor environment in advance to generate an image set of the indoor environment, and then a preset clustering center is determined according to the acquired image set. That is, in a possible implementation form of the embodiment of the present application, before the step 102, the method may further include:
acquiring images of the indoor environment according to a preset acquisition rule to obtain an image set corresponding to the indoor environment;
extracting the characteristics of each image in the image set, and determining a line characteristic set of a structural line corresponding to the image set;
and clustering the line feature set to determine each preset clustering center.
The preset collection rule may include parameters such as a collection mode, a collection frequency, a movement speed of the image collection device, and a movement mode, which are used when the image collection is performed on the indoor environment, but is not limited thereto. It should be noted that, when the preset acquisition rule is used to acquire the image of the indoor environment, the acquired image set can basically cover the indoor environment; alternatively, the acquired image set may cover images of various positions of the indoor environment as required in actual application.
For example, the preset collection rule may be "collection mode: carrying on a vehicle; acquisition frequency: 10 times/second; moving speed of the image capturing apparatus: 5 m/s; the moving mode is as follows: surrounding the indoor environment ".
It should be noted that the above examples are only illustrative and should not be construed as limiting the present application. In actual use, each parameter included in the preset acquisition rule and the specific value of each parameter can be determined according to actual needs, which is not limited in the embodiment of the present application.
In the embodiment of the application, the image acquisition device may be controlled to acquire images of the indoor environment according to a preset acquisition rule, so that an image set corresponding to the indoor environment is generated by using the images of all the indoor environments acquired by the image acquisition device. After an image set corresponding to an indoor environment is obtained, feature extraction can be performed on each image in the image set, a structural line included in each image in the image set is determined, line features corresponding to each structural line are determined by utilizing gray values of pixel points around the structural line, and then line features of the structural line included in each image are utilized to form a line feature set corresponding to the image set. The manner of extracting the features of each image in the image set corresponding to the indoor environment should be the same as the manner of extracting the features of the currently acquired image.
After determining the line feature set corresponding to the image set of the indoor environment, sequentially inputting each line feature in the line feature set into a clustering device to perform clustering processing on the line feature set, further generating a plurality of clustering centers corresponding to the line feature set, and taking the determined plurality of clustering centers as preset clustering centers.
And 103, determining a difference vector corresponding to each clustering center according to the line characteristics of each clustering center and each corresponding structural line.
The difference vector corresponding to the cluster center is a vector that can represent the difference between the cluster center and the line feature of the corresponding structural line. Alternatively, the distance vector between the cluster center and the line feature of each corresponding structural line may be determined.
As a possible implementation manner, the difference vector corresponding to the cluster center may be a sum of distance vectors between the cluster center and the line feature of each structure line corresponding to the cluster center. That is, in a possible implementation form of the embodiment of the present application, the step 103 may include:
determining a difference value sub-vector corresponding to each clustering center according to the characteristics of each clustering center and the line characteristics of each corresponding structure line;
and performing summation operation on the difference sub-vectors corresponding to each clustering center to determine the difference vector corresponding to each clustering center.
The difference sub-vector corresponding to the cluster center is a vector that can represent the difference between the cluster center and the line feature of a corresponding structural line. Optionally, the difference sub-vector corresponding to the cluster center may be determined according to a distance vector between the cluster center and a line feature of a corresponding structural line.
As a possible implementation manner, after the cluster center corresponding to each structure line in the currently acquired image is determined, each difference sub-vector corresponding to the cluster center may be determined according to a distance vector between the cluster center and a line feature of each structure line corresponding to the cluster center, and then a sum of each difference sub-vector corresponding to the cluster center is determined as a difference vector corresponding to the cluster center, so as to represent a difference between the line feature of the structure line corresponding to the cluster center and the cluster center.
As a possible implementation manner, after determining the cluster center corresponding to each structure line in the currently acquired image, the mean value of the line features of the structure lines corresponding to each cluster center may also be determined first, and then the distance vector between the mean value of the line features of the structure lines corresponding to the cluster center and the cluster center is determined, and the distance vector is determined as the difference vector corresponding to the cluster center.
It should be noted that, by using the above method, the difference vector between the line features of all the structural lines in the currently acquired image and the corresponding cluster centers thereof can be determined, so as to represent the overall structural features of the currently acquired image.
And 104, fusing the difference vectors corresponding to each clustering center to generate structural line features corresponding to the currently acquired image.
The structural line feature corresponding to the currently acquired image is a vector that can represent the overall structural feature of the currently acquired image.
In this embodiment of the application, because the difference vector corresponding to the cluster center can only represent the difference between the line feature of the structural line corresponding to the cluster center and the cluster center, the difference vector corresponding to each cluster center can be fused, so that the fused difference vector can represent the features of all structural lines in the currently acquired image, that is, the fused difference vector is determined to be the structural line feature corresponding to the currently acquired image.
Optionally, the difference vectors corresponding to each clustering center are subjected to fusion processing, and the difference vectors corresponding to each clustering center may be spliced. That is, in a possible implementation form of the embodiment of the present application, the step 104 may include:
and splicing the difference vectors corresponding to each clustering center to generate the structural line characteristics corresponding to the currently acquired image.
In a possible implementation form of the embodiment of the application, the difference vectors corresponding to each clustering center may be spliced, and the generated vectors after splicing are determined as the structural line features corresponding to the currently acquired image, so that the structural line features corresponding to the currently acquired image are fused with the difference features of all the structural lines in the currently acquired image and the clustering centers, so as to represent the overall structural features of the currently acquired image.
Optionally, the difference vectors corresponding to each cluster center are fused, or the difference vectors corresponding to each cluster center are added, and the sum of the difference vectors corresponding to each cluster center is determined as the structural line feature corresponding to the currently acquired image, so that the structural line feature corresponding to the currently acquired image fuses the difference features of all the structural lines in the currently acquired image and the cluster centers to represent the overall structural feature of the currently acquired image.
Optionally, the difference vector corresponding to each cluster center is fused, or the difference vector corresponding to each cluster center is averaged, and the mean value of the difference vector corresponding to each cluster center is determined as the structural line feature corresponding to the currently acquired image, so that the structural line feature corresponding to the currently acquired image fuses the difference features of all the structural lines in the currently acquired image and the cluster centers to represent the overall structural feature of the currently acquired image.
It should be noted that, the manner of performing the fusion processing on the difference vector corresponding to each cluster center may include, but is not limited to, the above-listed cases. In actual use, a mode of performing fusion processing on the difference vector corresponding to each cluster center can be selected according to actual needs, and the embodiment of the present application does not limit this.
And 105, determining an image similar to the currently acquired image in the image library according to the structural line characteristics corresponding to the currently acquired image and the similarity between the structural line characteristics corresponding to the images in the image library.
It should be noted that the image library may include images acquired in an indoor environment in advance, and the structural line feature corresponding to each image. The method for determining the structural line feature corresponding to each image in the image library is the same as the method for determining the structural line feature corresponding to the currently acquired image, and is not described herein again.
In this embodiment of the application, because the structure line feature of the currently acquired image may represent the overall structure feature of the currently acquired image, and the structure line feature corresponding to each image in the image library may represent the overall structure feature of each image, the similarity between the structure line feature corresponding to the currently acquired image and the structure line feature corresponding to each image in the image library may reflect the similarity between the currently acquired image and each image in the image library. Specifically, the greater the similarity between the structure line feature corresponding to the currently acquired image and the structure line feature corresponding to an image in the image library, the greater the similarity between the currently acquired image and the image may be determined, so that the image similar to the currently acquired image in the image library may be determined according to the similarity between the structure line feature corresponding to the currently acquired image and the structure line feature corresponding to each image in the image library.
As a possible implementation manner, the similarity between the structural line feature corresponding to the currently acquired image and the structural line feature corresponding to the image in the image library may be determined according to the distance between the structural line feature corresponding to the currently acquired image and the structural line feature corresponding to the image in the image library. Specifically, the greater the distance between the structure line feature corresponding to the currently acquired image and the structure line feature corresponding to the image in the image library, the greater the difference between the structure line feature corresponding to the currently acquired image and the structure line feature corresponding to the image, that is, the smaller the similarity between the structure line feature corresponding to the currently acquired image and the structure line feature corresponding to the image; conversely, the smaller the distance between the structure line feature corresponding to the currently acquired image and the structure line feature corresponding to the image in the image library is, the smaller the difference between the structure line feature corresponding to the currently acquired image and the structure line feature corresponding to the image is, that is, the greater the similarity between the structure line feature corresponding to the currently acquired image and the structure line feature corresponding to the image is.
Optionally, after determining the structural line features corresponding to the currently acquired image and the similarity between the structural line features corresponding to the images in the image library, according to a preset second threshold, determining an image with the similarity between the structural line features corresponding to the currently acquired image being greater than or equal to the second threshold as an image similar to the currently acquired image.
Optionally, the number N of the acquired images similar to the currently acquired image may be preset, so that after determining the similarity between the structural line features corresponding to the currently acquired image and the structural line features corresponding to the images in the image library, the N images with the maximum similarity between the structural line features corresponding to the currently acquired image may be determined as the images similar to the currently acquired image.
Optionally, an image similar to the currently acquired image may be determined according to a preset second threshold and the number N of images. Specifically, the first N images, in which the similarity between the structural line features corresponding to the currently acquired image is greater than or equal to the second threshold, may be determined as images similar to the currently acquired image. That is, if the number of images in which the similarity between the structural line features corresponding to the currently acquired image is greater than or equal to the second threshold is greater than or equal to N, the N images in which the similarity between the structural line features corresponding to the currently acquired image is the greatest may be determined as the images similar to the currently acquired image; if the number of images with the similarity between the structural line features corresponding to the currently acquired image being greater than or equal to the second threshold is less than N, all images with the similarity between the structural line features corresponding to the currently acquired image being greater than or equal to the second threshold may be determined as images similar to the currently acquired image.
Wherein N is an integer greater than or equal to 1. In practical use, the specific value of N may be determined according to actual needs, which is not limited in the embodiments of the present application.
It should be noted that the manner of determining the image similar to the currently acquired image may include, but is not limited to, the above-listed situations. In actual use, a mode of determining an image similar to the currently acquired image may be selected according to actual needs, which is not limited in the embodiment of the present application.
According to the technical scheme of the embodiment of the application, the line characteristics of all structural lines in the image are obtained by extracting the characteristics of the currently acquired image, the clustering center corresponding to each structural line is determined according to the similarity between the line characteristics of all structural lines and each preset clustering center, the difference vector corresponding to each clustering center is determined according to the line characteristics of each clustering center and each corresponding structural line, then the difference vectors corresponding to each clustering center are subjected to fusion processing to generate the structural line characteristics corresponding to the currently acquired image, and then the image similar to the currently acquired image in the image library is determined according to the structural line characteristics corresponding to the currently acquired image and the similarity between the structural line characteristics corresponding to all images in the image library. Therefore, the structural line characteristics of the currently acquired image are determined according to the line characteristics of the structural lines included in the currently acquired image and the preset clustering center, and then the images similar to the currently acquired image in the image library are determined by utilizing the structural line characteristics of the currently acquired image.
In one possible implementation form of the present application, the image library may further include location information of each image, so that the location information of the currently acquired image may be determined according to the location information of an image similar to the currently acquired image, so as to implement indoor positioning.
The indoor image retrieval method provided by the embodiment of the present application is further described below with reference to fig. 2.
Fig. 2 is a schematic flowchart of another indoor image retrieval method according to an embodiment of the present disclosure.
As shown in fig. 2, the indoor image retrieval method includes the following steps:
And step 203, determining a difference vector corresponding to each clustering center according to the line characteristics of each clustering center and each corresponding structural line.
And 204, fusing the difference vectors corresponding to each clustering center to generate structural line features corresponding to the currently acquired image.
The detailed implementation process and principle of the step 201 and the step 205 can refer to the detailed description of the above embodiments, and are not described herein again.
And step 206, determining the acquisition position of the currently acquired image according to the acquisition position of the image similar to the currently acquired image.
As a possible implementation manner, the image library may further include an acquisition position of each image. When the indoor environment is subjected to image acquisition to construct an image library, the current position of the image acquisition equipment can be adopted to label the image currently acquired by the image acquisition equipment, namely, the current position of the image acquisition equipment is determined as the acquisition position of the image currently acquired by the image acquisition equipment, so that the constructed image library comprises a plurality of images of the indoor environment and the acquisition positions of the images.
It can be understood that when the image acquisition device is located at the same position, the acquired images are also the same; correspondingly, if the two images are the same or similar, the acquisition positions of the two images can be determined to be the same or similar. Therefore, in the embodiment of the present application, the acquisition position of the currently acquired image may be determined according to the acquisition position of an image similar to the currently acquired image in the image library.
Alternatively, if only one determined image similar to the currently acquired image is present, the acquisition position of the image similar to the currently acquired image may be determined as the acquisition position of the currently acquired image.
Optionally, if there are multiple determined images similar to the currently acquired image, the acquisition position of the image with the maximum similarity between the structure line features corresponding to the currently acquired image may be determined as the acquisition position of the currently acquired image; or, the central point of the acquisition positions of the plurality of similar images can be determined according to the acquisition positions of the plurality of images similar to the currently acquired image, and the central point is determined as the acquisition position of the currently acquired image.
It should be noted that, the manner of determining the acquisition position of the currently acquired image according to the acquisition positions of the similar images may include, but is not limited to, the above-listed situations. In actual use, the acquisition position of the currently acquired image can be selected and determined according to actual needs, which is not limited in the embodiment of the application.
According to the technical scheme of the embodiment of the application, the line characteristics of each structural line in the image are obtained by extracting the characteristics of the currently acquired image, and determining the clustering center corresponding to each structural line according to the similarity between the line characteristic of each structural line and each preset clustering center, determining a difference vector corresponding to each clustering center according to the line characteristics of each clustering center and each corresponding structure line, then performing fusion processing on the difference vectors corresponding to each clustering center to generate the structure line characteristics corresponding to the currently acquired image, further determining the image similar to the current collected image in the image library according to the structural line characteristics corresponding to the current collected image and the similarity between the structural line characteristics corresponding to the images in the image library, so as to determine the acquisition position of the currently acquired image according to the acquisition position of the image similar to the currently acquired image. Therefore, the structural line characteristics of the currently acquired image are determined according to the line characteristics of the structural lines included in the currently acquired image and the preset clustering center, and then the images similar to the currently acquired image in the image library are determined by utilizing the structural line characteristics of the currently acquired image.
In order to implement the above embodiments, the present application further provides an indoor image retrieval device.
Fig. 3 is a schematic structural diagram of an indoor image retrieval device according to an embodiment of the present application.
As shown in fig. 3, the indoor image search device 30 includes:
the first obtaining module 31 is configured to perform feature extraction on a currently acquired image, and obtain line features of each structural line in the image;
the first determining module 32 is configured to determine a clustering center corresponding to each structural line according to a similarity between a line feature of each structural line and each preset clustering center;
the second determining module 33 is configured to determine a difference vector corresponding to each clustering center according to the line feature of each clustering center and each corresponding structure line;
the generating module 34 is configured to perform fusion processing on the difference vector corresponding to each clustering center to generate a structural line feature corresponding to the currently acquired image;
the third determining module 35 is configured to determine an image similar to the currently acquired image in the image library according to the structural line feature corresponding to the currently acquired image and the similarity between the structural line features corresponding to the images in the image library.
In practical use, the indoor image retrieval device provided in the embodiment of the present application may be configured in any electronic device to execute the indoor image retrieval method.
According to the technical scheme of the embodiment of the application, the line characteristics of all structural lines in the image are obtained by extracting the characteristics of the currently acquired image, the clustering center corresponding to each structural line is determined according to the similarity between the line characteristics of all structural lines and each preset clustering center, the difference vector corresponding to each clustering center is determined according to the line characteristics of each clustering center and each corresponding structural line, then the difference vectors corresponding to each clustering center are subjected to fusion processing to generate the structural line characteristics corresponding to the currently acquired image, and then the image similar to the currently acquired image in the image library is determined according to the structural line characteristics corresponding to the currently acquired image and the similarity between the structural line characteristics corresponding to all images in the image library. Therefore, the structural line characteristics of the currently acquired image are determined according to the line characteristics of the structural lines included in the currently acquired image and the preset clustering center, and then the images similar to the currently acquired image in the image library are determined by utilizing the structural line characteristics of the currently acquired image.
In one possible implementation form of the present application, the indoor image search device 30 further includes:
the second acquisition module is used for acquiring images of the indoor environment according to a preset acquisition rule to acquire an image set corresponding to the indoor environment;
the fourth determining module is used for extracting the characteristics of each image in the image set and determining the line characteristic set of the structural line corresponding to the image set;
and the fifth determining module is used for clustering the linear feature set and determining each preset clustering center.
Further, in another possible implementation form of the present application, the second determining module 33 is specifically configured to:
determining a difference value sub-vector corresponding to each clustering center according to the characteristics of each clustering center and the line characteristics of each corresponding structure line;
and performing summation operation on the difference sub-vectors corresponding to each clustering center to determine the difference vector corresponding to each clustering center.
Further, in another possible implementation form of the present application, the generating module 34 is specifically configured to:
and splicing the difference vectors corresponding to each clustering center to generate the structural line characteristics corresponding to the currently acquired image.
Further, in another possible implementation form of the present application, the image library further includes an acquisition position of each image;
accordingly, the indoor image retrieval device 30 further includes:
and the sixth determining module is used for determining the acquisition position of the currently acquired image according to the acquisition position of the image similar to the currently acquired image.
It should be noted that the foregoing explanation of the embodiment of the indoor image retrieval method shown in fig. 1 and fig. 2 is also applicable to the indoor image retrieval device 30 of this embodiment, and is not repeated here.
According to the technical scheme of the embodiment of the application, the line characteristics of each structural line in the image are obtained by extracting the characteristics of the currently acquired image, and determining the clustering center corresponding to each structural line according to the similarity between the line characteristic of each structural line and each preset clustering center, determining a difference vector corresponding to each clustering center according to the line characteristics of each clustering center and each corresponding structure line, then performing fusion processing on the difference vectors corresponding to each clustering center to generate the structure line characteristics corresponding to the currently acquired image, further determining the image similar to the current collected image in the image library according to the structural line characteristics corresponding to the current collected image and the similarity between the structural line characteristics corresponding to the images in the image library, so as to determine the acquisition position of the currently acquired image according to the acquisition position of the image similar to the currently acquired image. Therefore, the structural line characteristics of the currently acquired image are determined according to the line characteristics of the structural lines included in the currently acquired image and the preset clustering center, and then the images similar to the currently acquired image in the image library are determined by utilizing the structural line characteristics of the currently acquired image.
According to an embodiment of the present application, an electronic device and a readable storage medium are also provided.
As shown in fig. 4, the present disclosure is a block diagram of an electronic device according to an embodiment of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 4, the electronic apparatus includes: one or more processors 401, memory 402, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each electronic device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). In fig. 4, one processor 401 is taken as an example.
The memory 402, as a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the indoor image retrieval method in the embodiment of the present application (for example, the first acquisition module 31, the first determination module 32, the second determination module 33, the generation module 34, and the third determination module 35 shown in fig. 3). The processor 401 executes various functional applications of the server and data processing by running non-transitory software programs, instructions, and modules stored in the memory 402, that is, implements the indoor image retrieval method in the above-described method embodiment.
The memory 402 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the electronic device of the indoor image retrieval method, and the like. Further, the memory 402 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 402 may optionally include a memory remotely located from the processor 401, and these remote memories may be connected to the electronics of the indoor image retrieval method through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device of the indoor image retrieval method may further include: an input device 403 and an output device 404. The processor 401, the memory 402, the input device 403 and the output device 404 may be connected by a bus or other means, and fig. 4 illustrates an example of a connection by a bus.
The input device 403 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic apparatus of the indoor image retrieval method, such as an input device of a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointing stick, one or more mouse buttons, a track ball, a joystick, or the like. The output devices 404 may include a display device, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibrating motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the application, the line characteristics of all structural lines in the image are obtained by extracting the characteristics of the currently acquired image, the clustering center corresponding to each structural line is determined according to the similarity between the line characteristics of all structural lines and each preset clustering center, the difference vector corresponding to each clustering center is determined according to the line characteristics of each clustering center and each corresponding structural line, then the difference vectors corresponding to each clustering center are subjected to fusion processing to generate the structural line characteristics corresponding to the currently acquired image, and then the image similar to the currently acquired image in the image library is determined according to the structural line characteristics corresponding to the currently acquired image and the similarity between the structural line characteristics corresponding to all images in the image library. Therefore, the structural line characteristics of the currently acquired image are determined according to the line characteristics of the structural lines included in the currently acquired image and the preset clustering center, and then the images similar to the currently acquired image in the image library are determined by utilizing the structural line characteristics of the currently acquired image.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (12)
1. An indoor image retrieval method, comprising:
extracting the characteristics of the currently acquired image to acquire the line characteristics of each structural line in the image;
determining a clustering center corresponding to each structural line according to the similarity between the line characteristic of each structural line and each preset clustering center;
determining a difference vector corresponding to each clustering center according to the line characteristics of each clustering center and each corresponding structure line;
fusing the difference vector corresponding to each clustering center to generate a structural line characteristic corresponding to the currently acquired image;
and determining the images similar to the currently acquired images in the image library according to the structural line characteristics corresponding to the currently acquired images and the similarity between the structural line characteristics corresponding to the images in the image library.
2. The method according to claim 1, wherein before determining the cluster center corresponding to each structural line according to the similarity between the line feature of each structural line and each preset cluster center feature, the method further comprises:
acquiring images of an indoor environment according to a preset acquisition rule to obtain an image set corresponding to the indoor environment;
performing feature extraction on each image in the image set, and determining a line feature set of a structural line corresponding to the image set;
and clustering the line feature set to determine each preset clustering center.
3. The method of claim 1, wherein determining the difference vector corresponding to each cluster center based on the line features of each cluster center and the corresponding line features of the respective structure lines comprises:
determining a difference value sub-vector corresponding to each clustering center according to the characteristics of each clustering center and the line characteristics of each corresponding structure line;
and performing summation operation on the difference sub-vectors corresponding to each clustering center to determine the difference vector corresponding to each clustering center.
4. The method according to claim 1, wherein the fusing the difference vectors corresponding to each cluster center to generate the structural line features corresponding to the currently acquired image comprises:
and splicing the difference vectors corresponding to each clustering center to generate the structural line characteristics corresponding to the currently acquired image.
5. The method of any of claims 1-4, wherein the image library further comprises an acquisition location for each image;
after determining the image similar to the currently acquired image in the image library, the method further includes:
and determining the acquisition position of the currently acquired image according to the acquisition position of the image similar to the currently acquired image.
6. An indoor image retrieval apparatus, comprising:
the first acquisition module is used for extracting the characteristics of the currently acquired image and acquiring the line characteristics of each structural line in the image;
the first determining module is used for determining the clustering center corresponding to each structural line according to the similarity between the line characteristic of each structural line and each preset clustering center;
the second determining module is used for determining a difference vector corresponding to each clustering center according to the line characteristics of each clustering center and each corresponding structural line;
the generating module is used for carrying out fusion processing on the difference vector corresponding to each clustering center to generate the structural line characteristics corresponding to the currently acquired image;
and the third determining module is used for determining the images similar to the currently acquired images in the image library according to the similarity between the structural line features corresponding to the currently acquired images and the structural line features corresponding to the images in the image library.
7. The apparatus of claim 6, further comprising:
the second acquisition module is used for acquiring images of an indoor environment according to a preset acquisition rule to acquire an image set corresponding to the indoor environment;
the fourth determining module is used for extracting the features of each image in the image set and determining a line feature set of a structural line corresponding to the image set;
and the fifth determining module is used for clustering the line feature set and determining each preset clustering center.
8. The apparatus of claim 6, wherein the second determining module is specifically configured to:
determining a difference value sub-vector corresponding to each clustering center according to the characteristics of each clustering center and the line characteristics of each corresponding structure line;
and performing summation operation on the difference sub-vectors corresponding to each clustering center to determine the difference vector corresponding to each clustering center.
9. The apparatus of claim 6, wherein the generation module is specifically configured to:
and splicing the difference vectors corresponding to each clustering center to generate the structural line characteristics corresponding to the currently acquired image.
10. The apparatus according to any one of claims 6-9, wherein the image library further comprises an acquisition location of each image;
the device, still include:
and the sixth determining module is used for determining the acquisition position of the currently acquired image according to the acquisition position of the image similar to the currently acquired image.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
12. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010116190.9A CN111339344B (en) | 2020-02-25 | 2020-02-25 | Indoor image retrieval method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010116190.9A CN111339344B (en) | 2020-02-25 | 2020-02-25 | Indoor image retrieval method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111339344A true CN111339344A (en) | 2020-06-26 |
| CN111339344B CN111339344B (en) | 2023-04-07 |
Family
ID=71181812
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010116190.9A Active CN111339344B (en) | 2020-02-25 | 2020-02-25 | Indoor image retrieval method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111339344B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112381877A (en) * | 2020-11-09 | 2021-02-19 | 北京百度网讯科技有限公司 | Positioning fusion and indoor positioning method, device, equipment and medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2003071687A1 (en) * | 2002-02-20 | 2003-08-28 | Tadahiro Ohmi | Data processing device |
| US20060204105A1 (en) * | 2005-03-11 | 2006-09-14 | Kabushiki Kaisha Toshiba | Image recognition method |
| WO2012114401A1 (en) * | 2011-02-23 | 2012-08-30 | 日本電気株式会社 | Feature point matching device, feature point matching method, and non-temporary computer-readable medium having feature point matching program stored thereon |
| CN103473545A (en) * | 2013-08-01 | 2013-12-25 | 西安交通大学 | Text-image similarity-degree measurement method based on multiple features |
| CN103810299A (en) * | 2014-03-10 | 2014-05-21 | 西安电子科技大学 | Image retrieval method on basis of multi-feature fusion |
| JP2015099536A (en) * | 2013-11-20 | 2015-05-28 | 東芝テック株式会社 | Chart area detection device and chart area detection method |
| CN109033308A (en) * | 2018-07-16 | 2018-12-18 | 安徽江淮汽车集团股份有限公司 | A kind of image search method and device |
| CN110297935A (en) * | 2019-06-28 | 2019-10-01 | 京东数字科技控股有限公司 | Image search method, device, medium and electronic equipment |
| CN110751209A (en) * | 2019-10-18 | 2020-02-04 | 北京邮电大学 | Intelligent typhoon intensity determination method integrating depth image classification and retrieval |
-
2020
- 2020-02-25 CN CN202010116190.9A patent/CN111339344B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2003071687A1 (en) * | 2002-02-20 | 2003-08-28 | Tadahiro Ohmi | Data processing device |
| US20060204105A1 (en) * | 2005-03-11 | 2006-09-14 | Kabushiki Kaisha Toshiba | Image recognition method |
| WO2012114401A1 (en) * | 2011-02-23 | 2012-08-30 | 日本電気株式会社 | Feature point matching device, feature point matching method, and non-temporary computer-readable medium having feature point matching program stored thereon |
| CN103473545A (en) * | 2013-08-01 | 2013-12-25 | 西安交通大学 | Text-image similarity-degree measurement method based on multiple features |
| JP2015099536A (en) * | 2013-11-20 | 2015-05-28 | 東芝テック株式会社 | Chart area detection device and chart area detection method |
| CN103810299A (en) * | 2014-03-10 | 2014-05-21 | 西安电子科技大学 | Image retrieval method on basis of multi-feature fusion |
| CN109033308A (en) * | 2018-07-16 | 2018-12-18 | 安徽江淮汽车集团股份有限公司 | A kind of image search method and device |
| CN110297935A (en) * | 2019-06-28 | 2019-10-01 | 京东数字科技控股有限公司 | Image search method, device, medium and electronic equipment |
| CN110751209A (en) * | 2019-10-18 | 2020-02-04 | 北京邮电大学 | Intelligent typhoon intensity determination method integrating depth image classification and retrieval |
Non-Patent Citations (3)
| Title |
|---|
| V.F. VASILIEV: "Recognition of 2-D object on geometrical quasisimilarity of polygonal representations", 《SMC\"98 CONFERENCE PROCEEDINGS. 1998 IEEE INTERNATIONAL CONFERENCE ON SYSTEMS, MAN, AND CYBERNETICS (CAT. NO.98CH36218)》 * |
| 张俊: "基于局部特征集合的图像匹配技术研究与应用", 《中国优秀硕士学位论文全文数据库（电子期刊）》 * |
| 魏怡等: "形状描述法在图像检索中的应用综述", 《系统工程与电子技术》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112381877A (en) * | 2020-11-09 | 2021-02-19 | 北京百度网讯科技有限公司 | Positioning fusion and indoor positioning method, device, equipment and medium |
| CN112381877B (en) * | 2020-11-09 | 2023-09-01 | 北京百度网讯科技有限公司 | Positioning fusion and indoor positioning method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111339344B (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112036509A (en) | Method and apparatus for training image recognition models | |
| CN111241819B (en) | Word vector generation method and device and electronic equipment | |
| CN111539514A (en) | Method and apparatus for generating structure of neural network | |
| CN112270711B (en) | Model training and posture prediction method, device, equipment and storage medium | |
| CN111259671A (en) | Semantic description processing method, device and equipment for text entity | |
| CN111242306A (en) | Method, apparatus, electronic device, and computer-readable storage medium for quantum principal component analysis | |
| CN111507355A (en) | Character recognition method, device, equipment and storage medium | |
| CN111241838B (en) | Semantic relation processing method, device and equipment for text entity | |
| CN111582477A (en) | Training method and device of neural network model | |
| CN111611990A (en) | Method and device for identifying table in image | |
| CN112241716B (en) | Training sample generation method and device | |
| CN111708477B (en) | Key identification method, device, equipment and storage medium | |
| CN111090991A (en) | Scene error correction method and device, electronic equipment and storage medium | |
| CN111640103A (en) | Image detection method, device, equipment and storage medium | |
| CN110532415B (en) | Image search processing method, device, equipment and storage medium | |
| CN111967304A (en) | Method and device for acquiring article information based on edge calculation and settlement table | |
| CN113157829A (en) | Method and device for comparing interest point names, electronic equipment and storage medium | |
| CN115359308A (en) | Model training method, apparatus, device, storage medium, and program for identifying difficult cases | |
| CN111339344B (en) | Indoor image retrieval method and device and electronic equipment | |
| CN111563541B (en) | Training method and device of image detection model | |
| CN112488126A (en) | Feature map processing method, device, equipment and storage medium | |
| CN111767990A (en) | Neural network processing method and device | |
| CN112328896A (en) | Method, apparatus, electronic device, and medium for outputting information | |
| CN111488972A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN111814651A (en) | Method, device and equipment for generating lane line |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |