CN110704659B - Image list ordering method and device, storage medium and electronic device - Google Patents
Image list ordering method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN110704659B CN110704659B CN201910945807.5A CN201910945807A CN110704659B CN 110704659 B CN110704659 B CN 110704659B CN 201910945807 A CN201910945807 A CN 201910945807A CN 110704659 B CN110704659 B CN 110704659B
- Authority
- CN
- China
- Prior art keywords
- image
- distance
- list
- initial
- similarity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000004590 computer program Methods 0.000 claims description 12
- 238000004422 calculation algorithm Methods 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 4
- 238000012163 sequencing technique Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 6
- 238000005259 measurement Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/538—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Image Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a device for ordering an image list, a storage medium and an electronic device. Wherein the method comprises the following steps: acquiring a first initial image list associated with a target image from a first image database; acquiring at least one second distance between each image in the first initial image list and the target image; determining a second similarity between each image in the first initial image list and the target image according to the second distance; and sorting the first initial image list according to the second similarity to obtain a target image list. The method for adjusting the first initial image list by acquiring the second distance between the target image and each image in the first image database is adopted, and the similarity degree between the images is determined from more angles, so that the accuracy of image sorting is greatly improved, and the technical problem of inaccurate sorting results caused by incapability of accurately acquiring the similarity degree between the images is solved.
Description
Technical Field
The present invention relates to the field of image recognition, and in particular, to a method and apparatus for sorting an image list, a storage medium, and an electronic apparatus.
Background
In the related art, human Re-identification (Person Re-ID) is a challenging task in computer vision. In general, the face re-recognition Person re-ID can be regarded as a retrieval problem. Face recognition is desirable to enable tracking across cameras using computer vision algorithms, i.e., to find the same person under different cameras. Given a face search image, it is desirable to search the database for images that contain the same pedestrian in the cross-camera mode. After the initial ordering list is obtained, a practical scheme with good ordering effect comprises the steps of adding re-ordering, and images with higher correlation are expected to obtain higher ranks, so that the important point of face re-identification in the related technology is the accuracy of re-ordering.
The re-ranking method in the related art is used for improving the target retrieval precision. The related art utilizes a k-nearest neighbor algorithm (k-nearest neighbors) to explore similarity relationships to solve the re-ranking problem.
The scheme of reordering re-ranking in the related art depends largely on the quality of the initial ordered list. In the related technical scheme, similar relations among the top-ranked images in the initial list are utilized. Assuming that the reordered returned image is within k nearest neighbors of the search image, the returned image may be an image that matches the search image more closely, but if the wrong matching image is contained in k nearest neighbors, the return may deviate from the optimal case.
Aiming at the problem that the sorting result is inaccurate due to the fact that the similarity degree between the images cannot be accurately obtained, no effective solution is proposed at present.
Disclosure of Invention
The embodiment of the invention provides a method and a device for ordering an image list, a storage medium and an electronic device, which at least solve the technical problem of inaccurate ordering results caused by incapability of accurately acquiring the similarity between images.
According to an aspect of an embodiment of the present invention, there is provided a method for sorting an image list, including: acquiring a first initial image list associated with a target image from a first image database, wherein images in the first initial image list are ordered according to first similarity, and the first similarity is a first distance between the images in the first initial image list and the target image; acquiring at least one second distance between each image in the first initial image list and the target image; determining a second similarity between each image in the first initial image list and the target image according to the second distance; and sorting the first initial image list according to the second similarity to obtain a target image list.
According to another aspect of the present invention, there is also provided an image list sorting apparatus including: the first acquisition module is used for acquiring a first initial image list associated with a target image from a first image database, wherein images in the first initial image list are ordered according to first similarity, and the first similarity is a first distance between the images in the first initial image list and the target image; a second acquisition module, configured to acquire at least one second distance between each image in the first initial image list and the target image; a determining module, configured to determine a second similarity between each image in the first initial image list and the target image according to the second distance; and the ordering module is used for ordering the first initial image list according to the second similarity to obtain a target image list.
Optionally, the determining module is further configured to traverse the first initial image list, take each image as a first image, and perform the following steps: determining the first distance and the second distance between a first image and the target image in the first initial image list; inputting the first distance and the second distance into a distance combination model, wherein the distance combination model is obtained after training by using a plurality of sample data, and the distance combination model is used for carrying out weighted summation on the first distance and the second distance to obtain a second similarity; and obtaining an output result output by the distance combination model, wherein the output result comprises a second similarity between the first image and the target image.
Optionally, the determining module is further configured to determine, by using the distance combination model, a first weight corresponding to the first distance and a second weight corresponding to the second distance; and weighting and summing the first distance and the second distance by using the first weight and the second weight to obtain a second similarity between the first image and the target image.
Optionally, the second acquisition module further comprises at least one of: acquiring the sequencing Rank-order distance between each image and the target image; and acquiring the Jaccard distance between each image and the target image.
Optionally, the second obtaining module obtaining the Rank-order distance between each image and the target image includes: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring a first initial image list of the target image and a second initial image list of the first image, wherein a second initial image list associated with the first image is acquired from a second image database, the images in the second initial image list are ordered according to first similarity, the first similarity is a first distance between the images in the second initial image list and the first image, and the second image database comprises the first image database and the target image; acquiring a first position serial number of an image in the first initial image list in the second initial image list, a second position serial number of an image in the second initial image list in the first initial image list, a third position serial number of the target image in the second initial image list, and a fourth position serial number of the first image in the first initial image list; and obtaining a Rank-order distance between the first image and the target image according to the first position sequence number, the second position sequence number, the third position sequence number and the fourth position sequence number.
Optionally, a Rank-order distance d between the first image and the target image is obtained by the following formula R (q，g i ) The target image is denoted as q, and the first image is denoted as g i ：
wherein ,fq (j) A j-th image in the first initial image list representing the target image q,representing image f q (j) At the first image g i Is a sequence number bit in the second initial image list,/or->Representing the first image g i J-th image in said second initial image list,/and->Representation of image->A sequence number bit, O, in the first initial image list of the target image q q (g i ) Representing the first image g i A sequence number bit in said first initial image list of said target image q +.>Representing the target image q in the first image g i The sequence number bit in the second initial image list of (c), wherein j is a positive integer.
Optionally, the second acquiring module acquiring the Jaccard distance between each image and the target image includes: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring a first k-nearest neighbor image set corresponding to the target image and a second k-nearest neighbor image set corresponding to the first image; acquiring an intersection and a union between the second k-nearest neighbor image set and the first k-nearest neighbor image set; and determining the Jaccard distance between the first image and the target image according to the intersection set and the union set. Optionally, the Jaccard coefficients of the two images are determined according to the ratio of the intersection set and the union set, and the difference between 1 and the Jaccard coefficients is the Jaccard distance.
Optionally, the second obtaining module is further configured to obtain a jaccard distance between each image and the target image by the following formulaWherein the target image is denoted as q and the first image is denoted as g i ：
wherein ,/>
wherein ,/>
wherein ,is the first k-nearest neighbor image set of the target image q,/and>is the first image g i Is said +.>The k is an integer, which is a target threshold for filtering noise data in k-nearest neighbor algorithm results.
According to a further aspect of embodiments of the present invention, there is also provided a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the above-described method of ordering image lists when run.
According to still another aspect of the embodiments of the present invention, there is further provided an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the above-mentioned image list sorting method through the computer program.
In the embodiment of the invention, a first initial image list associated with a target image is acquired from a first image database, wherein images in the first initial image list are ordered according to first similarity, and the first similarity is a first distance between the images in the first initial image list and the target image; acquiring at least one second distance between each image in the first initial image list and the target image; determining a second similarity between each image in the first initial image list and the target image according to the second distance; and sorting the first initial image list according to the second similarity to obtain a target image list. The method comprises the steps of determining the similarity degree between images from more angles by adopting a mode of adjusting a first initial image list by acquiring a second distance between a target image and each image in a first image database, so that a more accurate similarity degree judgment result is obtained, the accuracy of image sorting is greatly improved, and the technical problem of inaccurate sorting result caused by the fact that the similarity degree between the images cannot be accurately acquired is solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
fig. 1 is a schematic view of an application scenario of a method for sorting an image list according to an embodiment of the present application;
FIG. 2 is a flow chart of a method of ordering image lists according to an embodiment of the application;
FIG. 3 is a schematic view of a scenario of a method of ordering an image list according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a sorting apparatus for image lists according to an embodiment of the present application;
FIG. 5 is a second schematic structural diagram of an image list sorting apparatus according to an embodiment of the present application;
FIG. 6 is a schematic diagram III of a structure of an image list sorting apparatus according to an embodiment of the present application;
fig. 7 is a schematic diagram of a structure of an image list sorting apparatus according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device implementing a method for sorting an image list according to an embodiment of the present application.
Detailed Description
In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In order to reduce interference by noise data, the application introduces Rank-order to restrict, and determines the similarity of two neighbors if and only if both are in k-neighbors of each other and their neighbors are also in k-neighbors of each other. Under the simple Euclidean distance measurement according to the original characteristics in the related art, the Jaccard distance is introduced, and the consideration of the context information is added. Further, in order to reduce noise interference, rank-order distance is introduced, and the three are combined to determine the true distance of the two images to a certain extent.
After measuring various distances, the application can directly adopt a mode of artificial judgment to remove the combination, and can carry out supervised learning through a two-class network so as to obtain a finer weighting scheme, thereby more truly reflecting the influence of three distances on the similarity degree between images.
In one embodiment of the application, a scheme for sorting according to various distances is provided, aiming at improving the hit rate of the Re-identification in Re-ID Re-identification application scenes. Re-ranking is a key step in improving the accuracy of face re-recognition person re-ID when it is considered as a retrieval process. However, in the re-identification re-ID retrieval process in the face of a complex scene, the use of only euclidean distance is far from sufficient, and the sorting effect is poor. Therefore, on the basis of Euclidean distance, the embodiment of the application introduces the sorting Rank-order distance and the Jacquard distance. In an alternative embodiment, for the three distances, the image distance measurement mode with higher robustness is learned through the shallow neural network, so that the image sorting is completed.
According to an aspect of the embodiment of the present application, there is provided a method for sorting an image list, optionally, as an optional implementation manner, the method for sorting an image list may be applied, but is not limited to, in the environment shown in fig. 1. The above-described method of ranking image lists may be, but is not limited to being, applied to the server 104 for assisting an application client in performing a search process for similar images on published target images. The application client may be, but not limited to, running in the user device 102, and the user device 102 may be, but not limited to, a terminal device supporting running of the application client, such as a mobile phone, a tablet computer, a notebook computer, a PC, etc. The server 104 and the user device 102 may implement data interaction through a network, which may include, but is not limited to, a wireless network or a wired network. Wherein the wireless network comprises: bluetooth, WIFI, and other networks that enable wireless communications. The wired network may include, but is not limited to: wide area network, metropolitan area network, local area network. The above is merely an example, and is not limited in any way in the present embodiment.
The application environment in fig. 1 comprises the following steps:
step one, user equipment sends a retrieved target image to a server;
step two, the server retrieves images similar to the target image according to the first similarity in a first image database, and a first initial image list is generated;
step three, the server adjusts the first initial image list according to the second distance between the image in the first initial image list and the target image to obtain a target image list;
and step four, the server feeds back the target image list to the user equipment.
Optionally, as an optional embodiment, as shown in fig. 2, the method for sorting the image list includes the following steps:
step S202, a first initial image list associated with a target image is obtained from a first image database, wherein images in the first initial image list are ordered according to first similarity, and the first similarity is a first distance between the images in the first initial image list and the target image;
the first image database may be a gamma in the related art. The first similarity may be calculated based on the euclidean distance between the two images.
The query person of the target image is marked as q;
First image database set g= { G i I=1, 2, …, N }, the feature of the image is extracted by the resnet101Characterizing and measuring the target image q and the image g in the first image database i Euclidean distance d between E (q，g i ) Wherein, the resnet101 is a depth residual network model obtained after optimization based on a residual neural network resnet (Residual Nerural Netwrok, simply referred to as resnet):
wherein ,xp ，The feature vectors of the two images are respectively sorted from small to large according to the distance, and a first initial image list L (q, G) is generated:
the goal of the subsequent flow of fig. 2 is to reorder the first initial image list so that more positive samples appear in the front section of the list.
Step S204, at least one second distance between each image in the first initial image list and the target image is acquired;
the second distance may be a Jaccard distance between the k-nearest neighbor image of each image and the k-nearest neighbor image of the target image, or a Jaccard coefficient, or a Rank-order distance of each image from the target image. The k-nearest neighbor image is an image set which is obtained by executing a k-nearest neighbor algorithm on an image and is most similar to the image in a feature space. The k-nearest neighbor algorithm may include the steps of: and calculating the distance between the points of the image and other images, sequentially sorting according to the distance, selecting k images of other images with the minimum distance from the image, and taking the k images as k adjacent images.
When the second distance is a Jaccard distance, acquiring the Jaccard distance between each image and the target image by: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring a first k-nearest neighbor image set corresponding to the target image and a second k-nearest neighbor image set corresponding to the first image; acquiring an intersection set and a union set between the second k-nearest neighbor image set and the first k-nearest neighbor image set; determining the Jaccard distance between the first image and the target image according to the intersection set and the union set;
when the second distance is a Rank-order distance, acquiring the Rank-order distance between each image and the target image by the following method comprises the following steps: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring the first initial image list of the target image and a second initial image list of the first image, wherein a second initial image list associated with the first image is acquired from a second image database, the images in the second initial image list are ordered according to a first similarity, the first similarity is a first distance between the images in the second initial image list and the first image, and the second image database comprises the first image database and the target image; acquiring a first position serial number of an image in the first initial image list in the second initial image list, a second position serial number of an image in the second initial image list in the first initial image list, a third position serial number of the target image in the second initial image list, and a fourth position serial number of the first image in the first initial image list; and obtaining a Rank-order distance between the first image and the target image according to the first position sequence number, the second position sequence number, the third position sequence number and the fourth position sequence number.
The Jaccard distance is used to compare dissimilarities between limited sample sets. The greater the Jaccard distance, the lower the sample similarity. The index related to the Jaccard distance is called a Jaccard coefficient, and is used to describe similarity and difference between sets, and the larger the Jaccard coefficient value is, the higher the sample similarity is.
Rank-order distance is used to measure the similarity of two faces, and is based on an interesting observation: two faces of the same person have many neighbors in common, but neighbors from faces of different people often vary widely.
Step S206, determining a second similarity between each image in the first initial image list and the target image according to the second distance;
when the first distance is a euclidean distance and the second distance is a Jaccard distance and a Rank-order distance, the second similarity may be a value obtained by performing weighted calculation based on the euclidean distance, the Jaccard distance and the Rank-order distance.
Step S208, sorting the first initial image list according to the second similarity to obtain a target image list.
Optionally, on the basis of similarity calculation for all features of the two images in the scheme of the flow of fig. 2, similarity calculation can be performed on local features of the two images, so as to assist in increasing or decreasing the accuracy of image sorting.
And adjusting the image sequence in the first initial image list to obtain a target image list, and ensuring that the image most similar to the target image appears in front of the target image list.
In the embodiment of the invention, the first initial image list is adjusted by acquiring the second distance between the target image and each image in the first image database, and the similarity between the images is determined from more angles, so that a more accurate similarity judgment result is obtained, the accuracy of image sorting is greatly improved, and the technical problem of inaccurate sorting result caused by the fact that the similarity between the images cannot be accurately acquired is solved.
Optionally, the determining, according to the second distance, a second similarity between each image in the first initial image list of the initial image list and the target image includes: traversing the first initial image list, taking each image as a first image, and executing the following steps: determining the first distance and the second distance between the first image and the target image in the first initial image list; inputting the first distance and the second distance into a distance combination model, wherein the distance combination model is obtained after training by using a plurality of sample data, and the distance combination model is used for carrying out weighted summation on the first distance and the second distance to obtain a second similarity; and obtaining an output result output by the distance combination model, wherein the output result comprises a second similarity between the first image and the target image. The distance combination model can be obtained by using a classification method for supervision training.
The input of the distance combination model is Euclidean distance between two images, jaccard distance and Rank-order distance, and the distance combination model is used for judging whether the objects in the two images are the same object or different objects. In the training stage, the training is carried out by using simple two classifications, the labeling result can be 0 or 1,0 is represented as that two images are different, 1 is represented as that the two images are the same, and the optimal combination weighting mode of three distances is obtained by monitoring the training mode. Because the network is simple, very little annotation data is needed, and once the weighting mode is determined, retraining in a new scene is not needed, and the data is not dependent.
Alternatively, the distance combination model may be designed by a lightweight linear support vector machine (Support Vector Machine, abbreviated as SVM), logistic regression, or other machine learning model.
The distance combination model may be:
d * (q，g i )＝f(d E (q，g i )，d J (q，g i )，d R (q，g i ))；
d * (q，g i ) For the target image q and the image g in the first image database i Is a second degree of similarity of (2); d, d E (q，g i ) Is the Euclidean distance between two images; d, d J (q，g i ) Is the Jaccard distance, d, between two images R (q，g i ) Is the Rank-order distance between the two images.
Optionally, the inputting the first distance and the second distance into the distance combination model includes: determining a first weight corresponding to the first distance and a second weight corresponding to the second distance through the distance combination model; and weighting and summing the first distance and the second distance by using the first weight and the second weight to obtain a second similarity between the first image and the target image. When the second distance is Jaccard distance and Rank-order distance, the Jaccard distance and Rank-order distance are correspondingly provided with different second weights. By adopting the scheme, different weights are set for different types of distances, so that a more accurate numerical value for describing the second similarity of the similarity degree between the two images is obtained, and the image reordering result is ensured.
Optionally, the acquiring at least one second distance between each image in the first initial image list and the target image includes at least one of: acquiring a Rank-order distance between each image and the target image; and acquiring the Jaccard distance between each image and the target image. The second distance may also include Jaccard coefficients or the like that describe the degree of similarity between different images.
Optionally, acquiring a Rank-order distance between the each image and the target image includes: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring the first initial image list of the target image and a second initial image list of the first image, wherein a second initial image list associated with the first image is acquired from a second image database, the images in the second initial image list are ordered according to a first similarity, the first similarity is a first distance between the images in the second initial image list and the first image, and the second image database comprises the first image database and the target image; acquiring a first position serial number of an image in the first initial image list in the second initial image list, a second position serial number of an image in the second initial image list in the first initial image list, a third position serial number of the target image in the second initial image list, and a fourth position serial number of the first image in the first initial image list; and obtaining a Rank-order distance between the first image and the target image according to the first position sequence number, the second position sequence number, the third position sequence number and the fourth position sequence number.
Optionally, aObtaining a Rank-order distance d between the first image and the target image by the following formula R (q，g i ) The target image is denoted as q, and the first image is denoted as g i ：
wherein ,fq (j) A j-th image in the first initial image list representing the target image q,representing image f q (j) In the first image g i The number bits in the second initial image list correspond to the first position numbers in the above-described embodiment. />Representing the first image g i A j-th image in the second initial image list,representation of image->The sequence number bits in the first initial image list of the target image q correspond to the second position sequence numbers in the above-described embodiment. O (O) q (g i ) Representing the first image g i The number bits in the first initial image list of the target image q correspond to the fourth position numbers in the above-described embodiment. />Representing the target image q in the first image g i The number bits in the second initial image list correspond to the third position numbers in the above-described embodiment. Wherein j is a positive integer. />The smallest of the two values is indicated. By adopting the scheme, the Rank-order distance is determined according to the image ordering condition in the initial image list between the two images, and a basis is provided for calculating the second similarity.
Optionally, acquiring the Jaccard distance between the each image and the target image includes: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring a first k-nearest neighbor image set corresponding to the target image and a second k-nearest neighbor image set corresponding to the first image; acquiring an intersection set and a union set between the second k-nearest neighbor image set and the first k-nearest neighbor image set; the Jaccard distance between the first image and the target image is determined from the intersection and the union. By adopting the scheme, the Jaccard distance is calculated by using the k neighbor image sets of the two images, and a basis is provided for calculating the second similarity.
The k-nearest neighbor image set may be obtained by performing a k-nearest neighbor algorithm on the image.
Optionally, the jaccard distance between the each image and the target image is obtained by the following formulaWherein the target image is denoted as q and the first image is denoted as g i ：
wherein ,/>
wherein ,/>
wherein ,the first k-nearest neighbor image set, which is the target image q,>is the first image g i Is the second k nearest neighbor image set of (1), the +.>The k is an integer, which is a target threshold for filtering noise data in the k-nearest neighbor algorithm result. In order to avoid noise interference, a threshold value +. >Noise data is filtered out.
The scheme scene diagram of the embodiment of the application is shown in figure 3, and comprises a target image q, a first image database and an image g in a first initial image list i And q and g i Is further calculated q and g i Jaccard distance d of (V) J Rank-order distance d R European distance d E The final distance d (also referred to as a second similarity) obtained by the distance combination model is finally obtained. And reordering the initial image list according to the final distance d to obtain a target image list.
By adopting the scheme, the following technical effects are realized: various distances such as Rank-order, jaccard and Euclidean distance are introduced, so that the investigation of the distances is more diversified, and the similarity between images is truly reflected. And (3) analyzing the relevant information of k neighbor images of each pair of images through Jaccard distance, and analyzing the relevant information between initial image lists of each pair of images through Rank-order distance, and establishing a combined distance model of re-ranking. By jointly considering the characteristics and the context information, fuzzy samples are effectively removed, re-identification re-ID performance is improved, more consideration is introduced to neighbor information, and the method is very effective to service scenes with large data difference in re-identification re-ID category; aiming at the traditional manually designed weighting measurement mode, a shallow two-class network learning combination weighting mode for multiple distances is designed by using supervised learning, so that the distance measurement mode can be designed more finely, and the retrieval performance is improved. In the combined distance measurement mode, a more comprehensive weighting mode can be provided by using a small amount of supervision information, and the retrieval precision of re-identification re-ID is greatly improved.
It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present invention is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present invention. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present invention.
According to another aspect of the embodiment of the present invention, there is also provided an image list sorting apparatus for implementing the above image list sorting method. As shown in fig. 4, the apparatus 40 includes:
a first obtaining module 42, configured to obtain a first initial image list associated with a target image from a first image database, where images in the first initial image list are ranked according to a first similarity, and the first similarity is a first distance between the images in the first initial image list and the target image;
a second obtaining module 44, configured to obtain at least one second distance between each image in the first initial image list and the target image;
A determining module 46, configured to determine a second similarity between each image in the first initial image list and the target image according to the second distance;
the sorting module 48 is configured to sort the first initial image list according to the second similarity, so as to obtain a target image list.
In the embodiment of the invention, the first initial image list is adjusted by acquiring the second distance between the target image and each image in the first image database, and the similarity between the images is determined from more angles, so that a more accurate similarity judgment result is obtained, the accuracy of image sorting is greatly improved, and the technical problem of inaccurate sorting result caused by the fact that the similarity between the images cannot be accurately acquired is solved.
Optionally, as shown in fig. 5, the determining module 46 is further configured to traverse the first initial image list, and perform the following steps in the sub-units with each image being the first image:
a first determining subunit 52, configured to determine the first distance and the second distance between the first image and the target image in the first initial image list;
an input subunit 54, configured to input the first distance and the second distance into a distance combination model, where the distance combination model is obtained after training using a plurality of sample data, and the distance combination model is configured to weight and sum the first distance and the second distance to obtain a second similarity;
A first obtaining subunit 56, configured to obtain an output result output by the distance combination model, where the output result includes a second similarity between the first image and the target image.
Optionally, the input subunit 54 is further configured to determine, through the distance combination model, a first weight corresponding to the first distance, and a second weight corresponding to the second distance; and the first weighting and the second weighting are used for carrying out weighted summation on the first distance and the second distance so as to obtain a second similarity between the first image and the target image.
Optionally, the second obtaining module 44 is further configured to obtain a Rank-order distance between the each image and the target image; and/or further used for acquiring the Jaccard distance between each image and the target image.
Optionally, as shown in fig. 6, the second obtaining module 44 obtains a Rank-order distance between the each image and the target image, including: traversing the first initial image list, taking each image as a first image, and executing the following steps in the subunit:
a second obtaining subunit 62, configured to obtain the first initial image list of the target image and a second initial image list of the first image, where the second initial image list associated with the first image is obtained from a second image database, and images in the second initial image list are ordered according to a first similarity, where the first similarity is a first distance between an image in the second initial image list and the first image, and the second image database includes the first image database and the target image;
A third acquiring subunit 64 that acquires a first position number of an image in the first initial image list in the second initial image list, a second position number of an image in the second initial image list, a third position number of the target image in the second initial image list, and a fourth position number of the first image in the first initial image list; and obtaining a Rank-order distance between the first image and the target image according to the first position sequence number, the second position sequence number, the third position sequence number and the fourth position sequence number. The third acquisition subunit 64 can calculate the Rank-order distance between the two images by the formula in the above embodiment.
Optionally, as shown in fig. 7, the second obtaining module 44 obtaining the Jaccard distance between each image and the target image includes: traversing the first initial image list, taking each image as a first image, and executing the following steps in the subunit:
a fourth obtaining subunit 72, configured to obtain a first k-nearest neighbor image set corresponding to the target image, and a second k-nearest neighbor image set corresponding to the first image;
A fifth acquisition subunit 74 configured to acquire an intersection and a union between the second k-nearest neighbor image set and the first k-nearest neighbor image set;
a sixth acquisition subunit 76 is configured to determine the Jaccard distance between the first image and the target image based on the intersection and the union.
Optionally, the second obtaining module 44 is further configured to obtain a jaccard distance between the each image and the target image by the following formulaWherein the target image is denoted as q and the first image is denoted as g i ：/>
wherein ,/>
wherein ,/>
wherein ,the first k-nearest neighbor image set, which is the target image q,>is the first image g i Is the second k nearest neighbor image set of (1), the +.>The k is an integer, which is a target threshold for filtering noise data in the k-nearest neighbor algorithm result.
According to a further aspect of the embodiments of the present invention, there is also provided an electronic device for implementing the above-mentioned image list sorting method, as shown in fig. 8, the electronic device comprising a memory 802 and a processor 804, the memory 802 storing a computer program, the processor 804 being arranged to execute the steps of any of the method embodiments described above by means of the computer program.
Alternatively, in this embodiment, the electronic apparatus may be located in at least one network device of a plurality of network devices of the computer network.
Alternatively, in the present embodiment, the above-described processor may be configured to execute the following steps by a computer program:
s1, acquiring a first initial image list associated with a target image from a first image database, wherein images in the first initial image list are ordered according to first similarity, and the first similarity is a first distance between the images in the first initial image list and the target image;
s2, acquiring at least one second distance between each image in the first initial image list and the target image;
s3, determining a second similarity between each image in the first initial image list and the target image according to the second distance;
and S4, sorting the first initial image list according to the second similarity to obtain a target image list.
Alternatively, it will be understood by those skilled in the art that the structure shown in fig. 8 is only schematic, and the electronic device may also be a terminal device such as a smart phone (e.g. an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a mobile internet device (Mobile Internet Devices, MID), a PAD, etc. Fig. 8 is not limited to the structure of the electronic device. For example, the electronic device may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 8, or have a different configuration than shown in FIG. 8.
The memory 802 may be used to store software programs and modules, such as program instructions/modules corresponding to the image list sorting method and apparatus in the embodiment of the present invention, and the processor 804 executes the software programs and modules stored in the memory 802, thereby executing various functional applications and data processing, that is, implementing the image list sorting method described above. Memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 802 may further include memory remotely located relative to processor 804, which may be connected to the terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 802 may be, but is not limited to, information such as a distance value between the first image database and the image. As an example, as shown in fig. 8, the memory 802 may include, but is not limited to, a first acquiring module 42, a second acquiring module 44, a determining module 46, and a sorting module 48 in the sorting apparatus including the image list. In addition, other module units in the sorting device of the image list may be included but not limited to, which is not described in detail in this example.
Optionally, the transmission device 806 is used to receive or transmit data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission means 806 includes a network adapter (Network Interface Controller, NIC) that can connect to other network devices and routers via a network cable to communicate with the internet or a local area network. In one example, the transmission device 806 is a Radio Frequency (RF) module for communicating wirelessly with the internet.
In addition, the electronic device further includes: a display 808 for displaying the target image and the images in the first image database; and a connection bus 810 for connecting the respective module parts in the above-described electronic device.
According to a further aspect of embodiments of the present invention, there is also provided a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.
Alternatively, in the present embodiment, the above-described computer-readable storage medium may be configured to store a computer program for executing the steps of:
S1, acquiring a first initial image list associated with a target image from a first image database, wherein images in the first initial image list are ordered according to first similarity, and the first similarity is a first distance between the images in the first initial image list and the target image;
s2, acquiring at least one second distance between each image in the first initial image list and the target image;
s3, determining a second similarity between each image in the first initial image list and the target image according to the second distance;
and S4, sorting the first initial image list according to the second similarity to obtain a target image list.
Alternatively, in this embodiment, it will be understood by those skilled in the art that all or part of the steps in the methods of the above embodiments may be performed by a program for instructing a terminal device to execute the steps, where the program may be stored in a computer readable storage medium, and the storage medium may include: flash disk, read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), magnetic or optical disk, and the like.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
The integrated units in the above embodiments may be stored in the above-described computer-readable storage medium if implemented in the form of software functional units and sold or used as separate products. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing one or more computer devices (which may be personal computers, servers or network devices, etc.) to perform all or part of the steps of the method of the various embodiments of the present application.
In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.
In several embodiments provided by the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiment of the apparatus is merely exemplary, and the division of the unit is merely a logic function division, and there may be another division manner in actual implementation, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The foregoing is merely a preferred embodiment of the invention, and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.
Claims (11)
1. A method for ordering a list of images, comprising:
acquiring a first initial image list associated with a target image from a first image database, wherein images in the first initial image list are ordered according to first similarity, the first similarity is a first distance between the images in the first initial image list and the target image, and the first distance is a Euclidean distance;
Acquiring at least one second distance between each image in the first initial image list and the target image, wherein the at least one second distance is Jaccard distance and/or Rank-order distance;
determining a second similarity between each image in the first initial image list and the target image according to the second distance, wherein the second similarity is obtained by inputting the first distance and the second distance into a distance combination model, the distance combination model is used for determining a first weight corresponding to the first distance and a second weight corresponding to the second distance, and weighting and summing the first distance and the second distance by using the first weight and the second weight to obtain the second similarity, wherein when the at least one second distance is the Jaccard distance and the Rank-order distance, the Jaccard distance and the Rank-order distance correspond to different second weights;
and sorting the first initial image list according to the second similarity to obtain a target image list.
2. The method of claim 1, wherein determining a second similarity between each image in the first initial image list of the initial image list and the target image based on the second distance comprises:
Traversing the first initial image list, taking each image as a first image, and executing the following steps:
determining the first distance and the second distance between a first image and the target image in the first initial image list;
inputting the first distance and the second distance into a distance combination model, wherein the distance combination model is obtained after training by using a plurality of sample data, and the distance combination model is used for carrying out weighted summation on the first distance and the second distance to obtain a second similarity;
and obtaining an output result output by the distance combination model, wherein the output result comprises a second similarity between the first image and the target image.
3. The method of claim 2, wherein the inputting the first distance and the second distance into a distance combining model comprises:
determining a first weight corresponding to the first distance and a second weight corresponding to the second distance through the distance combination model;
and weighting and summing the first distance and the second distance by using the first weight and the second weight to obtain a second similarity between the first image and the target image.
4. The method of claim 1, wherein the acquiring at least one second distance between each image in the first initial image list and the target image comprises at least one of:
acquiring the sequencing Rank-order distance between each image and the target image; and acquiring the Jaccard distance between each image and the target image.
5. The method of claim 4, wherein obtaining a Rank-order distance between each image and the target image comprises:
traversing the first initial image list, taking each image as a first image, and executing the following steps:
acquiring a first initial image list of the target image and a second initial image list of the first image, wherein a second initial image list associated with the first image is acquired from a second image database, the images in the second initial image list are ordered according to first similarity, the first similarity is a first distance between the images in the second initial image list and the first image, and the second image database comprises the first image database and the target image;
Acquiring a first position serial number of an image in the first initial image list in the second initial image list, a second position serial number of an image in the second initial image list in the first initial image list, a third position serial number of the target image in the second initial image list, and a fourth position serial number of the first image in the first initial image list;
and obtaining a Rank-order distance between the first image and the target image according to the first position sequence number, the second position sequence number, the third position sequence number and the fourth position sequence number.
6. The method of claim 5, wherein the Rank-order distance d between the first image and the target image is obtained by the following formula R (q,g i ) The target image is denoted as q, and the first image is denoted as g i ：
wherein ,fq (j) A j-th image in the first initial image list representing the target image q,representing image f q (j) At the first image g i Is a sequence number bit in the second initial image list,/or->Representing the first image g i J-th image in said second initial image list,/and- >Representation of image->A sequence number bit, O, in the first initial image list of the target image q q (g i ) Representing the first image g i A sequence number bit in said first initial image list of said target image q +.>Representing the target image q in the first image g i The sequence number bit in the second initial image list of (c), wherein j is a positive integer.
7. The method of claim 4, wherein acquiring the Jaccard distance between each image and the target image comprises:
traversing the first initial image list, taking each image as a first image, and executing the following steps:
acquiring a first k-nearest neighbor image set corresponding to the target image and a second k-nearest neighbor image set corresponding to the first image;
acquiring an intersection and a union between the second k-nearest neighbor image set and the first k-nearest neighbor image set;
and determining the Jaccard distance between the first image and the target image according to the intersection set and the union set.
8. The method of claim 7, wherein the jaccard distance between each image and the target image is obtained by the following formula Wherein the target image is denoted as q and the first image is denoted as g i ：
wherein ,/>
wherein ,/>
wherein ,is the first k-nearest neighbor image set of the target image q,
is the first image g i Is said +.>The k is an integer, which is a target threshold for filtering noise data in k-nearest neighbor algorithm results.
9. An apparatus for sorting an image list, comprising:
the first acquisition module is used for acquiring a first initial image list associated with a target image from a first image database, wherein images in the first initial image list are ordered according to first similarity, and the first similarity is a first distance between the images in the first initial image list and the target image;
the second acquisition module is used for acquiring at least one second distance between each image in the first initial image list and the target image, wherein the first distance is a Euclidean distance;
the determining module is used for determining a second similarity between each image in the first initial image list and the target image according to the second distance, and the at least one second distance is a Jaccard distance and/or a Rank-order distance;
The sorting module is configured to sort the first initial image list according to the second similarity to obtain a target image list, where the second similarity is a similarity obtained by inputting the first distance and the second distance into a distance combination model, the distance combination model is configured to determine a first weight corresponding to the first distance and a second weight corresponding to the second distance, and weight and sum the first distance and the second distance by using the first weight and the second weight to obtain the second similarity, where when the at least one second distance is the Jaccard distance and the Rank-order distance, the Jaccard distance and the Rank-order distance correspond to different second weights.
10. A computer readable storage medium comprising a stored program, wherein the program when run performs the method of any one of the preceding claims 1 to 8.
11. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method according to any of the claims 1 to 8 by means of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910945807.5A CN110704659B (en) | 2019-09-30 | 2019-09-30 | Image list ordering method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910945807.5A CN110704659B (en) | 2019-09-30 | 2019-09-30 | Image list ordering method and device, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110704659A CN110704659A (en) | 2020-01-17 |
| CN110704659B true CN110704659B (en) | 2023-09-26 |
Family
ID=69196710
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910945807.5A Active CN110704659B (en) | 2019-09-30 | 2019-09-30 | Image list ordering method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110704659B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112417197B (en) * | 2020-12-02 | 2022-02-25 | 云从科技集团股份有限公司 | Sorting method, sorting device, machine readable medium and equipment |
| CN112784086A (en) * | 2021-01-28 | 2021-05-11 | 北京有竹居网络技术有限公司 | Picture screening method and device, storage medium and electronic equipment |
| CN113343008A (en) * | 2021-06-29 | 2021-09-03 | 维沃软件技术有限公司 | Image sorting method, image sorting device, electronic device and readable storage medium |
| CN113505723B (en) * | 2021-07-23 | 2024-05-24 | 北京京东尚科信息技术有限公司 | Pedestrian re-recognition method and device, storage medium and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105303150A (en) * | 2014-06-26 | 2016-02-03 | 腾讯科技（深圳）有限公司 | Method and system for implementing image processing |
| CN106708943A (en) * | 2016-11-22 | 2017-05-24 | 安徽睿极智能科技有限公司 | Image retrieval reordering method and system based on arrangement fusion |
| CN109165612A (en) * | 2018-08-31 | 2019-01-08 | 苏州大学 | Pedestrian's recognition methods again based on depth characteristic and two-way KNN sorting consistence |
| DE102017222916A1 (en) * | 2017-12-15 | 2019-06-19 | Rohde & Schwarz Gmbh & Co. Kg | Device and measuring method for the identification of network devices |
| CN110059198A (en) * | 2019-04-08 | 2019-07-26 | 浙江大学 | A kind of discrete Hash search method across modal data kept based on similitude |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10120879B2 (en) * | 2013-11-29 | 2018-11-06 | Canon Kabushiki Kaisha | Scalable attribute-driven image retrieval and re-ranking |
| KR101910581B1 (en) * | 2017-06-16 | 2018-10-22 | 라인 가부시키가이샤 | Method and system of chat room list sorting based on conversational activeness and contextrual information |
-
2019
- 2019-09-30 CN CN201910945807.5A patent/CN110704659B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105303150A (en) * | 2014-06-26 | 2016-02-03 | 腾讯科技（深圳）有限公司 | Method and system for implementing image processing |
| CN106708943A (en) * | 2016-11-22 | 2017-05-24 | 安徽睿极智能科技有限公司 | Image retrieval reordering method and system based on arrangement fusion |
| DE102017222916A1 (en) * | 2017-12-15 | 2019-06-19 | Rohde & Schwarz Gmbh & Co. Kg | Device and measuring method for the identification of network devices |
| CN109165612A (en) * | 2018-08-31 | 2019-01-08 | 苏州大学 | Pedestrian's recognition methods again based on depth characteristic and two-way KNN sorting consistence |
| CN110059198A (en) * | 2019-04-08 | 2019-07-26 | 浙江大学 | A kind of discrete Hash search method across modal data kept based on similitude |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110704659A (en) | 2020-01-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110704659B (en) | Image list ordering method and device, storage medium and electronic device | |
| CN111523621B (en) | Image recognition method and device, computer equipment and storage medium | |
| TWI677852B (en) | A method and apparatus, electronic equipment, computer readable storage medium for extracting image feature | |
| CN108595585B (en) | Sample data classification method, model training method, electronic equipment and storage medium | |
| US8724910B1 (en) | Selection of representative images | |
| JP2018205937A (en) | Image retrieval device and program | |
| CN111275060B (en) | Identification model updating processing method and device, electronic equipment and storage medium | |
| EP3028184A1 (en) | Method and system for searching images | |
| CN107330750A (en) | A kind of recommended products figure method and device, electronic equipment | |
| US11126826B1 (en) | Machine learning system and method for recognizing facial images | |
| CN111401344A (en) | Face recognition method and device and training method and device of face recognition system | |
| CN106557728B (en) | Query image processing and image search method and device and monitoring system | |
| JP7036401B2 (en) | Learning server, image collection support system for insufficient learning, and image estimation program for insufficient learning | |
| CN112785441B (en) | Data processing method, device, terminal equipment and storage medium | |
| CN114139013A (en) | Image searching method and device, electronic equipment and computer readable storage medium | |
| WO2015146113A1 (en) | Identification dictionary learning system, identification dictionary learning method, and recording medium | |
| CN112818148B (en) | Visual retrieval sequencing optimization method and device, electronic equipment and storage medium | |
| CN113591629B (en) | Finger tri-modal fusion recognition method, system, device and storage medium | |
| CN110276283B (en) | Picture identification method, target identification model training method and device | |
| CN111782774B (en) | Method and device for recommending problems | |
| Biswas et al. | Large scale image clustering with active pairwise constraints | |
| CN113641855A (en) | Video recommendation method, device, equipment and storage medium | |
| KR101896177B1 (en) | An Image Searching System Providing Multiplex Result | |
| Boparai et al. | A hybrid approach for improving content based image retrieval systems | |
| CN111191119A (en) | Neural network-based scientific and technological achievement self-learning method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40018620 Country of ref document: HK |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |