CN110704659A - Image list sorting method and device, storage medium and electronic device - Google Patents
Image list sorting method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN110704659A CN110704659A CN201910945807.5A CN201910945807A CN110704659A CN 110704659 A CN110704659 A CN 110704659A CN 201910945807 A CN201910945807 A CN 201910945807A CN 110704659 A CN110704659 A CN 110704659A
- Authority
- CN
- China
- Prior art keywords
- image
- distance
- list
- initial
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000012163 sequencing technique Methods 0.000 claims abstract description 12
- 230000015654 memory Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 12
- 238000004422 calculation algorithm Methods 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 5
- 238000005259 measurement Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/538—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Image Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a device for sorting an image list, a storage medium and an electronic device. Wherein, the method comprises the following steps: acquiring a first initial image list associated with a target image from a first image database; acquiring at least one second distance between each image in the first initial image list and the target image; determining a second similarity between each image in the first initial image list and the target image according to the second distance; and sequencing the first initial image list according to the second similarity to obtain a target image list. The method for adjusting the first initial image list by obtaining the second distance between the target image and each image in the first image database is adopted, the similarity degree between the images is determined from more angles, the image sorting accuracy is greatly improved, and the technical problem that the sorting result is inaccurate due to the fact that the similarity degree between the images cannot be accurately obtained is solved.
Description
Technical Field
The invention relates to the field of image recognition, in particular to a method and a device for sorting an image list, a storage medium and an electronic device.
Background
In the related art, face Re-identification (Person Re-ID) is a challenging subject in computer vision. In general, face re-recognition Person re-ID can be considered a retrieval problem. Face re-recognition is a desire to utilize computer vision algorithms for cross-camera tracking, i.e., finding the same person under different cameras. Given a face search image, it is desirable to search the database for images containing the same pedestrian in a cross-camera mode. After the initial ranking list is obtained, the practical scheme with good ranking effect includes the step of adding re-ranking, and images with higher relevance are expected to obtain higher ranking, so the emphasis of face re-recognition in the related technology is on the accuracy of re-ranking.
The re-ranking method in the related art is used for improving the target retrieval precision. In the related art, a k-nearest neighbors algorithm (k-nearest neighbors) is utilized to explore similarity relations so as to solve the re-ranking problem.
The scheme of reordering re-ranking in the related art depends to a large extent on the quality of the initial ordered list. In the related art scheme, the similarity relation between the images ranked at the top in the initial list is utilized. Assuming that the returned image after reordering is within the k nearest neighbor of the search image, the returned image may be an image having a high degree of matching with the search image, but the return may also deviate from the best case if an erroneous matching image is contained in the k nearest neighbor.
In order to solve the problem that the sequencing result is inaccurate due to the fact that the similarity degree between the images cannot be accurately acquired, an effective solution is not provided at present.
Disclosure of Invention
The embodiment of the invention provides a method and a device for ordering an image list, a storage medium and an electronic device, which at least solve the technical problem of inaccurate ordering result caused by the fact that the similarity degree between images cannot be accurately acquired.
According to an aspect of the embodiments of the present invention, there is provided a method for sorting an image list, including: acquiring a first initial image list associated with a target image from a first image database, wherein the images in the first initial image list are sorted according to a first similarity, and the first similarity is a first distance between the image in the first initial image list and the target image; acquiring at least one second distance between each image in the first initial image list and the target image; determining a second similarity between each image in the first initial image list and the target image according to the second distance; and sequencing the first initial image list according to the second similarity to obtain a target image list.
According to another aspect of the present invention, there is also provided an apparatus for sorting an image list, including: the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a first initial image list associated with a target image from a first image database, images in the first initial image list are sorted according to a first similarity, and the first similarity is a first distance between the image in the first initial image list and the target image; a second obtaining module, configured to obtain at least one second distance between each image in the first initial image list and the target image; the determining module is used for determining a second similarity between each image in the first initial image list and the target image according to the second distance; and the sequencing module is used for sequencing the first initial image list according to the second similarity to obtain a target image list.
Optionally, the determining module is further configured to traverse the first initial image list, regarding each image as a first image, and perform the following steps: determining the first distance and the second distance between a first image in the first initial image list and the target image; inputting the first distance and the second distance into a distance combination model, wherein the distance combination model is obtained after training by using a plurality of sample data, and the distance combination model is used for weighting and summing the first distance and the second distance to obtain a second similarity; and acquiring an output result output by the distance combination model, wherein the output result comprises a second similarity between the first image and the target image.
Optionally, the determining module is further configured to determine a first weight corresponding to the first distance and a second weight corresponding to the second distance through the distance combination model; and carrying out weighted summation on the first distance and the second distance by using the first weight and the second weight so as to obtain a second similarity between the first image and the target image.
Optionally, the second obtaining module further includes at least one of: acquiring a Rank-order distance between each image and the target image; acquiring a Jacard Jaccard distance between each image and the target image.
Optionally, the acquiring, by the second acquisition module, a Rank-order distance between each image and the target image includes: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring the first initial image list of the target image and a second initial image list of the first image, wherein a second initial image list associated with the first image is acquired from a second image database, the images in the second initial image list are sorted according to a first similarity, the first similarity is a first distance between the images in the second initial image list and the first image, and the second image database comprises the first image database and the target image; acquiring a first position sequence number of an image in the first initial image list in the second initial image list, a second position sequence number of an image in the second initial image list in the first initial image list, a third position sequence number of the target image in the second initial image list, and a fourth position sequence number of the first image in the first initial image list; and obtaining the Rank-order distance between the first image and the target image according to the first position sequence number, the second position sequence number, the third position sequence number and the fourth position sequence number.
Optionally, the Rank-order distance d between the first image and the target image is obtained by the following formulaR(q，gi) The target image is recorded as q, and the first image is recorded as gi：
wherein ,fq(j) A jth image in said first initial image list representing said target image q,
representing image fq(j) In the first image giThe sequence number bits in the second initial image list,
representing said first image giThe jth image in the second initial image list,
representing images
Number bit, O, in the first initial image list of the target image qq(gi) Representing said first image giThe sequence number bits in the first initial image list of the target image q,
represents the aboveThe target image q is in the first image giWherein j is a positive integer.
Optionally, the second acquiring module acquiring the Jaccard distance between each image and the target image comprises: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring a first k adjacent image set corresponding to the target image and a second k adjacent image set corresponding to the first image; acquiring an intersection and a union between the second k neighbor image set and the first k neighbor image set; determining the Jaccard distance between the first image and the target image according to the intersection and the union. Optionally, the Jaccard coefficients of the two images are determined according to the ratio of the intersection to the union, and the difference between 1 and the Jaccard coefficients is the Jaccard distance.
Optionally, the second obtaining module is further configured to obtain a jaccard distance between each image and the target image according to the following formula
Wherein the target image is recorded as q, and the first image is recorded as gi：
wherein ,
wherein ,
is the first k set of neighboring images of the target image q,
is the first image giThe second k neighbor image set of
Is a target threshold for filtering noise data in k-nearest neighbor algorithm results, k being an integer.
According to still another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the above sorting method of the image list when running.
According to another aspect of the embodiments of the present invention, there is also provided an electronic apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above sorting method for the image list through the computer program.
In the embodiment of the invention, a first initial image list associated with a target image is acquired from a first image database, wherein the images in the first initial image list are sorted according to a first similarity, and the first similarity is a first distance between the image in the first initial image list and the target image; acquiring at least one second distance between each image in the first initial image list and the target image; determining a second similarity between each image in the first initial image list and the target image according to the second distance; and sequencing the first initial image list according to the second similarity to obtain a target image list. The method for adjusting the first initial image list by obtaining the second distance between the target image and each image in the first image database is adopted, the similarity degree between the images is determined from more angles, so that a more accurate similarity degree judgment result is obtained, the accuracy of image sorting is greatly improved, and the technical problem that the sorting result is inaccurate due to the fact that the similarity degree between the images cannot be accurately obtained is solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a schematic view of an application scenario of a sorting method for an image list according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method of ordering a list of images according to an embodiment of the invention;
FIG. 3 is a schematic diagram of a scenario of a method of ordering a list of images according to an embodiment of the present invention;
FIG. 4 is a first schematic structural diagram of an apparatus for sorting an image list according to an embodiment of the present invention;
FIG. 5 is a second schematic structural diagram of an apparatus for sorting an image list according to an embodiment of the present invention;
FIG. 6 is a third schematic structural diagram of an apparatus for sorting a list of images according to an embodiment of the present invention;
FIG. 7 is a fourth schematic structural diagram of an image list sorting apparatus according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device implementing a method for sorting an image list according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In order to reduce the interference of noisy data, the method introduces Rank-order de-constraint, and the similarity of two is judged if and only if the two are in k-neighbors of each other and the neighbors are also in k-neighbors of each other. In the related art, under the simple Euclidean distance measurement based on the original features, the Jaccard distance is introduced, and the consideration of the context information is added. Further, in order to reduce noise interference, a Rank-order distance is introduced, and the three determine the real distance of the two images to a certain extent.
After measuring various distances, the method can be used for directly adopting a manual judgment mode to combine, and supervised learning can be carried out through a two-classification network so as to obtain a more detailed weighting scheme, so that the influence of the three distances on the similarity degree between the images can be reflected more truly.
In one embodiment of the application, a scheme for sorting according to various distances is provided, and the aim is to improve the hit rate of Re-sorting in the Re-ID Re-identification application scene. When people re-identify person re-ID is regarded as a retrieval process, re-ranking is a key step for improving the accuracy of the re-identify person re-ID. However, when a re-identification re-ID retrieval process of a complex scene is faced, the Euclidean distance is far from sufficient, and the sorting effect is poor. Therefore, on the basis of the Euclidean distance, the sequencing Rank-order distance and the Jaccard Jacxed distance are introduced in the embodiment of the application. In an alternative embodiment, image ranking is accomplished by learning a more robust image distance metric through a shallow neural network for these three distances.
According to an aspect of the embodiments of the present invention, there is provided a method for sorting an image list, and optionally, as an optional implementation manner, the method for sorting an image list may be applied to, but is not limited to, an environment as shown in fig. 1. The above-described image list sorting method can be applied, but not limited to, in the server 104, for assisting the application client in performing a search process of similar images on the published target images. The application client may be but not limited to run in the user equipment 102, and the user equipment 102 may be but not limited to a mobile phone, a tablet computer, a notebook computer, a PC, and other terminal equipment supporting running of the application client. The server 104 and the user device 102 may, but are not limited to, enable data interaction via a network, which may include, but is not limited to, a wireless network or a wired network. Wherein, this wireless network includes: bluetooth, WIFI, and other networks that enable wireless communication. Such wired networks may include, but are not limited to: wide area networks, metropolitan area networks, and local area networks. The above is merely an example, and this is not limited in this embodiment.
The application environment in fig. 1 comprises the following steps:
firstly, user equipment sends a retrieved target image to a server;
secondly, the server retrieves images similar to the target image in a first image database according to the first similarity, and generates a first initial image list;
step three, the server adjusts the first initial image list according to a second distance between the image in the first initial image list and the target image to obtain a target image list;
and step four, the server feeds back the target image list to the user equipment.
Optionally, as an optional implementation manner, as shown in fig. 2, the method for sorting the image list includes the following steps:
step S202, a first initial image list associated with a target image is obtained from a first image database, wherein the images in the first initial image list are sorted according to a first similarity, and the first similarity is a first distance between the image in the first initial image list and the target image;
the first image database may be a gallery in the related art. The first similarity may be calculated from a euclidean distance between the two images.
The query person of the target image is marked as q;
the first image database set, G ═ Gi1, 2, …, N, extracting the features of the image through resnet101, and measuring the target image q and the image g in the first image databaseiHas a Euclidean distance d betweenE(q，gi) The resnet101 is a depth Residual error network model obtained by performing optimization based on a Residual neural network resnet (Residual neural network, abbreviated as resnet):
wherein ,xp，
Feature vectors of two images, respectively, and G is sorted from small to large according to this distance to generate a first initial image list L (q, G):
the goal of the subsequent flow of fig. 2 is to reorder the first initial picture list so that more positive samples appear in the front of the list.
Step S204, acquiring at least one second distance between each image in the first initial image list and the target image;
the second distance may be a Jaccard distance between k-nearest neighbor images of each image and k-nearest neighbor images of the target image, or a Jaccard coefficient, or a Rank-order distance of each image from the target image. The k-nearest neighbor image is an image set which is obtained by performing a k-nearest neighbor algorithm on one image and is most similar to the image in the feature space. The k-nearest neighbor algorithm may include the steps of: and calculating the distances between the points of the image and other images, sequencing the points in sequence according to the distances, selecting k images of other images with the minimum distance from the image, and taking the k images as k adjacent images.
When the second distance is the Jaccard distance, acquiring the Jaccard distance between each image and the target image by: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring a first k adjacent image set corresponding to the target image and a second k adjacent image set corresponding to the first image; acquiring the intersection and union of the second k adjacent image set and the first k adjacent image set; determining the Jaccard distance between the first image and the target image according to the intersection and the union;
when the second distance is a Rank-order distance, acquiring the Rank-order distance between each image and the target image by the following method comprises the following steps: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring a first initial image list of the target image and a second initial image list of the first image, wherein the second initial image list associated with the first image is acquired from a second image database, the images in the second initial image list are sorted according to a first similarity, the first similarity is a first distance between the images in the second initial image list and the first image, and the second image database comprises the first image database and the target image; acquiring a first position sequence number of an image in the first initial image list in the second initial image list, a second position sequence number of an image in the second initial image list in the first initial image list, a third position sequence number of the target image in the second initial image list and a fourth position sequence number of the first image in the first initial image list; and obtaining the Rank-order distance between the first image and the target image according to the first position sequence number, the second position sequence number, the third position sequence number and the fourth position sequence number.
The Jaccard distance is used to compare dissimilarity between a finite set of samples. The larger the Jaccard distance, the lower the sample similarity. The index related to the Jaccard distance is called Jaccard coefficient and is used for describing similarity and difference between sets, and the larger the Jaccard coefficient value is, the higher the sample similarity is.
The Rank-order distance is used to measure the similarity of two faces, and is based on an interesting observation: there are many shared neighbors to two faces of the same person, but the neighbors from faces of different persons are usually very different.
Step S206, determining a second similarity between each image in the first initial image list and the target image according to the second distance;
when the first distance is the Euclidean distance and the second distance is the Jaccard distance and the Rank-order distance, the second similarity can be a numerical value obtained by carrying out weighted calculation on the basis of the Euclidean distance, the Jaccard distance and the Rank-order distance.
And step S208, sequencing the first initial image list according to the second similarity to obtain a target image list.
Optionally, on the basis that the scheme in the flow in fig. 2 performs similarity calculation on all features of the two images, the similarity calculation may be performed on local features of the two images to assist in increasing or decreasing the precision of image ranking.
And adjusting the image sequence in the first initial image list to obtain a target image list, and ensuring that the image most similar to the target image appears in front of the target image list.
In the embodiment of the invention, the similarity degree between the images is determined from more angles by adopting a mode of adjusting the first initial image list by acquiring the second distance between the target image and each image in the first image database, so that a more accurate similarity degree judgment result is obtained, the accuracy of image sorting is greatly improved, and the technical problem of inaccurate sorting result caused by the fact that the similarity degree between the images cannot be accurately acquired is solved.
Optionally, the determining the second similarity between each image in the initial image list and the target image according to the second distance comprises: traversing the first initial image list, taking each image as a first image, and executing the following steps: determining the first distance and the second distance between the first image and the target image in the first initial image list; inputting the first distance and the second distance into a distance combination model, wherein the distance combination model is obtained after training by utilizing a plurality of sample data, and the distance combination model is used for weighting and summing the first distance and the second distance to obtain a second similarity; and acquiring an output result output by the distance combination model, wherein the output result comprises a second similarity between the first image and the target image. The distance combination model can be obtained by using a two-classification method for supervised training.
The input of the distance combination model is the Euclidean distance, the Jaccard distance and the Rank-order distance between two images, and the distance combination model is used for judging whether the objects in the two images are the same object or different objects. In the training stage, simple classification is used for training, the labeling result can be 0 or 1, 0 indicates that the two images are different, 1 indicates that the two images are the same, and the optimal combined weighting mode of the three distances is obtained through a supervision training mode. Because the network is simple, little marking data is needed, and once the weighting mode is determined, retraining under a new scene is not needed, and the data is not relied on.
Alternatively, the distance combination model may be designed by a machine learning model such as a lightweight linear Support Vector Machine (SVM), logistic regression, or the like.
The distance combination model may be:
d*(q，gi)＝f(dE(q，gi)，dJ(q，gi)，dR(q，gi))；
d*(q，gi) Is the target image q and the image g in the first image databaseiA second degree of similarity; dE(q，gi) Is the Euclidean distance between the two images; dJ(q，gi) Is the Jaccard distance, d, between the two imagesR(q，gi) The Rank-order distance between two images.
Optionally, the inputting the first distance and the second distance into the distance combination model includes: determining a first weight corresponding to the first distance and a second weight corresponding to the second distance through the distance combination model; and performing weighted summation on the first distance and the second distance by using the first weight and the second weight to obtain a second similarity between the first image and the target image. And when the second distance is the Jaccard distance and the Rank-order distance, the Jaccard distance and the Rank-order distance have different second weights correspondingly. By adopting the scheme, different weights are set for different types of distances so as to obtain a numerical value of the second similarity for more accurately describing the similarity between the two images and ensure the result of image reordering.
Optionally, the obtaining at least one second distance between each image in the first initial image list and the target image includes at least one of: acquiring a Rank-order distance between each image and the target image; and acquiring the Jaccard distance between each image and the target image. The second distance may also include a parameter such as a Jaccard coefficient describing the degree of similarity between different images.
Optionally, the obtaining a Rank-order distance between each image and the target image includes: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring a first initial image list of the target image and a second initial image list of the first image, wherein the second initial image list associated with the first image is acquired from a second image database, the images in the second initial image list are sorted according to a first similarity, the first similarity is a first distance between the images in the second initial image list and the first image, and the second image database comprises the first image database and the target image; acquiring a first position sequence number of an image in the first initial image list in the second initial image list, a second position sequence number of an image in the second initial image list in the first initial image list, a third position sequence number of the target image in the second initial image list and a fourth position sequence number of the first image in the first initial image list; and obtaining the Rank-order distance between the first image and the target image according to the first position sequence number, the second position sequence number, the third position sequence number and the fourth position sequence number.
Optionally, the Rank-order distance d between the first image and the target image is obtained by the following formulaR(q，gi) The target image is denoted as q and the first image is denoted as gi：
wherein ,fq(j) The jth image in the first initial image list representing the target image q,
representing image fq(j) In the first image giThe sequence number bit in the second initial image list corresponds to the first position sequence number in the above embodiment.
Representing the first image giThe jth image in the second initial image list,
representing images
The sequence number bit in the first initial image list of the target image q corresponds to the second position sequence number in the above embodiment. O isq(gi) Representing the first image giThe sequence number bit in the first initial image list of the target image q corresponds to the fourth position number in the above embodiment.
Indicating that the target image q is in the first image giThe sequence number bit in the second initial image list corresponds to the third position sequence number in the above embodiment. Wherein j is a positive integer.
Representing the smallest of the two values. By adopting the scheme, the Rank-order distance is determined according to the image sorting condition in the initial image list between the two images, and a basis is provided for calculating the second similarity.
Optionally, acquiring the Jaccard distance between each image and the target image comprises: traversing the first initial image list, taking each image as a first image, and executing the following steps: acquiring a first k adjacent image set corresponding to the target image and a second k adjacent image set corresponding to the first image; acquiring the intersection and union of the second k adjacent image set and the first k adjacent image set; determining the Jaccard distance between the first image and the target image according to the intersection and the union. By adopting the scheme, the Jaccard distance is calculated by using the k neighbor image sets of the two images, and a basis is provided for calculating the second similarity.
The k-nearest neighbor image set may be obtained by performing a k-nearest neighbor algorithm on the images.
Optionally, the jaccard distance between each image and the target image is obtained by the following formula
Wherein the target image is recorded as q, and the first image is recorded as gi：
wherein ,
is the first k set of neighboring images of the target image q,
is the first image giThe second k neighboring image set of
Is a target threshold for filtering noise data in the k-nearest neighbor algorithm results, k being an integer. To avoid noise interference, the threshold value may be set appropriately
Noise data is filtered out.
The schematic view of a scenario of the embodiment of the application is shown in fig. 3, and includes a target image q, a first image database, and an image g in a first initial image listiAnd q and giK neighbor image set ofCalculate q and g in one stepiJaccard distance d ofJRank-order distance dREuclidean distance dEFinally, the final distance d (also called the second similarity) is obtained by the distance combination model. And reordering the initial image list according to the final distance d to obtain a target image list.
By adopting the scheme, the following technical effects are realized: and introducing various distances such as Rank-order, Jaccard, Euclidean distance and the like, so that the distance investigation is more diversified, and the similarity between the images can be reflected truly. And analyzing the related information of the k adjacent images of each pair of images through the Jaccard distance, and analyzing the related information between the initial image lists of each pair of images through the Rank-order distance, and establishing a re-ranking and ranking combined distance model. By jointly considering the characteristics and the context information, the fuzzy sample is effectively removed, the re-identification re-ID performance is improved, more consideration is brought to neighbor information, and the method is extremely effective to the service scene with larger data difference in the re-identification re-ID category; aiming at the traditional manually designed weighting measurement mode, a shallow two-classification network learning combination weighting mode for various distances is designed by using supervised learning, so that the distance measurement mode can be more finely designed, and the retrieval performance is improved. In a combined distance measurement mode, a more comprehensive weighting mode can be provided by using a small amount of supervision information, and the retrieval precision of re-identification re-ID is greatly improved.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
According to another aspect of the embodiments of the present invention, there is also provided an image list sorting apparatus for implementing the above image list sorting method. As shown in fig. 4, the apparatus 40 includes:
a first obtaining module 42, configured to obtain a first initial image list associated with a target image from a first image database, where images in the first initial image list are sorted according to a first similarity, and the first similarity is a first distance between an image in the first initial image list and the target image;
a second obtaining module 44, configured to obtain at least one second distance between each image in the first initial image list and the target image;
a determining module 46, configured to determine a second similarity between each image in the first initial image list and the target image according to the second distance;
and a sorting module 48, configured to sort the first initial image list according to the second similarity, so as to obtain a target image list.
In the embodiment of the invention, the similarity degree between the images is determined from more angles by adopting a mode of adjusting the first initial image list by acquiring the second distance between the target image and each image in the first image database, so that a more accurate similarity degree judgment result is obtained, the accuracy of image sorting is greatly improved, and the technical problem of inaccurate sorting result caused by the fact that the similarity degree between the images cannot be accurately acquired is solved.
Optionally, as shown in fig. 5, the determining module 46 is further configured to traverse the first initial image list, regarding each image as a first image, and perform the following steps in the sub-units:
a first determining subunit 52, configured to determine the first distance and the second distance between the first image in the first initial image list and the target image;
an input subunit 54, configured to input the first distance and the second distance into a distance combination model, where the distance combination model is obtained after training by using multiple sample data, and the distance combination model is used to perform weighted summation on the first distance and the second distance to obtain a second similarity;
the first obtaining subunit 56 is configured to obtain an output result output by the distance combination model, where the output result includes a second similarity between the first image and the target image.
Optionally, the input subunit 54 is further configured to determine a first weight corresponding to the first distance and a second weight corresponding to the second distance through the distance combination model; and the second distance calculating unit is used for performing weighted summation on the first distance and the second distance by utilizing the first weight and the second weight so as to obtain a second similarity between the first image and the target image.
Optionally, the second obtaining module 44 is further configured to obtain a Rank-order distance between each image and the target image; and/or, further for obtaining the Jaccard distance between each image and the target image.
Optionally, as shown in fig. 6, the acquiring, by the second acquiring module 44, a Rank-order distance between each image and the target image includes: traversing the first initial image list, taking each image as a first image, and executing the following steps in the sub-units:
a second obtaining subunit 62, configured to obtain the first initial image list of the target image and a second initial image list of the first image, where the second initial image list associated with the first image is obtained from a second image database, and the images in the second initial image list are sorted according to a first similarity, the first similarity is a first distance between the images in the second initial image list and the first image, and the second image database includes the first image database and the target image;
a third obtaining subunit 64, configured to obtain a first position number of the image in the first initial image list in the second initial image list, a second position number of the image in the second initial image list in the first initial image list, a third position number of the target image in the second initial image list, and a fourth position number of the first image in the first initial image list; and the Rank-order distance between the first image and the target image is obtained according to the first position sequence number, the second position sequence number, the third position sequence number and the fourth position sequence number. The third acquisition subunit 64 may calculate the Rank-order distance between the two images by the formula in the above-described embodiment.
Optionally, as shown in fig. 7, the acquiring, by the second acquiring module 44, the Jaccard distance between each image and the target image includes: traversing the first initial image list, taking each image as a first image, and executing the following steps in the sub-units:
a fourth obtaining subunit 72, configured to obtain a first k neighboring image set corresponding to the target image and a second k neighboring image set corresponding to the first image;
a fifth obtaining subunit 74, configured to obtain an intersection and a union between the second k-neighbor image set and the first k-neighbor image set;
a sixth obtaining subunit 76, configured to determine the Jaccard distance between the first image and the target image according to the intersection and the union.
Optionally, the second obtaining module 44 is further configured to obtain a jaccard distance between each image and the target image according to the following formula
Wherein the target image is recorded as q, and the first image is recorded as gi：
wherein ,
is the first k set of neighboring images of the target image q,
is the first image giThe second k neighboring image set of
Is a target threshold for filtering noise data in the k-nearest neighbor algorithm results, k being an integer.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device for implementing the above method for sorting an image list, as shown in fig. 8, the electronic device includes a memory 802 and a processor 804, the memory 802 stores a computer program, and the processor 804 is configured to execute the steps in any one of the above method embodiments through the computer program.
Optionally, in this embodiment, the electronic apparatus may be located in at least one network device of a plurality of network devices of a computer network.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, acquiring a first initial image list associated with a target image from a first image database, wherein the images in the first initial image list are sorted according to a first similarity, and the first similarity is a first distance between the image in the first initial image list and the target image;
s2, acquiring at least one second distance between each image in the first initial image list and the target image;
s3, determining a second similarity between each image in the first initial image list and the target image according to the second distance;
and S4, sorting the first initial image list according to the second similarity to obtain a target image list.
Alternatively, it can be understood by those skilled in the art that the structure shown in fig. 8 is only an illustration, and the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 8 is a diagram illustrating a structure of the electronic device. For example, the electronic device may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 8, or have a different configuration than shown in FIG. 8.
The memory 802 may be used to store software programs and modules, such as program instructions/modules corresponding to the method and apparatus for sorting an image list in the embodiment of the present invention, and the processor 804 executes various functional applications and data processing by running the software programs and modules stored in the memory 802, that is, implements the above-described method for sorting an image list. The memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 802 can further include memory located remotely from the processor 804, which can be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 802 may be, but not limited to, information such as a distance value between the first image database and the image. As an example, as shown in fig. 8, the memory 802 may include, but is not limited to, a first obtaining module 42, a second obtaining module 44, a determining module 46, and a sorting module 48 in the sorting apparatus including the image list. In addition, other module units in the sorting apparatus for the image list may also be included, but are not limited to these, and are not described in detail in this example.
Optionally, the transmitting device 806 is configured to receive or transmit data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 806 includes a Network adapter (NIC) that can be connected to a router via a Network cable and other Network devices to communicate with the internet or a local area Network. In one example, the transmission device 806 is a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
In addition, the electronic device further includes: a display 808 for displaying the target image and the images in the first image database; and a connection bus 810 for connecting the respective module parts in the above-described electronic apparatus.
According to a further aspect of an embodiment of the present invention, there is also provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
Alternatively, in the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:
s1, acquiring a first initial image list associated with a target image from a first image database, wherein the images in the first initial image list are sorted according to a first similarity, and the first similarity is a first distance between the image in the first initial image list and the target image;
s2, acquiring at least one second distance between each image in the first initial image list and the target image;
s3, determining a second similarity between each image in the first initial image list and the target image according to the second distance;
and S4, sorting the first initial image list according to the second similarity to obtain a target image list.
Alternatively, in this embodiment, a person skilled in the art may understand that all or part of the steps in the methods of the foregoing embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, network devices, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the unit is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The above is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, it is possible to make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.
Claims (11)
1. A method for ordering a list of images, comprising:
acquiring a first initial image list associated with a target image from a first image database, wherein the images in the first initial image list are sorted according to a first similarity, and the first similarity is a first distance between the image in the first initial image list and the target image;
acquiring at least one second distance between each image in the first initial image list and the target image;
determining a second similarity between each image in the first initial image list and the target image according to the second distance;
and sequencing the first initial image list according to the second similarity to obtain a target image list.
2. The method of claim 1, wherein determining a second similarity between each image in the initial image list, first initial image list, and the target image according to the second distance comprises:
traversing the first initial image list, taking each image as a first image, and executing the following steps:
determining the first distance and the second distance between a first image in the first initial image list and the target image;
inputting the first distance and the second distance into a distance combination model, wherein the distance combination model is obtained after training by using a plurality of sample data, and the distance combination model is used for weighting and summing the first distance and the second distance to obtain a second similarity;
and acquiring an output result output by the distance combination model, wherein the output result comprises a second similarity between the first image and the target image.
3. The method of claim 2, wherein the inputting the first distance and the second distance into a distance combination model comprises:
determining a first weight corresponding to the first distance and a second weight corresponding to the second distance through the distance combination model;
and carrying out weighted summation on the first distance and the second distance by using the first weight and the second weight so as to obtain a second similarity between the first image and the target image.
4. The method of claim 1, wherein said obtaining at least one second distance between each image in the first initial image list and the target image comprises at least one of:
acquiring a Rank-order distance between each image and the target image;
acquiring a Jacard Jaccard distance between each image and the target image.
5. The method of claim 4, wherein obtaining a Rank-order distance between the each image and the target image comprises:
traversing the first initial image list, taking each image as a first image, and executing the following steps:
acquiring the first initial image list of the target image and a second initial image list of the first image, wherein a second initial image list associated with the first image is acquired from a second image database, the images in the second initial image list are sorted according to a first similarity, the first similarity is a first distance between the images in the second initial image list and the first image, and the second image database comprises the first image database and the target image;
acquiring a first position sequence number of an image in the first initial image list in the second initial image list, a second position sequence number of an image in the second initial image list in the first initial image list, a third position sequence number of the target image in the second initial image list, and a fourth position sequence number of the first image in the first initial image list;
and obtaining the Rank-order distance between the first image and the target image according to the first position sequence number, the second position sequence number, the third position sequence number and the fourth position sequence number.
6. The method of claim 5, wherein the Rank-order distance d between the first image and the target image is obtained by the following formulaR(q，gi) The target image is recorded as q, and the first image is recorded as gi：
wherein ,fq(j) A jth image in said first initial image list representing said target image q,
representing image fq(j) In the first image giThe sequence number bits in the second initial image list,
representing said first image giThe jth image in the second initial image list,
representing imagesNumber bit, O, in the first initial image list of the target image qq(gi) Representing said first image giThe sequence number bits in the first initial image list of the target image q,
representing said target image q in said first image giWherein j is a positive integer.
7. The method of claim 4, wherein obtaining the Jaccard distance between each image and the target image comprises:
traversing the first initial image list, taking each image as a first image, and executing the following steps:
acquiring a first k adjacent image set corresponding to the target image and a second k adjacent image set corresponding to the first image;
acquiring an intersection and a union between the second k neighbor image set and the first k neighbor image set;
determining the Jaccard distance between the first image and the target image according to the intersection and the union.
8. The method of claim 7, wherein the jaccard distance between each image and the target image is obtained by the following formula
Wherein the target image is recorded as q, and the first image is recorded as gi：
wherein ,
is the first k set of neighboring images of the target image q,is the first image giThe second k neighbor image set of
Is a target threshold for filtering noise data in k-nearest neighbor algorithm results, k being an integer.
9. An apparatus for sorting a list of images, comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a first initial image list associated with a target image from a first image database, images in the first initial image list are sorted according to a first similarity, and the first similarity is a first distance between the image in the first initial image list and the target image;
a second obtaining module, configured to obtain at least one second distance between each image in the first initial image list and the target image;
the determining module is used for determining a second similarity between each image in the first initial image list and the target image according to the second distance;
and the sequencing module is used for sequencing the first initial image list according to the second similarity to obtain a target image list.
10. A computer-readable storage medium comprising a stored program, wherein the program when executed performs the method of any of claims 1 to 8.
11. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 8 by means of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910945807.5A CN110704659B (en) | 2019-09-30 | 2019-09-30 | Image list ordering method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910945807.5A CN110704659B (en) | 2019-09-30 | 2019-09-30 | Image list ordering method and device, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110704659A true CN110704659A (en) | 2020-01-17 |
| CN110704659B CN110704659B (en) | 2023-09-26 |
Family
ID=69196710
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910945807.5A Active CN110704659B (en) | 2019-09-30 | 2019-09-30 | Image list ordering method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110704659B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112417197A (en) * | 2020-12-02 | 2021-02-26 | 云从科技集团股份有限公司 | Sorting method, sorting device, machine readable medium and equipment |
| CN112784086A (en) * | 2021-01-28 | 2021-05-11 | 北京有竹居网络技术有限公司 | Picture screening method and device, storage medium and electronic equipment |
| CN113343008A (en) * | 2021-06-29 | 2021-09-03 | 维沃软件技术有限公司 | Image sorting method, image sorting device, electronic device and readable storage medium |
| CN113505723A (en) * | 2021-07-23 | 2021-10-15 | 京东科技控股股份有限公司 | Pedestrian re-identification method and device, storage medium and electronic equipment |
| CN113505723B (en) * | 2021-07-23 | 2024-05-24 | 北京京东尚科信息技术有限公司 | Pedestrian re-recognition method and device, storage medium and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150154229A1 (en) * | 2013-11-29 | 2015-06-04 | Canon Kabushiki Kaisha | Scalable attribute-driven image retrieval and re-ranking |
| CN105303150A (en) * | 2014-06-26 | 2016-02-03 | 腾讯科技（深圳）有限公司 | Method and system for implementing image processing |
| CN106708943A (en) * | 2016-11-22 | 2017-05-24 | 安徽睿极智能科技有限公司 | Image retrieval reordering method and system based on arrangement fusion |
| US20180367325A1 (en) * | 2017-06-16 | 2018-12-20 | Line Corporation | Method and system for sorting chatroom list based on conversational activeness and contextual information |
| CN109165612A (en) * | 2018-08-31 | 2019-01-08 | 苏州大学 | Pedestrian's recognition methods again based on depth characteristic and two-way KNN sorting consistence |
| DE102017222916A1 (en) * | 2017-12-15 | 2019-06-19 | Rohde & Schwarz Gmbh & Co. Kg | Device and measuring method for the identification of network devices |
| CN110059198A (en) * | 2019-04-08 | 2019-07-26 | 浙江大学 | A kind of discrete Hash search method across modal data kept based on similitude |
-
2019
- 2019-09-30 CN CN201910945807.5A patent/CN110704659B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150154229A1 (en) * | 2013-11-29 | 2015-06-04 | Canon Kabushiki Kaisha | Scalable attribute-driven image retrieval and re-ranking |
| CN105303150A (en) * | 2014-06-26 | 2016-02-03 | 腾讯科技（深圳）有限公司 | Method and system for implementing image processing |
| CN106708943A (en) * | 2016-11-22 | 2017-05-24 | 安徽睿极智能科技有限公司 | Image retrieval reordering method and system based on arrangement fusion |
| US20180367325A1 (en) * | 2017-06-16 | 2018-12-20 | Line Corporation | Method and system for sorting chatroom list based on conversational activeness and contextual information |
| DE102017222916A1 (en) * | 2017-12-15 | 2019-06-19 | Rohde & Schwarz Gmbh & Co. Kg | Device and measuring method for the identification of network devices |
| CN109165612A (en) * | 2018-08-31 | 2019-01-08 | 苏州大学 | Pedestrian's recognition methods again based on depth characteristic and two-way KNN sorting consistence |
| CN110059198A (en) * | 2019-04-08 | 2019-07-26 | 浙江大学 | A kind of discrete Hash search method across modal data kept based on similitude |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112417197A (en) * | 2020-12-02 | 2021-02-26 | 云从科技集团股份有限公司 | Sorting method, sorting device, machine readable medium and equipment |
| CN112784086A (en) * | 2021-01-28 | 2021-05-11 | 北京有竹居网络技术有限公司 | Picture screening method and device, storage medium and electronic equipment |
| CN113343008A (en) * | 2021-06-29 | 2021-09-03 | 维沃软件技术有限公司 | Image sorting method, image sorting device, electronic device and readable storage medium |
| CN113505723A (en) * | 2021-07-23 | 2021-10-15 | 京东科技控股股份有限公司 | Pedestrian re-identification method and device, storage medium and electronic equipment |
| CN113505723B (en) * | 2021-07-23 | 2024-05-24 | 北京京东尚科信息技术有限公司 | Pedestrian re-recognition method and device, storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110704659B (en) | 2023-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111523621B (en) | Image recognition method and device, computer equipment and storage medium | |
| TWI677852B (en) | A method and apparatus, electronic equipment, computer readable storage medium for extracting image feature | |
| CN108595585B (en) | Sample data classification method, model training method, electronic equipment and storage medium | |
| CN109190023B (en) | Collaborative recommendation method and device and terminal equipment | |
| US8724910B1 (en) | Selection of representative images | |
| CN110704659B (en) | Image list ordering method and device, storage medium and electronic device | |
| US20150039583A1 (en) | Method and system for searching images | |
| CN110008397B (en) | Recommendation model training method and device | |
| CN110765882B (en) | Video tag determination method, device, server and storage medium | |
| CN107330750A (en) | A kind of recommended products figure method and device, electronic equipment | |
| CN112395487A (en) | Information recommendation method and device, computer-readable storage medium and electronic equipment | |
| WO2023185925A1 (en) | Data processing method and related apparatus | |
| CN112818148B (en) | Visual retrieval sequencing optimization method and device, electronic equipment and storage medium | |
| CN110851737A (en) | Recommendation method and device, electronic equipment and computer storage medium | |
| CN114817655A (en) | Cross-modal retrieval method, network training method, device, equipment and medium | |
| CN110276283B (en) | Picture identification method, target identification model training method and device | |
| CN115840831A (en) | Commodity retrieval method, device, equipment and storage medium | |
| CN113076487B (en) | User interest characterization and content recommendation method, device and equipment | |
| JP2016014990A (en) | Moving image search method, moving image search device, and program thereof | |
| CN111026935B (en) | Cross-modal retrieval reordering method based on adaptive measurement fusion | |
| CN112488863B (en) | Dangerous seed recommendation method and related equipment in user cold start scene | |
| CN111539536B (en) | Method and device for evaluating service model hyper-parameters | |
| CN109886299B (en) | User portrait method and device, readable storage medium and terminal equipment | |
| CN107358266B (en) | Mobile terminal with recognition function | |
| CN117332107A (en) | Face image searching and sorting method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40018620 Country of ref document: HK |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |