WO2021219117A1 - 图像检索方法、图像检索装置、图像检索系统及图像显示系统 - Google Patents
图像检索方法、图像检索装置、图像检索系统及图像显示系统 Download PDFInfo
- Publication number
- WO2021219117A1 WO2021219117A1 PCT/CN2021/091308 CN2021091308W WO2021219117A1 WO 2021219117 A1 WO2021219117 A1 WO 2021219117A1 CN 2021091308 W CN2021091308 W CN 2021091308W WO 2021219117 A1 WO2021219117 A1 WO 2021219117A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- painting
- target
- library
- search result
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 78
- 238000010422 painting Methods 0.000 claims abstract description 343
- 238000000605 extraction Methods 0.000 claims description 108
- 238000012545 processing Methods 0.000 claims description 30
- 238000004590 computer program Methods 0.000 claims description 16
- 230000004044 response Effects 0.000 claims description 15
- 238000012549 training Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 16
- 230000006870 function Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 13
- 238000004422 calculation algorithm Methods 0.000 description 9
- 230000009286 beneficial effect Effects 0.000 description 8
- 238000012937 correction Methods 0.000 description 6
- RKWPMPQERYDCTB-UHFFFAOYSA-N ethyl n-[4-[benzyl(2-phenylethyl)amino]-2-(4-nitrophenyl)-1h-imidazo[4,5-c]pyridin-6-yl]carbamate Chemical compound N=1C(NC(=O)OCC)=CC=2NC(C=3C=CC(=CC=3)[N+]([O-])=O)=NC=2C=1N(CC=1C=CC=CC=1)CCC1=CC=CC=C1 RKWPMPQERYDCTB-UHFFFAOYSA-N 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 230000003993 interaction Effects 0.000 description 4
- 230000006835 compression Effects 0.000 description 3
- 238000007906 compression Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 238000010428 oil painting Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000001174 ascending effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/532—Query formulation, e.g. graphical querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/538—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Definitions
- the present invention relates to the field of image processing technology, in particular to an image retrieval method, an image retrieval device, an image retrieval system, an image display system and a computer-readable storage medium.
- the picture screen can display images (in this article, the painting is taken as an example), and can restore the real texture of the painting, so it has been widely used.
- the user can input the text description of the related information of the painting (for example, the name or author of the painting, etc.) to make the painting screen display the corresponding painting for the user to enjoy.
- the user finds a painting that does not know the name, author, and other related information, it is difficult for the user to directly search and display the painting through the painting screen, and it is difficult for the user to understand the information of the painting.
- an image retrieval method including: receiving a first original image; extracting image features of the first original image to obtain a first feature code; and obtaining first target information according to the first feature code; According to the first target information, a first target painting set corresponding to the first target information is searched in a painting library and/or a knowledge graph library; and the first target painting set is output.
- the extracting the image feature of the first original image to obtain the first feature code includes: using a first image feature extraction model to extract the image feature of the first original image to obtain the first feature code.
- One feature code One feature code.
- the image retrieval method further includes: in response to a user's operation on the first target painting collection, obtaining user operation information; and according to the user's operation information, updating the first image feature extraction model to obtain A second image feature extraction model; receiving a second original image; using the second image feature extraction model to extract image features of the second original image to obtain a second feature code; according to the second feature code, to obtain a second feature code 2.
- Target information according to the second target information, search for a second target painting set corresponding to the second target information in the painting library and the knowledge graph database; output the second target painting set .
- the updating the first image feature extraction model according to the user's operation information to obtain the second image feature extraction model includes: classifying the user's operation information and calculating The proportion of user operation information corresponding to various tags; adjust the weights of image features corresponding to the various tags according to the proportions of the various tags; according to the first image feature extraction model and the adjusted The weight is trained to form the second image feature extraction model; the first image feature extraction model is replaced with the second image feature extraction model.
- the obtaining the first target information according to the first feature code includes: calculating the first feature code and each feature code in the first feature code library and the second feature code library. And obtain the feature code corresponding to each distance within the preset range, and use each obtained feature code as the first target feature code set; determine the first target according to the first target feature code set information.
- the first feature encoding library is a feature encoding library obtained by performing image feature extraction on a plurality of images in the painting library through the first image feature extraction model
- the second feature encoding library is a feature encoding library obtained by
- the first image feature extraction model is a feature encoding library obtained by performing image feature extraction on multiple images in the knowledge map library.
- the obtaining the second target information according to the second feature code includes: calculating the second feature code and each feature code in the third feature code library and the fourth feature code library. And obtain the feature codes corresponding to the distances within the preset range, and use the obtained feature codes as the second target feature code set; determine the second target according to the second target feature code set information.
- the third feature encoding library is a feature encoding library obtained by performing image feature extraction on a plurality of images in the painting library through the second image feature extraction model
- the fourth feature encoding library is a feature encoding library obtained through the
- the second image feature extraction model is a feature encoding library obtained by performing image feature extraction on multiple images in the knowledge map library.
- the searching for a first target painting set corresponding to the first target information in a painting library and/or a knowledge graph library according to the first target information includes: according to the first target information The target information is searched in the painting library to obtain a first search result; according to the first target information, a second search result is obtained in the knowledge graph library; in the case that the first search result includes a hit painting, The first search result is taken as the first target painting set; the hit painting is the one with the highest similarity to the original image among the paintings corresponding to the first target information; the first search result is not In the case where the hit painting is included and the second search result includes the hit painting, the first search result and the hit painting are used together as the first target painting set; in the first search result In the case that neither the second search result includes the hit painting, the first search result is used as the first target painting set.
- the searching for a first target painting set corresponding to the first target information in a painting library and/or a knowledge graph library according to the first target information includes: according to the first target information The target information is searched in the painting library to obtain a first search result; according to the first target information, a second search result is obtained in the knowledge graph library; in the first search result and the second search result In the same situation, use the first search result as the first target painting collection; in the case where the first search result and the second search result are not completely the same, use the first search result and the The collection of the second search results serves as the first target painting collection.
- an image retrieval device configured as the image retrieval method described in any of the above embodiments.
- the image retrieval device includes: a processor and a memory; the memory stores program instructions, and the program instructions can be executed by the processor.
- an image retrieval device including: a first server and a second server; the first server stores a painting library, and the second server stores a knowledge graph library.
- the first server is configured to: receive a first original image, and send the first original image to the second server; the second server is configured to: receive a message sent by the first server A first original image; extracting image features of the first original image to obtain a first feature code; and obtaining first target information according to the first feature code, and transmitting the first target information to the first server;
- the first server is also configured to: receive the first target information transmitted by the second server; according to the first target information, search for the first target information in the painting library and/or the knowledge graph library The first target painting set corresponding to the target information; outputting the first target painting set.
- the first server is further configured to perform data processing on the received first original image, and send the data-processed first original image to the second server.
- the second server is further configured to extract the image features of the first original image by using the first image feature extraction model to obtain the first feature code.
- the second server is further configured to: in response to a user's operation on the first target painting collection, obtain user operation information;
- the feature extraction model is updated to obtain the second image feature extraction model.
- the first server is further configured to receive a second original image and send the second original image to the second server.
- the second server is further configured to: use the second image feature extraction model to extract image features of the second original image to obtain a second feature code; and obtain second target information according to the second feature code ; Transmit the second target information to the first server.
- the first server is further configured to: according to the second target information, search for a second target painting collection corresponding to the second target information in the painting library and the knowledge graph library; The second target painting collection.
- the second server is further configured to: classify the user's operation information, calculate the proportion of the user operation information corresponding to various tags; and according to the proportion of the various tags , Adjust the weights of the image features corresponding to the various tags; train and form the second image feature extraction model according to the first image feature extraction model and the image features after adjusting the weights; combine the first image feature The extraction model is replaced with the second image feature extraction model.
- the first server is further configured to: search in the painting library according to the first target information to obtain a first search result; according to the first target information in the knowledge graph library The search obtains a second search result; in the case that the first search result includes a hit painting, the first search result is used as the first target painting set; the hit painting is used as each painting corresponding to the first target information The one with the highest similarity to the original image; in the case that the first search result does not include the hit painting and the second search result includes the hit painting, use the first search result and The hit paintings collectively serve as the first target painting set; in the case that neither the first search result nor the second search result includes the hit paintings, the first search result is used as the first target painting set; A collection of target paintings.
- the first server is further configured to: search in the painting library according to the first target information to obtain a first search result; according to the first target information in the knowledge graph library The search obtains a second search result; in the case where the first search result and the second search result are the same, one of the first search result and the second search result is used as the first target painting collection In the case where the first search result and the second search result are not completely the same, the collection of the first search result and the second search result is used as the first target painting collection.
- an image retrieval system including: an image retrieval device and terminal equipment.
- the image retrieval device is configured to execute the image retrieval method described in any of the above embodiments;
- the terminal device is configured to: collect a first original image, and upload the first original image to the image retrieval device; and receive the image retrieval The first target painting set output by the device; in response to the user's operation, the first target painting set is output.
- an image display system including: a terminal device and a painting display terminal.
- the terminal device is configured to: collect a first original image, and upload the first original image to the image retrieval device; receive the first target painting collection output by the image retrieval device;
- the display terminal outputs the first target painting collection;
- the image retrieval device is configured to execute the image retrieval method according to any one of the above-mentioned embodiments of the claims;
- the painting display terminal is configured to display the first A collection of target paintings.
- a computer-readable storage medium stores computer program instructions that, when the computer program instructions run on a processor, cause the processor to execute the instructions described in any of the above embodiments.
- the method of image retrieval is provided, and the computer-readable storage medium stores computer program instructions that, when the computer program instructions run on a processor, cause the processor to execute the instructions described in any of the above embodiments. The method of image retrieval.
- Fig. 1 is a structure diagram of a knowledge graph according to some embodiments of the present invention.
- Fig. 2 is a structural diagram of another knowledge graph according to some embodiments of the present invention.
- Fig. 3 is a flowchart of an image retrieval method according to some embodiments of the present invention.
- FIG. 4 is a flowchart of S300 in the image retrieval method shown in FIG. 3;
- FIG. 5 is a flowchart of S400 in the image retrieval method shown in FIG. 3;
- Fig. 6 is a flowchart of another image retrieval method according to some embodiments of the present invention.
- FIG. 7 is a flowchart of S700 in the image retrieval method shown in FIG. 6;
- FIG. 8 is a flowchart of S1000 in the image retrieval method shown in FIG. 6;
- FIG. 9 is a diagram of an interface for displaying a first target painting collection on a terminal device according to some embodiments of the present invention.
- FIG. 10 is an interface diagram of a terminal device displaying information of a first target painting collection according to some embodiments of the present invention.
- FIG. 11 is an interface diagram of a terminal device according to some embodiments of the present invention displaying the second target information corresponding to the key field of the question;
- Fig. 12A is a structural diagram of an image retrieval device according to some embodiments of the present invention.
- FIG. 12B is a flowchart of an image retrieval method performed by the image retrieval device in some embodiments of the present invention.
- FIG. 13A is a structural diagram of a painting retrieval system according to some embodiments of the present invention.
- FIG. 13B is a structural diagram of a painting retrieval system and an image retrieval device according to some embodiments of the present invention.
- Figure 13C is a diagram of an operation interface of a terminal device according to some embodiments of the present invention.
- FIG. 14 is a structural diagram of another painting retrieval system according to some embodiments of the present invention.
- Figure 15 is a structural diagram of a cloud server deployment solution in some embodiments of the present invention.
- Fig. 16 is a structural diagram of a solution for user feedback, knowledge graph query and log storage in some embodiments of the present invention.
- first and second are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined with “first” and “second” may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present disclosure, unless otherwise specified, “plurality” means two or more.
- the expressions “coupled” and “connected” and their extensions may be used.
- the term “connected” may be used when describing some embodiments to indicate that two or more components are in direct physical or electrical contact with each other.
- the term “coupled” may be used when describing some embodiments to indicate that two or more components have direct physical or electrical contact.
- the term “coupled” or “communicatively coupled” may also mean that two or more components are not in direct contact with each other, but still cooperate or interact with each other.
- the embodiments disclosed herein are not necessarily limited to the content of this document.
- At least one of A, B, and C has the same meaning as “at least one of A, B, or C", and both include the following combinations of A, B, and C: only A, only B, only C, A and B The combination of A and C, the combination of B and C, and the combination of A, B and C.
- a and/or B includes the following three combinations: A only, B only, and the combination of A and B.
- the term “if” is optionally interpreted as meaning “when” or “when” or “in response to determination” or “in response to detection.”
- the phrase “if it is determined" or “if [the stated condition or event] is detected” is optionally interpreted to mean “when determining" or “in response to determining" Or “when [stated condition or event] is detected” or “in response to detecting [stated condition or event]”.
- Some embodiments of the present invention provide an image retrieval method.
- the technical elements involved in the image retrieval method are schematically described below.
- the knowledge graph is a structured semantic knowledge base, and its basic components include the first entity, the relationship and the second entity.
- the knowledge graph is usually composed of multiple nodes and multiple edges. Each node is represented as a first entity or a second entity, and each edge is represented as a relationship. In the knowledge graph, multiple edges connect each node to form a network structure. Each node corresponds to a unique identity, and each edge corresponds to a unique identity.
- the knowledge graph can be applied to knowledge reasoning, search, question and answer and other related scenarios, and can make precise and refined answers.
- FIG. 1 shows the basic structure of the knowledge graph.
- the knowledge graph includes a first node 11 and a second node 13, and the first node 11 and the second node 13 are connected by an edge 12.
- the first node 11 is represented as a first entity
- the edge 12 is represented as a relationship
- the second node 13 is represented as a second entity.
- the first node 11, the edge 12, and the second node 13 form a triple structure of "first entity, relationship, and second entity".
- the first entity may be a character name or a place name, etc.
- the second entity may be a character name or a place name, etc.
- the relationship can be father-son, father-son, mother-son, spouse, or geographic affiliation.
- the character A and the character C have a father-daughter relationship, that is, the character A corresponds to the character C through the father-daughter relationship.
- the character A is the first entity
- the character C is the second entity
- the character C is the first entity
- the character A is the second entity.
- the character A and the character B have a husband and wife relationship, that is, the character A corresponds to the character B through the husband and wife relationship.
- the character A is the first entity
- the character B is the second entity
- the character B is the first entity
- the character A is the second entity.
- the person B and the place D have a place of birth relationship, that is, the person B corresponds to the place D through the place of birth relationship.
- the person B is the first entity
- the place D is the second entity.
- the image retrieval method provided by some embodiments of the present invention includes: S100-S500.
- the image retrieval method further includes: the terminal device collects the first original image.
- the first original image is taken through a terminal device.
- the source of the object corresponding to the first original image includes multiple sources.
- the subject may be derived from an image displayed on some electronic display device (such as a television), or from a printed matter (such as a poster, a magazine, or a calendar, etc.).
- the subject that receives the first original image and performs retrieval based on the first original image may be, for example, an image retrieval device.
- the aforementioned image features include multiple categories.
- the image characteristics may include categories such as color, shape (for example, plant shape or animal shape, etc.), texture, type, or time.
- each category corresponds to a label, which represents a dimension of image features. Therefore, it can also be said that image features have many dimensions.
- the first feature is encoded as a string of binary numbers, and the length of the first feature can be set according to the complexity of the image feature of the first original image.
- the first feature code may be 2048 dimensions, 1024 dimensions or 512 dimensions.
- extracting the image feature of the first original image to obtain the first feature code includes: using the first image feature extraction model to extract the image feature of the first original image to obtain the first feature code.
- the aforementioned first image feature extraction model may be a feature extraction model of a deep learning model.
- the first target information is information that can correspond to at least one image in the painting library and/or the knowledge graph library.
- the first target information may include the identification of multiple images (for example, name, serial number, storage address, etc., the name or storage address is taken as an example in the following), and each name is correspondingly unique in the painting library and/or knowledge graph library Image.
- the first target information is obtained according to the first feature encoding, including S310-S330.
- the first feature encoding library and the second feature encoding library are obtained by performing image feature extraction on the images in the painting library and the knowledge map library by the first image feature extraction model.
- the first feature encoding library is a feature encoding library obtained by performing image feature extraction on a plurality of images in the painting library through a first image feature extraction model;
- the second feature encoding library is a pair of the first image feature extraction model A feature code library obtained by extracting image features from multiple images in the knowledge map library.
- the types and numbers of the tags of the extracted image features are the same as the types and numbers of the tags of the image features extracted from the first original image.
- the dimension of each feature code in the first feature code library and the dimension of each feature code in the second feature code library are the same as the dimension of the first feature code.
- S320 Calculate the distance between the first feature code and each feature code in the first feature code library and the second feature code library, and obtain the feature code corresponding to each distance within a preset range, and combine the obtained Each feature code is used as the first target feature code set.
- the above distance is the Hamming distance, or the above distance is the Euclidean distance.
- the above-mentioned preset range has different meanings according to different types of distance settings.
- the above distance is the Hamming distance, and in this case, the preset range may be 0-5, for example. Since the Hamming distance refers to the number of different characters at the corresponding positions of two character strings, that is, the Hamming distance is a natural number, at this time, the Hamming distance can be 0, 1, 2, 3, 4, or 5.
- the foregoing distance is Euclidean distance
- the preset range may be, for example, 0-5. Since the Euclidean distance refers to the true distance between two points in the m (m is a positive integer) dimensional space, at this time, the Euclidean distance can be any value between 0-5. For example, 0, 1, 2, 3 or 5 etc.
- the calculation of the distance between the first feature code and each feature code in the first feature code library is taken as an example to illustrate schematically.
- the first feature code can be sequentially compared with each feature code in the first feature code library to determine the corresponding position of the first feature code and each feature code in the first feature code library Whether the characters are the same, and record the different numbers.
- the number of differences is the Hamming distance between the first feature code and each feature code in the first feature code library.
- the first feature code is 1011101, and one feature code in the first feature code library is 1001001. It can be seen from the two feature codes that the third bit of the first feature code is different from the third bit of the feature code in the first feature code library, and the fifth bit of the first feature code is different from the first feature code library. The fifth bit of the feature code of is different, and the characters at the other corresponding positions are the same, the Hamming distance between the first feature code 1011101 and the feature code 1001001 in the first feature code library is 2.
- the first feature code is 1011101, and one feature code in the first feature code library is 0001001. It can be seen from the two feature codes that the first bit of the first feature code is different from the first bit of the feature code in the first feature code library, and the third bit of the first feature code is different from the first bit of the feature code in the first feature code library. The third bit of the feature code is different, the fifth bit of the first feature code is different from the fifth bit of the feature code in the first feature code library, and the characters in the remaining corresponding positions are the same, then the second feature code 1011101 is the same as the first feature
- the Hamming distance between feature codes 000001001 in the code library is 3.
- the Hamming distance between the first feature code 1011101 and the feature code 1001001 in the first feature code library is smaller than the Hamming distance between the first feature code 1011101 and the feature code 0001001 in the first feature code library.
- the distance that is, the number of identical characters in the corresponding position between the first feature code 1011101 and the feature code 1001001 in the first feature code library is greater than the correspondence between the first feature code 1011101 and the feature code 0001001 in the first feature code library
- the number of characters in the position is the same.
- the feature code 1001001 in the first feature code library is more similar to the first feature code 1011101.
- the feature code in the first feature code library The image corresponding to 1001001 is more similar to the first original image.
- the greater the distance between the first feature code and the feature code in the first feature code library the greater the distance between the object corresponding to the first original image and the image corresponding to the feature code in the first feature code library The lower the similarity between.
- the smaller the distance between the first feature code and the feature code in the first feature code library, the similarity between the object corresponding to the first original image and the image corresponding to the feature code in the first feature code library The higher the degree.
- the distance between the first feature code and the feature code in the first feature code library is 0, the object corresponding to the first original image is the same as the image corresponding to the feature code in the first feature code library.
- the feature corresponding to each distance within the preset range can be obtained. Encoding, and use the acquired feature codes as the first target feature code set.
- the aforementioned preset range refers to the preset range of the distance between the first feature code and each feature code in the first feature code library and the second feature code library.
- the upper limit of the preset range can be selected and set according to actual needs. Exemplarily, the upper limit of the preset range is 5, that is, the preset range is 0-5.
- the feature code corresponding to the distance between the first feature code and each feature code in the first feature code library and the second feature code library within 0 to 5 can be extracted as the first feature code.
- Target feature encoding set can be extracted as the first feature code.
- the above S320 further includes: after calculating the distance between the first feature code and each feature code in the first feature code library and the second feature code library, pairing the calculated each feature code according to the size of the distance Sort by distance.
- Sorting the calculated distances according to the size of the distance includes: sorting the calculated distances in ascending order, or sorting the calculated distances in descending order. In this way, the distances within the preset range and the feature codes corresponding to the distances can be directly obtained from the beginning or end of the sequence table obtained by sorting. In addition, the similarity between the image corresponding to each feature code and the object corresponding to the first original image can be understood intuitively according to the arrangement sequence.
- the above S320 before sorting the sizes of the distances, further includes: a process of calculating the distance between the first feature code and each feature code in the first feature code library and the second feature code library , Delete the first feature code and the feature code corresponding to the distance between the first feature code library and the feature code in the second feature code library that is greater than the upper limit of the preset range. This helps to reduce the workload of sorting the distances later and improve the efficiency of the painting retrieval method.
- S330 Determine first target information according to the first target feature encoding set.
- the above-mentioned first target information may be a list composed of the names of multiple images, and the corresponding image may be found in the painting library and/or the knowledge graph library according to the first information.
- the name of each image corresponds to an image storage address, and the image can be found according to the image storage address.
- the above-mentioned first target information may also be a link address where the image is stored.
- the image stored in each link address is an image corresponding to the first target feature code set.
- the first target feature code set since the first target feature code set is obtained according to the first feature code library and the second feature code library, the first target feature code set may only be the feature codes in the first feature code library.
- the first target information corresponds to the names (or stored link addresses) of some images in the painting library; or the first target feature code set may only be the feature codes in the second feature code library.
- the first target information Corresponds to the names (or stored link addresses) of some images in the knowledge graph library; or one part of the first target feature code set is the feature code in the first feature code library, and the other part is the feature code in the second feature code library Feature encoding.
- the first target information corresponds to the name (or the stored link address) of a part of the image in the painting library and the name (or the stored link address) of the part of the image in the knowledge graph library.
- a first feature code library and a second feature code library are established, and after the image feature of the first original image is extracted to obtain the first feature code, the first feature code and the first feature code library and the second feature code are calculated.
- the distance between each feature code in the code library is filtered to obtain each feature code corresponding to each distance within the preset range as the first target feature code set, and then the first target information is determined according to the first target feature code set Compared with directly comparing the original image with the paintings in the painting library and the knowledge map library, this is helpful to reduce the difficulty of comparison and improve the accuracy of retrieval.
- S400 According to the first target information, search for a first target painting set corresponding to the first target information in the painting library and the knowledge graph library.
- the searched image is the first target painting collection.
- the number of images included in the first target painting collection can be selected and set according to actual needs. Exemplarily, the number of images included in the first target painting set is 5, 10, or 20.
- one image may be the same as the subject corresponding to the first original image.
- This image may be called a hit painting, and the remaining images are the same as those of the first original image.
- the corresponding subjects are similar; or, the images included in the first target painting set are all similar to the subjects corresponding to the first original image, and the hit paintings are not included in the first target painting set at this time.
- the first target painting collection if there is an image that is the same as the subject corresponding to the first original image (that is, including the hit painting), this also realizes that the name of the subject is not known
- related information such as, author, etc., search for the subject
- the first target paintings obtained by the search are concentrated, and the remaining images are similar to the subject corresponding to the first original image, which can provide users with appreciation and
- the opportunity of similar images of the subject corresponding to the first original image is convenient for users to compare and study multiple images.
- the first target painting set corresponding to the first target information is searched in the painting library and the knowledge graph library, including S410 to S430.
- the process of finding the first search result in the painting library according to the first target information may include finding the name of the corresponding image in the painting library according to the first target information, and finding the corresponding image according to the name of the corresponding image The storage address, and then find the corresponding image according to the storage address of each image, and form the first search result.
- the second search result can also be formed by the above-mentioned method, which will not be repeated here.
- the first search result is a part of the image in the painting library, and the second search result is without.
- the first search result is none, and the second search result is in the knowledge graph library Part of the image.
- the first target information not only corresponds to the name (or the stored link address) of a part of the image in the painting library, but also corresponds to the name (or the stored link address) of the part of the image in the knowledge graph library
- the first search result is part of the image in the painting library
- the second search result is part of the image in the knowledge graph library.
- S410 and S420 may be executed at the same time, or S410 may be executed first and then S420 may be executed, or S420 may be executed first and then S410 may be executed, which is not limited in this example.
- the part of the image searched from the painting library is exactly the same as the part of the image searched from the knowledge graph library.
- you can search from the painting library The obtained partial images are used as the first target painting collection. This can avoid the repetition of the images in the first target painting set.
- the part of the image searched from the painting library is not the same as the part of the image searched from the knowledge graph library.
- you can use the painting library The collection of the partial images searched in and the partial images searched from the knowledge graph library is used as the first target painting set. In this way, the integrity of the images in the first target painting set can be ensured.
- the first search result is a part of the image in the painting library
- the second search result is a part of the image in the knowledge graph library
- the two parts are the same and the part is different
- the same part of one of the two can be filtered.
- the remaining part of the two images are collectively used as the first target painting set.
- the first target painting set can be determined by the following method:
- the first search result includes the hit painting
- the first search result is used as the first target painting set
- the hit painting is the one with the highest similarity to the original image among the paintings corresponding to the first target information.
- the first target The subject included in the painting is the same as the subject in the first original image;
- the first search result and the hit painting are used together as the first target painting set;
- the first search result is used as the first target painting set.
- the hit paintings in the first target painting set determined by the above method only come from the painting library.
- whether the first search result is the same as the second search result can be obtained by comparing the names of the images.
- the first target painting collection After the first target painting collection is found, the first target painting collection can be output.
- the image retrieval method provided by some embodiments of the present invention is combined with a painting library and a knowledge graph library.
- the first feature code can be obtained.
- the painting library and the knowledge map library the target paintings that are the same and/or similar to the subject corresponding to the first original image can be retrieved. This enables the user to find the correct picture without knowing the name, author, and other related information of the painting. Retrieval of paintings.
- the image retrieval method further includes: performing data processing on the first original image.
- data processing can include multiple methods.
- the data processing includes correction processing.
- the correction processing includes using a background removal algorithm and a gradient correction algorithm to correct a situation in which the first original image does not match the actual object caused by the acquisition shooting angle or the acquisition location.
- the data processing includes compression processing. By compressing the first original image, the transmission speed of the first original image sent from the terminal device to the cloud device can be increased, and the distortion of the first original image can be avoided.
- the data processing includes correction processing and compression processing.
- the first original image may be corrected first, and then the corrected first original image may be compressed.
- beneficial effects of performing data processing on the first original image reference may be made to the beneficial effects of the above two examples.
- both of them in addition to storing multiple images in the above-mentioned painting library and knowledge graph library, both of them also store information corresponding to the respective stored images.
- the information includes related information of the first target painting collection, for example, it may include the author, style, or collection location of the first target painting collection.
- the terminal device may display the first target painting collection.
- the display of the first target painting collection by the terminal device may be referred to as a primary display.
- the terminal device may, in response to the user's selection operation, search for information corresponding to the selected image from the painting library or the knowledge graph library, and display the information of the selected image.
- the terminal device only displays at least one image in the first target painting set.
- the user can select from at least one image displayed on the terminal device (for example, by clicking on the display screen of the terminal device to select) The image.
- the terminal device can respond to the user's selection operation to cause the screen displayed on the display screen to jump and display the information of the selected image.
- the display of the information of the selected image by the terminal device may be referred to as a secondary display.
- the terminal device may also respond to the user's question operation by sending the questioned key field to the image retrieval apparatus.
- the information of each image includes at least one key field, and each key field in the at least one key field includes an information entity.
- the information of a certain image includes "The author is Zhang San, the style is oil painting, and it is collected in the National Art Museum of China", and the key fields include Zhang San, oil painting or the National Art Museum of China.
- Zhang San, oil painting or National Art Museum of China are the information entities.
- the user can select a key field from it (that is, the information you want to know Ask questions).
- the terminal device can respond to the user's questioning operation and send the key field questioned to the image retrieval device.
- the image retrieval apparatus may obtain information corresponding to the key field being asked according to the knowledge graph library, and send the information to the terminal device.
- the image retrieval device After the image retrieval device receives the key field being asked, it can search and obtain information that constitutes a relationship with the key field being asked according to the knowledge graph database, that is, information corresponding to the key field being asked.
- the information corresponding to the key field being asked is a detailed introduction of the key field being asked.
- the information corresponding to the questioned key field corresponds to the questioned key field means that the information corresponding to the questioned key field corresponds to the questioned key field through a relationship. This relationship includes but is not limited to date of birth, place of birth, work or style of work, etc.
- the key field being questioned is Zhang San
- the information corresponding to the key field being questioned includes: xx year xx month xx day corresponding to Zhang San through the date of birth, and one corresponding to Zhang San through the place of birth In Beijing, through works corresponding to Zhang San's "One", “Two” and “Three”, etc., or through works style corresponding to Zhang San's scenic style, and so on.
- the terminal device after receiving the information corresponding to the key field being asked, displays the information. In this way, the retrieval of the image and the retrieval and understanding of the information of the image are realized without knowing the related information such as the name and author of the image.
- the display of the information by the terminal device may be referred to as a three-level display.
- the image retrieval method provided by some embodiments of the present invention, by combining with the knowledge graph database, can feed back the information related to the selected image when the user’s question operation about the selected image is received.
- the information corresponding to the key field being questioned enables the user to gain a further understanding of the relevant information of the retrieved image, and thus to obtain a further understanding of the retrieved image.
- the terminal device After the terminal device displays the information corresponding to the key field being asked, it can further respond to the user's question operation, that is, perform the above steps in turn, and perform the information corresponding to the key field being asked. Learn more. In this way, the embodiment of the present invention can respond to and feed back the user's question operation step by step through the combined knowledge graph library, so that the user can obtain a fuller understanding of the retrieved image.
- the above-mentioned image retrieval method further includes: the image retrieval device stores at least one of the first original image and the key field being questioned.
- the above-mentioned key field being asked includes the key field being asked in at least one question operation.
- the user’s retrieval of at least one of the first original image and the questioned key field can be used as the user’s log, so that by storing at least one of the first original image and the questioned key field, This can realize the storage of the user's log, and facilitate subsequent users to re-search and understand the retrieved content.
- a user name needs to be registered. While the image retrieval device stores the user's log, it also stores the user's user name, so that the user name of each user corresponds to the log, which facilitates the management of each user's log.
- the above-mentioned image retrieval method further includes: obtaining relevant information corresponding to at least one of the stored first original image and the questioned key field according to the knowledge graph, and sending the relevant information To the terminal device; wherein the related information includes similar target paintings and/or similar target information similar to at least one of the stored original image and the key field being questioned.
- the image retrieval device After the image retrieval device stores at least one of the first original image and the questioned key field, the image retrieval device can obtain the relationship between the first original image and the questioned key field according to the relationship in the knowledge graph database. Relevant information corresponding to at least one of. In this way, the recommendation of information related to the user's log can be realized.
- the related information corresponding to the first original image can be obtained through the relationship in the knowledge graph database, and the related information includes similarities similar to the first original image.
- Target paintings and/or similar target information can be obtained through the relationship in the knowledge graph database, and the related information includes similarities similar to the first original image.
- the above-mentioned image retrieval method further includes S600-S1200.
- the user's operation information refers to the content record of the information interaction between the user and the image retrieval device through the terminal device.
- the user's operation on the first target painting collection is, for example, the above-mentioned selection operation or question operation.
- the user's operation information is, for example, the user's log.
- the acquired user operation information may be user operation information in a certain period of time.
- the time period can be one month, two weeks, one week, or one day, which can be set according to actual needs.
- S700 Update the first image feature extraction model according to the user's operation information to obtain a second image feature extraction model.
- S800 Receive a second original image.
- the second image feature extraction model is used to extract the image feature of the second original image to obtain the second feature code.
- the process of finding the second target painting set and outputting the second target painting set in the above S800 ⁇ S1200 can refer to the process of finding the first target painting set and outputting the first target painting set in the above S100 ⁇ S500 Note, I won’t repeat it here.
- the first image feature extraction model is updated according to the user's operation information to obtain the second image feature extraction model, including S710-S740.
- S710 Classify user operation information, and calculate the proportion of user operation information corresponding to various tags.
- categorizing the user's operation information may include: setting multiple tags, and marking the corresponding user operation information with at least one tag.
- each tag represents a type of operation information, for example, the multiple tags may be: more related works, author profile, works market price, etc.
- the proportion of user operation information corresponding to different types of tags may be different.
- the proportion of user operation information corresponding to tags that are of great interest to the user may be relatively large, and the proportion of user operation information corresponding to tags that are of little interest to the user may be relatively small.
- the sum of the weights of the image features corresponding to various tags is 1.
- the weight of the image feature corresponding to the tag "type” is 50%
- the weight of the image feature corresponding to the tag “content” is 50%.
- the proportion of user operation information corresponding to the label "type” is 70%
- the proportion of user operation information corresponding to the label "content” is 30%.
- the weight of the corresponding image feature is adjusted from 50% to 30%.
- S730 Train and form a second image feature extraction model according to the first image feature extraction model and the adjusted weight.
- the first image feature extraction model can be combined with training to form a second image feature extraction model.
- the second image feature extraction model is a feature extraction model of a deep learning model.
- the first image feature extraction model may be replaced according to the path of storing the first image feature extraction model.
- the second target information is obtained according to the second feature encoding, including S1010 to S1030.
- S1010 Acquire a third feature code library corresponding to the painting library and a fourth feature code library corresponding to the knowledge graph library.
- the third feature encoding library and the fourth feature encoding library are obtained by performing image feature extraction on the images in the painting library and the knowledge map library by the second image feature extraction model.
- the third feature encoding library is a feature encoding library obtained by performing image feature extraction on a plurality of images in the painting library through a second image feature extraction model;
- the fourth feature encoding library is a feature encoding library obtained by using the second image feature extraction model A feature code library obtained by extracting image features from multiple images in the knowledge map library.
- the weight of the image feature corresponding to the label "type” is 70%, and the weight of the image feature corresponding to the label “content” is 30%.
- the feature code corresponding to the tag "type” can account for 70%, and the feature code corresponding to the tag "content” The proportion of the code can be 30%.
- S1020 Calculate the distance between the second feature code and each feature code in the third feature code library and the fourth feature code library, and obtain feature codes corresponding to each distance within a preset range, and compare the obtained Each feature code is used as the second target feature code set.
- S1030 Determine second target information according to the second target feature encoding set.
- the painting library can be updated multiple times to make the paintings stored in the painting library more comprehensive.
- the image retrieval method further includes: the terminal device responds to the user's update operation, sending an update request to the image retrieval device, and the retrieval device updates the painting library according to the update request.
- the first feature encoding library is obtained by extracting the image features of each painting in the painting library, after the painting library is updated, the first feature encoding library can also be updated.
- the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered as going beyond the scope of the present invention.
- an image retrieval device that implements the above image retrieval method and an image retrieval system including the image retrieval device and terminal equipment are also provided.
- the image retrieval device will be described below, and An image retrieval system including an image retrieval device and terminal equipment is introduced as an example.
- the embodiment of the present invention can divide the image retrieval apparatus into functional modules according to the above method embodiments.
- each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module.
- the above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiment of the present invention is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.
- the image retrieval device 100 and the image retrieval system 300 are schematically described below.
- the image retrieval device 100 includes a processor 11 and a memory 12.
- the memory 12 stores program instructions, which can be executed by the processor 11 to cause the retrieval device 100 to perform the following operations: S100-S500.
- the processor 11 may include one or more processing units.
- the processor 11 may include an application processor (AP), a modem processor, and a graphics processing unit (GPU).
- AP application processor
- GPU graphics processing unit
- Image signal processor image signal processor, ISP
- controller memory
- video codec digital signal processor
- DSP digital signal processor
- baseband processor baseband processor
- neural network processor neural-network processing unit
- the memory 12 may be provided independently of the processor 11 and coupled to the processor 11, and the memory 12 may also be provided in the processor 11 for storing instructions and data.
- the memory in the processor is a cache memory.
- the memory can store instructions or data that the processor has just used or recycled. If the processor needs to use the instruction or data again, it can be called directly from the memory. It avoids repeated access and reduces the waiting time of the processor, thus improving the efficiency of the system.
- the beneficial effects that can be achieved by the image retrieval device 100 provided by some embodiments of the present invention are the same as the beneficial effects that can be achieved by the image retrieval methods provided in some of the foregoing embodiments, and will not be repeated here.
- the image retrieval device 100 further performs the following operations: extracting the image features of the first original image by using the first image feature extraction model to obtain the first feature code.
- the image retrieval apparatus 100 in the process of obtaining the first target information according to the first feature encoding, the image retrieval apparatus 100 further performs the following operations: S310-S330.
- the image retrieval device 100 in the process of finding the first target painting collection corresponding to the first target information in the painting library and the knowledge graph library, the image retrieval device 100 also performs the following operations: S410 to S430.
- the above-mentioned image retrieval apparatus 100 further performs the following operations: S600-S1200.
- the first image feature extraction model is updated according to the user's operation information to obtain the second image feature extraction model
- the image retrieval apparatus 100 further performs the following operations: S710 ⁇ S740.
- the image retrieval apparatus 100 in the process of obtaining the second target information according to the second feature encoding, the image retrieval apparatus 100 further performs the following operations: S1010 to S1030.
- the image retrieval apparatus 100 includes a first server and a second server.
- the first server may be, for example, a middle-stage server
- the second server may be a cloud server. Setting two servers can ensure that different functions are set on different servers, so that when one of the servers has a problem, the functions on the other servers can still work normally.
- the first server may store a painting library
- the second server may store a knowledge graph library
- the first server 110 is configured to receive the first original image and send the first original image to the second server 120; or, the first server 110 is configured to receive the first original image.
- Image data processing is performed on the first original image, and the first original image after the data processing is sent to the second server 120.
- the second server 120 is configured to receive the first original image sent by the first server 110, extract the image feature of the first original image to obtain the first feature code, and obtain the first target information according to the first feature code.
- the first server 110 is configured to search for a first target painting set corresponding to the first target information in the painting library and/or the knowledge graph library according to the first target information, and output the first target painting set.
- the first server is further configured to receive the first target information sent by the second server, and according to the first target information, find a first search result corresponding to the first target information in the painting library.
- the second server is further configured to search and obtain a second search result corresponding to the first target information in the knowledge graph database according to the first target information.
- the first search result and the second search result are used as the first target painting collection.
- the collection of the first search result and the second search result is used as the first target painting collection.
- the first server is further configured to: search in the painting library according to the first target information to obtain a first search result; search in the knowledge graph library according to the first target information to obtain a second search result .
- the first search result is used as the first target painting set; in the case where the first search result does not include the hit painting and the second search result includes the hit painting, the first search result is It is used as the first target painting set together with the hit painting; in the case that neither the first search result nor the second search result includes the hit painting, the first search result is used as the first target painting set.
- the second server is further configured to compare the first search result with the second search result.
- the image retrieval apparatus 100 is further configured to obtain the user's operation information in response to the user's operation on the first target painting collection.
- the user's operation information refers to the user's operation of asking questions about the displayed painting and painting information.
- the user's operation information includes the keyword clicked by the user and the number of times the corresponding keyword is clicked.
- the image retrieval device 100 is also configured to update the first image feature extraction model according to the user's operation information to obtain the second image feature extraction model.
- the image retrieval device 100 is also configured to receive the second original image, use the second image feature extraction model to extract the image features of the second original image to obtain the second feature code, and according to the second target information, in the painting library and the knowledge map
- the second target painting set corresponding to the second target information is found in the library, and the second target painting set is output.
- the second server is further configured to: in response to the user's operation on the first target painting collection, obtain the user's operation information, and update the first image feature extraction model according to the user's operation information to obtain The second image feature extraction model.
- the first server is also configured to receive the second original image and send the second original image to the second server.
- the second server is also configured to: use the second image feature extraction model to extract the image features of the second original image to obtain the second feature code, and obtain the second target information according to the second feature code, and to convert the second target information Transfer to the first server.
- the first server is further configured to: according to the second target information, find a second target painting collection corresponding to the second target information in the painting library and the knowledge graph library, and output the second target painting collection.
- the second server is further configured to classify user operation information, calculate the proportion of user operation information corresponding to various tags, and adjust various tags according to the proportions of various tags.
- a painting retrieval system 300 includes: a terminal device 200 and an image retrieval apparatus 100 as provided in some of the above embodiments.
- the foregoing terminal device may be a mobile phone, a tablet computer, a PC (Personal Computer), or a notebook, or the foregoing terminal device may be a camera, a mobile phone, a tablet computer, or a PC (Personal Computer) that is electrically connected to the screen. , Personal computer) or notebook and other equipment.
- the terminal device is configured to: collect a first original image, and upload the first original image to the image retrieval device; receive the first target painting collection output by the image retrieval device; Operation, output the first target painting collection.
- the terminal device can be coupled with the painting display terminal to form a display system to display the first target painting collection on the image retrieval device.
- the painting display terminal refers to a device that can display paintings.
- the size of the painting display terminal is larger than that of the terminal device, and illustratively, the display effect of the painting display terminal is better than the display effect of the terminal device, for example, painting display
- the terminal is a drawing screen.
- the terminal device 200 may be coupled to the image retrieval apparatus 100 and configured to send the first original image to the image retrieval apparatus 100 and Receive first target information, etc.
- the terminal device 200 can also be coupled with the picture screen P to form a display system 400 to display the retrieved paintings on the picture screen P.
- the embodiment of the present disclosure does not limit the coupling manner between the terminal device 200 and the picture screen P.
- the terminal device 200 and the picture screen P may be coupled via Bluetooth or Wifi.
- the user can realize the functions of searching for paintings and acquiring related information by operating the terminal device 200.
- the terminal device 200 may include a painting appreciation applet. See FIG. 13C.
- the terminal device 200 will display an initial interface such as shown in (a) in FIG. 13C.
- the user can do so by doing the following: first click the "add image" button, for example, click the finger in (a) in Figure 13C Icon to enter the image acquisition interface shown in (b) of Figure 13C.
- the user can acquire the photos of the image that he wants to enjoy from the storage system of the terminal device 200, such as an album, or take pictures of the desired image on the spot.
- the storage system of the terminal device 200 such as an album
- take pictures of the desired image on the spot For the photo of the image to be enjoyed, take on-site shooting as an example. Click the "camera" icon in (b) in FIG. 13C to enter the shooting interface shown in (c) in FIG. 13C, and take a photo.
- the painting appreciation applet will automatically identify the paintings in the photos, and search for related paintings in the image retrieval device 300.
- the retrieved target paintings and similar paintings are displayed on the terminal device 200, as shown in FIG. 9, and then, The user can also continue to click on the text in the painting or the introduction of the painting to obtain more relevant information, that is, to ask questions about the displayed painting and its related information.
- the terminal device 200 and the image retrieval apparatus 100 may communicate through a network, and the network may include various connection types, such as wired, wireless communication links, or fiber optic cables.
- the terminal device 200 is configured to collect a first original image, and upload the first original image to the image retrieval apparatus 100.
- the image retrieval device 200 is configured to receive the first original image and extract the image feature of the first original image to obtain the first feature code.
- the image retrieval device 200 is further configured to obtain first target information according to the first feature encoding.
- the image retrieval device 200 is further configured to search for the first target painting set corresponding to the first target information in the painting library and the knowledge graph library according to the first target information.
- the image retrieval device 200 is also configured to output the first target collection of paintings.
- the terminal device 100 is also configured to display the first target collection of paintings.
- the beneficial effects that can be achieved by the painting retrieval system 300 provided by some embodiments of the present disclosure are the same as the beneficial effects that can be achieved by the image retrieval apparatus 100 and the terminal device 200 provided in some of the above embodiments, and will not be repeated here.
- the terminal device 200 is also configured to display the second target collection of paintings.
- the image retrieval apparatus 100 is further configured to classify user operation information, and calculate the proportion of user operation information corresponding to various tags.
- the image retrieval device 100 is also configured to adjust the weights of the image features corresponding to the various tags according to the proportions of the various tags.
- the image retrieval device 100 is further configured to train and form a second image feature extraction model according to the first image feature extraction model and the image features after adjusting the weight.
- the image retrieval device 100 is also configured to replace the first image feature extraction model with a second image feature extraction model.
- the image retrieval device 100' and the image retrieval system 300' including the terminal device 200' and the image retrieval device 100' will be schematically described below. .
- the image retrieval device 100' includes: a painting retrieval module 1', a search module 2', a knowledge graph library 3', and a painting library 4'.
- the terminal device 200' includes: an image acquisition module 5'and a display module 6'.
- the image acquisition module 5' is configured to acquire a first original image, and upload the first original image to the image retrieval device 100'.
- the painting retrieval module 1' is configured to receive the first original image, extract the image feature of the first original image to obtain the first feature code, and obtain the first target information according to the first feature code.
- the search module 2' is configured to find the first target painting set corresponding to the first target information in the painting library 4'and the knowledge graph library 3'according to the first target information, and output the first target painting set corresponding to the first target information.
- the display module 6' is configured to display the first set of target paintings.
- the display module 6' may be a display screen.
- the display screen may be an LCD display screen or an OLED display screen or the like.
- the display screen may be a display screen with a touch function, so that it is convenient for the user to perform selection operations and question operations by touching the display screen.
- the image retrieval device 100' further includes: a feature code library 7'.
- the feature code library 7' includes a first feature code library and a second feature code library, and is configured to store the feature code obtained by the first image feature extraction model in each painting in the painting library and the knowledge map library .
- the painting retrieval module 1' is further configured to calculate the distance between the first feature code and each feature code in the first feature code library and the second feature code library, and obtain the distance within the preset range
- the feature codes corresponding to the distances of, the acquired feature codes are used as the first target feature code set; the first target information is determined according to the first target feature code set.
- the search module 2' is further configured to obtain a search result list according to the first target information, and search for the first target painting collection in the painting library 4'and the knowledge graph library 3'according to the search result list.
- the search result list includes painting names corresponding to the first target painting collection.
- the search module 2' is further configured to search for information corresponding to the first target painting set in the knowledge graph library 3'.
- the display module 6' is further configured to display information corresponding to the first set of target paintings.
- the terminal device 200' further includes a feedback module 8'.
- the feedback module 8' is configured to, in response to the user's questioning operation, send the questioned key field to the image retrieval device 100'.
- the image retrieval apparatus 100' further includes: a questioning module 9'.
- the questioning module 9' is configured to receive the key field being questioned and send the key field being questioned to the searching module 2'.
- the image retrieval apparatus 100' further includes: a knowledge query module 10'.
- the knowledge query module 10' is configured to receive the questioned key field sent by the search module 2', find the information corresponding to the questioned key field according to the knowledge graph database 3', and send the information To find module 2'.
- the search module 2' is further configured to search for the information corresponding to the key field in question in the painting library 4'.
- the search module 2' is further configured to send information corresponding to the key field being questioned to the terminal device 200'.
- the display module 6' is also configured to display information corresponding to the key field being asked.
- the image retrieval apparatus 100' further includes: a log storage module 11'.
- the log storage module 11' is configured to receive the first original image sent by the painting retrieval module 1', receive the questioned key field sent by the questioning module 9', and compare the first original image and the questioned key field At least one of them is stored.
- the image retrieval apparatus 100' further includes: a data processing module 12'.
- the data processing module 12' is configured to receive the first original image sent by the image acquisition module 5', perform data processing on the first original image, and send the first original image after the data processing to the painting retrieval module 1' .
- the data processing includes correction processing and/or compression processing.
- the image retrieval device 100' further includes: a painting library update module 13'.
- the painting library update module 13' is configured to update the painting library 4'.
- the image retrieval device 100' further includes: a feature code library update module 14'.
- the feature code library update module 14' is configured to update the first feature code library in the feature code library 7'after updating the painting library 4'.
- the feature code library update module 14' is further configured to update the second feature code library in the feature code library 7'after updating the knowledge graph library 3'.
- the image retrieval apparatus 100' further includes: a feature extraction model update module 15', and the feature extraction model update module 15' is configured to update the feature encoding library module 14' according to the information in the log storage module 11' Update.
- the structure of the image retrieval apparatus 100' provided in some of the foregoing embodiments includes multiple structures, which are not limited in the present invention, and the functions mentioned in the foregoing embodiments shall prevail.
- the image retrieval apparatus 100' includes a first server 110' and a second server 120'.
- the first server 110' may include a search module 2', a painting library 4', a questioning module 9', and a data processing module 12'
- the second server 120' may include a painting retrieval module 1', a knowledge graph library 3', and features
- the second server 120' as a cloud server in which a GPU Docker and/or a CPU Docker are built in the cloud server as an example, the structure of the second server 120' is schematically described.
- the deployment plan of the cloud server is as follows.
- GPU Docker and CPU Docker are jointly built on the container cloud platform.
- the painting retrieval algorithm, the painting library update algorithm, the log storage algorithm, and the input or output application programming interface of the call of the knowledge graph library are deployed on the GPU Docker and CPU Docker. , API for short).
- the number of GPU Docker and CPU Docker used can be increased and decreased accordingly according to the load capacity and performance indicators of hardware resources.
- the deployment of each algorithm adopts a Docker deployment plan that combines GPU Docker and CPU Docker, which can effectively evaluate the utilization of hardware resources and facilitate algorithm migration and resource expansion.
- Nginx load balancing is set up between GPU Docker and CPU Docker. Its function is: the greater the number of visits per unit time of a server, the greater the pressure on the server, and the pressure on the server When it exceeds its own capacity, the server will crash. By setting up Nginx load balancing, you can share the pressure on the server, avoid server crashes, and let users have a better experience.
- the method of setting up Nginx load balancing to share the pressure on the server is: the component includes a server with multiple servers (for example, the server is a Docker including the painting retrieval module 1', or a Docker including the painting library update module 13', etc.) Clusters, and intermediate servers (for example, Nginx load balancing).
- the access request can be sent to the intermediate server first, and the intermediate server can select a server with less pressure in the server cluster and introduce the access request to the server.
- the pressure on each server in the server cluster can be balanced, and a situation where a certain server is under too much pressure and crashes can be avoided.
- the image retrieval device 100' can use Nginx load balancing to select a server with less pressure ( That is, the Docker that includes the painting retrieval module 1'), and sends the original image to the server to retrieve the original image. In this way, it can be avoided that the request for retrieving the first original image generated by the terminal device 200' is concentrated in one server, causing the server to crash.
- the update of the painting library and the update of the first feature encoding library can be implemented through interaction with the Redis cluster. In this way, in the process of updating the painting library and/or updating the first feature code library, there is no need to restart the server.
- the storage of the user's log (that is, the user's retrieval record) can be implemented through Kafka cluster interaction.
- the user's feedback request and/or question request for the information of the first original image displayed by the terminal device can call the knowledge in the knowledge graph library 3'through the knowledge query module 10', and feed the called knowledge back to the user.
- the log storage module 11' the user's feedback request and the user's username can be bundled and stored through Kafka cluster interaction.
- the shared storage in FIG. 15 and FIG. 16 is used to store the code data of each algorithm and the model file for each algorithm to call.
- the above-mentioned multiple functional modules included in the terminal device 200' may be integrated in the mobile phone software (Application, APP for short) in the terminal device 200'.
- the application program in the APP By running the application program in the APP, the functions required by the terminal device 200' can be realized.
- Some embodiments of the present invention provide a computer-readable storage medium (for example, a non-transitory computer-readable storage medium) in which computer program instructions are stored, and when the computer program instructions run on a processor , Causing the processor to execute one or more steps in the image retrieval method described in any one of the above embodiments.
- the aforementioned computer-readable storage medium may be, for example, a non-transitory computer-readable storage medium.
- the foregoing computer-readable storage medium may include, but is not limited to: magnetic storage devices (for example, hard disks, floppy disks, or magnetic tapes, etc.), optical disks (for example, CD (Compact Disk), DVD (Digital Versatile Disk, Digital universal disk), etc.), smart cards and flash memory devices (for example, EPROM (Erasable Programmable Read-Only Memory), cards, sticks or key drives, etc.).
- Various computer-readable storage media described in the present invention may represent one or more devices and/or other machine-readable storage media for storing information.
- the term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and/or carrying instructions and/or data.
- Some embodiments of the present invention also provide a computer program product.
- the computer program product includes computer program instructions, and when the computer program instructions are executed on a computer, the computer program instructions cause the computer to execute one or more steps in the image retrieval method described in the foregoing embodiment.
- Some embodiments of the present invention also provide a computer program.
- the computer program When the computer program is executed on a computer, the computer program causes the computer to execute one or more steps in the image retrieval method described in the above-mentioned embodiments.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Library & Information Science (AREA)
- Multimedia (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
Claims (18)
- 一种图像检索方法,包括:接收第一原始图像;提取所述第一原始图像的图像特征,得到第一特征编码;根据所述第一特征编码,得到第一目标信息;根据所述第一目标信息,在画作库和/或知识图谱库中查找得到与所述第一目标信息相对应的第一目标画作集;输出所述第一目标画作集。
- 根据权利要求1所述的图像检索方法,其中,所述提取所述第一原始图像的图像特征,得到第一特征编码,包括:利用第一图像特征提取模型,提取所述第一原始图像的图像特征,得到所述第一特征编码;所述图像检索方法,还包括:响应于用户对所述第一目标画作集的操作,获取用户的操作信息;根据所述用户的操作信息,对所述第一图像特征提取模型进行更新,得到第二图像特征提取模型;接收第二原始图像;利用所述第二图像特征提取模型,提取所述第二原始图像的图像特征,得到第二特征编码;根据所述第二特征编码,得到第二目标信息;根据所述第二目标信息,在所述画作库和所述知识图谱库中查找得到与所述第二目标信息相对应的第二目标画作集;输出所述第二目标画作集。
- 根据权利要求2所述的图像检索方法,其中,所述根据所述用户的操作信息,对所述第一图像特征提取模型进行更新,得到第二图像特征提取模型,包括:对所述用户的操作信息进行归类,计算各类标签所对应的用户操作信息的占比;根据所述各类标签的占比,调整所述各类标签所对应的图像特征的权重;根据所述第一图像特征提取模型以及调整后的权重,训练形成所述第二图像特征提取模型;将所述第一图像特征提取模型替换为所述第二图像特征提取模型。
- 根据权利要求1至3中的任一项所述的图像检索方法,其中,所述根据所述第一特征编码,得到第一目标信息,包括:计算所述第一特征编码与第一特征编码库和第二特征编码库中的每个特征编码之间的距离,并获取在预设范围内的各距离所对应的特征编码,将所获取的各特征编码作为第一目标特征编码集;根据所述第一目标特征编码集确定所述第一目标信息;其中,所述第一特征编码库为通过所述第一图像特征提取模型对所述画作库中的多个图像进行图像特征提取获得的特征编码库;所述第二特征编码库为通过所述第一图像特征提取模型对所述知识图谱库中的多个图像进行图像特征提取获得的特征编码库。
- 根据权利要求2至4中的任一项所述的图像检索方法,其中,所述根据所述第二特征编码,得到第二目标信息,包括:计算所述第二特征编码与第三特征编码库和第四特征编码库中的每个特征编码之间的距离,并获取在预设范围内的各距离所对应的特征编码,将所获取的各特征编码作为第二目标特征编码集;根据所述第二目标特征编码集确定所述第二目标信息;其中,所述第三特征编码库为通过所述第二图像特征提取模型对所述画作库中的多个图像进行图像特征提取获得的特征编码库;所述第四特征编码库为通过所述第二图像特征提取模型对所述知识图谱库中的多个图像进行图像特征提取获得的特征编码库。
- 根据权利要求1至5中的任一项所述的图像检索方法,其中,所述根据所述第一目标信息,在画作库和/或知识图谱库中查找得到与所述第一目标信息相对应的第一目标画作集包括:根据所述第一目标信息在所述画作库中查找得到第一查找结果;根据所述第一目标信息在所述知识图谱库中查找得到第二查找结果;在所述第一查找结果包括命中画作的情况下,以所述第一查找结果作为第一目标画作集;所述命中画作为所述第一目标信息对应的各画作中与所述原始图像相似度最高的一幅;在所述第一查找结果不包含所述命中画作且所述第二查找结果包括所述命中画作的情况下,以所述第一查找结果和所述命中画作共同作为所述第一目标画作集;在所述第一查找结果和所述第二查找结果均不包括所述命中画作的情况下,以所述第一查找结果作为所述第一目标画作集。
- 根据权利要求1至5中的任一项所述的图像检索方法,其中,所述根据所述第一目标信息,在画作库和/或知识图谱库中查找得到与所述第一目标信息相对应的第一目标画作集包括:根据所述第一目标信息在所述画作库中查找得到第一查找结果;根据所述第一目标信息在所述知识图谱库中查找得到第二查找结果;在所述第一查找结果和所述第二查找结果相同的情况下,以所述第一查找结果作为第一目标画作集;在所述第一查找结果和所述第二查找结果不完全相同的情况下,以所述第一查找结果和所述第二查找结果的合集作为所述第一目标画作集。
- 一种图像检索装置,所述图像检索装置被配置为执行权利要求1至7中的任一项所述的图像检索方法;其中,所述图像检索装置包括:处理器和存储器;所述存储器存储有程序指令,所述程序指令可由所述处理器执行。
- 一种图像检索装置,包括:第一服务器和第二服务器;所述第一服务器存储有画作库,所述第二服务器存储有知识图谱库;其中,所述第一服务器被配置为:接收第一原始图像,并将所述第一原始图像发送至所述第二服务器;所述第二服务器被配置为:接收所述第一服务器发送的第一原始图像;提取所述第一原始图像的图像特征得到第一特征编码;并根据所述第一特征编码得到第一目标信息,将所述第一目标信息传输至所述第一服务器;所述第一服务器还被配置为:接收所述第二服务器传输的所述第一目标信息;根据所述第一目标信息,在画作库和/或知识图谱库中查找得到与所述第一目标信息相对应的第一目标画作集;输出所述第一目标画作集。
- 根据权利要求9所述的图像检索装置,其中,所述第一服务器还被配置为,对所接收的所述第一原始图像进行数据处理,将数据处理后的第一原始图像发送至所述第二服务器。
- 根据权利要求9或10所述的图像检索装置,其中,所述第二服务器还被配置为:利用第一图像特征提取模型,提取所述第一原始图像的图像特征,得到第一特征编码。
- 根据权利要求9至11中的任一项所述的图像检索装置,其中,所述第二服务器还被配置为:响应于用户对所述第一目标画作集的操作,获取用户的操作信息;根据所述用户的操作信息,对所述第一图像特征提取模型进行更新,得到第二图像特征提取模型;所述第一服务器还被配置为:接收第二原始图像并将所述第二原始图像发送至所述第二服务器;所述第二服务器还被配置为:利用所述第二图像特征提取模型,提取所述第二原始图像的图像特征,得到第二特征编码;根据所述第二特征编码,得到第二目标信息;将所述第二目标信息传输至所述第一服务器;所述第一服务器还被配置为:根据所述第二目标信息,在所述画作库和所述知识图谱库中查找得到与所述第二目标信息相对应的第二目标画作集;输出所述第二目标画作集。
- 根据权利要求12所述的图像检索装置,其中,所述第二服务器还被配置为:对所述用户的操作信息进行归类,计算各类标签所对应的用户操作信息的占比;根据所述各类标签的占比,调整所述各类标签所对应的图像特征的权重;根据所述第一图像特征提取模型以及调整权重后的图像特征,训练形成所述第二图像特征提取模型;将所述第一图像特征提取模型替换为所述第二图像特征提取模型。
- 根据权利要求9至13中的任一项所述的图像检索装置,其中,所述第一服务器还被配置为:根据所述第一目标信息在所述画作库中查找得到第一查找结果;根据所述第一目标信息在所述知识图谱库中查找得到第二查找结果;在所述第一查找结果包括命中画作的情况下,以所述第一查找结果作为第一目标画作集;所述命中画作为所述第一目标信息对应的各画作中与所述原始图像相似度最高的一幅;在所述第一查找结果不包含所述命中画作且所述第二查找结果包括所述命中画作的情况下,以所述第一查找结果和所述命中画作共同作为所述第一目标画作集;在所述第一查找结果和所述第二查找结果均不包括所述命中画作的情况下,以所述第一查找结果作为所述第一目标画作集。
- 根据权利要求9至13中的任一项所述的图像检索装置,其中,所述第一服务器还被配置为:根据所述第一目标信息在所述画作库中查找得到第一查找结果;根据所述第一目标信息在所述知识图谱库中查找得到第二查找结果;在所述第一查找结果和所述第二查找结果相同的情况下,以所述第一查找结果和所述第二查找结果中的一者作为第一目标画作集;在所述第一查找结果和所述第二查找结果不完全相同的情况下,以所述 第一查找结果和所述第二查找结果的合集作为所述第一目标画作集。
- 一种图像检索系统,包括:图像检索装置,被配置为执行权利要求1至8任一项所述的图像检索方法;终端设备,被配置为:采集第一原始图像,并将所述第一原始图像上传至所述图像检索装置;接收所述图像检索装置输出的第一目标画作集;响应于用户的操作,输出所述第一目标画作集。
- 一种图像显示系统,包括:终端设备和画作显示终端;终端设备被配置为:采集第一原始图像,并将所述第一原始图像上传至图像检索装置;接收所述图像检索装置输出的第一目标画作集;响应于用户的操作,向所述画作显示终端输出所述第一目标画作集;其中,所述图像检索装置被配置为:执行权利要求1至8任一项所述的图像检索方法;所述画作显示终端被配置为:显示所述第一目标画作集。
- 一种计算机可读存储介质,存储有计算机程序指令,在所述计算机程序指令在处理器上运行时,使得处理器执行如权利要求1至8中任一项所述的图像检索的方法。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/781,093 US12026192B2 (en) | 2020-04-30 | 2021-04-30 | Image retrieval method, image retrieval devices, image retrieval system and image display system |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010368319.5A CN111581421B (zh) | 2020-04-30 | 2020-04-30 | 图像检索方法、图像检索装置及图像检索系统 |
CN202010368319.5 | 2020-04-30 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021219117A1 true WO2021219117A1 (zh) | 2021-11-04 |
Family
ID=72124657
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/091308 WO2021219117A1 (zh) | 2020-04-30 | 2021-04-30 | 图像检索方法、图像检索装置、图像检索系统及图像显示系统 |
Country Status (3)
Country | Link |
---|---|
US (1) | US12026192B2 (zh) |
CN (1) | CN111581421B (zh) |
WO (1) | WO2021219117A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2023131016A1 (zh) * | 2022-01-04 | 2023-07-13 | 北京字节跳动网络技术有限公司 | 教程数据的展示方法、装置、计算机设备以及存储介质 |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111581421B (zh) * | 2020-04-30 | 2024-06-04 | 京东方科技集团股份有限公司 | 图像检索方法、图像检索装置及图像检索系统 |
CN112329797B (zh) * | 2020-11-13 | 2024-08-13 | 杭州海康威视数字技术股份有限公司 | 目标对象的检索方法、装置、服务器和存储介质 |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107885861A (zh) * | 2017-11-21 | 2018-04-06 | 桂林爱家购股份有限公司 | 一种图像检索系统和方法 |
US20190147292A1 (en) * | 2017-11-13 | 2019-05-16 | Hitachi, Ltd. | Image retrieving apparatus, image retrieving method, and setting screen used therefor |
US20190180101A1 (en) * | 2016-07-30 | 2019-06-13 | Huawei Technologies Co., Ltd. | Image Recognition Method and Terminal |
CN110209866A (zh) * | 2019-05-30 | 2019-09-06 | 苏州浪潮智能科技有限公司 | 一种图像检索方法、装置、设备及计算机可读存储介质 |
CN110688516A (zh) * | 2019-10-08 | 2020-01-14 | 北京旷视科技有限公司 | 图像检索方法、装置、计算机设备和存储介质 |
CN111581421A (zh) * | 2020-04-30 | 2020-08-25 | 京东方科技集团股份有限公司 | 图像检索方法、图像检索装置及图像检索系统 |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107562805B (zh) * | 2017-08-08 | 2020-04-03 | 浙江大华技术股份有限公司 | 一种以图搜图的方法及装置 |
CN109918532B (zh) * | 2019-03-08 | 2023-08-18 | 苏州大学 | 图像检索方法、装置、设备及计算机可读存储介质 |
-
2020
- 2020-04-30 CN CN202010368319.5A patent/CN111581421B/zh active Active
-
2021
- 2021-04-30 WO PCT/CN2021/091308 patent/WO2021219117A1/zh active Application Filing
- 2021-04-30 US US17/781,093 patent/US12026192B2/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190180101A1 (en) * | 2016-07-30 | 2019-06-13 | Huawei Technologies Co., Ltd. | Image Recognition Method and Terminal |
US20190147292A1 (en) * | 2017-11-13 | 2019-05-16 | Hitachi, Ltd. | Image retrieving apparatus, image retrieving method, and setting screen used therefor |
CN107885861A (zh) * | 2017-11-21 | 2018-04-06 | 桂林爱家购股份有限公司 | 一种图像检索系统和方法 |
CN110209866A (zh) * | 2019-05-30 | 2019-09-06 | 苏州浪潮智能科技有限公司 | 一种图像检索方法、装置、设备及计算机可读存储介质 |
CN110688516A (zh) * | 2019-10-08 | 2020-01-14 | 北京旷视科技有限公司 | 图像检索方法、装置、计算机设备和存储介质 |
CN111581421A (zh) * | 2020-04-30 | 2020-08-25 | 京东方科技集团股份有限公司 | 图像检索方法、图像检索装置及图像检索系统 |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2023131016A1 (zh) * | 2022-01-04 | 2023-07-13 | 北京字节跳动网络技术有限公司 | 教程数据的展示方法、装置、计算机设备以及存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN111581421B (zh) | 2024-06-04 |
US12026192B2 (en) | 2024-07-02 |
CN111581421A (zh) | 2020-08-25 |
US20230004595A1 (en) | 2023-01-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021219117A1 (zh) | 图像检索方法、图像检索装置、图像检索系统及图像显示系统 | |
US12001475B2 (en) | Mobile image search system | |
US11899681B2 (en) | Knowledge graph building method, electronic apparatus and non-transitory computer readable storage medium | |
US9430585B2 (en) | Media object query submission and response | |
US9495385B2 (en) | Mixed media reality recognition using multiple specialized indexes | |
US9607014B2 (en) | Image tagging | |
US8385660B2 (en) | Mixed media reality indexing and retrieval for repeated content | |
US8676810B2 (en) | Multiple index mixed media reality recognition using unequal priority indexes | |
US10380461B1 (en) | Object recognition | |
US20090070302A1 (en) | Mixed Media Reality Recognition Using Multiple Specialized Indexes | |
US20240273136A1 (en) | Systems and methods for mobile image search | |
US20210406324A1 (en) | System and method for providing a content item based on computer vision processing of images | |
WO2021155682A1 (zh) | 一种多模态数据检索方法、系统、终端及存储介质 | |
CN111708942B (zh) | 多媒体资源推送方法、装置、服务器及存储介质 | |
JP7307607B2 (ja) | メディアベースのコンテンツシェアを容易にする方法、コンピュータプログラム及びコンピューティングデバイス | |
CN113204691B (zh) | 一种信息展示方法、装置、设备及介质 | |
CN112765387A (zh) | 图像检索方法、图像检索装置和电子设备 | |
US9424338B2 (en) | Clustering queries for image search | |
CN111538859B (zh) | 一种动态更新视频标签的方法、装置及电子设备 | |
CN114329028A (zh) | 一种数据处理方法、设备以及计算机可读存储介质 | |
US20230325391A1 (en) | Method and system of retrieving assets from personalized asset libraries | |
WO2021051562A1 (zh) | 人脸特征点定位方法、装置、计算设备和存储介质 | |
US20210263960A1 (en) | Information processing device, information processing system, and non-transitory recording medium | |
US10140518B1 (en) | Content information auditing service | |
WO2023168997A1 (zh) | 一种跨模态搜索方法及相关设备 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21796487 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21796487 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 23/06/2023) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21796487 Country of ref document: EP Kind code of ref document: A1 |