CN108288208B - Display object determination method, device, medium and equipment based on image content - Google Patents

Display object determination method, device, medium and equipment based on image content Download PDF

Info

Publication number
CN108288208B
CN108288208B CN201710685418.4A CN201710685418A CN108288208B CN 108288208 B CN108288208 B CN 108288208B CN 201710685418 A CN201710685418 A CN 201710685418A CN 108288208 B CN108288208 B CN 108288208B
Authority
CN
China
Prior art keywords
classification
nodes
node
leaf
selecting
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710685418.4A
Other languages
Chinese (zh)
Other versions
CN108288208A (en
Inventor
欧彦麟
芦清林
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN201710685418.4A priority Critical patent/CN108288208B/en
Publication of CN108288208A publication Critical patent/CN108288208A/en
Priority to PCT/CN2018/100009 priority patent/WO2019029714A1/en
Priority to US16/537,885 priority patent/US11200444B2/en
Application granted granted Critical
Publication of CN108288208B publication Critical patent/CN108288208B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0641—Shopping interfaces
    • G06Q30/0643—Graphical representation of items or shoppers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G06F16/48—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/483—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/51—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/901—Indexing; Data structures therefor; Storage structures
    • G06F16/9027—Trees
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24147—Distances to closest patterns, e.g. nearest neighbour classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/2431—Multiple classes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/24323—Tree-organised classifiers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0631—Item recommendations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Finance (AREA)
  • Accounting & Taxation (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Development Economics (AREA)
  • Library & Information Science (AREA)
  • Multimedia (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Image Analysis (AREA)

Abstract

The application provides a display object determining method, device, medium and equipment based on image content, relates to the technical field of information retrieval, and particularly relates to a method for solving the problem that the efficiency of determining similar objects of objects in the prior art is low. In the application, the ordered management of each display object in the display object set is realized by constructing the search tree, so that the search can be better performed when similar objects (namely, display objects to be recommended) are searched for objects in the images. In addition, in the embodiment of the application, when the display object to be recommended is retrieved, the retrieval is preferentially performed from the classification with high similarity to the object in the image, and the whole display object set is not traversed, so that the calculation amount can be reduced, and the processing resource can be saved. Furthermore, in the embodiment of the application, the retrieval is terminated when the specified quantity is reached, so that the retrieval efficiency can be further effectively improved.

Description

Display object determination method, device, medium and equipment based on image content
Technical Field
The present application relates to the field of information retrieval technologies, and in particular, to a method, an apparatus, a medium, and a device for determining a display object based on image content.
Background
Digital images have become a primary means of communicating information to users by networks and intelligent devices. For example, when shopping, the object is presented in the form of an image so that the user can understand the appearance of the object. When the video is viewed, each frame of image in the video also contains objects of various colors, wherein some objects may be interested by users. In order to facilitate a user to know an object and similar objects in a picture, in the prior art, features of the object in the picture are usually extracted, and then, according to the extracted features, a presentation object similar to the object is searched in a set of presentation objects and recommended to the user.
Generally, when searching for a display object in the prior art, it is necessary to calculate the similarity between an object in an image and each display object in a set of display objects according to the features of the object, and then select and recommend the display object with the top N (N is a positive integer greater than 0) of similarity ranking to a user.
Therefore, in the prior art, the whole set of display objects is circularly traversed when the display objects are searched each time, when the number of the display objects in the set of display objects is large, the whole set of display objects is traversed when similar objects of one object are searched each time, the calculation amount is huge, the time consumption is long, and the processing resources are wasted.
Disclosure of Invention
The embodiment of the application provides a display object determining method, device, medium and equipment based on image content, and when similar objects of objects in an image are determined to be display objects, the whole set of the display objects needs to be traversed, so that the problems of huge calculation amount, long time consumption, waste of processing resources and the like are solved.
The display object determining method based on the image content provided by the embodiment of the application comprises the following steps:
extracting the characteristics of at least one object contained in the image to be processed;
for each object, determining the feature similarity of the feature of the object and the feature similarity of classification nodes in a pre-constructed search tree; the search tree is used for representing a set of display objects, each classification node is used for representing one class of objects, each classification node comprises at least one leaf node, and the classification nodes and the leaf nodes in the search tree are used for representing the display objects;
selecting classification nodes according to the sequence of the feature similarity from high to low of the object, and calculating the feature similarity between leaf nodes contained in the selected classification nodes and the object until the number of the calculated leaf nodes reaches the specified number;
and selecting the first N classification nodes and leaf nodes as display objects to be recommended according to the sequence of the feature similarity from large to small, wherein N is a positive integer and is less than the designated number.
Another embodiment of the present application further provides an apparatus for determining a display object based on image content, including:
the characteristic extraction module is used for extracting the characteristics of at least one object contained in the image to be processed;
the classification node comparison module is used for determining the feature similarity of the object and the classification nodes in the pre-constructed search tree aiming at each object; the search tree is used for representing a set of display objects, each classification node is used for representing one class of objects, each classification node comprises at least one leaf node, and the classification nodes and the leaf nodes in the search tree are used for representing the display objects;
the similarity determining module is used for selecting classification nodes according to the sequence of the feature similarity of the classification nodes to the object from high to low, and calculating the feature similarity of leaf nodes contained in the selected classification nodes and the object until the number of the calculated leaf nodes reaches the specified number;
and the display object determining module is used for selecting the first N classification nodes and leaf nodes as display objects to be recommended according to the sequence of the feature similarity from large to small, wherein N is a positive integer and is less than the designated number.
Another embodiment of the present application further provides a computing device, which includes a memory and a processor, where the memory is configured to store program instructions, and the processor is configured to call the program instructions stored in the memory, and execute any display object determination method based on image content in the embodiments of the present application according to the obtained program instructions.
Another embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and when the processor loads the computer-executable instructions, the processor executes any display object determination method based on image content in the embodiments of the present application.
In the embodiment of the application, the ordered management of each display object in the display object set is realized by constructing the search tree, so that the search can be better performed when similar objects (namely, display objects to be recommended) are searched for objects in the images. In addition, in the embodiment of the application, when the display object to be recommended is retrieved, the retrieval is preferentially performed from the classification with high similarity to the object in the image, and the whole display object set is not traversed, so that the calculation amount can be reduced, and the processing resource can be saved. Furthermore, in the embodiment of the application, the retrieval is terminated when the specified quantity is reached, so that the retrieval efficiency can be further effectively improved.
Drawings
Fig. 1 is a schematic view of an application scenario of a display object determination method based on image content according to an embodiment of the present application;
fig. 2 is a schematic flowchart illustrating a method for determining a display object based on image content according to an embodiment of the present application;
fig. 3 is a schematic diagram illustrating classification of a display object according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a search tree according to an embodiment of the present application;
fig. 5 is a schematic flowchart illustrating a method for determining a display object based on image content according to a second embodiment of the present application;
fig. 6 is a schematic structural diagram illustrating a display object determination apparatus based on image content according to a third embodiment of the present application;
fig. 7 is a schematic structural diagram of a computing device according to a fourth embodiment of the present application.
Detailed Description
In order to facilitate understanding of technical solutions provided by the embodiments of the present application, the embodiments of the present application are described in further detail below with reference to the drawings of the specification.
Reference is first made to fig. 1, which is one of application scene schematic diagrams of a display object determination method based on image content according to an embodiment of the present application. The scenario may include, for example, a user 10, a smart terminal 11, and a server 12. Various clients capable of displaying images, such as a web browser, a news-about video, a shopping application client, etc., may be installed in the intelligent terminal 11. The user may perform an operation on the client of the smart terminal 11, requesting the client of the smart terminal 11 to display an image. For example, the Tencent video in the intelligent terminal 11 is requested to play the video. After the client of the intelligent terminal 11 and the server 12 establish communication connection; the image is acquired from the server 12 and displayed according to the user's request. Taking the displayed image as a to-be-processed image, the intelligent terminal 11 or the server 12 performs the following operations:
extracting the characteristics of at least one object contained in the image to be processed;
extracting the characteristics of at least one object contained in the image to be processed;
for each object, determining the feature similarity of the feature of the object and the feature similarity of classification nodes in a pre-constructed search tree; the search tree is used for representing a set of display objects, each classification node is used for representing one class of objects, each classification node comprises at least one leaf node, and the classification nodes and the leaf nodes in the search tree are used for representing the display objects;
selecting classification nodes according to the sequence of the feature similarity from high to low of the object, and calculating the feature similarity between leaf nodes contained in the selected classification nodes and the object until the number of the calculated leaf nodes reaches the specified number;
and selecting the first N classification nodes and leaf nodes as display objects to be recommended according to the sequence of the feature similarity from large to small, wherein N is a positive integer and is less than the designated number.
Of course, in the implementation, the above steps may be partially executed by the smart terminal 11 and partially executed by the server 12. For example, after extracting the features of at least one object included in the image to be processed, the intelligent terminal may send the extracted features of each object to the server 12, and then the server 12 performs subsequent operations to determine the display object to be recommended.
The intelligent terminal 11 and the server 12 may be communicatively connected through a communication network, which may be a local area network, a wide area network, or the like. The intelligent terminal 11 may be a mobile phone, a tablet computer, a notebook computer, a personal computer, etc., and the server 12 may be any device capable of supporting the corresponding method for determining the display object based on the image content.
To facilitate understanding of the display object determining method based on image content provided in the embodiments of the present application, the following embodiments further illustrate this.
Example one
Referring to fig. 2, a flowchart of a display object determination method based on image content according to an embodiment of the present application includes the following steps:
step 201: features of at least one object contained in the image to be processed are extracted.
Step 202: for each object, determining the feature similarity of the feature of the object and the feature similarity of classification nodes in a pre-constructed search tree; the search tree is used for representing a set of display objects, each classification node is used for representing one class of objects, each classification node comprises at least one leaf node, and the classification nodes and the leaf nodes in the search tree are used for representing the display objects.
Step 203: and selecting classification nodes according to the sequence of the feature similarity with the object from high to low, and calculating the feature similarity between leaf nodes contained in the selected classification nodes and the object until the number of the calculated leaf nodes reaches the specified number.
For example, the classification nodes are N1, N2, N3, and N4 in the descending order of similarity to the features of the objects in the image. Then, the feature similarity between each leaf node and the object is calculated from the leaf node of N1, and if the number of the calculated leaf nodes does not reach the specified number after the calculation of the leaf node of N1 is completed, the feature similarity between each leaf node and the object is calculated from the leaf node of the classification node N2 continuously until the number of the calculated leaf nodes reaches the specified number. Therefore, the reasonable display object to be recommended can be found well only by comparing the feature priority of the object with the display object under the classification node with the highest feature similarity. Moreover, the feature similarity of the object does not need to be compared with all display objects in the search tree, and the retrieval efficiency can be improved.
Step 204: and selecting the first N classification nodes and leaf nodes as display objects to be recommended according to the sequence of the feature similarity from large to small, wherein N is a positive integer and is less than the designated number.
Therefore, in the embodiment of the application, the search tree is constructed to realize the ordered management of the display objects in the display object set, so that the search can be better performed when similar objects (namely, display objects to be recommended) are searched for the objects in the images. In addition, in the embodiment of the application, when the display object to be recommended is retrieved, the retrieval is preferentially performed from the classification with high similarity to the object in the image, and the whole display object set is not traversed, so that the calculation amount can be reduced, and the processing resource can be saved. Furthermore, in the embodiment of the application, the retrieval is terminated when the specified quantity is reached, so that the retrieval efficiency can be further effectively improved.
In one embodiment, the presentation object may also be represented by only leaf nodes, and the classification nodes may exist in the form of tags. For example, if the extracted object is a garment, the tag may be a garment, and if the extracted object is a face, the tag is a face. In specific implementation, the search tree can be constructed according to the following method:
step A1: and determining labels corresponding to the display objects, wherein one label represents one classification.
In specific implementation, the tags corresponding to the display objects may be manually calibrated, and certainly, the probability that each display object belongs to each tag may also be determined by an artificial intelligence learning method through a pre-constructed tag-based classification model, and the tag with the highest probability is selected as the tag corresponding to the display object, or the tag with the probability greater than the preset probability is selected as the tag of the display object. When the label with the probability greater than the preset probability is selected as the label of the display object, one display object may correspond to a plurality of labels.
Step A2: and taking the label as a classification node, and taking a display object corresponding to the label as a leaf node of the label.
In a specific implementation, for each extracted object, determining feature similarity between the object and a classification node in a pre-constructed search tree based on the feature of the object in step 202 may specifically include:
and for each extracted object, determining the probability that the object belongs to each label based on a pre-constructed label-based classification model, and taking the probability as the feature similarity of the object and each classification node.
In addition, during specific implementation, the semantic closest to the feature of the object can be determined according to the extracted feature of the object and the conversion relation between the feature and the preset semantic space, and then the label corresponding to the object can be determined according to the semantic. For example, if the semantic meaning closest to the extracted object feature is clothing, the tag to which the object belongs is determined to be clothing. And taking the distance between the object feature and the closest semantic meaning as the feature similarity between the feature of the object and the corresponding label. It should be noted that how the extracted features of the object are mapped to the preset semantic space may be determined according to the prior art, which is not limited in this application.
Further, in this embodiment of the present application, in order to facilitate management and retrieval of the display object, if the classification node and the leaf node are both used to represent the display object, the search tree may be constructed according to the following method:
step B1: and classifying the display objects in the display object set based on a cluster analysis method, wherein the display objects with the characteristic similarity greater than the specified similarity between the display objects are classified into one class.
As shown in fig. 3, the elliptical area on the left side is provided with various dispersed display objects, after clustering analysis, the display objects are clustered, and the clustering result is shown in the diagram on the right side in fig. 3 and is divided into a plurality of categories.
The cluster analysis method may adopt methods in the prior art, such as a direct clustering method, a shortest distance clustering method, and a farthest distance clustering method.
Step B2: and for each type of display object, determining the display object closest to the cluster center in the type of display object as a classification node for representing the type of display object.
Step B3: and taking the display objects except the classification nodes in the display objects of the type as leaf nodes of the classification nodes.
In this way, in the embodiment of the present application, each classification node and leaf node represent an actual display object, so that when calculating the feature similarity with an object, the calculation of the classification node is compared with the calculation of the actual display object, and each calculation of the feature similarity is compared with the display object in the set of display objects, so that each calculation of the feature similarity is meaningful.
Further, in the embodiment of the present application, in order to further ensure that the calculation of the feature similarity is a comparison with an actual display object, the search tree in the embodiment of the present application may include a root node, and the root node is also an actual display object. Specifically, the method further comprises:
step C1: and calculating the clustering center of the classification node according to the characteristics of the classification node.
Step C2: selecting a classification node closest to the clustering center distance of the classification nodes from the classification nodes as a root node in a search tree; taking classification nodes except the root node as child nodes of the root node, and taking the child nodes as classification nodes in the search tree; wherein, the root node, the classification node and the leaf node all represent the display object.
In this way, the root node is also an actual display object, and in the implementation, after extracting the features of the object included in the image to be processed, the method further includes: and calculating the feature similarity of the features of the object and the root node.
Therefore, the root node participates in the calculation of the feature similarity, and the root node can also consider the display object to be recommended when the display object to be recommended is selected finally.
As shown in fig. 4, the structure of the established search tree is schematically illustrated. The left oval area in fig. 4 represents a set of presentation objects, which includes each presentation object. Fig. 4 shows a structure of a search tree, where nodes R, a, 1, b, c, and d are all display objects. a. b, c and d are in a category, and a is closest to the cluster center of the category, so that a is used for representing classification nodes, and b, c and d are leaf nodes of a. R, a, A and 1 are in a category, R is closest to the clustering center of the category, so R is used as a root node, and the rest a, A and 1 are classified nodes.
Further, in a specific implementation, in order to improve the retrieval efficiency by using the number of iterations as a termination retrieval condition, step 203 in this embodiment of the present application (i.e., selecting classification nodes according to a sequence from high to low of feature similarity of the object, and calculating feature similarity between leaf nodes included in the selected classification nodes and the object until the number of calculated leaf nodes reaches a specified number) may specifically include the following steps:
step D1: and taking the classification nodes as classification sets.
Step D2: and selecting a classification node with the maximum characteristic similarity with the object from the classification set.
Step D3: and selecting a leaf node from the selected classification nodes, calculating the feature similarity between the leaf node and the object, and increasing the times of selecting the leaf node by a specified value.
In specific implementation, the designated value may be 1, that is, the number of times of selecting the leaf node is increased by 1 every time the feature similarity is calculated.
Wherein, when selecting the leaf node, the leaf node can be selected randomly. Further, in order to ensure that the display object with high feature similarity can be retrieved as far as possible, in the embodiment of the present application, a leaf node with the maximum feature similarity to the classification node is selected from designated leaf nodes of the selected classification node, where the designated leaf node is a leaf node that does not calculate the feature similarity to the object. Thus, when searching for leaf nodes under a certain classification node, leaf nodes with high feature similarity with the classification node should have high feature similarity with an object. Therefore, the display object with high feature similarity can be preferentially searched as much as possible.
Step D4: and judging whether the number of the selected leaf nodes reaches the specified number.
Step D5: if the specified number is reached, step 204 is performed.
Step D6: if the specified number is not reached, judging whether leaf nodes which do not have the feature similarity with the object exist in the selected classification nodes or not.
Step D7: if yes, the process returns to step D3.
Step D8: if not, the selected classification node is deleted from the classification set, and the step D2 is executed.
Therefore, the classification nodes selected each time are the classification nodes with high feature similarity, and the leaf nodes selected each time are the leaf nodes with the highest possibility of high feature similarity with the object. Therefore, within limited comparison times, the display object with high feature similarity can be selected as much as possible, and the display object to be recommended selected from the display objects is more reasonable.
In specific implementation, the feature similarity may be classified into a grade, for example, the grade of each classification node is determined according to a corresponding relationship between the grade and the feature similarity range, where the feature similarity and the grade are in a positive correlation. In specific implementation, the feature similarity between the classification node and the object may be calculated preferentially from the classification node with the high level of feature similarity.
Of course, in specific implementation, the classification nodes with the feature similarity greater than the designated similarity may be selected first, and then the feature similarity between the leaf nodes included in each classification node and the object may be calculated from the selected classification nodes according to the descending order of the feature similarity.
In order to further understand the method for determining a display object based on image content provided in the embodiment of the present application, the following takes the search of similar faces as an example, and the embodiment will be described in detail.
Example two
As shown in fig. 5, taking the example of retrieving a face similar to a video as an example, the method provided by the embodiment of the present application is further explained, including the following steps:
step 501: and taking a currently displayed frame of image in the played video as an image to be processed, and extracting the characteristics of at least one human face contained in the image to be processed.
Certainly, in specific implementation, an image of a specified frame after the currently displayed frame of image may also be used as the image to be processed to compensate for the situation that the similar human face is displayed only after the currently displayed frame of image is played due to long retrieval time. In specific implementation, the method can be determined according to actual requirements, and the method is not limited in the application.
Step 502: and aiming at each extracted face, calculating the feature similarity between the features of the face and the root nodes in the face search tree constructed in advance based on the features of the face.
The construction method of the face search tree is the same as the construction method in steps B1-B3 and steps C1-C2 in the first embodiment, and will not be described herein again. It should be noted that the root node in the face search tree is also the face in the set of display objects.
Step 503: and determining the similarity between the characteristics of the human face and the characteristics of the classification nodes in the human face search tree.
Step 504: and taking the classification nodes as classification sets.
Step 505: and selecting a classification node with the maximum similarity to the characteristics of the human face from the classification set.
Step 506: and selecting the leaf node with the maximum feature similarity with the classification node from the designated leaf nodes of the selected classification nodes, wherein the designated leaf node is the leaf node which does not calculate the feature similarity with the face.
Step 507: and calculating the feature similarity between the selected leaf nodes and the face, and increasing the number of times of selecting the leaf nodes by 1.
Step 508: and judging whether the number of the selected leaf nodes reaches the specified number, if not, executing a step 509, and if so, executing a step 511.
Step 509: and judging whether leaf nodes which do not have the feature similarity with the face exist in the selected classification nodes, if so, executing step 506, and if not, executing step 510.
Step 510: the selected classification node is deleted from the classification set and the process returns to step 505.
Step 511: and selecting the first N nodes as the faces to be recommended according to the sequence of the feature similarity from large to small from the root node, the classification node and the leaf node of which the feature similarity is calculated.
If the root node and the feature similarity of a certain classification node and the face are ranked in the top N, the selected N nodes comprise the root node and the certain classification node.
In the embodiment of the application, similar faces are selected for the faces in the video based on the video and recommended to the user. Subsequent users may select similar faces of interest to learn the resume of the actor.
Certainly, in specific implementation, the extracted object can also be a commodity, so that similar commodities can be recommended to the user through the video, and the shopping requirement of the user can be met through the recommended commodities when the user needs the commodity, and the user does not need to search for the commodities in the video.
In addition, in the specific implementation, the search tree may include one large class of display objects, for example, all display objects are clothes, or may include a plurality of large classes of display objects, for example, including both clothes and shoes. In specific implementation, the search tree can be constructed according to actual requirements and the method provided by the embodiment of the application. For example, multiple search trees may be constructed, one search tree representing clothing and another search tree representing shoes. In particular, the root node of the search tree may be used to determine under which search tree the subsequent search is performed. For example, the clothes and the root node of the shoe search tree have low similarity and the clothes search tree has high similarity, so that when the display object to be recommended is determined subsequently, the display object to be recommended is preferentially retrieved under the clothes search tree based on a depth-first search method.
EXAMPLE III
Based on the same inventive concept, the embodiment of the present application further provides a device for determining a display object based on image content, and the principle and the beneficial effects of the device are similar to those described in the above method embodiment, and are not repeated herein.
As shown in fig. 6, is a schematic structural diagram of the apparatus, including:
a feature extraction module 601, configured to extract features of at least one object included in the image to be processed;
a classification node comparison module 602, configured to determine, for each object, a feature similarity between a feature of the object and a feature of a classification node in a search tree that is constructed in advance; the search tree is used for representing a set of display objects, each classification node is used for representing one class of objects, each classification node comprises at least one leaf node, and the classification nodes and the leaf nodes in the search tree are used for representing the display objects;
a similarity determining module 603, configured to select classification nodes according to a sequence from high to low of feature similarity with the object, and calculate feature similarity between leaf nodes included in the selected classification nodes and the object until the number of the calculated leaf nodes reaches a specified number;
the display object determining module 604 is configured to select, according to the order of the feature similarity from large to small, the first N classification nodes and leaf nodes as display objects to be recommended, where N is a positive integer and is smaller than the specified number.
Further, the apparatus further comprises:
the search tree construction module is used for constructing a search tree according to the following method if the classification nodes and the leaf nodes are used for representing the display objects:
classifying the display objects in the display object set based on a cluster analysis method, wherein the display objects with the characteristic similarity greater than the specified similarity among the display objects are classified into one class;
for each type of display object, determining the display object closest to the clustering center in the type of display object as a classification node for representing the type of display object; and the number of the first and second electrodes,
and taking the display objects except the classification nodes in the display objects of the type as leaf nodes of the classification nodes.
Further, the similarity determining module specifically includes:
a classification set determining unit, configured to use the classification node as a classification set;
the classification node selection unit is used for selecting a classification node with the maximum characteristic similarity with the object from the classification set;
the leaf node selecting unit is used for selecting a leaf node from the selected classification nodes, calculating the feature similarity between the leaf node and the object, and increasing the times of selecting the leaf node by a specified value;
the iteration judging unit is used for judging whether the times of selecting the leaf nodes reach the specified number;
the first triggering unit is used for triggering the display object determining module to execute the selection of the first N classification nodes and the leaf nodes as the display objects to be recommended according to the descending order of the feature similarity if the judgment result of the iteration judging unit reaches the specified number;
the leaf node judging unit is used for judging whether leaf nodes which do not have the characteristic similarity with the object exist in the selected classification nodes or not if the judgment result of the iteration judging unit is that the specified number is not reached;
the third triggering unit is used for triggering the leaf node selecting unit to select a leaf node from the selected classification nodes if the judgment result of the leaf node judging unit is positive, calculating the feature similarity between the leaf node and the object, and increasing the times of selecting the leaf node by a specified value;
and the third triggering unit is used for deleting the selected classification node from the classification set if the judgment result of the leaf node judging unit is no, and triggering the classification node selecting unit to select the classification node with the maximum characteristic similarity with the object from the classification set.
Further, the leaf node selecting unit is specifically configured to select, from designated leaf nodes of the selected classification nodes, a leaf node with the largest feature similarity to the classification node, where the designated leaf node is a leaf node for which the feature similarity to the object is not calculated.
Further, the apparatus further comprises:
the classification center determining module is used for calculating the clustering center of the classification node according to the characteristics of the classification node;
the root node determining module is used for selecting the classification node closest to the clustering center distance of the classification nodes from the classification nodes as the root node in the search tree; taking classification nodes except the root node as child nodes of the root node, and taking the child nodes as classification nodes in the search tree; wherein, the root node, the classification node and the leaf node all represent display objects;
the device further comprises:
and the root node similarity determining module is used for calculating the feature similarity between the features of the object and the root node after the features of the object contained in the image to be processed are extracted by the feature extracting module.
Example four
The fourth embodiment of the present application further provides a computing device, which may specifically be a smart phone, a tablet computer, a Personal Digital Assistant (PDA), or the like. As shown in fig. 7, the computing device may include a Central Processing Unit (CPU) 701, a memory 702, an input device 703, an output device 704, etc., the input device may include a keyboard, a mouse, a touch screen, etc., and the output device may include a Display device such as a Liquid Crystal Display (LCD), a Cathode Ray Tube (CRT), etc.
The memory may include Read Only Memory (ROM) and Random Access Memory (RAM), and provides the processor with program instructions and data stored in the memory. In an embodiment of the present application, the memory may be configured to store program instructions of a method for determining a presentation object based on image content. The processor is used for executing the following steps according to the obtained program instructions by calling the program instructions stored in the memory: extracting the characteristics of at least one object contained in the image to be processed;
for each object, determining the feature similarity of the feature of the object and the feature similarity of classification nodes in a pre-constructed search tree; the search tree is used for representing a set of display objects, each classification node is used for representing one class of objects, each classification node comprises at least one leaf node, and the classification nodes and the leaf nodes in the search tree are used for representing the display objects;
selecting classification nodes according to the sequence of the feature similarity from high to low of the object, and calculating the feature similarity between leaf nodes contained in the selected classification nodes and the object until the number of the calculated leaf nodes reaches the specified number;
and selecting the first N classification nodes and leaf nodes as display objects to be recommended according to the sequence of the feature similarity from large to small, wherein N is a positive integer and is less than the designated number.
EXAMPLE five
A fifth embodiment of the present application provides a computer-readable storage medium, which is used for storing computer program instructions for a processor, and which includes a program for executing the method for determining a display object based on image content.
The computer-readable storage medium may be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic memory (e.g., floppy disks, hard disks, magnetic tape, magneto-optical disks (MOs), etc.), optical memory (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor memory (e.g., ROMs, EPROMs, EEPROMs, non-volatile memories (NANDFLASHs), Solid State Disks (SSDs)), etc.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims (10)

1. A display object determination method based on image content is characterized by comprising the following steps:
extracting the characteristics of at least one object contained in the image to be processed;
for each object, determining the feature similarity of the feature of the object and the feature similarity of classification nodes in a pre-constructed search tree; the search tree is used for representing a set of display objects, each classification node is used for representing one class of objects, each classification node comprises at least one leaf node, and the classification nodes and the leaf nodes in the search tree are used for representing the display objects;
selecting classification nodes according to the sequence of the feature similarity of the object from high to low, and calculating the feature similarity of leaf nodes contained in the selected classification nodes and the object until the number of the calculated feature similarities reaches a specified number;
selecting the first N classification nodes and leaf nodes as display objects to be recommended according to the sequence of the feature similarity from large to small, wherein N is a positive integer and is smaller than the designated number;
wherein the method further comprises:
the search tree is constructed according to the following method:
classifying the display objects in the display object set based on a cluster analysis method, wherein the display objects with the characteristic similarity greater than the specified similarity among the display objects are classified into one class;
for each type of display object, determining the display object closest to the clustering center in the type of display object as a classification node for representing the type of display object; and the number of the first and second electrodes,
and taking the display objects except the classification nodes in the display objects of the type as leaf nodes of the classification nodes.
2. The method according to claim 1, wherein the classification nodes are selected according to the sequence of the feature similarity with the object from high to low, and the feature similarity between the leaf nodes included in the selected classification nodes and the object is calculated until the number of the calculated leaf nodes reaches a specified number, specifically comprising:
taking the classification nodes as a classification set;
selecting a classification node with the maximum characteristic similarity with the object from the classification set;
selecting a leaf node from the selected classification nodes, calculating the feature similarity between the leaf node and the object, and increasing the times of selecting the leaf node by a specified value;
judging whether the number of times of selecting the leaf nodes reaches the specified number;
if the number of the nodes reaches the specified number, performing operation of selecting the nodes with the preset number as the display objects to be recommended according to the descending order of the feature similarity from the nodes which are used for representing the display objects and have the calculated feature similarity;
if the number does not reach the specified number, judging whether leaf nodes which do not have the feature similarity calculated with the object exist in the selected classification nodes or not;
if yes, returning to execute the operation of selecting a leaf node from the selected classification nodes, calculating the feature similarity of the leaf node and the object, and increasing the times of selecting the leaf node by a specified value;
and if not, deleting the selected classification node from the classification set, and returning to the step of selecting the classification node with the maximum characteristic similarity with the object from the classification set.
3. The method of claim 2, wherein selecting a leaf node from the selected classification nodes comprises:
and selecting the leaf node with the maximum feature similarity with the classification node from the designated leaf nodes of the selected classification nodes, wherein the designated leaf node is the leaf node which does not calculate the feature similarity with the object.
4. The method according to any one of claims 1-3, further comprising:
calculating the clustering center of the classification node according to the characteristics of the classification node;
selecting a classification node closest to the clustering center distance of the classification nodes from the classification nodes as a root node in a search tree; taking classification nodes except the root node as child nodes of the root node, and taking the child nodes as classification nodes in the search tree; wherein, the root node, the classification node and the leaf node all represent display objects;
after extracting the features of the object contained in the image to be processed, the method further comprises:
and calculating the feature similarity of the features of the object and the root node.
5. An apparatus for determining a display object based on image content, the apparatus comprising:
the characteristic extraction module is used for extracting the characteristics of at least one object contained in the image to be processed;
the classification node comparison module is used for determining the feature similarity of the object and the classification nodes in the pre-constructed search tree aiming at each object; the search tree is used for representing a set of display objects, each classification node is used for representing one class of objects, each classification node comprises at least one leaf node, and the classification nodes and the leaf nodes in the search tree are used for representing the display objects;
the similarity determining module is used for selecting classification nodes according to the sequence of the feature similarity of the object from high to low, calculating the feature similarity of leaf nodes contained in the selected classification nodes and the object until the number of the calculated feature similarities reaches the specified number;
the display object determining module is used for selecting the first N classification nodes and leaf nodes as display objects to be recommended according to the sequence of the feature similarity from large to small, wherein N is a positive integer and is smaller than the designated number;
wherein the apparatus further comprises:
the search tree construction module is used for constructing a search tree according to the following method:
classifying the display objects in the display object set based on a cluster analysis method, wherein the display objects with the characteristic similarity greater than the specified similarity among the display objects are classified into one class;
for each type of display object, determining the display object closest to the clustering center in the type of display object as a classification node for representing the type of display object; and the number of the first and second electrodes,
and taking the display objects except the classification nodes in the display objects of the type as leaf nodes of the classification nodes.
6. The apparatus according to claim 5, wherein the similarity determining module specifically includes:
a classification set determining unit, configured to use the classification node as a classification set;
the classification node selection unit is used for selecting a classification node with the maximum characteristic similarity with the object from the classification set;
the leaf node selecting unit is used for selecting a leaf node from the selected classification nodes, calculating the feature similarity between the leaf node and the object, and increasing the times of selecting the leaf node by a specified value;
the iteration judging unit is used for judging whether the times of selecting the leaf nodes reach the specified number;
the first triggering unit is used for triggering the display object determining module to execute the selection of the first N classification nodes and the leaf nodes as the display objects to be recommended according to the descending order of the feature similarity if the judgment result of the iteration judging unit reaches the specified number;
the leaf node judging unit is used for judging whether leaf nodes which do not have the characteristic similarity with the object exist in the selected classification nodes or not if the judgment result of the iteration judging unit is that the specified number is not reached;
the third triggering unit is used for triggering the leaf node selecting unit to select a leaf node from the selected classification nodes if the judgment result of the leaf node judging unit is positive, calculating the feature similarity between the leaf node and the object, and increasing the times of selecting the leaf node by a specified value;
and the third triggering unit is used for deleting the selected classification node from the classification set if the judgment result of the leaf node judging unit is no, and triggering the classification node selecting unit to select the classification node with the maximum characteristic similarity with the object from the classification set.
7. The apparatus according to claim 6, wherein the leaf node selecting unit is specifically configured to select, from designated leaf nodes of the selected classification nodes, a leaf node having a largest feature similarity with the classification node, where the designated leaf node is a leaf node for which the feature similarity with the object is not calculated.
8. The apparatus of any of claims 5-7, further comprising:
the classification center determining module is used for calculating the clustering center of the classification node according to the characteristics of the classification node;
the root node determining module is used for selecting the classification node closest to the clustering center distance of the classification nodes from the classification nodes as the root node in the search tree; taking classification nodes except the root node as child nodes of the root node, and taking the child nodes as classification nodes in the search tree; wherein, the root node, the classification node and the leaf node all represent display objects;
the device further comprises:
and the root node similarity determining module is used for calculating the feature similarity between the features of the object and the root node after the features of the object contained in the image to be processed are extracted by the feature extracting module.
9. A computing device comprising a memory for storing program instructions and a processor for calling the program instructions stored in the memory and executing the method for determining a display object based on image content according to any one of claims 1 to 4 according to the obtained program instructions.
10. A computer-readable storage medium storing computer-executable instructions, and when loaded by a processor, performing the method for determining the display object based on the image content according to any one of claims 1 to 4.
CN201710685418.4A 2017-08-11 2017-08-11 Display object determination method, device, medium and equipment based on image content Active CN108288208B (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN201710685418.4A CN108288208B (en) 2017-08-11 2017-08-11 Display object determination method, device, medium and equipment based on image content
PCT/CN2018/100009 WO2019029714A1 (en) 2017-08-11 2018-08-10 Image content-based display object determination method, device, medium, and apparatus
US16/537,885 US11200444B2 (en) 2017-08-11 2019-08-12 Presentation object determining method and apparatus based on image content, medium, and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710685418.4A CN108288208B (en) 2017-08-11 2017-08-11 Display object determination method, device, medium and equipment based on image content

Publications (2)

Publication Number Publication Date
CN108288208A CN108288208A (en) 2018-07-17
CN108288208B true CN108288208B (en) 2020-08-28

Family

ID=62831602

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710685418.4A Active CN108288208B (en) 2017-08-11 2017-08-11 Display object determination method, device, medium and equipment based on image content

Country Status (3)

Country Link
US (1) US11200444B2 (en)
CN (1) CN108288208B (en)
WO (1) WO2019029714A1 (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108288208B (en) 2017-08-11 2020-08-28 腾讯科技（深圳）有限公司 Display object determination method, device, medium and equipment based on image content
CN109002537B (en) * 2018-07-19 2021-05-14 南昌工程学院 Information processing system and method based on deep learning
CN109190680A (en) * 2018-08-11 2019-01-11 复旦大学 The detection and classification method of Medicines image based on deep learning
CN111126102A (en) * 2018-10-30 2020-05-08 富士通株式会社 Personnel searching method and device and image processing equipment
CN109840129A (en) * 2019-01-30 2019-06-04 维沃移动通信有限公司 A kind of display control method and electronic equipment
CN110458292A (en) * 2019-07-04 2019-11-15 西安工程大学 A kind of clothes recommended method based on expertise
CN113449548A (en) * 2020-03-24 2021-09-28 华为技术有限公司 Method and apparatus for updating object recognition model
CN117156200B (en) * 2023-06-06 2024-08-02 青岛尘元科技信息有限公司 Method, system, electronic equipment and medium for removing duplication of massive videos

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102057371A (en) * 2008-06-06 2011-05-11 汤姆逊许可证公司 System and method for similarity search of images
CN104239566A (en) * 2014-09-28 2014-12-24 小米科技有限责任公司 Method and device for searching videos
CN105468781A (en) * 2015-12-21 2016-04-06 小米科技有限责任公司 Video query method and device

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8812333B2 (en) * 2007-09-17 2014-08-19 Amadeus S.A.S. Automated split ticketing
CN102402508A (en) * 2010-09-07 2012-04-04 华东师范大学 Similar image search device and search method thereof
JP5596628B2 (en) * 2011-06-17 2014-09-24 トヨタ自動車株式会社 Object identification device
US9424648B2 (en) * 2011-07-07 2016-08-23 Siemens Aktiengesellschaft Method and system for device detection in 2D medical images
CN103324949B (en) * 2012-03-21 2016-09-28 阿里巴巴集团控股有限公司 The method and apparatus identifying object from image
CN103559504B (en) 2013-11-04 2016-08-31 北京京东尚科信息技术有限公司 Image target category identification method and device
CN106354735A (en) 2015-07-22 2017-01-25 杭州海康威视数字技术股份有限公司 Image target searching method and device
US10540683B2 (en) * 2017-04-24 2020-01-21 Microsoft Technology Licensing, Llc Machine-learned recommender system for performance optimization of network-transferred electronic content items
CN108288208B (en) * 2017-08-11 2020-08-28 腾讯科技（深圳）有限公司 Display object determination method, device, medium and equipment based on image content

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102057371A (en) * 2008-06-06 2011-05-11 汤姆逊许可证公司 System and method for similarity search of images
CN104239566A (en) * 2014-09-28 2014-12-24 小米科技有限责任公司 Method and device for searching videos
CN105468781A (en) * 2015-12-21 2016-04-06 小米科技有限责任公司 Video query method and device

Also Published As

Publication number Publication date
WO2019029714A1 (en) 2019-02-14
US11200444B2 (en) 2021-12-14
CN108288208A (en) 2018-07-17
US20190385012A1 (en) 2019-12-19

Similar Documents

Publication Publication Date Title
CN108288208B (en) Display object determination method, device, medium and equipment based on image content
US9460117B2 (en) Image searching
CN110909182B (en) Multimedia resource searching method, device, computer equipment and storage medium
US10482146B2 (en) Systems and methods for automatic customization of content filtering
CN112052387B (en) Content recommendation method, device and computer readable storage medium
WO2019120023A1 (en) Gender prediction method and apparatus, storage medium and electronic device
CN109255000B (en) Dimension management method and device for label data
CN113987119B (en) Data retrieval method, and cross-modal data matching model processing method and device
CN108268510B (en) Image annotation method and device
CN113657087B (en) Information matching method and device
CN109471978A (en) A kind of e-sourcing recommended method and device
US20230004595A1 (en) Image retrieval method, image retrieval devices, image retrieval system and image display system
JP7499946B2 (en) Method and device for training sorting model for intelligent recommendation, method and device for intelligent recommendation, electronic device, storage medium, and computer program
US20170286522A1 (en) Data file grouping analysis
CN106649380A (en) Hot spot recommendation method and system based on tag
CN113435523B (en) Method, device, electronic equipment and storage medium for predicting content click rate
WO2023168997A9 (en) Cross-modal retrieval method and related device
US20230180917A1 (en) Method for processing makeup and electronic device
CN110442614A (en) Searching method and device, electronic equipment, the storage medium of metadata
CN115878874A (en) Multimodal retrieval method, device and storage medium
CN111797765B (en) Image processing method, device, server and storage medium
CN115168609A (en) Text matching method and device, computer equipment and storage medium
CN114461822A (en) Resource processing method, device, equipment and storage medium
CN103164504A (en) Smartphone refined picture searching system and method
CN113515701A (en) Information recommendation method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant