KR20200073967A - Method and apparatus for determining target object in image based on interactive input - Google Patents

Abstract

Disclosed are a method for determining a target object in an image based on interactive input and an apparatus thereof. The method for determining a target object obtains first feature information corresponding to an image and second feature information corresponding to interactive input. The target object corresponding to the interactive input is determined from objects of the image in accordance with the first feature information and the second feature information.

Description

Method and apparatus for determining a target object from an image based on interactive input {METHOD AND APPARATUS FOR DETERMINING TARGET OBJECT IN IMAGE BASED ON INTERACTIVE INPUT}

The following embodiments relate to the field of human-machine interactive technology, and specifically, to a method and apparatus for determining a target object in an image based on interactive input.

Determining target objects based on interactive inputs is an important area of research on human-machine interactive technology. According to one side, in the field of computer vision, object detection technology is widely applied.

Target object detection is a type of technology that detects a specific type of object from an image or image (eg, one frame of an image) in the computer vision field. Specifically, for the input image, a bounding box of each object included in the image may be presented, and a corresponding object type tag may be presented.

According to the other side, in man-machine interaction, the computer can understand the user's interactive input according to human interaction habits. For example, for a voice input by the user, the computer first uses a voice identification technology to convert the voice command issued by the user into text, thereby making it convenient for the computer to understand the user's command. Then, the computer can extract nouns of user commands through natural language processing-related techniques such as grammar analysis. Combining the two with human-machine interactive technology that determines the target object according to the interactive input, so that at a certain level, the computer understands which object the user refers to (determines what the target object is), so that a given image or video (video In one frame), a determination is made on an object that the user refers to.

1A to 1C are diagrams illustrating an example of determining a target object in an image according to the prior art.

1A to 1C, for example, FIG. 1A shows an example in which there is only one "airplane" in an image. If the user says "airplane", the computer understands the object the user refers to, and can present a bounding box 110 corresponding to the object as shown in FIG. 1B.

Such man-machine technology may face difficulties when there are multiple instances of the same type as the object the user refers to in the scene, and simply uses the target detection technology to distinguish the object that the user specifically refers to It is impossible. For example, if a user says "Motorcycler", and there are many people in the scene, the object detection technology cannot determine which case (person) the user specifically refers to, resulting in accurate results. Cannot be imported.

For this type of problem, one solution of the conventional technique is to simultaneously show a plurality of cases (111, 112, 113) detected as shown in FIG. 1C below, and give a sequence number, allowing the user to select a specific sequence number again, It is a way to realize the decision of the target object. However, this kind of method requires a separate confirmation selection step. Thereby, interactive efficiency can be reduced. In addition, when a relatively large number of cases exist in the scene (for example, a group photo taken by many people), the tags are too dense, which is disadvantageous for the user to select.

Another type of solution of the existing technology treats this type of problem as one kind of fine-grained object detection, and when training the detection model, separates object attribute information with a separate tag (eg For example, it is considered to be treated as a short man, a person with glasses, a red car, etc.). The drawback of this approach is that a large amount of separate annotations are required when training the model. In addition, the accuracy of object detection for types that have never appeared in the training set in actual use is greatly reduced.

A method of determining a target object from an image based on the interactive input according to an embodiment includes: obtaining first characteristic information corresponding to the image and second characteristic information corresponding to the interactive input; And determining a target object corresponding to the interactive input among objects included in the image according to the first characteristic information and the second characteristic information.

In this case, acquiring the first characteristic information corresponding to the image and the second characteristic information corresponding to the interactive input include: when acquiring the first characteristic information corresponding to the image, included in the image And obtaining semantic characteristic information between each object and at least one other object included in the image.

At this time, obtaining the semantic characteristic information between each object included in the image and at least one other object included in the image, based on the location information of each object included in the image, included in the image And obtaining the semantic feature information between each object and at least one other object.

In this case, obtaining the semantic characteristic information between each object included in the image and at least one other object included in the image may include at least one of the objects included in the image and at least one other object. Determining a candidate region; Obtaining classification feature information of an object in the candidate region; Obtaining region semantic feature information between objects in the candidate region; And generating the semantic feature information between each object included in the image and at least one other object based on the classification feature information and the area semantic feature information.

At this time, before generating the semantic feature information between each object included in the image and at least one other object, based on the classification feature information and the area semantic feature information, the classification feature information and the area semantic feature The method may further include performing joint correction on the information.

At this time, before generating the semantic feature information between each object included in the image and at least one other object, determining a reference area according to the candidate area; Obtaining region feature information of the reference region; And performing joint modification on the classification feature information, the region semantic feature information, and the region feature information based on the classification feature information, the region semantic feature information, and the region feature information.

In this case, the candidate region may include one of the objects included in the image and one of at least one other object included in the image.

In this case, the first feature information includes global visual feature information corresponding to the image, visual feature information corresponding to each object included in the image, relative location information between objects included in the image, and included in the image It may include at least one of the relative size feature information between the objects and the semantic feature information between the objects included in the image.

At this time, determining the target object corresponding to the interactive input among the objects included in the image, before determining the target object, further comprising the step of performing a fusion processing (fusion processing) on the first feature information can do.

In this case, a method of determining a target object from an image based on the interactive input includes: obtaining training data including a sample image; Determining at least one candidate region based on each object included in the sample image and at least one other object included in the sample image; Determining a reference area according to the candidate area, and obtaining area feature information of the reference area; Generating an area title according to the area feature information; And performing training on a neural network model for obtaining semantic feature information between objects included in the image, using the supervised training data with the area title.

At this time, obtaining the first feature information corresponding to the image and the second feature information corresponding to the interactive input may include: performing a word vector transformation on the interactive input; And obtaining the second feature information corresponding to the interactive input based on the word vector.

In this case, acquiring the first characteristic information corresponding to the image and the second characteristic information corresponding to the interactive input may be performed before the step of performing the word vector transformation on the interactive input. The method further includes determining whether the word belongs to the set first word, and performing the word vector conversion on the interactive input includes: when the word of the interactive input belongs to the set first word; The method may include using a word vector of the second word having high similarity to the word vector of the first word as a word vector corresponding to the first word.

In this case, the first word may indicate a word whose frequency of use is lower than the first set value, and the second word may indicate a word whose frequency of use is higher than the second set value.

In this case, the interactive input may include at least one of voice input and text input.

An apparatus for determining a target object from an image based on an interactive input according to an embodiment includes: a feature acquiring unit that acquires first feature information corresponding to an image and second feature information corresponding to an interactive input; And a target determination unit that determines a target object corresponding to the interactive input among objects included in the image, according to the first characteristic information and the second characteristic information.

In this case, the first feature information includes global visual feature information corresponding to the image, visual feature information corresponding to each object included in the image, relative location information between objects included in the image, and included in the image It may include at least one of the relative size feature information between the objects and the semantic feature information between the objects included in the image.

At this time, the feature acquiring unit determines at least one candidate region based on each object included in the image and at least one other object, obtains classification feature information of an object in the candidate region, and objects in the candidate region Obtaining area semantic characteristic information between them, determining a reference area according to the candidate area, obtaining area characteristic information of the reference area, and based on the classification characteristic information, the area semantic characteristic information, and the area characteristic information , Performing a joint change on the classification feature information, the region semantic feature information, and the region feature information, and based on the corrected classification feature information, the corrected region semantic feature information, and the corrected region feature information The semantic feature information may be generated between each object included in at least one other object.

In this case, the feature acquiring unit may perform a word vector transformation on the interactive input, and obtain the second feature information corresponding to the interactive input based on the word vector.

In this case, when the word vector conversion is performed on the interactive input, the feature obtaining unit determines whether the word of the interactive input belongs to the set first word, and the first word in which the word of the interactive input is set. If it belongs to, the word vector of the second word having high similarity to the word vector of the first word is used as a word vector corresponding to the first word, and the first word is a word whose frequency of use is lower than the first set value And the second word may indicate a word whose frequency of use is higher than the second set value.

According to a technical method according to an embodiment, by obtaining first feature information including semantic feature information between objects, and matching the second feature information corresponding to the first feature information and the interactive input, the image based on the interactive input To improve the ability to understand the feature representation of the object expressed by the user of the human-machine interactive system by determining the target object, and to make the human-machine interactive system more accurately and faster to determine the target object. At the same time, by mapping words that are rarely seen (less frequently used) to words that are frequently seen (more frequently used) with the same meaning, the human-machine interactive system improves the ability to adapt to the less frequently used words. And this helps to determine the target object more accurately and faster.

1A to 1C are diagrams illustrating an example of determining a target object in an image according to the prior art.
2 is a flowchart illustrating a method of determining a target object in an image based on interactive input according to an embodiment;
3 is a diagram exemplarily showing a process of determining a target object from an image based on interactive input according to an embodiment.
4 is a diagram illustrating an exemplary process for obtaining visual feature information of an object according to an embodiment.
5 is a diagram illustrating an exemplary process for obtaining semantic feature information between objects according to an embodiment.
6 is a diagram illustrating another example of a method of determining a target object in an image based on interactive input according to an embodiment.
7 is a diagram showing a distribution diagram showing the frequency of use of words.
8 is a diagram illustrating an example of applying a method of determining a target object in an image based on interactive input according to an embodiment.
9 is a diagram illustrating a configuration of an apparatus for determining a target object in an image based on interactive input according to an embodiment.
10 is a diagram exemplarily illustrating an apparatus for determining a target object from an image based on interactive input according to an embodiment.
In the accompanying drawings, the same or similar structure may be indicated by the same or similar attached drawing notation.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

A method of determining a target object in an image based on interactive input according to an embodiment mainly includes a step of obtaining feature information and a step of determining the target object. Specifically, the first characteristic information corresponding to the image and the second characteristic information corresponding to the interactive input are obtained, and according to the first characteristic information and the second characteristic information, a target object of the image is determined for the interactive input in the object of the image do.

In a specific embodiment, the first characteristic information is visual characteristic information (also referred to as global visual characteristic information) of the entire image corresponding to the image and visual characteristic information (also referred to as visual characteristic information of a single object) corresponding to each object of the image. ), at least one of relative position information and/or relative size feature information between each object in the image and at least one other object proximate to the object, and semantic feature information between objects in the image.

At this time, the definition close to any object is based on the location information of each object in the image. For example, when a distance between an object and one other object is smaller than a preset distance, it may be defined that the object and one other object are close.

In a specific embodiment, the step of obtaining the first feature information corresponding to the image includes: first, visual feature information of a complete image, visual feature information of a single object, between each object in the image and at least one other object proximate to the object And obtaining semantic feature information between the object of the image and the relative location information and the relative size feature information, respectively. Then, fusion processing is performed on each of the pieces of information to obtain first feature information corresponding to the image.

When the first feature information includes semantic feature information between objects in the image, FIG. 2, which is a flowchart according to a method of determining a target object in the image based on the interactive input of the embodiment, may be followed.

2 is a flowchart illustrating a method of determining a target object in an image based on interactive input according to an embodiment;

Referring to FIG. 2, in step 210, first feature information corresponding to an image and second feature information corresponding to an interactive input are acquired. Here, the first feature information may include semantic feature information between objects in the image.

Then, in step 220, the target object of the image is determined for the interactive input from the object of the image according to the first characteristic information and the second characteristic information.

In a specific embodiment, when the first feature information includes semantic feature information between objects in the image, and when obtaining the first feature information corresponding to the image in step 210, each object in the image and at least one object close to the object And obtaining semantic feature information between different objects.

In a specific embodiment, the step of acquiring semantic characteristic information between each object in the image included in the first characteristic information corresponding to the image and at least one other object proximate to the object is first performed on each object and object in the image. At least one candidate region is determined based on at least one other object in proximity. Then, the semantic feature information of the object in the candidate region and the classification feature information of the object in the candidate region are respectively obtained. Finally, semantic feature information between each object in the image and at least one other object proximate to the object may be generated based on the classification feature information and the area semantic feature information.

In another specific embodiment, the semantic feature information between each object in the image and at least one other object is used to make the semantic feature information between each object in the acquired image and at least one other object proximate to the object more accurate. Before generation, joint correction may be performed on the classification feature information and the region semantic feature information based on the acquired classification feature information and the region language feature information. Here, the joint change refers to correcting the corresponding characteristic information using other related types of characteristic information.

Additionally, in another specific embodiment, before acquiring semantic feature information between each object in the image and at least one other object, a reference region including the candidate region is determined according to the candidate region, and region feature information of the reference region It is possible to obtain, and based on the classification characteristic information, the area semantic characteristic information and the area characteristic information, joint change can be performed on the classification characteristic information, the area semantic characteristic information and the area characteristic information.

In addition, when training on the semantic feature information and the neural network model between each object included in the image and at least one other object, an area title is acquired using a reference area, and the area title is used to obtain a model. You can also perform supervised training on This is beneficial for improving the quality of the model. Below, it will be described in more detail through specific examples.

In one embodiment, the basic network visual feature information, a single object visual feature information, and the relative location information and relative size feature information between each object in the image and other objects proximate to each object, using the basic network To extract. Semantic feature information can be extracted using a pair-wise visual relationship network (VRN). This network can be obtained through a special structure and training using a convolutional neural network of a basic network (for example, VGG-Net). The VRN is only used to perform differentiation from other basic networks, and does not limit the embodiment.

According to a method of determining a target object from an image according to an embodiment, any specific case (eg, person, animal, object, etc.) described in interactive input (eg, natural language input or language input) in an image or scene The same object) can be determined. In addition, in the method of determining the target object in the image according to an embodiment, even if there are a plurality of cases belonging to the same type in the image or scene, the specific case referred to through semantic characteristics such as properties, location, and motion related to interactive input may be determined. To improve the interactive capabilities of the human-machine interactive system. Hereinafter, a specific example of determining a target object in an image based on the interactive input will be described using an example of a human-machine interactive system of natural language input or voice input.

3 is a diagram exemplarily showing a process of determining a target object from an image based on interactive input according to an embodiment.

Referring to FIG. 3, first, object detection 312 was performed on an object of the input image 310 (or any one frame of a video). Object detection 312 can be used to detect all objects in an image, and provides a bounding box for each object (including object location and size information).

In this case, object detection may be performed using a Faster R-CNN network. However, this function can be realized using other networks, and the embodiment is not limited to using the Faster R-CNN network. At this time, the Faster R-CNN network is a prior art and represents a network that detects objects in images and classifies them.

The visual feature extraction 322 is performed on a region of the object by using a basic network for each detected object included in the image 320 in which the object is detected. In addition, the same visual feature extraction 322 is performed for other adjacent object regions and the entire image. That is, through the visual feature extraction, the image 330 from which the visual feature has been extracted includes visual feature information 331 of a single object (each object), visual feature information 332 of a complete image, and relative location information between objects of the image ( 333) and/or size feature information 334 between objects of the image, respectively.

In this case, the visual feature information of the complete image may be, for example, the image 310 input as the overall visual feature of the image, such as a park or daytime, as the visual feature information of the complete image.

And, the visual characteristic information of a single object is a visual characteristic of each object included in the image. For example, the input image 310 may be a bicycle, a person, a skateboard, a hat, a box, etc. Color and location may also be included in the visual feature information.

In addition, the semantic feature information indicates a meaningful relationship between an object and another object in close proximity. For example, in the input image 310, a “riding state” corresponding to a relationship between a bicycle and a person's object in a person riding a bicycle It can be semantic feature information. That is, the semantic feature information indicates how the object is connected to the object.

In the above embodiment, visual feature information of the object region and the entire image is extracted using the last layers of the convolutional layers of the third and fourth groups using Faster R-CNN. In the same way, feature extraction can be performed using other layers of different networks (eg, VGG-16, ResNet-101, etc.). However, it is not limited to the above embodiment.

4 is a diagram illustrating an exemplary process for obtaining visual feature information of an object according to an embodiment.

Referring to FIG. 4, the input image 410 as listed above acquires the bounding box of each object of the image 420 where the object is detected through the object detection 412. The current bounding box (corresponding to the area of the current object), the adjacent bounding box (corresponding to other adjacent object areas), and then using any feature layer of the basic network (e.g., VGG-16, ResNet-101, etc.) and The visual feature information 431, 432, 433 is extracted 430 using the entire image. At the same time, the relative location information and the relative size feature information of the bounding box adjacent to the current bounding box are treated as separate features and connected to the visual feature information. It is possible to obtain visual feature information corresponding to each object (each bounding frame) of the image by sequentially executing the extraction of the visual feature information for each object in the image.

In this case, the normalized values (x/W,y/H) of the upper left coordinates of the bounding box (and/or the bounding box of a single object) of the current and proximity objects and the normalized values of width and height (w/W, h/H) and area normalization values (w*h/W*H) can also be treated as part of the visual feature information. Specifically, the width (w) and height (h) of the bounding box for the bounding box (and/or the bounding box of a single object) of the current object and the proximity object are equally equal to the width (W) and height (H) of the entire image. Divide. Again, the area of the entire image is divided by the area of the bounding box to obtain a 5D feature vector.

The position information and size information for an object configured in this kind of method and the relative position and relative size information of the object and other objects adjacent thereto include left/right/upper/lower sides in the description of the currently entered word, or When including the largest/smallest/highest/lowest etc description, the target object described by the language can be determined.

Additionally, at least a pair of objects are created by selecting at least one object close to it from around the current object. Specifically, with respect to the detected object, an object close to it is selected, and a pair of objects are sequentially formed. Here, the implications of'proximity' are as listed above. When composing a pair of objects, typically, a pair of objects are created by selecting some of the closest objects according to the positional relationship between the objects. There are a series of interactive relationships between adjacent objects, such as transport, placement, viewing, wearing, riding, and leaning. In addition, the selected adjacent objects may not exceed a preset number (eg, 5).

With respect to the created pair of objects, the network VRN is identified through a visual relationship to extract semantic feature information of the corresponding pair of objects (that is, between objects).

5 is a diagram illustrating an exemplary process for obtaining semantic feature information between objects according to an embodiment.

Referring to FIG. 5, in the corresponding case, the VGG-Net convolutional network may be used as the basic network 520 of the VRN. That is, feature extraction is performed on the image 510 input using VGG-Net (eg, VGG-16) to obtain a shared feature 530 of the image. Subsequently, a pair of objects is detected from the object detection result, and a candidate region region proposal is generated based on the selected pair of objects. That is, each candidate region includes a pair of objects. That is, it includes one of the (current) objects and at least one of the (current) objects and one of the other objects. Thereby, a combination relationship of objects is obtained. Then, three different computer visual tasks are processed through three parallel branches 541, 542, and 543. Specifically, on the basis of the obtained shared features, two object bounding boxes of the candidate region and a bounding box corresponding to the candidate region are divided into three branches 541, 542, and 543 (branch) to perform feature extraction.

(1) Selecting an area bounding box (corresponding to a reference area) whose range is larger than the bounding box corresponding to the candidate area, and extracting area feature information for the bounding box (the purpose is to perform area heading generation later)

(2) For each bounding box of each of the two objects, extraction is performed to classify feature information (objects are then classified as objects).

(3) Object region semantic feature information extraction is performed on the bounding box corresponding to the candidate region (the purpose is to subsequently perform semantic feature identification between objects, specifically, relationships between objects such as, for example, motion relations)

As shown in FIG. 5, before generating semantic feature information between each object and other objects proximate to each object based on the classification feature information, the region feature information, and the region feature information, the classification feature information, the region semantic feature information, and The method further includes constructing a dynamic graph based on the region feature information, and performing correction (which may be referred to as performing a joint change) according to the classification feature information, the region semantic feature information, and the region feature information according to the dynamic graph.

The dynamic graph connects different inspiration areas of different branches (which can be understood as different bounding boxes) into one through semantic and spatial conventions, and changes the content according to the modification process. By jointly changing different branch features through passing messages between different branches, the features between each branch are correlated with each other to obtain semantic feature information more accurately.

At this time, the region feature information corresponds to the feature information of the reference region including the candidate region, and can modify the classification feature information and the region semantic feature information using the region feature information, which takes into account the neural network model output accuracy. Can. After completing the modification, object classification is performed using the modified feature classification, semantic feature information between objects is identified, and an area title 570 is generated.

Specifically, an area title 570 may be generated based on the feature information of the branch 1 561 after modification through a long short-term memory (LSTM) included in the basic network 520. In other words, in response to the area of the image, "one person wears a hat and flies a kite in the park."

At this time, the image area that is easy to understand may not generate the area title. A scene graph 580 is generated according to the characteristics of the modified branch 2 562 and the modified branch 3 563. The scene graph 580 can be understood as one matrix, and The size is N*N, and N is the number of objects in the detected image, each row and each column of the corresponding matrix respectively corresponds to one object, and semantic feature information between each element and object of the matrix is To respond.

As shown in FIG. 5, each term in the scene matrix corresponds to one object, such that the first row of the scene graph 580 may correspond to the object "person". In the scene graph 580, each column of the matrix also corresponds to one object. For example, the first column of the scene diagram may correspond to the "hat" of the object, the second column of it may correspond to the "yeon" of the object, and the third column to the "park" of the object. Can.

The position at which the row and column of the scene graph 580 intersect (that is, the position of each element in the matrix) corresponds to semantic feature information between objects. For example, the position where the first row and the first column intersect correspond to the semantic feature information between the object "person" and the object "hat", and the element is "wear", the object "person" and the object "hat" Expresses the semantic characteristic information of "living "a person wears a hat". For the same reason, the position at which the first row and the second column intersect is "fly", and expresses the semantic feature information of "person flies a kite" between object "person" and object "kite". The element at the position where the first row and the third column intersect is “inside”, which represents the semantic characteristic information of “the person is in the park” between the object “person” and the object “park”. That is, the generated scene image can clearly express semantic feature information between objects of the image.

In addition, in the online test process of the VRN, before the scene graph 580 is generated according to the characteristics of the modified branch 2 562 and the modified branch 3 563, that is, the VRN generates a scene graph 580. Output of the last one fully connected layer before (i.e., before generating semantic feature information between each object in the image and at least one other object based on the classification feature information, region semantic feature information, and region feature information) Extract the result and use it to describe the semantic feature information between two objects. Similarly, according to demand and test results, output results of different layers of a corresponding network may be used to express object semantic feature information.

In another embodiment, it is possible to provide a VRN training method for acquiring an area title using a reference area title and performing supervised training on a model using the area title. In this training method, first, training data of an image including a sample is acquired, and at least one candidate region is determined based on each object and at least one other object of the sample image. Then, a reference area is determined according to the candidate area, and area characteristic information of the reference area is obtained. Again, an area title is generated according to the area feature information. When performing training on the neural network model, in addition to performing supervised training on the classification feature information of the branch 1 and branch 3 and the region semantic feature information, the training data of the director is used as the director's training data. Perform training on. In the inverse distribution process, it is possible to obtain better object classification and relationship identification networks by helping to update network weights of branches 1 and 2. In this way, better classification and feature information can be extracted in the test stage.

Continuing with reference to FIG. 3, a method of determining the target object of FIG. 3 includes visual characteristic information 331 of a single object, visual characteristic information 332 of a complete image, relative location information 333 between objects, and an object After obtaining one or a plurality of the relative size feature information 334 between the objects and the semantic feature information 335 between the objects in sequence, the obtained information is subjected to feature fusion processing to obtain the first feature information 350.

In a specific embodiment, the fusion process may perform dimensionality reduction through a fully connected layer (FC) 340 representing a completely connected layer after concatenating each input information. It can be realized by first reducing the dimension through a fully connected layer (FC) 340 and then reconnecting it, and then processing the partial information of the information first, and then connecting with other information to perform the dimension reduction process. However, different convergence processing methods may be used depending on design requirements and actual demands.

Here, the dimensional reduction means that many objects are connected to adjacent objects and extracted as meaningful visual features. For example, when generating the first characteristic information of the cyclist by connecting the visual characteristic information of the single object called bicycle and the human and the semantic characteristic information of the riding state, the bicycle, the person, and the riding state are connected as one. It can be called dimensional reduction.

According to the other side, what is processed in the input language is converted into a text by using a voice identification tool, and then encoded using the LSTM to encode the entire sentence, so that the linguistic characteristics of the phrase (language feature), that is, second feature information is acquired.

In the process of matching (visual language matching) to the first feature information and the second feature information, the obtained first feature information and the second feature information are respectively set in a predetermined feature space (ie, an embedded space). This includes mapping. When performing mapping on the first characteristic information, when performing mapping through the FC on the first characteristic information of each object (ie, each bounding box), in the new feature space, each corresponding to each bounding box Acquires the first characteristic information. Here, the first characteristic information and the second characteristic information of one of the bounding boxes include a pair of characteristics, and the first characteristic information and the second characteristics of the bounding box corresponding to the object (ie, the target object) specified by the user language. The set of information features is a pair of language-visual features that are related to each other. A pair of language-initiating features related to each other has a greater similarity in the mapping space when compared to a pair of language-visual features not related to each other. Accordingly, based on the similarity size, one object having the highest similarity (or a pair of objects having similarities within a predetermined range) may be determined as a result of the target object according to the user's voice.

One object having the highest similarity (bounding box, that is, the bounding box with the highest score) or similarity within a certain range through the similarity between the first characteristic information of each object (bounding box) and the second characteristic information of the input language One group of objects (a plurality of bounding boxes, i.e., a plurality of bounding boxes with relatively high scores) may be taken as the final result output.

In another embodiment, some objects having the highest similarity may be selected according to demand, and these objects may be output and provided to the user for selection.

In the prior art, based on the extraction of the visual features of the object of the image, it was possible to extract the visual features for any bounding box or the entire image, and only the location and/or size information of each object. Features of this type include only the actual location and/or size between objects. However, it did not include relatively high-level semantic information, such as relationships between images. It could handle the tallest room, the second dish on the left, for example. If there are two people in the image, one holding one box, one sitting on top of the box, or each riding a bike/bicycle hitter or wearing a hat/hat If the high-level semantic information between objects cannot be understood, when the user inputs "the person sitting on the box or the person lighting the box", if the high-level semantic information between objects is not understood, the system refers to the user. You cannot make accurate judgments about who a person is.

Depending on the embodiment, it is possible to acquire image characteristics that correspond to relatively high level language information. For example, it is an action or relationship between different objects, such as riding, holding, facing, kicking, and the like. Embodiments can understand the relationship between objects. For example, people-ride-car, people-crash-car, people-disembark-car. By identifying relationships between objects, visual information and language information can be better matched.

In an embodiment, a system for mutually fusing visual feature information of an object with language feature information between objects has been disclosed, and a problem in which the prior art cannot distinguish an object that a user refers to according to location and size information has been solved. It is beneficial to improve the interactive performance of mechanical interactive systems.

According to an embodiment, when classifying an object using language feature information (eg, a relationship between objects) between two objects, a result of determining a target object more accurately may be obtained.

Existing human-machine interactive systems may face the following problems. When the interactive input is a voice input, expressions of the same object of different people (users) cannot be completely matched. For example, if there are multiple plates in the image, there is a pineapple on one of them, and most users faced by the system earlier call it pineapple. However, one new user may have different language habits to call pineapple differently. If he asks the system for "a plate with pineapple on top", the system will not be able to understand what the object named "pineapple" is.

In the actual application of the human-machine interactive system, since different people have different linguistic habits, the probability of occurrence of different words differs greatly from each other. As a result, the LSTM model cannot learn relatively good feature expressions for uncommon words, making it impossible to understand sentences expressed by the user.

The following example proposes a solution to the above problem.

6 is a diagram illustrating another example of a method of determining a target object in an image based on interactive input according to an embodiment.

Referring to FIG. 6, a method of determining a target object in an image based on the interactive input is largely obtained by obtaining 610 first feature information, 610 obtaining second feature information 620, and first feature information The second feature information may be divided into matching step 630.

Looking at step 610 of FIG. 6 in more detail, the method of determining the target object receives an image (or a frame of a video) in step 611, detects an object for the object of the image received in step 612, and step 613 In step 614, visual feature extraction is performed on the bounding box of the object included in the image in which the object is detected, and in step 614, visual feature information of a single object (each object), visual feature information of a complete image, and relative position between objects in the image The size feature information 334 between objects of the information and/or image is respectively obtained, and in step 615, relative size feature information between objects is obtained.

The method of determining the target object may include one or more of visual characteristic information of a single object, visual feature information of a complete image, relative location information between objects, relative size feature information between objects, and semantic feature information between objects in step 616. After obtaining, in turn, the acquired information is subjected to feature fusion processing to obtain first feature information. At this time, a more detailed description of obtaining the first characteristic information is as described above with reference to FIGS. 3 to 5.

Looking at step 620 of FIG. 6 in more detail, the method of determining the target object converts the voice information input in step 621 into characters, converts each word of the input phrase into a word vector in step 622, and inputs in step 623 It is determined whether a word is a word whose usage frequency is lower than a preset value, and if the result of the determination is a word having a low usage frequency, the word having the closest usage frequency is replaced with each word of the entire voice information input in step 624. A word vector having a high frequency of use is obtained, and the second sentence information (that is, language feature) is obtained by processing the entire sentence input in step 625.

At this time, in step 623, replacing the word with a low usage frequency with a word with a high usage frequency having a close meaning searches for a word vector of a word having a similar usage frequency that is higher than a preset value of a cosine similarity of a word vector of a word having a low usage frequency. And, it may represent that the word vector of the word with low usage frequency is replaced with the word vector of the word with high usage frequency.

7 is a diagram showing a distribution diagram showing the frequency of use of words.

Referring to FIG. 7, the use of words when people describe objects has a very prominent long-tail effect. Taking the RefCOCO+ data set as an example, the data set has a total of 2627 different words (displaying words on the horizontal axis, but not specifically showing the coordinate points of each word), where 10 words with the highest frequency appear. , On average, appeared 13000 times (intensively distributed around the origin, which is the same as the dotted frame in FIG. 7 ), but more than half of the words (1381) appear less than 20 times. In the following embodiment, a word whose frequency of use is lower than the first setting value is defined as a first word, and a word whose frequency of use is higher than the second setting value is defined as a second word.

Since the frequency of use of the first word is relatively low, the quantity of samples is very unbalanced. Thus, when training the model, for a sample with very few occurrences, the LSTM module cannot learn a relatively good feature expression of the word. This affects the extraction of the second feature information (ie semantic feature) of this sentence. For example, for a scene with a poison line (a small boat made by digging a single log) in the image, if the user asks the system to say "middle dock line", for the word "road line", a training set appears Because the number of times is relatively small, the model cannot correlate its semantic features with visual features of the target image area. Accordingly, the object that the user refers to cannot be understood. If the "poisonous line" is replaced with a word having a relatively high number of appearances and a close meaning (for example, a small boat), the system can "understand" the object the user refers to and output the correct result. Table 1 presents a series of word substitution examples.

Existing word refrigerator totally Puppy grin blackest loaf wallet Mapping word fridge completely dog smile darkest bread purse

In this embodiment, the extraction of the second feature information (that is, the language feature) includes function modules such as a language identification unit (or identification unit), a word vector unit, a word determination unit, a word replacement unit, and a feature extraction unit. Can be realized through

First, the input voice information is converted into text using a voice identification unit. Then, each word of the input phrase is converted into a word vector using the word vector unit. Subsequently, it is determined whether or not the input word is the first word using the word determination unit, and when the determination result is the first word, the word replacement unit will replace the corresponding first word by selecting the second word having a similar meaning. The feature extraction unit is one LSTM language coding unit, which completes coding for the entire sentence by inputting words one by one, thereby obtaining second feature information (ie, language features).

Note that after converting each word of the input phrase into a word vector using the word vector unit, the system jointly stores each word and word vector of the input phrase and does not belong to the first word. When judging whether or not, the words used by the word judgment unit are also stored words, not word vectors.

Based on the meaning of the word vector proposed in the embodiment and a method of substituting words close to each other, a relatively low frequency sample (using the first word with a relatively low frequency) that cannot extract relatively good features can extract relatively good features. It can be replaced with a high-frequency sample (using a second word with a relatively high frequency), and basically the meaning of the original sentence does not change even if replaced. For example, when the word described by the user is "black shirt with pinkish emblem", after the word substitution, it can be converted into a "black shirt with reddish logo", which is basically the same as the original input meaning.

In this case, the interactive input of each of the above-described embodiments is not limited to natural language input, and the interactive input may be text directly input by a user in a text format.

Accordingly, as shown in FIG. 6, when extracting the first feature information, complete information and/or the same size feature information and language feature information between objects are respectively obtained. Accordingly, according to the embodiment, a problem with no high-level semantic features including the above-mentioned object relationships and a relatively good language feature using words that are not frequently seen in a training set or words that have never appeared in a training set are relatively low. It can solve the problem that can not be extracted.

The VRN proposed in the embodiment can be applied to a pair of objects in an image, semantic feature information (relationship between objects) between two objects can be extracted, and better features can be extracted. By replacing words with relatively high frequency of use, better language features are obtained, allowing the human-machine interactive system to find objects described in the user language in the image more accurately and faster.

In addition, in the embodiment, a second word having a relatively high frequency of use is used to replace the first word having a relatively low frequency of use, so that a human-machine interactive scheme can be realized or used alone.

In a specific embodiment, visually, object detection is performed on an input image to obtain each object bounding box. Using the basic network (VGG-16, ResNet-101, etc.), we extract the visual feature information of the current object's bounding box and the entire image of any specific layer, and the location and size information of the current object's bounding box and the current object's The relative position and relative size information of the bounding box of the bounding box and the proximity object are treated as separate features and connected to the visual feature information to obtain visual feature information corresponding to each bounding box. At this time, VGG-16 and ResNet-101 represent a network for large-scale image recognition in the prior art.

In addition, without performing extraction on the semantic feature information between objects, if the first word of the interactive input exists, a process of replacing it with the second word may be performed. In this case, for a simple human-machine interactive scene, the interactive input is not primarily for a high-level semantic understanding scene, and this removal can be performed on the system to lower the system cost, and the system processing speed and processing accuracy It can be balanced. More detailed operations can be obtained by referring to the above-described respective embodiments. Here, this will not be described in detail anymore.

Each embodiment can be applied to various other man-machine interactive scenes, and it is understood that object information in a scene described by people using a language plays a very important role for a man-machine interactive system. Users can select any object in the scene without using their hands. If another instance such as a corresponding object type exists in the image, the user cannot accurately determine the object described based on the image classification method, or a separate confirmation/selection step is required, resulting in lower man-machine interactive performance. Using an embodiment, this type of problem can be solved and this process can be completed quickly and accurately. Based on the similarity, when determining the target object, it is possible to detect certain types of objects according to demand.

The method of determining the target object from the image provided in the embodiment can be widely applied to cases designated by the user as interactive input (eg, voice input or text input), including price, evaluation, translation, encyclopedia, navigation, etc. However, it is not limited to this. It can be widely used in systems such as head-up display systems (AR HUD), augmented reality glasses (AR glasses), and smart furniture.

For example, a user may make a request to translate the text on the billboard to the right of the XX display in the image where a large number of billboards are present in the system. Although there is a large number of billboards in the scene, the user can determine a specific target object referred to by the user using the first characteristic information of the image, and after the determination of the target object is completed, the user may use text recognition technology and machine translation technology. You can perform the translation of billboards filed by.

Again, for example, the user asks the system who is the shortest person in the image. Using the first feature information of the image, the system can determine a specific target object referred to by the user, thereby identifying the face of the determined target object using the human face identification system to answer the user's problem.

Again, for example, the user asks the system how many shoes the person who jumped from the right side wore. The system uses the first feature information of the image (including semantic features between objects, for example, jumps, etc.) to determine a specific target object that the user refers to, and retrieves an image using a shoe image of the target object. It can be combined again to obtain price information of the corresponding product.

8 is a diagram illustrating an example of applying a method of determining a target object in an image based on interactive input according to an embodiment.

Referring to FIG. 8, information such as a man wearing jeans, a white laptop computer on a desk, and a woman wearing a black shirt can be identified. This has significantly improved the input the system can identify.

On the performance index, a system for determining the target object with training was run in the RefCOCO+ public data set, and tests were performed. The data set contains over 17,000 images, contains 42,000 objects referred to instead, and phrases describing 120,000 objects. Performance tests were performed on the verification set, test set A, and test set B of the corresponding data set, and when compared with the existing method through algorithm comparison, the embodiments in the other test sets were upgraded relatively prominently. In test set A, the performance was over 1.5%. At this time, RefCOCO+ includes a data set generated by training images collected in the prior art.

In another embodiment, an apparatus for determining a target object in an image based on interactive input is further provided.

9 is a diagram illustrating a configuration of an apparatus for determining a target object in an image based on interactive input according to an embodiment.

Referring to FIG. 9, an apparatus 900 for determining a target object from an image based on an interactive input includes a feature acquisition unit 910 and a target determination unit 920. The feature acquisition unit 910 acquires first feature information corresponding to the image and second feature information corresponding to the interactive input. The target determination unit 920 determines the target object for the interactive input from the object of the image according to the first characteristic information and the second characteristic information. More detailed operations of the feature acquisition unit 910 and the target determination unit 920 may be obtained by referring to the above-described embodiment. This will not be described in detail anymore.

10 is a diagram exemplarily illustrating an apparatus for determining a target object from an image based on interactive input according to an embodiment.

Referring to FIG. 10, the apparatus 1000 according to the embodiment includes a processor 1010 such as, for example, a digital signal processing apparatus (DSP). The processor 1010 may be a single device or a plurality of devices for performing other operations according to an embodiment. The device 1000 may further include an input/output (I/O) device 1030 and may be used to receive a signal from another device (or person) or to send a signal to another device (or person). do.

In addition, the device 1000 includes a memory 1020, and the memory 1020 may have a format such as nonvolatile or volatile memory. Examples include electronically erasable and programmable read only memory (EEPROM) and flash memory devices. The memory 1020 stores computer readable instructions, and when the processor 1010 executes the computer readable instructions, the computer readable instructions cause the processor to implement the invention according to the embodiment.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

Obtaining first feature information corresponding to the image and second feature information corresponding to an interactive input; And
Determining a target object corresponding to the interactive input among objects included in the image according to the first characteristic information and the second characteristic information
A method of determining a target object in an image based on an interactive input including a.
According to claim 1,
The obtaining of the first characteristic information corresponding to the image and the second characteristic information corresponding to the interactive input may include:
When acquiring the first characteristic information corresponding to the image,
Obtaining semantic feature information between each object included in the image and at least one other object included in the image
A method of determining a target object in an image based on an interactive input including a.
According to claim 2,
The obtaining of the semantic feature information between each object included in the image and at least one other object included in the image may include:
Obtaining the semantic feature information between each object included in the image and at least one other object based on the location information of each object included in the image
A method of determining a target object in an image based on an interactive input including a.
According to claim 2,
The obtaining of the semantic feature information between each object included in the image and at least one other object included in the image may include:
Determining at least one candidate region based on each object included in the image and at least one other object;
Obtaining classification feature information of an object in the candidate region;
Obtaining region semantic feature information between objects in the candidate region; And
Generating the semantic feature information between each object included in the image and at least one other object based on the classification feature information and the area semantic feature information.
A method of determining a target object in an image based on an interactive input including a.
According to claim 4,
Before generating the semantic feature information between each object included in the image and at least one other object,
Performing joint correction on the classification feature information and the region semantic feature information based on the classification feature information and the region semantic feature information.
A method of determining a target object in an image based on the interactive input further comprising a.
According to claim 4,
Before generating the semantic feature information between each object included in the image and at least one other object,
Determining a reference area according to the candidate area;
Obtaining region feature information of the reference region; And
Performing a joint change on the classification feature information, the region semantic feature information, and the region feature information based on the classification feature information, the region semantic feature information, and the region feature information
A method of determining a target object in an image based on the interactive input further comprising a.
According to claim 4,
The candidate region,
One of the objects included in the image and one of the at least one other object included in the image
How to determine a target object from an image based on interactive input.
According to claim 1,
The first feature information,
Global visual feature information corresponding to the image,
Visual feature information corresponding to each object included in the image,
Relative location information between objects included in the image,
Relative size feature information between objects included in the image and
Semantic feature information between objects included in the image
Containing at least one of
How to determine a target object from an image based on interactive input.
The method of claim 8,
Determining a target object corresponding to the interactive input among the objects included in the image,
Performing fusion processing on the first characteristic information before determining the target object
A method of determining a target object in an image based on the interactive input further comprising a.
According to claim 1,
Obtaining training data including a sample image;
Determining at least one candidate region based on each object included in the sample image and at least one other object included in the sample image;
Determining a reference area according to the candidate area, and obtaining area feature information of the reference area;
Generating an area title according to the area feature information; And
Using the area title as supervised training data, performing training on a neural network model to obtain semantic feature information between objects included in an image
A method of determining a target object in an image based on the interactive input further comprising a.
According to claim 1,
The obtaining of the first characteristic information corresponding to the image and the second characteristic information corresponding to the interactive input may include:
Performing a word vector transformation on the interactive input; And
Obtaining the second feature information corresponding to the interactive input based on the word vector
A method of determining a target object in an image based on an interactive input including a.
The method of claim 11,
The obtaining of the first characteristic information corresponding to the image and the second characteristic information corresponding to the interactive input may include:
Prior to the step of performing the word vector transformation on the interactive input,
Determining whether a word of the interactive input belongs to a first word set
Further comprising,
The step of performing the word vector transformation on the interactive input,
When the word of the interactive input belongs to the set first word, using the word vector of the second word having high similarity to the word vector of the first word as a word vector corresponding to the first word
Containing
How to determine a target object from an image based on interactive input.
The method of claim 12,
The first word,
The frequency of use indicates a word lower than the first set value,
The second word,
A word whose frequency of use is higher than the second setting value,
How to determine a target object from an image based on interactive input.
According to claim 1,
The interactive input,
At least one of voice input and text input,
How to determine a target object from an image based on interactive input.
A feature acquiring unit that acquires first feature information corresponding to an image and second feature information corresponding to an interactive input; And
A target determination unit that determines a target object corresponding to the interactive input among objects included in the image according to the first characteristic information and the second characteristic information
Device for determining a target object in the image based on the interactive input including a.
The method of claim 15,
The first feature information,
Global visual feature information corresponding to the image,
Visual feature information corresponding to each object included in the image,
Relative location information between objects included in the image,
Relative size feature information between objects included in the image and
Semantic feature information between objects included in the image
Containing at least one of
A device that determines a target object from an image based on interactive input.
The method of claim 16,
The feature acquisition unit,
Determining at least one candidate region based on each object included in the image and at least one other object,
Obtain classification feature information of an object in the candidate region,
Area semantic feature information between objects in the candidate area is acquired,
A reference area is determined according to the candidate area,
Acquire region feature information of the reference region,
Performing a joint change on the classification feature information, the region semantic feature information, and the region feature information based on the classification feature information, the region semantic feature information, and the region feature information; And
Generating the semantic feature information between each object included in the image and at least one other object based on the corrected classification feature information, the corrected region semantic feature information, and the corrected region feature information
A device that determines a target object from an image based on interactive input.
The method of claim 15,
The feature acquisition unit,
Perform word vector transformation on the interactive input,
Acquiring the second characteristic information corresponding to the interactive input based on the word vector
A device that determines a target object from an image based on interactive input.
The method of claim 15,
The feature acquisition unit,
When performing the word vector conversion on the interactive input,
It is determined whether the word of the interactive input belongs to the set first word, and when the word of the interactive input belongs to the set first word, the word vector of the second word having high similarity to the word vector of the first word Is used as a word vector corresponding to the first word,
The first word,
The frequency of use indicates a word lower than the first set value,
The second word,
A word whose frequency of use is higher than the second setting value,
A device that determines a target object from an image based on interactive input.
A computer-readable recording medium, characterized in that a program for executing the method of any one of claims 1 to 14 is recorded.

Images