JP7227799B2 - Image retrieval device, image retrieval method and computer program - Google Patents

Description

The present invention relates to an image search device, an image search method, and a computer program.

2. Description of the Related Art Conventionally, an image search device that uses a computer to search for an image desired by a user searches for images based on an input keyword. The image search device compares the keywords associated with each of the images stored in the server with the keywords input by the user, and presents the search results to the user.

In the technique of Patent Document 1, a plurality of thumbnail images related to a set keyword are displayed. In this case, the server computer reflects the detection rate of each keyword in the number of displayed thumbnail images for each keyword. The detection rate for each keyword is input by the user when searching for images. Accordingly, a server computer is disclosed that can reflect the degree of importance of each keyword that a user considers to a search.

JP 2013-003727 A

In Patent Literature 1, image retrieval is performed based on text described in a Web page and an input keyword. However, since the text described in the Web page is subjectively set by a person, the technique disclosed in Japanese Patent Laid-Open No. 2002-200013 cannot perform a search based on information that is not recognized by the person.

Furthermore, since the relationships between the objects indicated by the keywords are not set in each image, in Patent Literature 1, it is not possible to perform a search including the relationships between the objects indicated in the images.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image search apparatus, an image search method, and a computer program capable of improving the accuracy of image search.

The image search device comprises: a feature extraction unit for calculating search data including object name information of a plurality of objects included in an image and relationship name information indicating a predetermined relationship between the plurality of objects included in the image; a storage unit for storing search data in association with an image; an input unit for accepting input of at least one of object name information and related name information as a search query; searching the storage unit based on the search query; A search unit for extracting a predetermined image corresponding to a search query, and an output unit for outputting search results.

According to the present invention, the accuracy of image retrieval can be improved.

1 is a schematic diagram of an image search device according to a first embodiment; FIG. FIG. 2 is a hardware configuration diagram of an image search device; Explanatory drawing of a search query input part. Explanatory drawing of object name information. Explanatory drawing of related name information. Explanatory drawing of the data for a search. Explanatory drawing of a search result output part. 4 is a flowchart of processing of a search unit; 4 is a flowchart of processing of a feature extraction unit; Enlarged view of the A region. Explanatory drawing of the search result output part which highlights an object. Schematic diagram of an image search device according to a second embodiment. The flow chart of learning processing. Schematic diagram of an image search device according to a third embodiment. Explanatory drawing of a search query input part. 4 is a flowchart of image search processing; Schematic diagram of an image search device according to a fourth embodiment. Explanatory drawing of a search query input part. 4 is a flowchart of image search processing; Explanatory drawing of the object name information based on 5th Example. The schematic of the station which concerns on 6th Example. Explanatory drawing of object name information. FIG. 11 is a hardware configuration diagram of an image search device according to the seventh embodiment; FIG. 11 is a hardware configuration diagram of an image search system according to an eighth embodiment;

Although the present embodiment will be described below with reference to the accompanying drawings, the present invention is not limited to the configuration described in the drawings. The present embodiment relates to an image retrieval device for retrieving images. The image retrieval device 1 of this embodiment can be used in a monitoring system for monitoring visitors at various facilities such as airports, stations, ports, department stores, hotels, and event venues. The image search device 1 of this embodiment, unlike a normal image search engine on the web, automatically identifies a plurality of objects included in one image and the relationships (physical relationships) between the objects. Calculate and save. As a result, according to the present embodiment, it is possible to accurately extract an image related to the search purpose from among the images accumulated in the storage unit, and the usability is improved.

A surveillance system to which the image retrieval device 1 according to the present embodiment is applied generally shoots a specific surveillance target area without any particular theme or motif, rather than intentionally shooting a specific subject. . Therefore, in the image acquired as a result of monitoring, a plurality of miscellaneous objects appear regardless of their master and subordinates. The image retrieval device 1 according to this embodiment determines physical relationships between a plurality of objects based on their positions on the image, and stores them in association with the images.

Not limited to surveillance systems, it is also possible to automatically extract the physical relationships between objects in still images or moving images in which a plurality of objects are captured, and store them in association with the images. .

Image search services provided on websites, etc., are associated with descriptive text (e.g., "entrance ceremony", "wedding", etc.) that matches the intention of the photographer, or the image analysis result by the analyst. Descriptions (for example, "students going to school", "busy beach at the opening of the sea", etc.) are simply associated with the images. That is, the image search engines offered on the Web do not consider the physical relationships between objects and use only a few words or phrases to search for the images. As will be apparent from the following description, it should be noted that the image search device 1 according to this embodiment is completely different from the prior art including image search engines on the web.

FIG. 1 is a schematic diagram of an image retrieval device 1. As shown in FIG. The image retrieval apparatus 1 in this embodiment retrieves a user's target image (hereinafter sometimes referred to as a predetermined image) from among a plurality of images stored in the image data storage unit 123 .

The image search device 1 includes a search query input unit 120 as an example of an "input unit", a search unit 121, a feature extraction unit 122, an image data storage unit 123, a feature data storage unit 124, and an "output unit". and an image data acquisition unit 126 .

The search query input unit 120 is a function that receives search queries from users. The search query includes at least one of object name information 1242 (see FIG. 4) or relationship name information 1244 (see FIG. 5). Note that "object" may be indicated as "object" in the drawings. The object name information 1242 indicates multiple objects included in the image. The relationship name information 1244 indicates relationships between multiple objects.

The search query input unit 120 is, for example, a UI (User Interface) displayed on the output device 11 (hereinafter sometimes referred to as the monitor 11). The search query input unit 120 is connected to the search unit 121 so as to be unidirectionally communicable. Note that the search query input unit 120 will be described later with reference to FIG.

The search unit 121 has a function of searching the storage unit 12 (described later with reference to FIG. 2) based on the search query and extracting a predetermined image corresponding to the search query. The search unit 121 is connected to the image data storage unit 123, the feature data storage unit 124, and the search result output unit 125 so as to be unidirectionally communicable. The search unit 121 may be connected to the feature extraction unit 122 so as to be able to communicate bidirectionally. The search unit 121 will be described later with reference to FIG.

The feature extraction unit 122 has a function of calculating search data (see FIGS. 4 to 6) from an image. The search data includes object name information 1242 and relationship name information 1244 . The feature extraction unit 122 is connected to the image data storage unit 123 so as to be unidirectionally communicable. The feature extraction unit 122 is connected to the feature data storage unit 124 so as to be able to communicate bidirectionally. The feature extraction unit 122 will be described later with reference to FIG.

The image data storage unit 123 is a database that stores a plurality of images. The image data storage unit 123 is connected to the image data acquisition unit 126 so as to be unidirectionally communicable. The feature data storage unit 124 is a database that stores search data.

The search result output unit 125 has a function of outputting search results. The search result output unit 125 causes the monitor 11 to display the search result, for example. It should be noted that the search result output unit 125 is not limited to displaying the search results on the monitor 11, but rather displays the search results on an external terminal via a communication interface (communication I/F (InterFace) in the figure) 16 (see FIG. 2). may be output. The search result output unit 125 will be described later with reference to FIG.

The image data acquisition unit 126 has a function of storing a plurality of image data in the image data storage unit 123 . The image data acquisition unit 126 acquires moving image data captured by, for example, a surveillance camera or the like, and stores an image of each frame of the moving image in the image data storage unit 123 . Note that the image data acquisition unit 126 may acquire a plurality of image data (either still images or moving images) from the Internet or the like, not limited to moving image data captured by a surveillance camera or the like. The image data acquisition unit 126 acquires image 2, for example. The image data acquisition unit 126 stores image 2 in the image data storage unit 123, for example.

An image 2 is, for example, a one-frame image in moving image data captured by a surveillance camera. Image 2 displays, for example, the premises of station 5 . In the image 2, for example, an entrance 51 and people 21, 25, 29, 33, 37, 40 are displayed.

Person 21 is a child. A person 21 wears clothes 22 and shoes 23 and carries a bag 24 . Person 25 is an adult female. A person 25 wears a suit 26 and shoes 27 and carries a bag 28 . Person 29 is an adult male. A person 29 wears a suit 30 and shoes 31 and has a drink 32 . Person 33 is an adult male. A person 33 wears plain clothes 34 and shoes 35 and carries a bag 36 . Person 37 is a child. Person 37 wears plain clothes 38 and shoes 39 . Person 40 is an adult female. Person 40 wears plain clothes 41 and shoes 42 . The persons 33 and 37 hold hands and are in contact with each other, and similarly, the persons 40 and 37 hold hands and are in contact with each other.

FIG. 2 is a hardware configuration diagram of the image search device 1. As shown in FIG. The image search device 1 bidirectionally communicates with a monitor 11 (output unit 11 in the drawing), an input device 13, a storage unit 12, a CPU 14, a memory 15, a communication interface 16, and functions 11 to 16. and a data transmission line 17 for possible connection.

The monitor 11 is, for example, a display provided in a personal computer, a display provided in a mobile information terminal, a mobile phone (so-called smart phone), a wearable terminal, or the like. The input device 13 is, for example, a keyboard or mouse. Note that the input device 13 may be a microphone. In this case, the search query input unit 120 may have a function of converting data input by voice into text data representing the search query. Furthermore, the input device 13 and the output device 11 may be integrated like a tablet or an AR (Augmented Reality) display.

The storage unit 12 is, for example, a non-volatile storage device such as a hard disk or SSD (Solid State Drive). Any type of storage medium can be used. The storage unit 12 stores computer programs (hereinafter referred to as programs) for a search query input unit 120, a search unit 121, a feature extraction unit 122, a search result output unit 125, and an image data acquisition unit 126. FIG. The storage unit 12 also stores databases such as an image data storage unit 123 and a feature data storage unit 124 .

The CPU 14 reads each program from the storage unit 12 via the memory 15 and executes it. The memory 15 is, for example, a volatile storage device such as "RAM (Random Access Memory)".

The communication interface 16 is a device that communicates with an external device via a communication network such as a LAN (Local Area Network), the Internet, or a SAN (Storage Area Network).

FIG. 3 is an explanatory diagram of the search query input unit 120. As shown in FIG. The search query input unit 120 causes the monitor 11 to display a plurality of object name input fields 1201(1) and 1201(2), a relationship name input field 1202, and a search button 1203. FIG. The object name input fields 1201(1) and 1201(2) may be referred to as the object name input field 1201 when they are not distinguished from each other.

The object name input field 1201 is a function of receiving object name information 1242 . For example, "person" is entered in the object name input field 1201(1). For example, "bag" is entered in the object name input field 1201(2).

Note that the object name input field 1201 is not limited to the function of receiving the object name information 1242, and may have the function of receiving an "object ID (IDentification)" 1241 (see FIG. 4). The search query input unit 120 is not limited to displaying two object name input fields 1201(1) and 1201(2), and may display three or more object name input fields 1201 on the monitor 11. FIG.

The relationship name input field 1202 is a function for receiving relationship name information 1244 . In the relationship name input field 1202, for example, "has" is input.

Note that the relationship name input field 1202 may have a function of accepting the “relationship ID” 1243 (see FIG. 5) in addition to the function of accepting the relationship name information 1244 . The search query input unit 120 is not limited to displaying one relationship name input field 1202, and may display two or more relationship name information 1244. FIG.

That is, the user searches for an image in which a "person" "holds" a "bag". A search button 1203 is a button for executing an image search.

The search query input unit 120 displays a plurality of object name input fields 1201 and at least one relationship name input field 1202 side by side in the scroll direction of the monitor 11 . The scroll direction of the monitor 11 is, for example, the vertical direction of the monitor 11 (direction S in the figure).

That is, the search query input unit 120 displays the object name input field 1201(1) on the upper side of the monitor 11, for example. The search query input unit 120, for example, displays the object name input field 1201(2) downward from the object name input field 1201(1). The search query input unit 120, for example, causes the relationship name input field 1202 to be displayed downward from the object name input field 1201(2).

As a result, even when using a vertically long monitor 11 provided on a mobile phone or the like, the user can scroll down to select a plurality of object name input fields 1201 and at least one relationship name input field. 1202 and can be displayed. Thereby, the operability of the search query input unit 120 can be improved.

The information received by the object name input field 1201 and the relationship name input field 1202 may be selected using pull-down menus. Information displayed in the pull-down menu may be information stored in the feature data storage unit 124 .

FIG. 4 is an explanatory diagram of the object name information 1242. As shown in FIG. An “object ID” 1241 and object name information 1242 are stored in the feature data storage unit 124 .

The “object ID” 1241 stores information for identifying the object name information 1242 . The object name information 1242 stores the name of the object included in the image. The object name information 1242 stores, for example, "person", "car", "train", "bag" or "sticker".

FIG. 5 is an explanatory diagram of the relationship name information 1244. As shown in FIG. A “relationship ID” 1243 and relation name information 1244 are stored in the feature data accumulation unit 124 .

Information for identifying the relationship name information 1244 is stored in the “relationship ID” 1243 . The relationship name information 1244 stores information indicating relationships between objects. A relationship between each object indicates a physical relationship. The related name information 1244 stores, for example, "holding", "riding", "attaching", "touching", "hanging", and the like.

FIG. 6 is an explanatory diagram of search data. The feature data storage unit 124 stores an "image ID" 1245, a "combination ID" 1246, identification information 1241(1) and 1241(2) of each object, position information 1247(1) and 1247(1) of each object. 2), size information 1248(1) and 1248(2) of each object, and "relationship ID" 1243 are saved. In the drawing, the feature data storage unit 124 stores search data containing two objects for one image, but three or more objects for one image. Data for search that is obtained may be stored. In the figure, for "relationship ID" 1243, "image ID" 1245, and "combination ID" 1246, since the characters may not fit within the frame, the symbols are directly described.

“Image ID” 1245 stores information for identifying an image. For example, an image 2 is shown as an image whose “image ID” is “0”. The 'combination ID' 1246 stores information for identifying the combination of the 'object 1 ID' 1241(1), the 'object 2 ID' 1241(2), and the 'relationship ID' 1243 .

"Object 1 ID" 1241(1) and "Object 2 ID" 1241(2) store information identifying objects. “Object 1 ID” 1241(1) and “Object 2 ID” 1241(2) correspond to “Object ID” 1241 (see FIG. 4).

Object position information 1247(1) and 1247(2) stores object position information. Note that the position information may be calculated, for example, by counting the number of pixels from the edge of the image. The location information may be indicated using two-dimensional coordinates of x and y coordinates on the image. Note that the position information may be indicated using three-dimensional coordinates.

Object size information 1248(1) and 1248(2) stores object size information. Note that the size information may be calculated by, for example, the number of pixels on the image. The size information may be calculated by the length of the edge of each object from the centroid of each object. The size of an object may be indicated using the width w and height h of the object volume on the image.

FIG. 7 is an explanatory diagram of the search result output unit 125. As shown in FIG. The search result output unit 125 causes the monitor 11 to display the predetermined image searched by the search unit 121 . The search result output unit 125 displays, for example, image 2, which is a predetermined image.

FIG. 8 is a flow chart of processing of the search unit 121. As shown in FIG. After the image search device 1 is activated, the search unit 121 acquires a plurality of image data from the image data storage unit 123 and acquires search data from the feature data storage unit 124 (S11). When the search button 1203 of the search query input unit 120 is pressed (S12: Yes), the search unit 121 executes image search processing (S13 to S15).

When describing the processing of the search unit 121, a case where "person", "bag" and "have" shown in FIG.

The search unit 121 acquires the object name information 1242 indicating "person" and "bag" and the related name information 1244 indicating "have" from the search query input unit 120 (S13). The search unit 121 searches for a predetermined image from the plurality of images (S14). The predetermined image has search data including, for example, object name information 1242 of "person" and "bag" and related name information 1244 of "has".

As shown in the search data of FIG. 6, when the "combination ID" 1246 is "1", the "object 1 ID" 1241(1) indicates "0", and the "object 2 ID" 1241(2) indicates " 3”, and the “relationship ID” 1243 indicates “0”. "0" in the "object ID" 1241 indicates "person" (see FIG. 4). "3" in the "object ID" 1241 indicates "bag". "0" in the "relationship ID" 1243 indicates "has".

The search unit 121 determines that the image of "0" of the "image ID" 1245 includes the object name information 1242 of "person" and "bag" and the related name information 1244 of "has". The search unit 121 sets the image 2 as the predetermined image.

The search unit 121 transmits image 2 to the search result output unit 125 (S15). The search unit 121 transmits information on the objects included in the image 2 and information on relationships between the objects to the search result output unit 125 . The processing of the search unit 121 returns to the processing (S12) when continuing the search (S16: No). The processing of the search unit 121 ends when the search ends (S16: Yes).

FIG. 9 is a flow chart of processing of the feature extraction unit 122. As shown in FIG. The feature extraction unit 122 is executed, for example, when a new image is added to the image data accumulation unit 123 . Note that the feature extraction unit 122 may be executed by a user. The feature extraction unit 122 may be executed at a predetermined cycle set in a scheduler or the like.

The feature extraction unit 122 acquires at least one piece of image data from the image data storage unit 123 (S21). The image extraction unit 122 may acquire image data newly added to the image data storage unit 123, for example. The image extraction unit 122 may acquire all image data stored in the image data storage unit 123 .

The feature extraction unit 122 calculates object name information 1242 (see FIG. 4) (S22). The feature extraction unit 122 calculates the relationship name information 1244 (see FIG. 5) using the object name information 1242 calculated in the process (S22) (S23).

Note that the processing (S22, 23) of the feature extraction unit 122 calculates the object name information 1242 and the relationship name information 1244 by using, for example, a CNN (Convolutional Neural Network). The feature extraction unit 122 may calculate position information of the object and information on the size of the object. A calculation formula for the search data of the feature extraction unit 122 is shown in Formula 1 below.

(o1, x1, y1, w1, h1, c1, o2, x2, y2, w2, h2, c2, r, cr)=φ(I; θ) Equation (1)

In the right side of Equation 1, "φ" indicates the CNN equation. "I" indicates the data of the input image. “θ” indicates a parameter of CNN. The parameter “θ” is set in the feature extraction unit 122 to calculate data of multiple objects shown in the image and data of relevance between the multiple objects shown in the image. The feature extraction unit 122 calculates search data according to the formula “φ” using “I” and “θ”.

The left side of Equation 1 indicates search data. "o1" and "o2" indicate the name 1242 of each object. "x1", "x2", "y1" and "y2" indicate the position 1247 of each object. "w1", "w2", "h1" and "h2" indicate the size 1248 of each object. 'c1' and 'c2' indicate the reliability of the estimation of each object. “r” indicates the relationship name information 1244 . “cr” indicates the reliability of estimation of the relationship name information 1244 . Note that each search data may be abbreviated, such that "o1" and "o2" are indicated as "o" when there is no particular distinction between objects.

Note that the feature extraction unit 122 may calculate state name information indicating the state of each object without being limited to calculating the relationship name information 1244 between the objects. That is, the feature extraction unit 122 calculates state name information such as "smiling" by reading a person's facial expression or the like. The search unit may search for images based on the state name information.

The feature extraction unit 122 calculates relationship name information 1244 based on the distance between each object. FIG. 10 is an enlarged view of the A area. The feature extraction unit 122 calculates, for example, the position of the center of gravity of the object as positional information of the object (shown as a diamond-shaped mark in the figure). The position information is indicated by, for example, two-dimensional information of the y-coordinate and the x-coordinate of the image.

The feature extraction unit 122 calculates position information (x11, y11) of the person 21 . The feature extraction unit 122 calculates position information (x12, y12) of the bag 23 . The feature extraction unit 122 calculates the position information (x21, y21) of the person 25. FIG. The feature extraction unit 122 calculates the position information (x22, y22) of the bag 26. FIG.

The feature extraction unit 122 calculates the distance between each piece of position information. The feature extraction unit 122 determines objects within a predetermined distance as objects having a relationship. That is, for example, since the distance between the position information (x11, y11) and the position information (x12, y12) is within a predetermined distance, the feature extraction unit 122 extracts the relationship name information 1244 between the person 25 and the bag 26. calculate. Since the distance between the position information (x11, y11) and the position information (x22, y22) is longer than the predetermined distance, the feature extraction unit 122 does not calculate the relationship name information 1244 between the person 25 and the bag 23.

Note that the feature extraction unit 122 may calculate the distance of each object based on the shortest distance of the outer shape of each object instead of calculating the distance of each object based on the positional information of the center of gravity of the object. Position information of each object is not limited to two-dimensional space, and position information may be calculated in three-dimensional space.

Returning to FIG. 9, the feature extraction unit 122 transmits the calculated search data to the feature data accumulation unit 124 (S24). The feature extraction unit 122 ends after processing (S24).

The search unit 121 can acquire image data and search data from the image data storage unit 123 and the feature data storage unit 124 before the search button 1203 is pressed. As a result, the search unit 121 can reduce the frequency of communication between the image data storage unit 123 and the feature data storage unit 124 during image search (S13 to S15).

Note that the search unit 121 may execute the image data and search data acquisition process (S11) after the image search start process (S12: Yes). This allows the search unit 121 to reduce the usage of the memory 15 in the standby state.

The feature extraction unit 122 may calculate search data from a plurality of images in the image data storage unit 123 after the image search start process (S12: Yes). In this case, the search unit 121 may acquire search data from the feature extraction unit 122 . As a result, the amount of data stored in the feature data storage unit 124 can be reduced.

A user may input information on one object and information on one relationship into the search query input unit 120 . In this case, the search unit 121 selects data of any object from the feature data storage unit 124 . The search unit 121 acquires relevance data between the object of the search query and any selected object. The search unit 121 may perform an image search by comparing the obtained relevance data and the relevance data of the search query.

A plurality of objects included in a predetermined image may be highlighted in the search result output unit 125 . FIG. 11 is an explanatory diagram of the search result output unit 125 that highlights objects.

The search result output unit 125 displays the predetermined object included in the search query and the predetermined related name information included in the search query. The search result output unit 125 causes the monitor 11 to display, for example, the person 21, the bag 22, and "I have" 1244(1).

The search result output unit 125 separates and displays a plurality of predetermined objects. The search result output unit 125 displays predetermined related name information between a plurality of displayed predetermined objects. For example, the search result output unit 125 causes the person 21 and the bag 22 to be displayed separately, and displays "has" 1244(1) to indicate the relationship between the person 21 and the bag 22.

The search result output unit 125 displays other relationships among other related name information not included in the search query, predetermined objects, and other objects. The search result output unit 125 displays the sticker 43 and "attached" 1244(2).

The search result output unit 125 separates the other object from the displayed predetermined object and displays it. The search result output unit 125 displays other related name information between the displayed predetermined object and the other displayed object. The search result output unit 125 causes the bag 22 and the sticker 43 to be displayed separately, and displays "attached" 1244(2) between the bag 22 and the sticker 43 to indicate the relevance.

The search result output unit 125 arranges and displays a predetermined object, predetermined related name information, another object, and other related name information in the scroll direction of the screen. For example, in the scrolling direction (S direction) of the monitor 11, the search result output unit 125 displays the person 25 below the person 21, displays the bag 28 below the bag 22, and displays "I have" 1244. (1) Display "has" 1244 (3) downward.

The search result output unit 125 may display the search query. By displaying the search query, the information entered by the user can be verified.

The search result output unit 125 is not limited to the layouts shown in FIGS. 7 and 11, and may employ other display methods to improve visibility. The search result output unit 125 may set the size of the object image to be displayed according to the object estimation reliability “c” (see Equation 1). As a result, the image search device 1 can improve the visibility of search results.

The image retrieval apparatus 1 shown in this embodiment uses the relationship name information 1244 by including the input unit 120, the retrieval unit 121, the feature extraction unit 122, the output unit 125, and the storage unit 12. Images can be searched. As a result, the image retrieval device 1 can improve the accuracy of image retrieval.

Since the relationship name information 1244 indicates the physical relationship between each object, the feature extraction unit 122 can calculate the relationship name information 1244 based on the position information between each object.

The feature extraction unit 122 can calculate object name information 1242 included in the image. As a result, the feature extraction unit 122 can also store the object name information 1242 that is not recognized by humans in the feature data storage unit 1244 . As a result, it becomes unnecessary for the user to input the object name information by himself/herself, so that usability is improved.

In order to calculate the relationship name information 1244 between objects having a physical relationship based on the position information of each object, the feature extraction unit 122 calculates the relationship name information 1244 between objects at a predetermined distance or more. suppress As a result, the feature extraction unit 122 can suppress calculation of redundant related name information 1244 .

Since the search query input unit 120 has a function of inputting a search query in characters, the usability of the image search device 1 is improved.

Since the search query input unit 120 has a function of displaying the object name input field 1201 and the relationship name input field 1202 side by side in the scroll direction of the monitor 11 , it is possible to display fields that cannot be displayed on the monitor 11 . This improves usability of the image search device 11 .

The search result output unit 125 has a function of displaying a plurality of predetermined objects with more emphasis than other objects. This allows the user to easily find the desired object even when multiple objects are displayed in the image.

The search result output unit 125 has a function of displaying predetermined related name information. Thereby, the user can easily grasp which objects have a predetermined relationship with each other.

The search result output unit 125 has a function of displaying other objects and other related name information. This allows the user to grasp other objects and other related name information.

The search result output unit 125 has a function of displaying a plurality of predetermined objects separated from each other, a function of displaying predetermined relationship name information between the plurality of predetermined objects to be displayed, and a function to display other objects. a function to display other related name information between the specified object to be displayed and the other object to be displayed; a function to display the specified object and the specified relationship It has name information, other objects and other related name information, and a function of displaying them side by side in the scroll direction of the screen. Thereby, the search result output unit 125 can improve the visibility of the search results.

Since this embodiment corresponds to a modification of the first embodiment, differences from the first embodiment will be mainly described. FIG. 12 is a schematic diagram of the image retrieval device 1a. The image search device 1a in this embodiment learns parameters set in the feature extraction unit 122a by the learning unit 127. FIG.

The image search device 1a includes a search query input unit 120a, a search unit 121, a feature extraction unit 122a, an image data storage unit 123a, a feature data storage unit 124a, a search result output unit 125a, and an image data acquisition unit 126. , and a learning unit 127 . The search query input unit 120a is a function that receives search queries from users. The search query input unit 120a is a UI displayed on the monitor 11, for example. The search query input unit 120a is connected to the search unit 121 and the learning unit 127 so as to be unidirectionally communicable.

The feature extraction unit 122a has a function of calculating search data from an image. The feature extraction unit 122a is connected to the image data storage unit 123a so as to be unidirectionally communicable. The feature extraction unit 122a is connected to the feature data storage unit 124a and the learning unit 127 so as to be able to communicate bidirectionally. The feature extraction unit 122a may be connected to the search unit 121 so as to be able to communicate bidirectionally.

The image data storage unit 123a is a database that stores a plurality of images. Learning images used by the learning unit 127 are stored in the image data storage unit 123a. The feature data storage unit 124a is a database that stores search data for each image. The feature data accumulation unit 124a stores teacher data indicating search data included in the learning image.

The learning unit 127 has a function of learning the parameter "θ". The learning unit 127 is connected to the image data storage unit 123, the feature data storage unit 124a, and the search result output unit 125a so as to be unidirectionally communicable. Note that the learning process will be described later with reference to FIG.

The search result output unit 125a is a function of outputting the search result of the search unit 121 to the user. The search result output unit 125a is a UI displayed on the monitor 11, for example. The search result output unit 125 a may output the learning result of the learning unit 127 .

FIG. 13 is a flow chart of the learning process. The learning process is performed by the processes of the learning unit 127 (S30, S31, S36, S37) and the processes of the feature extraction unit 122a (S32 to S35, S38). The learning process is not limited to the user's operation, and may be executed at predetermined intervals.

The learning unit 127 acquires a plurality of data of images for learning from the image data storage unit 123 (S30). The learning unit 127 acquires teacher data corresponding to each of the plurality of learning images from the feature data accumulation unit 124a (S31).

The feature extraction unit 122a initializes the parameter "θ" (S32). When the feature extraction unit 122a is composed of a CNN, the feature extraction unit 122a initializes the parameter "θ" by, for example, randomly extracting values from a Gaussian distribution or a uniform distribution. The learning unit 127 transmits data of a plurality of learning images to the feature extraction unit 122a (S33).

The feature extraction unit 122a calculates search data from the learning image (S34). That is, the learning unit 127 causes the feature extraction unit 122 to calculate the data of the plurality of objects shown in the learning image and the data of the relevance between the plurality of objects shown in the learning image as pre-learning data. . The feature extraction unit 122a transmits a plurality of pieces of search data calculated for each learning image to the learning unit 127 (S35).

The learning unit 127 calculates updated values of the parameters based on the teacher data and the pre-learning search data calculated by the feature extraction unit 122a (S36). For example, the learning unit 127 associates a plurality of search data received from the feature extraction unit 122a with a plurality of teacher data received from the feature data accumulation unit 124 for each learning image. The learning unit 127 calculates the error between the search data and the teacher data. The learning unit 127 calculates update values for a plurality of parameters based on the calculated error.

The learning unit 127 calculates the errors of the positions “x” and “y” of each object and the sizes “w” and “h” of each object (see Equation 1), for example, by using a squared error calculation method. do. The learning unit 127 calculates the errors of the estimation reliability levels "c" and "cr" by using, for example, "Softmax cross entropy". The learning unit 127 calculates the update value of the parameter from each calculated error value by using, for example, the error backpropagation method.

The learning unit 127 transmits the updated parameter values to the feature extraction unit 122a (S37). The feature extraction unit 122a updates the parameter values (S38). The feature extractor 122a updates the parameters by using stochastic gradient descent, for example.

The learning unit 127 determines whether to continue or end the learning (S39). When learning is to be continued (S39: No), the processing of the learning unit 127 moves to processing (S33). If learning is to end (S39: Yes), the processing of the learning unit 127 ends.

Note that the end processing of the learning unit 127 may be operated by the user. The learning unit 127 monitors the difference between the error in the search data and teacher data calculated with the updated parameters and the error in the search data and teacher data calculated with the pre-update parameters. , may decide whether to continue learning.

Note that when the search start process (S12: Yes) (see FIG. 8) is executed, the feature extraction unit 122a acquires a plurality of images from the image data storage unit 123, and calculates search data. good. Thereby, the search unit 121 can perform an image search based on the search data calculated by the feature extraction unit 122a with updated parameters.

The present embodiment configured in this manner has the same effects as those of the first embodiment. Furthermore, according to the present embodiment, the image search device 1a includes the learning unit 127, so it is possible to update the parameters of the feature extraction unit 122a. Thereby, the feature extraction unit 122a can improve the accuracy of calculating the search data. As a result, the image retrieval device 1a can improve the accuracy of image retrieval.

Since the present embodiment corresponds to a modified example of the first and second embodiments, the differences from the first and second embodiments will be mainly described. FIG. 14 is a schematic diagram of the image search device 1b. The image search device 1b in this embodiment searches for a predetermined image based on an input image.

The image search device 1b includes a search query input unit 120b, a search unit 121b, a feature extraction unit 122b, an image data storage unit 123a, a feature data storage unit 124a, a search result output unit 125a, and an image data acquisition unit 126. , and a learning unit 127 .

The search query input unit 120b has a function of receiving an input image from the user. A plurality of objects that are the same as or similar to a plurality of objects included in a predetermined image are displayed in the input image. The search query input unit 120b is a UI displayed on the monitor 11, for example. The search query input unit 120b is connected to the search unit 121b and the learning unit 127 so as to be unidirectionally communicable. The search query input section 120b will be described later with reference to FIG.

The feature extraction unit 122b has a function of calculating search data from an image. The feature extraction unit 122b is connected to the image data storage unit 123a so as to be unidirectionally communicable. The feature extraction unit 122b is connected to the search unit 121b, the feature data storage unit 124b, and the learning unit 127 so as to be able to communicate bidirectionally.

The feature extraction unit 122b calculates a search query from data of the input image. The search unit 121b has a function of searching the storage unit 12 based on the search query calculated by the feature extraction unit 122 and extracting a predetermined image corresponding to the search query. The search unit 121b is connected to the image data storage unit 123a, the feature data storage unit 124a, and the search result output unit 125a so as to be unidirectionally communicable. Processing by the search unit 121b and the feature extraction unit 122b will be described later with reference to FIG.

FIG. 15 is an explanatory diagram of the search query input section 120b. An image input field 1204 and a search button 1203 are displayed in the search query input section 120b. An image input field 1204 is an area for inputting an input image. A user may input an image of an object contained in a given image. A user may input images of objects that are similar to objects contained in a given image. A user may input an image similar to a predetermined image.

The image input field 1204 is displayed in the center of the monitor 11, for example. The user is not limited to inputting one image, and may input a plurality of images. In this case, a plurality of image input fields 1204 may be displayed side by side in the scroll direction of the monitor 11, for example.

FIG. 16 is a flowchart of image search processing. Image search processing may be executed by the user pressing a search button 1203 . The search unit 121b acquires input image data from the search query input unit 120b (S41). The search unit 121b transmits data of the input image to the feature extraction unit 122b (S42).

The feature extraction unit 122b calculates a search query from the input image (S43). The feature extraction unit 122b transmits the search query to the search unit 121b (S44).

The search unit 121b compares the search query of the input image acquired from the feature extraction unit 122b with the search data of the plurality of images stored in the feature data storage unit 124, thereby obtaining a predetermined image from among the plurality of images. An image is searched (S45). The search unit 121b transmits a predetermined image to the search result output unit 125a (S46).

Note that the image search device 1b may search for a predetermined image based on the degree of similarity between the input image and a plurality of images. In this case, the feature extraction unit 122b calculates a feature vector from the image by using CNN, for example.

A feature vector is, for example, an m-dimensional (m is a predetermined constant) data group representing features shown in an image. A feature vector includes features of a plurality of objects included in an image and features of relationships between the plurality of objects included in the image. The feature extraction unit 122b may calculate the feature of the color of the object, the feature of the pattern of the object, etc. as a feature vector.

Based on the feature vector calculated from the input image and the feature vector calculated from the image stored in the storage unit 12, the feature extraction unit 122b calculates a plurality of degrees of similarity to the input image for each image stored in the storage unit 12. calculate. The search unit 121b searches the storage unit 12 based on a plurality of degrees of similarity and extracts a predetermined image.

Note that the feature extraction unit 122b may calculate information on the area where the object is displayed from the image using "Activation map" of CNN. The feature extraction unit 122b can improve the similarity regarding the same object shown in the input image and the plurality of images by calculating the feature vector of the calculated area.

The feature extraction unit 122b may calculate, for example, the degree of similarity between the feature vector of the input image and the feature vectors of a plurality of images using the Euclidean distance. That is, the feature extracting unit 122b calculates the degree of similarity by measuring the distance between the feature vector of the input image and the feature vectors of the plurality of images in m dimensions with m types of features as axes.

The learning unit 127 may set parameters for calculating the feature vector of the image in the feature extraction unit 122b. The learning unit 127 may cause the feature extraction unit 122b to learn processing of color-correcting a plurality of images and calculating feature vectors. Thereby, the search unit 121b can calculate a similarity higher than the similarity calculated without color correction.

The search result output unit 125b may output a predetermined image based on the degree of similarity. For example, the search result output unit 125b may arrange a plurality of predetermined images in descending order of similarity and display them on the monitor 11 . Thereby, the image search device 1b can improve the visibility.

The present embodiment configured in this manner has the same effects as those of the first and second embodiments. Furthermore, according to the present embodiment, the image search device 1b includes the search query input unit 120b and the feature extraction unit 122b that calculates a search query from an input image, so images similar to the input image can be searched. can be done. This allows the user to perform an image search by inputting an input image.

Further, the feature extraction unit 122b has a function of calculating a feature vector of an image, and a function of calculating a plurality of degrees of similarity between an input image and an image stored in the image data storage unit 123a for each image stored in the storage unit. and have The search unit has a function of searching the storage unit based on a plurality of degrees of similarity and extracting a predetermined image. Thus, the image retrieval device 1b can perform image retrieval based on the degree of similarity between the input image and the image data storage unit 123a.

Since this embodiment corresponds to modifications of the first to third embodiments, the differences from the first to third embodiments will be mainly described. FIG. 17 is a schematic diagram of the image search device 1c. The image retrieval device 1c in this embodiment performs an image retrieval using an input retrieval query, and rearranges and displays a plurality of predetermined images of the retrieval results based on the input image.

The image search device 1c includes a search query input unit 120c, a search unit 121c, a feature extraction unit 122c, an image data storage unit 123a, a feature data storage unit 124a, a search result output unit 125a, and an image data acquisition unit 126. , and a learning unit 127 .

The search query input unit 120c is a function that receives a search query and an input image from the user. The search query input unit 120c accepts a search query in text form. The search query input section 120c is a UI displayed on the monitor 11, for example. The search query input unit 120c is connected to the search unit 121c and the learning unit 127 so as to be unidirectionally communicable. The search query input section 120c will be described later with reference to FIG.

The feature extraction unit 122c has a function of calculating search data from an image and a function of calculating a feature vector from image data. The feature extraction unit 122c is connected to the image data storage unit 123c so as to be unidirectionally communicable. The feature extraction unit 122c is connected to the search unit 121c, the feature data storage unit 124a, and the learning unit 127 so as to be able to communicate bidirectionally.

The search unit 121c has a function of searching the storage unit 12 based on a search query and extracting a predetermined image corresponding to the search query. The search unit 121c is connected to the image data storage unit 123a, the feature data storage unit 124a, and the search result output unit 125a so as to be unidirectionally communicable. Processing of the search unit 121b and the feature extraction unit 122b will be described later with reference to FIG.

FIG. 18 is an explanatory diagram of the search query input section 120c. The search query input section 120c causes the monitor 11 to display an object name input field 1201, a relationship name input field 1202, a search button 1203, and an image input field 1204. FIG. In the object name input field 1201 and the relation name input field 1202, for example, object name information 1242 indicating "person" and "bag" and relation name information 1244d indicating "has" are input.

FIG. 19 is a flowchart of image search processing. The search unit 121c acquires an input image and a search query from the search query input unit 120c (S51). The search unit 121c searches for at least one predetermined image from the plurality of images (S52). The search unit 121c searches for a plurality of predetermined images from among the plurality of images, for example, in the same manner as the search processing (S12 to S15) (see FIG. 8) in the first embodiment.

The feature extraction unit 122c acquires data of the input image and data of a plurality of predetermined images from the search unit 121c. The feature extraction unit 121c calculates feature vectors included in the input image. The feature extraction unit 121c calculates feature vectors included in a predetermined image. The feature extraction unit 121c calculates the degree of similarity to the input image for each predetermined image based on the calculated feature vector (S53).

The feature extraction unit 122c transmits a plurality of similarity data to the search unit 121c (S54). The search unit 121c transmits data of a plurality of predetermined images and similarity data set to the predetermined images to the search result output unit 125c. The search result output unit 125c displays a plurality of predetermined images on the monitor 11 based on the degree of similarity (S55). For example, the search result output unit 125c preferentially displays on the monitor an image having a higher degree of similarity among the plurality of predetermined images than other images.

The present embodiment configured in this manner has the same effects as those of the first to third embodiments. Furthermore, according to this embodiment, the image search device 1c has the search query input unit 120c and the search unit 121c, so that the images searched by the search query can be rearranged and displayed. The image search device 1c can display images searched by the search unit 121c based on the degree of similarity with the input image. As a result, the image search device can improve the visibility of search results.

Since this embodiment corresponds to modifications of the first to fourth embodiments, differences from the first to fourth embodiments will be mainly described. FIG. 20 is an explanatory diagram of object information. The image search apparatus according to this embodiment calculates a lower keyword 1259 from the input upper keyword 1258 and performs an image search based on the lower keyword 1259 . The search unit 121d has processing for calculating a plurality of lower keywords 1259 from the upper keyword 1258 .

The high-level keyword 1258 is a keyword indicating characteristics of a given image. Top keywords 1258 are entered into the search query input by the user. A lower keyword 1259 is the name of an object that constitutes the upper keyword 1258 . It should be noted that the lower keyword may indicate the property of the object included in the predetermined image. Low level keywords 1259 may indicate the nature of high level keywords 1258 .

A user inputs, for example, "female employee" into the search query input section. The search unit 121d acquires data indicating "female employee" from the input unit.

The search unit 121d calculates a plurality of lower keywords 1259 corresponding to "female employee". The search unit 121d calculates data indicating, for example, "woman", "suit", "pumps" or "shoes". Note that the search unit 121d may estimate the lower keyword 1259 from the upper keyword 1258 based on the image search history.

The search unit 121d searches for at least one predetermined image by comparing the search data stored in the feature data storage unit with a plurality of lower keywords 1259 corresponding to "female employee". The search unit transmits data of a predetermined image to the search result output unit. The output unit displays a predetermined image.

The image retrieval apparatus shown in this embodiment can perform image retrieval using a plurality of lower keywords 1259 even when the user inputs one upper keyword 1258 . As a result, the image retrieval device improves the accuracy of image retrieval.

Since this embodiment corresponds to modifications of the first to fourth embodiments, differences from the first to fourth embodiments will be mainly described. The image retrieval apparatus according to the present embodiment performs image retrieval based on changes over time of a plurality of objects appearing in a moving image. The image retrieval apparatus of this embodiment will be described below by taking people 25 and 29 moving through the station 5 as an example.

21 is a schematic diagram of station 5. FIG. The station 5 is provided with an entrance 51, a ticket vending machine 52, and a ticket gate 53, for example. People 25 and 29 moving in the station 5 are photographed by equipment for photographing moving images such as surveillance cameras.

For example, the person 25 goes from the entrance 51 to the ticket vending machine 52 , purchases a ticket from the ticket vending machine 52 , and goes to the ticket gate 53 . After exiting the ticket gate 53 , the person 29 heads for the entrance 51 .

FIG. 22(1) is an explanatory diagram of information of persons 25 and 29 who move within the station 5 premises. The feature data accumulation unit 124 e has a “feature ID” 1261 , feature information 1262 , an earlier time feature 1263 and a later time feature 1264 . Information for identifying the feature information 1262 of the persons 25 and 29 is stored in the “feature ID” 1261 .

The feature information 1262 stores data indicating features of the persons 25 and 29 . The feature information 1262 stores, for example, "person getting on train" or "person getting off train". A pre-temporal feature 1263 and a post-temporal feature 1264 indicate the states of the persons 25 and 29 over time. That is, when the persons 25 and 29 change from the state of "having a wallet" to the state of "going from the ticket vending machine to the ticket gate", the persons 25 and 29 have the characteristics of "the person who gets on the train".

FIG. 22(2) is a diagram showing the details of the previous time feature 1263. FIG. The previous temporal feature 1263 has a “previous feature ID” 12631 , a first object 12632 , a second object 12633 and a relationship 12634 . Information for identifying the previous temporal feature 1263 is stored in the “previous feature ID” 12631 .

Object information is stored in the first object 12632 and the second object 12633 . The first object 12632 stores, for example, "person". The second object 12633 stores, for example, "purse" or "ticket gate".

Information on the relationship between the first object 12632 and the second object 12633 is saved in the relationship 12634 . Relationship 12634 stores, for example, "has" or "passes".

FIG. 22(3) is a diagram showing the details of the post-time features 1264. FIG. Post-temporal feature 1264 has 'post-feature ID' 12641 , first object 12642 , second object 12643 , and relationship 12644 . Information for identifying the post-temporal feature 1264 is stored in the “posterior feature ID” 12641 .

Object information is stored in the first object 12642 and the second object 12643 . The first object 12642 stores, for example, "person". The second object 12643 stores, for example, "ticket" or "entrance". The relationship 12644 stores the relationship between the first object 12642 and the second object 12643 . The relationship 12644 stores, for example, "has" or "passes".

The user inputs, for example, "person on train" to the input unit. The search unit acquires data indicating "a person on the train" from the input unit. The search unit searches for persons 25 and 29 who change from the state of "having a wallet" to the state of "going from the ticket vending machine to the ticket gate" in the moving image of the station 5.例文帳に追加

In the position information 291, the search unit recognizes that the person 25 is in a state of "having" a "wallet". In the position information 292, the search unit recognizes that the person 25 is in a state of "having" a "ticket". Since the person 25 moves from the position information 291 to the position information 292, the search unit determines the person 25 to be "a person who gets on the train".

The search unit extracts a predetermined image including the person 25 from the moving image data stored in the image data storage unit as a predetermined image. The search unit transmits a predetermined image to the output unit. The search result output unit displays a predetermined image on the monitor.

The image retrieval apparatus shown in this embodiment can retrieve a predetermined image based on objects that change with time or relationships between objects.

Since this embodiment corresponds to modifications of the first to fourth embodiments, differences from the first to fourth embodiments will be mainly described. The image retrieval device 1f in this embodiment has an image data storage unit 123 that is subjected to image retrieval by a plurality of users and is isolated for each user. FIG. 23 is a hardware configuration diagram of the image search device 1f. A plurality of terminals 6(1) to 6(n) (where n is an arbitrary integer) are connected to the image retrieval device 1f via a network 7. FIG. Terminals 6(1) to 6(n) may be referred to as terminal 6 when they are not distinguished from each other.

The image search device 1f has a storage unit 12f, a CPU 14, a memory 15, a communication interface 16, and a data transmission line 17. FIG. The storage unit 12f has a search unit 121, a feature extraction unit 122, an image data acquisition unit 126, and terminal databases 128(1) to 128(n). The terminal databases 128(1) to 128(n) may be referred to as the terminal database 128 when they are not distinguished from each other. The image search device 1f can be provided on one or more computers provided on a network, as is known as a cloud.

The terminal database 128 is a database that stores image data and search data for each terminal 6 . By isolating the data stored in each terminal database 128, users are restricted from accessing from terminals 6 other than the corresponding terminal 6. FIG. For example, terminal database 128(1) is accessible from terminal 6(1) and has restricted access from terminal 6(2). The terminal database 128 has image data storage units 123(1) to 123(n) and feature data storage units 124(1) to 124(n).

The terminal 6 is a computer having communication interfaces (indicated as communication I/F in the figure) 61(1) to 61(n), a search query input unit 120, and a search result output unit 125. The communication interfaces 61(1) to 61(n) may be referred to as the communication interface 61 unless otherwise distinguished.

The communication interface 61 is, for example, a LAN connection terminal, a SAN connection terminal, or a wireless communication connection device. The search query input section 120 and search result output section 125 are stored in the storage section of each terminal 6 .

A user uses, for example, terminal 6(1) of a plurality of terminals 6 . The user enters a search query into search query input section 125 . The terminal 6(1) transmits information identifying the search query and the terminal (1) to the image search device 1f via the communication interface 61(1).

The image search device 1f performs an image search using the search unit 121 based on the acquired search query. The search unit 121 selects the terminal database 128(1) based on the information identifying the terminal (1). The search unit 121 stores data in the image data storage unit 123(1) based on the search data stored in the feature data storage unit 124(1) and the search query acquired from the terminal 6(1). A predetermined image is retrieved from a plurality of images.

The image search device 1f transmits a predetermined image to the terminal 6(1) via the communication interface 16. FIG. The search result output unit 125 outputs the acquired predetermined image.

The image retrieval device 1f shown in this embodiment can be used by a plurality of terminals 6 in common. This allows multiple users to use the image search device 1f.

Since the image data and search data stored in the storage unit 12 are isolated for each terminal database 128, the image search device 1f can be used by a plurality of users with different attributes, but the data cannot be sent to other users. can be hidden.

Since this embodiment corresponds to modifications of the first to fourth embodiments, differences from the first to fourth embodiments will be mainly described. The image search system in this embodiment has a function of allowing the user to search for images from video data of a plurality of surveillance cameras 9 . FIG. 23 is an explanatory diagram of the image search system 8. As shown in FIG. The image retrieval system 8 has image retrieval devices 81, 82(1) to 82(n). "p" is an arbitrary constant. The image retrieval devices 82(1) to 82(p) may be referred to as the image retrieval device 82 when they are not distinguished from each other. The image retrieval device 82 is generated by the image retrieval device 81 additionally learning data for each user.

Each image retrieval device 82 is connected to a plurality of terminals 6 and monitoring cameras 9(1) to 9(q) via the network 7 so as to be able to communicate bidirectionally. "q" is an arbitrary constant. Surveillance cameras 9(1) to 9(q) may be referred to as surveillance camera 9 unless otherwise distinguished. It should be noted that the monitoring camera 9 may be another device for capturing moving images.

By operating the terminal 6 , the user searches for a predetermined image from the imaged data of the surveillance camera 9 . That is, the user selects terminal 6(1), for example. The user operates the terminal 6 ( 1 ) to select one of the surveillance cameras 9 . The user selects surveillance camera 9(1), for example.

The user searches for a predetermined image from the photographed data of the surveillance camera 9 ( 1 ) using the image search device 82 . In this embodiment, the image retrieval device 82 corresponding to the user is, for example, image retrieval device 82(1). The image retrieval device 82(1) performs image retrieval by acquiring photographed data from the monitoring camera 9(1). The image retrieval device 82(1) transmits the image of the retrieval result to the terminal 6(1). Terminal 6(1) outputs an image of the search result.

The image search system 8 shown in the present embodiment is connected to each terminal 6 and each monitor camera 9 via the network 7, so that the user can perform image search using the photographed data of each monitor camera.

In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. The above embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the described configurations. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Moreover, the configuration of another embodiment can be added to the configuration of one embodiment. Also, a part of the configuration of each embodiment can be added, deleted, or replaced with another configuration.

Some or all of the above configurations, functions, processing units, processing means, etc. may be realized by hardware, for example, by designing integrated circuits. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in recording devices such as memories, hard disks, SSDs (Solid State Drives), and recording media such as IC cards, SD cards, and DVDs.

Moreover, the technical features included in the above-described embodiments are not limited to the combinations specified in the claims, and can be combined as appropriate.

REFERENCE SIGNS LIST 1 image search device 2 image 120 search query input unit 121 search unit 122 feature extraction unit 123 image data storage unit 124 feature data storage unit 125 search result output unit 126 …Image data acquisition unit

Claims (15)

An image retrieval device,
a feature extraction unit that calculates search data including object name information of a plurality of objects included in an image and relationship name information indicating a predetermined relationship between the plurality of objects included in the image;
a storage unit that stores the search data in association with the image;
an input unit that receives input of at least one of the object name information and the relationship name information as a search query;
a search unit that searches the storage unit based on the search query and extracts a predetermined image corresponding to the search query;
and an output unit that outputs search results,
The input unit
a function of accepting the search query as text;
and a function of accepting an input image containing a plurality of objects,
The search unit has a function of extracting from the storage unit a plurality of predetermined images corresponding to the search query received in the form of characters,
The feature extraction unit is
A function of calculating a feature vector indicating features of a plurality of objects included in the image and features of relationships between the plurality of objects included in the image;
a function of calculating a plurality of degrees of similarity to the input image for each predetermined image based on the feature vector calculated from the predetermined image and the feature vector calculated from the input image;
Further, the image retrieval device, wherein the output unit has a function of outputting the predetermined image based on the degree of similarity. An image retrieval device,
a feature extraction unit that calculates search data including object name information of a plurality of objects included in an image and relationship name information indicating a predetermined relationship between the plurality of objects included in the image;
a storage unit that stores the search data in association with the image;
an input unit that receives input of at least one of the object name information and the relationship name information as a search query;
a search unit that searches the storage unit based on the search query and extracts a predetermined image corresponding to the search query;
an output unit that outputs search results;
a learning unit that learns parameters for calculating the search data from the image,
Furthermore, the feature extraction unit has the parameter,
Furthermore, in the storage unit,
a learning image used when the learning unit learns;
and teacher data indicating search data included in the learning image are stored,
The learning unit
A function of calculating, as pre-learning data, the feature extraction unit before parameter update, using search data included in the learning image;
a function of learning updated values of the parameters based on the teacher data and the pre-learning data;
and a function of updating the parameters of the feature extraction unit based on updated values of the parameters. An image retrieval device,
a feature extraction unit that calculates search data including object name information of a plurality of objects included in an image and relationship name information indicating a predetermined relationship between the plurality of objects included in the image;
a storage unit that stores the search data in association with the image;
an input unit that receives input of at least one of the object name information and the relationship name information as a search query;
a search unit that searches the storage unit based on the search query and extracts a predetermined image corresponding to the search query;
and an output unit that outputs search results,
the object name information included in the search query indicates a plurality of predetermined objects;
The output unit has a function of displaying the predetermined object included in the predetermined image in a more emphasized manner than other objects,
The related name information included in the search query indicates predetermined related name information,
other relationship name information indicates the relationship between the predetermined object and the other object;
Furthermore, the output unit
a function of displaying the other object and the other related name information;
a function of displaying the plurality of predetermined objects spaced apart from each other;
a function of displaying the predetermined relationship name information between the plurality of predetermined objects to be displayed;
a function of displaying the other object spaced apart from the predetermined object to be displayed;
a function of displaying the other related name information between the displayed predetermined object and the displayed other object;
An image search device comprising the predetermined object, the predetermined related name information, the other object, the other related name information, and a function of displaying them side by side in the scroll direction of the screen. 4. The image retrieval apparatus according to claim 1, wherein said predetermined relationship indicates a physical relationship between said objects. The feature extraction unit is
a function of calculating position information of each of the plurality of objects included in the image;
4. The image retrieval device according to claim 1, further comprising a function of calculating said related name information based on said position information. 4. The image retrieval apparatus according to claim 2, wherein said input unit has a function of accepting said retrieval query in characters. The input unit
a function of displaying a plurality of object name input fields for receiving the object name information and at least one relationship name input field for receiving the relationship name information;
7. The image retrieval device according to claim 1, further comprising a function of displaying the plurality of object name input fields and the at least one relationship name input field side by side in the scroll direction of the screen. The input unit has a function of accepting an input image containing a plurality of objects,
The feature extraction unit has a function of calculating a search query from the input image,
4. The image search device according to claim 1, wherein the search unit has a function of extracting from the storage unit a predetermined image corresponding to a search query calculated from the input image. Furthermore, the feature extraction unit
A function of calculating a feature vector indicating features of a plurality of objects included in the image and features of relationships between the plurality of objects included in the image;
a function of calculating a plurality of degrees of similarity to the input image for each image stored in the storage unit based on the feature vector calculated from the input image and the feature vector calculated from the image stored in the storage unit; , has
9. The image search device according to claim 8, wherein the search unit has a function of searching the storage unit based on a plurality of similarities and extracting a predetermined image. calculating search data including object name information of a plurality of objects included in an image and relationship name information indicating a predetermined relationship between the plurality of objects included in the image;
a step of storing the search data in association with an image;
a step of inputting at least one of the object name information and the related name information in characters as a search query, and inputting an input image including a plurality of objects;
searching a storage unit based on the search query;
a step of extracting a plurality of predetermined images from the storage unit corresponding to the search query accepted in the form of characters;
and outputting search results;
The step of calculating the search data includes:
calculating a feature vector indicating features of a plurality of objects included in the image and features of relationships between the plurality of objects included in the image;
calculating a plurality of degrees of similarity to the input image based on the feature vector calculated from the predetermined image and the feature vector calculated from the input image;
Further, in the image retrieval method, the step of outputting the retrieval result has a function of outputting the predetermined image based on the degree of similarity. calculating search data including object name information of a plurality of objects included in an image and relationship name information indicating a predetermined relationship between the plurality of objects included in the image;
a step of storing the search data in association with an image;
at least one of the object name information and the relationship name information is input as a search query;
searching a storage unit based on the search query;
extracting a predetermined image corresponding to the search query;
a step of outputting search results;
learning parameters for calculating the search data from the image;
Furthermore, in the storage unit,
a learning image used when a learning unit that learns the parameter learns;
and teacher data indicating search data included in the learning image are stored,
The step of learning the parameters includes:
calculating the search data included in the learning image as pre-learning data before updating the parameters;
learning updated values of the parameters based on the teacher data and the pre-learning data;
and updating the parameters based on updated values of the parameters. calculating search data including object name information of a plurality of objects included in an image and relationship name information indicating a predetermined relationship between the plurality of objects included in the image;
a step of storing the search data in association with an image;
at least one of the object name information and the relationship name information is input as a search query;
searching a storage unit based on the search query;
extracting a predetermined image corresponding to the search query;
and outputting search results;
the object name information included in the search query indicates a plurality of predetermined objects;
The related name information included in the search query indicates predetermined related name information,
other relationship name information indicates the relationship between the predetermined object and the other object;
Furthermore, the step of outputting the search results includes:
a step of displaying the predetermined object included in the predetermined image in a more emphasized manner than other objects;
a step of displaying the other object and the other related name information;
displaying the plurality of predetermined objects separately;
a step of displaying the predetermined relationship name information between the plurality of predetermined objects to be displayed;
displaying the other object spaced apart from the predetermined object to be displayed;
displaying the other related name information between the displayed predetermined object and the displayed other object;
and displaying the predetermined object, the predetermined related name information, the other object, and the other related name information side by side in a screen scroll direction. A computer program for causing a computer to function as an image retrieval device,
on the computer,
a feature extraction unit that calculates search data including object name information of a plurality of objects included in an image and relationship name information indicating a predetermined relationship between the plurality of objects included in the image;
a storage unit that stores the search data in association with an image;
an input unit that receives at least one of the object name information and the related name information as a search query in the form of characters, and receives an input image that includes a plurality of objects;
A search unit that searches the storage unit based on the search query, extracts a predetermined image corresponding to the search query, and extracts a plurality of predetermined images from the storage unit that correspond to the search query received in the form of characters. and,
Realize an output unit that outputs the search results, and
In the input section,
a function of accepting the search query as text;
and executing a function of accepting an input image containing a plurality of objects,
causing the search unit to execute a function of extracting a plurality of predetermined images from the storage unit corresponding to the search query received in the form of characters;
In the feature extraction unit,
A function of calculating a feature vector indicating features of a plurality of objects included in the image and features of relationships between the plurality of objects included in the image;
a function of calculating a plurality of degrees of similarity to the input image for each predetermined image based on the feature vector calculated from the predetermined image and the feature vector calculated from the input image;
Further, the computer program for causing the output unit to execute a function of outputting the predetermined image based on the degree of similarity. A computer program for causing a computer to function as an image retrieval device,
on the computer,
a feature extraction unit having parameters for calculating search data including object name information of a plurality of objects included in an image and relationship name information indicating a predetermined relationship between the plurality of objects included in the image; ,
a storage unit that stores the search data in association with an image;
an input unit that receives input of at least one of the object name information and the relationship name information as a search query;
a search unit that searches the storage unit based on the search query and extracts a predetermined image corresponding to the search query;
an output unit that outputs search results;
and a learning unit that learns the parameters,
Furthermore, in the storage unit,
a learning image used when the learning unit learns;
storing teacher data indicating search data included in the learning image;
to the learning unit,
A function of calculating, as pre-learning data, the feature extraction unit before parameter update, using search data included in the learning image;
a function of learning updated values of the parameters based on the teacher data and the pre-learning data;
A computer program for executing a function of updating the parameter of the feature extraction unit based on the updated value of the parameter. A computer program for causing a computer to function as an image retrieval device,
on the computer,
a feature extraction unit that calculates search data including object name information of a plurality of objects included in an image and relationship name information indicating a predetermined relationship between the plurality of objects included in the image;
a storage unit that stores the search data in association with an image;
an input unit that receives input of at least one of the object name information and the relationship name information as a search query;
a search unit that searches the storage unit based on the search query and extracts a predetermined image corresponding to the search query;
Realize an output unit that outputs the search results, and
the object name information included in the search query indicates a plurality of predetermined objects;
causing the output unit to display the predetermined object included in the predetermined image in a more emphasized manner than other objects;
The related name information included in the search query indicates predetermined related name information,
other relationship name information indicates the relationship between the predetermined object and the other object;
Furthermore, in the output unit,
a function of displaying the other object and the other related name information;
a function of displaying the plurality of predetermined objects spaced apart from each other;
a function of displaying the predetermined relationship name information between the plurality of predetermined objects to be displayed;
a function of displaying the other object spaced apart from the predetermined object to be displayed;
a function of displaying the other related name information between the displayed predetermined object and the displayed other object;
A computer program for executing the predetermined object, the predetermined related name information, the other object, the other related name information, and a function of displaying them side by side in the scroll direction of the screen.

Images