JP2016162414A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To create a search query capable of improving search accuracy.SOLUTION: An image processing device specifies a person area containing a person, in an input image, then divides the person area into plural partial areas, and divides each of plural partial areas into plural small areas. Then, the image processing device performs clustering of the plural small areas, for forming clusters, in each of plural partial areas. In each of the plural partial areas, based on attribute of the clusters, a cluster of a query candidate is selected from the clusters. In each of the plural partial areas, a query element is created from the small area of the selected cluster, then by combining the query elements of the plural partial areas, a search query for searching for the person is created.SELECTED DRAWING: Figure 1

Description

  The present invention relates to generating search queries for images.

  When searching for a person in image data or video data, select the person's clothes and characteristics of belongings one by one, specify multiple search queries, and search the person by integrating the search results of all queries. A method has been proposed. When a user specifies a search query, visual confirmation is required to select a feature such as clothing color, and it is often determined by the user's subjectivity. The manually selected query has a problem that the input contents vary depending on the user and the target person cannot be found.

  Patent Document 1 discloses a system that automatically determines a search query. Specifically, “an image retrieval system for retrieving an image similar to an image designated by a user, wherein an image acquired from a video or the like taken by a camera is divided into blocks of a predetermined size, and a color histogram Based on the block, the feature amount of the block is extracted, and first information indicating the correspondence between the identifier of the image, the information indicating the position of the block, and the feature amount of the block is generated, Based on this information, at least one block selected by the user among the divided blocks is used as a first search key, and the feature quantity of the first search key is compared with the feature quantity of another block. , A predetermined number of blocks similar to the first search key are detected as search results, and an image including the search results is displayed to the user ”(summary).

JP2011-18238

  In the known method for automatically registering human features and determining search queries, the area estimated to be a person is divided into a plurality of areas, and all the areas are used. There is a problem that the search accuracy becomes worse due to a mixture of objects. In human detection based on background subtraction, areas such as illumination fluctuations and human shadows simply remain in places where the environment is bad, and it is difficult to separate the background using this method alone.

  Therefore, a search query generation method that can improve the search accuracy is desired.

  A typical example of the present invention is an image processing device that includes a memory device that stores a program, an arithmetic device that operates according to the program, and an input / output device, and the arithmetic device detects a person in an input image. A human region including the human region is identified, the human region is divided into a plurality of partial regions, each of the plurality of partial regions is divided into a plurality of small regions, and the plurality of small regions are clustered in each of the plurality of partial regions. In each of the plurality of partial regions, a query candidate cluster is selected from the cluster based on the attribute of the cluster, and in each of the plurality of partial regions, a query is performed from a small region of the selected cluster. Generate an element and combine the query elements of the plurality of partial areas to generate a search query for searching for the person .

  According to one embodiment of the present invention, a search query that improves search accuracy can be generated. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

1 shows a configuration example of an image search device. The other structural example of an image search device is shown. The flowchart of the process which produces | generates a search query from the person image designated in the input image, and performs a person search using the produced | generated search query is shown. The flowchart of the query setting process in the flowchart shown to FIG. 3A is shown. The flowchart of the query production | generation process in the flowchart shown to FIG. 3B is shown. An example from when a person area is determined by a user until a search query is set is shown. An example of the relationship between the arrangement of small areas in a partial area and the arrangement of small areas in a search query is shown. The example of the cluster produced | generated from one partial area is shown. The cluster selected as a candidate for search query generation, and the cluster removed as a noise cluster are shown. An example of a noise reference area in the feature amount space is shown. An example of a humanoid mask is shown. An example of a block type mask is shown. An example in which a mask is formed along a person image in a person area will be described. A process of selecting one mask from a plurality of masks in the division of a person area into partial areas will be described. The navigation image for determination of a person area and a mask is shown.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. It should be noted that this embodiment is merely an example for realizing the present invention, and does not limit the technical scope of the present invention. In each figure, the same reference numerals are given to common configurations.

  FIG. 1 shows a configuration example of an image search apparatus that is an image processing apparatus. The image search apparatus 101 has a general computer hardware configuration. Specifically, the image search apparatus 101 includes an interface (I / F) 190, a CPU 191, a memory 192, and an input / output device 193, which are interconnected by a bus.

  The I / F 190 is connected to a network, receives data from an external device via the network, and transmits data to the external device. The input / output device 193 includes an input device for a user to give an instruction to the image search device 101 and an output device for outputting information to the user. The input device is, for example, a keyboard, a pointing device, a touch panel, and the like, and the output device is, for example, a monitor display, a printer, or the like.

  The memory 192 stores a program executed by the CPU 191. Specifically, the memory 192 stores an image input program 111, a person area specifying program 112, a partial area dividing program 121, a small area specifying program 122, an image search program 131, and a search result display program 132. The memory 192 further stores an image database 141.

  The image search device 101 may include an auxiliary storage device (not shown) including a nonvolatile storage medium, and may store programs and data loaded into the memory 192. The program and data stored in the memory 192 may be acquired from an external device via the I / F 190.

  The CPU 191 that is an arithmetic device operates according to a program stored in a memory 192 that is a memory device. The program is executed by the CPU 191 to perform a predetermined process. Therefore, in the present disclosure, the description with the program as the subject may be an explanation with the CPU (processor) as the subject. Alternatively, the process executed by the program is a process performed by a computer and a computer system on which the program operates.

  The processor operates as a functional unit (means) that realizes a predetermined function by operating according to a program. For example, the CPU 191 functions as a person region specifying unit, a partial region dividing unit, and a small region specifying unit by operating according to the person region specifying program 112, the partial region dividing program 121, and the small region specifying program 122, respectively. The above points are the same for other programs and other computers.

  The image input program 111 acquires image data 100 for generating a search query from an external device or the image database 141. The image data 100 acquired by the image input program 111 is a still image or a moving image. The image input program 111 outputs the acquired image data 100 to the person area specifying program 112.

  The person area specifying program 112 extracts still image data from the image data 100 and specifies a person area including the entire image of one person. The person area specifying program 112 outputs the data of the specified person area to the partial area dividing program 121.

  The partial area dividing program 121 divides the person area into a plurality of partial areas, and outputs data of each of the plurality of partial areas to the small area specifying program 122. The small area specifying program 122 divides each of the plurality of partial areas into a plurality of small areas. Further, the small region specifying program 122 clusters the small regions in each partial region, and selects a small region to be used for the query in each partial region based on the clustering result. By dividing into partial areas, noise can be removed more appropriately by clustering.

  The small area specifying program 122 outputs the data of the small area selected in each partial area to the small area integration program 123. The small area integration program 123 integrates the small areas selected for the query in each partial area, and generates a query element corresponding to each partial area. The image search query is composed of query elements in all partial areas.

  The image search program 131 performs an image search in the image database 141 using the generated search query, and outputs the search result to the search result display program 132. The search result display program 132 generates image data indicating the search result, and displays the search result on the display device of the input / output device 193.

  The image database 141 is generated from a person image in the image database 141 and stores data having the same configuration as the search query. This data is also called a search entry. The image input program 111 outputs a plurality of still images acquired from the image database 141 to the person area specifying program 112. The person area specifying program 112 extracts a person image in each still image according to a predetermined person detection algorithm.

  The partial area dividing program 121, the small area specifying program 122, and the small area integration program 123 generate a search entry from each of the extracted person images by the same method as the search query. The image database 141 stores the generated search entry in association with the person image. The image search program 131 searches the person image indicated by the search query by comparing the search query with the search entry in the image database 141.

  As described above, since the search entry can be generated by the same method as the search query, the generation of the search query will be described below. FIG. 2 shows another configuration example of the image search apparatus 101. The image search apparatus 101 includes a plurality of computers. In the example of FIG. 2, the image search apparatus 101 includes a plurality of user operation units 102, a query generation unit 103, and a search unit 104, which are interconnected via a network (not shown).

  The user operation unit 102 receives the image data 100 for generating a search query, further accepts input about the search query from the user, and presents the search result to the user. The query generation unit 103 generates a search query from the person image extracted from the image data 100. The search unit 104 searches the image database 141 according to the search query, and transmits the search result to the user operation unit 102.

  Each of the user operation unit 102, the query generation unit 103, and the search unit 104 is configured by a single computer. A plurality of users can use the image search apparatus 101 by a plurality of user operation units 102 (computers). Efficient search query creation and search processing can be realized by distributed processing by a plurality of computers.

  The user operation unit 102, the query generation unit 103, and the search unit 104 have a general computer configuration. Specifically, the user operation unit 102 includes a CPU 201, a memory 202, an I / F 203, and an input / output device 204. The query generation unit 103 includes a CPU 205, a memory 206, and an I / F 207. The search unit 104 includes a CPU 208, a memory 209, and an I / F 210. The CPUs 201, 205, and 208 are arithmetic devices in the image search device 101, and the memories 202, 206, and 209 are memory devices in the image search device 101.

  The memory 202 of the user operation unit 102 stores an image input program 111, a person area specifying program 112, and a person area transmission program 151. The operations of the image input program 111 and the person area specifying program 112 are as described above. The person area transmission program 151 acquires the image data of the person area from the person area specifying program 112 and transmits it to the query generation unit 103 via the I / F 203.

  The memory 206 of the query generation unit 103 stores a person area reception program 152, a partial area division program 121, a small area identification program 122, a small area integration program 123, and a query transmission program 153. The person area receiving program 152 receives image data of the person area from the user operation unit 102 via the I / F 207 and outputs the image data to the partial area dividing program 121.

  The operations of the partial area division program 121, the small area specifying program 122, and the small area integration program 123 are as described above. The query transmission program 153 acquires the search query generated from the small area integration program 123 and transmits it to the search unit 104 via the I / F 207.

  The memory 209 of the search unit 104 stores a query reception program 154, an image search program 131, a search result display program 132, and an image database 141. The query reception program 154 receives a search query from the query generation unit 103 via the I / F 210 and outputs it to the image search program 131.

  The operations of the image search program 131 and the search result display program 132 and the data stored in the image database 141 are as described above. The search result display program 132 transmits the search result to the user operation unit 102, and the user operation unit 102 presents the search result to the user in the input / output device 204.

  FIG. 3A shows a flowchart of a process for generating a search query from a person image specified in an input image and executing a person search using the generated search query. FIG. 3B shows a flowchart of the query setting process S12 in the flowchart shown in FIG. 3A. FIG. 3C shows a flowchart of the query generation process S22 in the flowchart shown in FIG. 3B. In the following description, the configuration of the image search apparatus 101 shown in FIG. 1 is referred to.

  In FIG. 3A, the image input program 111 acquires new image data 100 from the external device or the image database 141 (S11). The image input program 111 displays a still image in the input image data 100 on the input / output device 193. A search query setting process is executed for the displayed image (S12). The query setting process S12 will be described later with reference to FIG. 3B.

  The image search program 131 executes a search process using the set search query (S13). The image search program 131 searches the person database similar to the search query from the image database 141. The image search program 131 calculates the similarity between the search query and the person image, and selects image data whose similarity is greater than a threshold value. The similarity can be calculated from the distance between the search query and the person image search entry in the feature amount space.

  The search result display program 132 displays, on the input / output device 193, the person image selected in the search process S13 or the still image or moving image including the person image in descending order of similarity to the search query.

  The user refers to the search result displayed by the input / output device 193. In the displayed still image or moving image, when there is a person to be further searched (S15: YES), the user designates the person from the input / output device 193. The input / output device 193 transmits the user designation to the person area specifying program 112. Steps S12 to S15 are repeated for the designated person image. If no new person is designated (S15: NO), this process ends.

  The search query setting process S12 will be described with reference to FIG. 3B. The person area specifying program 112 receives the user designation of the person area including the person in the image displayed on the input / output apparatus 193 from the input / output apparatus 193 (S21).

  For example, the user uses a mouse or touch panel of the input / output device 193 to designate a person area by surrounding a region including a target person with a rectangle in the displayed image. Alternatively, the person area specifying program 112 may indicate the person area in the display image according to a predetermined person detection algorithm and accept the selection of the person area from the input / output device 193 by the user.

  The person area specifying program 112 outputs image data of the specified person area to the partial area dividing program 121. The partial area dividing program 121, the small area specifying program 122, and the small area integrating program 123 generate a search query from the image data of the person area (S22). The small area integration program 123 sets the generated search query in the image search program 131 (S23).

  The search query generation process S22 will be described with reference to FIG. 3C. The partial area dividing program 121 divides the person area into a plurality of partial areas (S31). The small area specifying program 122 divides each partial area into small areas (S32). The small area specifying program 122 groups small areas by clustering in each partial area. Further, the small region specifying program 122 removes unnecessary clusters corresponding to noise including the background from the generated clusters (S34). The small area specifying program 122 determines whether or not the cluster is a noise cluster based on the cluster attributes such as cluster size (number of small areas), color, and texture.

  The small area integration program 123 integrates the small areas remaining after noise removal in each partial area using statistical values such as the representative value, center value, and average value of the feature amount, and corresponds to each partial area. Generate query elements. The small area integration program 123 generates a search query by combining query elements of all partial areas.

  Details of the processing steps described with reference to FIGS. 3A to 3C will be described below. FIG. 4A shows an example from when the person area is determined by the user until the search query is set. The description will be divided into the user interface F01 and the internal processing F02. The user interface F01 displays the input image F11. The user interface F01 is provided by the input / output device 193.

  In the input image F11, a person area F12 is designated. The user determines a target person and designates the person area F12 on the input image F11 by drawing a rectangle using a pointing device, for example.

  The person area specifying program 112 acquires the person area F2 designated by the user from the user interface F0. As described above, the person area specifying program 112 may detect a person in the input image F11 by the hourly object detection algorithm and present a person area surrounding each person. The user selects a person area including the target person from the presented person areas.

  The person area specifying program 112 detects a person using a discriminator learned based on a machine learning method. The discriminator learns a reference for identifying a person from feature quantities extracted from a large number of samples. The person area specifying program 112 sets a detection window in the input image F11, extracts a feature amount while moving the detection window in the input image F11, and detects a person.

  For example, a luminance gradient direction histogram (HOG) that is a feature amount related to edge information can be used as a feature amount for human detection. In addition, feature amounts based on color information and motion information may be used. Various person detection methods are known, and detailed description thereof is omitted here.

  A person area F12 given to the internal process F02 is an area cut out from the input image F11. The person area F12 may be enlarged / reduced to a prescribed size. The partial area dividing program 121 divides the person area F12 into a plurality of partial areas F13. In the example of FIG. 4A, the area is divided into a partial area F13 of Top, Upper, and Bottom corresponding to the person area F12, the head, upper body, and lower body parts.

  For example, the partial area dividing program 121 divides the person area F12 into three partial areas F13 in the vertical direction at a preset ratio. By dividing into three parts in the vertical direction, the head, Tops, and Bottoms can be individually included in each of the partial regions F13. The feature quantities of these parts often differ greatly, and the feature quantities within each part are often similar.

  The adjacent partial regions F13 may partially overlap each other or may be completely different regions. By partially overlapping the adjacent partial regions F13, the entire regions can be more reliably included in the partial region F13.

  The division method shown in FIG. 4A is an example, and the person area F12 may be divided by other methods. The person area F12 may be divided in the left-right direction, or may be divided into two or more partial areas F13. In the example of FIG. 4A, the partial region F13 is rectangular, but the shape of the partial region F13 is not limited to this.

  Next, the small area specifying program 122 divides each of the partial areas F13 into a plurality of small areas F14. For example, the shape and size of the small region F14 are determined in advance. The small area specifying program 122 may determine the size of the person area F12 according to the size of the person area F12. The size or shape of the small region F14 may be set for each partial region F13. In the example of FIG. 4A, the small region F14 is rectangular, but the shape of the small region F14 is not limited to this.

  The partial area F13 and the corresponding small area F14 are stored in the memory 192 in association with each other. The small area specifying program 122 groups the divided small areas F14 by clustering in each partial area F13. The small region specifying program 122 calculates the feature vector of each small region F14 and performs clustering in the vector space of the feature vector. As the feature amount, a value obtained from color information or edge information can be used. For example, the feature amount may include an average color determined from the RGB values of the pixels in the small region F14.

  For clustering of the small area F14, the small area specifying program 122 can employ any clustering algorithm. For example, the small area specifying program 122 may use a k-average method. The small area specifying program 122 randomly arranges temporary cluster center points of the determined number of clusters n in the feature amount space, and associates each point with the closest temporary cluster center point. A point associated with each temporary cluster center point constitutes each temporary cluster.

  The small region specifying program 122 further calculates the center of gravity (coordinate average) in each temporary cluster, determines these as new temporary cluster center points, and generates a new temporary cluster based on the new temporary cluster center point. The small region specifying program 122 repeats the generation of the temporary cluster center point and the temporary cluster, and determines the final cluster.

  The small region specifying program 122 removes a cluster corresponding to noise from the generated cluster and selects a candidate cluster to be used for generating a search query. FIG. 5A shows an example of a cluster generated from one partial region F13. Three clusters are generated, cluster 1 is composed of three small areas A to C, cluster 2 is composed of one small area D, and cluster 3 is composed of two small areas E and F. .

  The small area specifying program 122 removes the noise cluster based on the number of small areas constituting the cluster. Thereby, a noise class can be excluded appropriately. For example, when the number of constituent small areas of the cluster is equal to or less than the threshold, the cluster is determined as a noise cluster. FIG. 5B shows a cluster selected as a candidate for generating a search query (query candidate cluster) and a cluster removed as a noise cluster. In this example, a cluster having two or more small regions is selected as a candidate for generating a search query.

  The small area specifying program 122 may accept a user setting of a threshold value for noise determination. The user inputs a threshold value via the input / output device 193, and the small region specifying program 122 removes noise clusters according to the threshold value.

  The small region specifying program 122 may select a predetermined number of clusters as candidates for generating a search query from clusters having a large number of constituent small regions, and may be determined in advance from clusters having a small number of constituent small regions. A number of clusters may be removed as noise.

  In another example, the small region specifying program 122 may remove a noise cluster based on the position of the cluster in the feature space. The position of the cluster in the feature space indicates the attribute of the cluster. The small region specifying program 122 determines a noise reference region to which the centroid of the noise cluster belongs from the learning data. FIG. 5C shows an example of the noise reference region 501 in the feature amount space. The small area specifying program 122 calculates the centroid position of the cluster, and when the centroid position is included in the noise reference area 501, the small area specifying program 122 determines the cluster as a noise cluster.

  The small area specifying program 122 may determine the noise cluster based on the distance from the noise reference area 501. For example, the small region specifying program 122 may select a predetermined number of clusters as candidates for generating a search query from the clusters having the shortest distance between the noise reference region 501 and the gravity center position.

  The small region specifying program 122 may determine a noise cluster based on a distance from a specific point in the noise region. For example, the small region specifying program 122 may select a predetermined number of clusters as candidates for generating a search query from clusters having a large distance between a specific point in the noise region and the center of gravity position. The small region specifying program 122 may prepare a plurality of noise reference points, and determine the noise cluster based on the distance between each point constituting the cluster and each noise reference point.

  The small area specifying program 122 may remove a predetermined part as noise in the person area F12. For example, a part of the edge area in the person area F12 is removed as a noise area.

  Returning to FIG. 4A, in the region F15, the small region F14 belonging to the noise cluster is marked with a cross. The small region integration program 123 calculates feature vectors of color information and texture information from all or part of the small region F14 selected as a query candidate in each partial region F13. The feature vector is a query element F16 corresponding to the partial region in the search query F17. The value of each element of the feature vector is a representative value, a center value, an average value, etc. of the feature amount of the small region F14.

  The small region integration program 123 may generate a query element F16 by selecting some query candidate small regions F14 from the partial region F13 and spatially arranging them. In FIG. 4A, a rectangular image composed of four small regions F14 is a query element F16. A person search (image search) is performed using the feature vectors of color information and texture information calculated from the query element F16.

  When selecting some query candidate small regions F14 from the partial region F13, the small region integration program 123, for example, from a small region close to the center of gravity of the query candidate small region F14 in the feature amount space, to a specified number of small regions F14. Select. Alternatively, the small area integration program 123 may select a part of the small areas F14 based on the arrangement of the small areas F14 in the partial area F13.

  For example, the small region integration program 123 selects the query candidate small region F14 that is closest to the center of gravity of the query candidate small region F14 in the feature amount space, or the query candidate small region F14 that is closest to the center position of the partial region F13. A query candidate small region F14 used for generating a query element is selected from the query candidate small regions F14 closest to the query candidate small region F14 selected in the region F13.

  The small area integration program 123 places the selected small area F14 in the query element F16 (search query F17) in the query element F16. In the query element F16, the selected small region F14 may be rearranged in a prescribed shape, and the shape (arrangement) in the partial region F13 of the selected small region F14 may be maintained as it is.

  The small area integration program 123 arranges small areas in the query element F16 in accordance with the arrangement of the small areas F14 in the partial area F13. Thereby, it is possible to generate a search query that enables a more accurate search. FIG. 4B shows an example of the relationship between the arrangement of the small area F14 in the partial area and the arrangement of the small area F14 in the query element F16.

  From the partial area F13, four small areas A to D are selected for query element generation. In the query element F16A, the small areas A to D are arranged so as to form a rectangle. The small area integration program 123 arranges the small areas A to D in the query element F16A so as to maintain the relative positional relationship of the small areas A to D in the partial area F13. That is, in the partial area 113 and the query element F16A, the small areas A, B, C, and D are arranged at the upper left, upper right, lower right, and lower left, respectively.

  In the query element F16B, the small areas A to D have the same positional relationship as the arrangement in the partial area F13. For example, in the search window 161, a feature vector is generated from the small areas A to D, and other areas in the search window 161 are not referenced in the feature vector generation.

  When the small area F14 is rearranged in the prescribed shape in the query element F16 like the query element F16A, the small area integration program 123 may select the small area F14 in accordance with the prescribed shape of the query element F16. For example, the small region integration program 123 selects a query candidate small region F14 corresponding to one small region F14 of the query element F16 from the partial region F13, and is closest to the relative position of each of the other small regions F14 in the query element F16. The query candidate small area F14 is selected from the partial area F13.

  For example, in the example of FIG. 4A, the first selected query candidate small region F14 corresponds to the lower left small region F14 of the query element F16, and the query candidate closest in the right, left, and upper right directions of the selected query candidate small region F14. The small area F14 is further selected from the partial area F13.

  The small area integration program 123 combines the query elements F16 of the partial areas F13 to generate a search query F17. In the search query F17, the query elements F16 are associated with the corresponding partial areas F13. The small region integration program 123 further generates a search query image F18 from the search query F17 and displays it on the user interface F01 (input / output device 193).

  The user can confirm the search query image F18 before inputting the search instruction. Upon receiving a search instruction from the user interface F01 (input / output device 193), the image search program 131 searches for a person in the image database 141 using the generated search query F17. The search query image F18 may not be displayed, and the image search program 131 may search for a person using the generated search query F17 without receiving a user instruction.

  6A to 6C show a method of dividing the person area 12 into a plurality of partial areas 13. The partial area dividing program 121 may divide the person area 12 into a plurality of partial areas 13 using a mask. By using a mask, background noise can be effectively removed.

  FIG. 6A shows an example of a humanoid mask F21. When the partial area dividing program 121 receives the person area F12, the partial area dividing program 121 changes the human mask F21 in accordance with the outer shape of the person area F12. This is the same for other masks.

  The humanoid mask F21 includes a humanoid transmission region and a shielding region surrounding the transmission region. The transmission region is divided into three transmission regions F211 to F213. The transmission regions F211 to F213 correspond to the partial region F13 of Top, Upper, and Bottom, respectively. The transmission region F212 may partially overlap with each of the adjacent transmission region F211 and transmission region F213.

  An area that overlaps with and is exposed in the transmissive areas F211 to F213, that is, an area included in the transmissive areas F211 to F213 is extracted as a partial area F13 of the person area F12. The small area specifying program 122 divides the partial areas F13 in the transparent areas F211 to F213 into a plurality of small areas F14 in the person area F12.

  FIG. 6B shows an example of the block mask F22. In the mask F22, the transmission area is composed of a plurality of rectangular transmission areas. Specifically, the transmissive region includes three transmissive regions F221 to F223. The transmission areas F221 to F223 correspond to the partial area F13 of Top, Upper, and Bottom, respectively. The periphery of the transmission areas F221 to F223 is a shielding area.

  The transmission region F222 partially overlaps with each of the adjacent transmission region F221 and transmission region F223. Thereby, the whole site | parts, such as bottoms and tops which change with a person, can be included in each of the transmission area | regions 221-223. The transmission regions 221 to 223 may be completely separated.

  FIG. 6C shows an example in which the mask F31 is formed along the person image in the person area F12. Thereby, more noise in the person area F12 can be removed before clustering. In the removal FIG. 6C, transmission regions F311 to F313 correspond to the partial region F13 of Top, Upper, and Bottom, respectively.

  For example, the partial area dividing program 121 superimposes the human mask F21 or the block mask F22 on the person area. The partial area dividing program 121 specifies the outline of a person from the edge information of the image in each of the images in the Top, Upper, and Bottom transparent areas. For example, the partial region dividing program 121 identifies the edge using the Watershed algorithm from the information on the luminance gradient of the image, and cuts out the transparent regions F311 to F313 as compared with the human body shape model.

  By forming the mask F31 along the person image, more noise in the person region F12 can be removed before clustering. If the user designates a mask area for the person area F12, each part can be determined by comparing the edge using the Watershed algorithm with the human body model without mask data in advance.

  FIG. 7 shows a process of selecting one mask from a plurality of masks in the division of the person area F12 into the partial areas F13. The memory 192 stores a posture / mask database 57. The partial area dividing program 121 specifies the posture of the person in the person area F12 and selects a mask corresponding to the posture of the person from the posture / mask database 57. Thereby, the area other than the person can be more appropriately removed from the person area F12.

  The posture / mask database 57 stores a plurality of masks and feature vectors associated with each mask. The feature vector is a posture feature vector corresponding to a different posture of the human body. The posture feature vector can be extracted from a human body model or a lot of learning data.

  The partial area dividing program 121 extracts a human body image in the person area F12 and calculates a feature vector of the human body image. The feature vector to be calculated is, for example, a HOG feature vector. The partial region dividing program 121 converts the calculated feature vector into a posture feature vector, and extracts the posture feature of the person region F12 (S51).

  The partial region dividing program 121 compares the calculated posture feature vector with each posture feature vector in the posture / mask database 57 (S52). The partial region dividing program 121 selects a mask corresponding to the most similar (smallest distance) posture feature vector in the posture / mask database 57 (S53). The partial area dividing program 121 divides the person area F12 into partial areas F13 using the selected mask.

  The posture / mask database 57 may store feature vectors converted from posture feature vectors. It is not necessary to convert the feature vector calculated from the person area F12 into the posture feature vector. The partial area dividing program 121 may accept selection of a mask by the user. The partial area dividing program 121 divides the person area F12 into partial areas F13 using the mask selected by the user.

  FIG. 8 shows a navigation image for determining the person region F12 and the mask. The navigation image includes an input image display area 80 for displaying an input image and a mask display area 81 for displaying a selectable mask. The mask is used in the division of the person area F12. The user inputs a frame 82 that defines a person area in the input image display area 80.

  When the user determines the person area F12, the person area specifying program 112 displays a mask, more specifically, an outline 83 of the transparent area of the mask in the frame 82 input by the user. The person area specifying program 112 deforms and displays one mask selected from the registered masks according to the frame input by the user.

  When the user selects a mask from the mask display area 81, the person area specifying program 112 displays the selected mask in the frame 82. When the user changes the shape of the frame 82, the person region specifying program 112 changes the shape of the mask 83 in the frame according to the deformed frame 82.

  When the user selects the enter button 84, the person area specifying program 112 outputs information about the person area F12 determined by the frame 82 when the enter button 58 is selected and the selected mask to the partial area dividing program 121. By the navigation, a search query can be generated using a person region and a mask that the user determines to be appropriate.

  In the navigation image, the mask display area 81 may be omitted. The person area specifying program 112 displays a pre-registered human figure such as the mask outline 83 in the frame 82. When the user changes the shape of the frame 82, the person area specifying program 112 changes the shape of the human figure in the frame according to the deformed frame 82.

  As described above, the image search device of the present disclosure can automatically generate a search query and improve search accuracy. The image search device according to the present disclosure automatically generates a search query that can be searched with high accuracy by dividing a region including the target person into a plurality of regions and removing noise including the background.

  In the image search device of the present disclosure, the operation time can be reduced by a simplified input operation, and even if the number of clothes patterns and color combinations increases, there is no need to input manually. Since the image search device of the present disclosure can automatically generate a search query from actual data, the accuracy of the search query can be improved and the risk of missing a target person can be reduced. Note that the image search device of the present disclosure is suitable for searching for a target person, but may be applied to search for other targets.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  Each of the above-described configurations, functions, processing units, and the like may be realized by hardware by designing a part or all of them, for example, with an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files for realizing each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card or an SD card.

  In addition, the control lines and information lines are those that are considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. In practice, it may be considered that almost all the components are connected to each other.

  DESCRIPTION OF SYMBOLS 100 ... Input image data, 101 ... Image search apparatus, 102 ... User operation part, 103 ... Query production | generation part, 104 ... Search part, 111 ... Image input program, 112 ... Person area identification program, 121 ... Partial area division program, 122 ... small area specifying program, 123 ... small area integration unit, 131 ... image search program, 132 ... search result display program, 141 ... image database, 151 ... person area transmission program, 152 ... person area reception program, 153 ... query transmission program 154: Query reception program, 190, 203, 207, 210 ... Interface, 191, 201, 205, 208 ... CPU, 192, 202, 206, 209 ... Memory, 193, 204 ... I / O device

Claims (12)

A memory device for storing the program;
An arithmetic device that operates according to the program;
An image processing apparatus including an input / output device,
The arithmetic unit is:
In the input image, specify the person area that contains the person,
Dividing the person area into a plurality of partial areas;
Dividing each of the plurality of partial regions into a plurality of small regions;
In each of the plurality of partial regions, a cluster is formed by clustering the plurality of small regions,
In each of the plurality of partial areas, a cluster of query candidates is selected from the clusters based on the attributes of the clusters,
In each of the plurality of partial areas, a query element is generated from a small area of the selected cluster,
An image processing apparatus that generates a search query for searching for the person by combining query elements of the plurality of partial regions. The image processing apparatus according to claim 1,
The said arithmetic unit is an image processing apparatus which selects a cluster of query candidates from the said cluster based on the number of the small area | regions which comprise the said cluster in each of these partial area | regions. The image processing apparatus according to claim 1, wherein:
The arithmetic device is an image processing device in which a mask is overlaid on the person area, and an area exposed from a transmission area of the mask is divided into a plurality of partial areas. The image processing apparatus according to claim 3,
The memory device further includes a mask database including a plurality of masks corresponding to different postures of the person;
The image processing device, wherein the arithmetic device estimates the posture of the person in the person region and selects a mask corresponding to the posture from the mask database. The image processing apparatus according to claim 1, wherein:
The arithmetic unit is:
In each of the plurality of partial areas, select a small area from the small area of the selected cluster,
An image processing apparatus that arranges the partial small area in the query element according to a positional relationship of the partial small area in the partial area. The image processing apparatus according to claim 1, wherein:
The arithmetic unit is:
In the input / output device, the input image is displayed,
Receiving a designation of a frame of the person area for the displayed input image from the input / output device;
In the input / output device, a humanoid area is displayed in the frame,
Receiving the change of the frame including the humanoid from the input / output device;
An image processing apparatus that divides the person area defined by a frame that has received a decision input from the input / output apparatus into the plurality of partial areas. A method for generating a search query for searching for an image,
In the input image, specify the person area that contains the person,
Dividing the person area into a plurality of partial areas;
Dividing each of the plurality of partial regions into a plurality of small regions;
In each of the plurality of partial regions, a cluster is formed by clustering the plurality of small regions,
In each of the plurality of partial areas, a cluster of query candidates is selected from the clusters based on the attributes of the clusters,
In each of the plurality of partial areas, a query element is generated from a small area of the selected cluster,
Generating a search query for searching for the person by combining query elements of the plurality of partial regions. The method of claim 7, comprising:
A method of selecting a query candidate cluster from the cluster based on the number of small areas constituting the cluster in each of the plurality of partial areas. The method according to claim 7 or 8, comprising:
A method in which a mask is overlaid on the person area, and an area exposed from a transmission area of the mask is divided into a plurality of partial areas. The method of claim 9, comprising:
Estimating the posture of the person in the person area;
A method of selecting a mask corresponding to the posture from a plurality of masks corresponding to different postures. The method according to claim 7 or 8, comprising:
In each of the plurality of partial areas, select a small area from the small area of the selected cluster,
A method of arranging the partial small area in the query element according to a positional relationship in the partial area of the partial small area. The method according to claim 7 or 8, comprising:
Displaying the input image;
Receiving a designation of the frame of the person area for the displayed input image;
Display a humanoid area in the frame,
Receiving a change of the frame containing the humanoid,
Dividing the person region defined by the frame receiving the decision input into the plurality of partial regions.