JP2011018238A - Image retrieval system and image retrieval method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently retrieve a target person by a retrieval key not depending on a person or a specific portion of the person without accurately detecting a person area.SOLUTION: An image retrieval system is provided for retrieving an image similar to an image designated by a user. An image acquired from a video or the like captured by a camera is divided into blocks each having a predetermined size. The characteristic amount of the block is extracted based on a color histogram. First information indicating correspondence relation among an identifier of the image, information on a position of the block, and the characteristic amount of the block is created. The characteristic amount of the first retrieval key and the characteristic amount of another block are compared with at least one block selected by the user of the divided blocks as the first retrieval key, based on the first information to detect a predetermined number of blocks similar to the first retrieval key as a retrieval result, and the image including the retrieval result is displayed to the user.

Description

  The present invention relates to image information retrieval, and more particularly to an image retrieval system and an image retrieval method for retrieving a person or an object from captured image information.

  In recent years, violent crime has increased and security awareness has improved. Along with this, many surveillance cameras are installed in places where people gather, such as stores and airports. Video information photographed by these surveillance cameras is stored in a storage device such as a surveillance recorder and viewed as necessary.

  In addition, with the widespread use of IP cameras, a large number of cameras are connected via a network, and the capacity of the storage device is increased. Therefore, a large amount of video information is stored in the storage device. Therefore, it is very difficult to visually check the stored video information as in the past.

  Therefore, various similar search techniques have been proposed in order to search for an image showing a specific person or object from such a large amount of video information and present the searched image. Here, the similarity search technique is a technique for detecting information similar to a search key designated by a user from target information and presenting the detected data. In particular, the similar image search technique is a technique for searching for an image having a high degree of similarity with the image of the search key based on feature quantities such as color and shape acquired from the image serving as the search key. When searching for a person, features such as facial images and clothing colors can be used as search keys.

  Here, a method has been proposed in which a frame image of an accumulated video is divided into a plurality of regions by clustering using edge information and color information (see, for example, Patent Document 1). In the method described in Patent Document 1, first, an edge image is created from a frame image by edge extraction and smoothing. Next, clustering is performed using the pixel values of the frame image, and each region is labeled. If there are no edges on the boundary between any two labeled regions, then the two regions are merged. Then, feature quantities such as a color histogram and a central moment are extracted from each region, and a search is executed using the extracted feature quantities.

  Further, a method has been proposed in which a background image of each camera is created from a frame image of a video, and a person area is extracted based on a difference between the created background image and an original image (for example, see Patent Document 2). . In the method described in Patent Document 2, the characteristics of a person are generated using a histogram created from pixel values of the entire extracted person area and a histogram created from each partial area obtained by dividing the person area into a plurality of areas. The amount is calculated, and the calculated feature amount is used for the search.

  Also proposed is a method of extracting a moving object area from a video frame image as a rectangular area, dividing the extracted rectangular area into a plurality of blocks, and then extracting a representative color from each of the divided blocks. (For example, Patent Document 3). As a method of dividing the extracted rectangular area into blocks, there are a method of simply dividing the rectangular area into four, and a method of dividing the extracted rectangular area into areas such as the head, upper body, lower body, and legs using color information. In the method described in Patent Document 3, a person or representative color designated by a user is extracted, and a person is searched for by comparing the representative colors of blocks in the same positional relationship. Specifically, the similarity is calculated by comparing the color of the upper body of each block, the color of shoes, and the like. That is, the person is identified in consideration of the relative positional relationship of each part of the person.

Japanese Patent Laid-Open No. 2002-13312 JP 2001-268657 A JP 2005-202938 A

  The conventional image search system for searching for a desired person or object from video has the following problems.

  FIG. 23 is an explanatory diagram showing a person area detected by a conventional image search system.

  The left figure (A) of FIG. 23 is the figure which extracted the person area | region strictly from the image. The middle figure (B) is a diagram in which the person area is cut out as a rectangular area, and the cut out rectangular area is divided into four equal parts. The right figure (C) is a diagram in which a person area is cut out as a rectangular area, and the cut out rectangular area is divided by a person's part.

  First, in the methods described in Patent Documents 1 and 2, a person area surrounding the shape of a person is extracted as indicated by the broken line in FIG. 23A. In the first place, the person area and the background are strictly separated. The accuracy of the extracted person area is low. For this reason, a desired search result is often not obtained.

  Therefore, there has been considered a method of extracting the feature amount of the region without strictly extracting the person region and using the extracted feature amount for the search. For example, in the method described in Patent Document 3, the extracted person area is surrounded by a rough rectangular area as indicated by a broken line in FIG. 23B or 23C, and the rectangular area is divided into a plurality of blocks. To divide. However, in this method, when a divided block is used as a search key, the positional relationship between the blocks must be considered.

  Specifically, for example, in the block shown in FIG. 23B, in order to compare an image including a person as a search key with an image including a person as a search target, The block, the lower left block, and the lower right block must be compared.

  For this reason, if the position of the person who is the search key and the person to be searched are different, or if the area of the person shown in the camera is different, the part of the person included in each block is different, so the accuracy of the search is descend. For example, it is difficult to compare a person standing and a person sitting. In addition, it is difficult to compare a person whose whole body is reflected with a person whose only upper body is reflected.

  In this way, in an image search system that searches for a person based on a person's feature quantity, it is difficult to accurately detect the area where the person exists in the first place. Even if is extracted, the accuracy of the search is low. Even if the person area is not strictly detected and the person area surrounded by a rough rectangular area is divided into areas having some meaning, such as the upper left part and the upper body, the accuracy of the search is And influenced by size.

  The present invention has been made in view of the above-described problems, and it is possible to efficiently search for a target person using a search key that does not strictly detect a person area and does not depend on a person or a specific part of the person. An object of the present invention is to provide an image search system that can be used. In addition, this image search system can search not only a person but the moving object.

A typical example of the present invention is as follows. That is, an image search system for searching for an image similar to an image designated by a user, comprising: at least one computer including a processor, a memory connected to the processor, and a storage device; Acquiring a video photographed by a camera or a video stored in the storage device, extracting a predetermined number of images per unit time from the acquired video, and extracting the extracted image to a predetermined size The feature amount of the divided block is extracted based on the color histogram, and the position of the block in the extracted image identifier and the region of the extracted image is extracted. First information indicating a correspondence relationship between the information indicating whether the information is present and the feature amount of the extracted block is created, and based on the first information, Among the divided blocks, at least one block selected by the user is used as a first search key, and the feature amount of the first search key is compared with the feature amount of another block, thereby obtaining the first block. A predetermined number of blocks similar to one search key are detected as search results,
An image including the detected search result is displayed to the user.

  According to an embodiment of the present invention, it is possible to efficiently search for a person or an object desired by a user.

It is a block diagram which shows the structure of the image search system of 1st Embodiment. It is a block diagram which shows the image registration process of 1st Embodiment. It is a flowchart which shows the block division | segmentation process of 1st Embodiment. It is explanatory drawing which shows the image acquired by the block division process of 1st Embodiment. It is a block diagram which shows the search process of 1st Embodiment. It is explanatory drawing which shows the image displayed at the time of the block selection of 1st Embodiment. It is a flowchart which shows the search result integration process of 1st Embodiment. It is a block diagram which shows the structure of the image search system of 2nd Embodiment. It is a block diagram which shows the image registration process of 2nd Embodiment. It is a flowchart which shows the block division | segmentation process of 2nd Embodiment. It is a block diagram which shows the search process of 2nd Embodiment. It is a flowchart which shows the search process of 2nd Embodiment. It is a block diagram which shows the search process of 3rd Embodiment. It is explanatory drawing which shows the image displayed at the time of multiple block selection of 3rd Embodiment. It is a flowchart which shows the search result integration process of 3rd Embodiment. It is a block diagram which shows the search process of 4th Embodiment. It is explanatory drawing which shows the image displayed when the search key area | region of 4th Embodiment is designated. It is explanatory drawing which shows the display screen of the search result of 4th Embodiment. It is explanatory drawing which shows the image management information of 1st Embodiment. It is explanatory drawing which shows the feature-value management information of 1st Embodiment. It is a flowchart which shows the search process of 1st Embodiment. It is a flowchart which shows the search result integration process of 4th Embodiment. It is explanatory drawing which shows the person area | region detected by the conventional image search system.

  Embodiments of the present invention will be described below with reference to the drawings.

[Embodiment 1]
Hereinafter, an image search system according to a first embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration of an image search system according to the first embodiment.

  The image search system of the first embodiment includes a server computer 110, a client computer 130, an image database 140, a feature amount database 150, and a camera 160. Each device is connected to each other by a communication infrastructure 120.

  The server computer 110 includes an external interface 111, a central processing unit (CPU) 112, a memory 113, and a large-capacity external storage device (HD) 114. The server computer 110 is a computer on which a search engine operates, and provides a search service to the client computer 130 via the communication infrastructure 120.

  The external interface 111 is an interface (I / F) for connecting the server computer 110 to the communication infrastructure 120. The CPU 112 is a processor that controls processing of the server computer. The memory 113 is a work area for control processing executed by the CPU 112, and stores various data and programs loaded from the HD 114. The HD 114 is a large-capacity storage device such as a hard disk, and stores programs executed by the CPU 112, data, and the like.

  The client computer 130 includes a display device such as a CRT, and displays an output screen and an interface screen created by the server computer 110 on the display device. The client computer 130 includes an input device such as a keyboard and a mouse, and outputs data, commands, and the like input by the user operating the input device to the server computer 110.

  The image search system according to the present embodiment has a configuration in which the server computer 110 and the client computer 130 connected via a network provide a service. However, a general personal computer can perform a search service using an image search application. May be provided.

  The image database 140 is a database for storing images or image thumbnails, and stores, for example, image management information (see FIG. 19). The feature amount database 150 is a database for storing feature amounts extracted from the target image and a search index, and stores feature amount management information (see FIG. 20), for example.

  FIG. 19 is an explanatory diagram illustrating the image management information 2000 according to the first embodiment.

  The image management information 2000 includes an image ID 2001, a camera ID 2002, a shooting time 2003, and an image 2004. An image ID 2001 is an identifier for identifying an image. The camera ID 2002 is an identifier for identifying the camera that has captured the image. The shooting time 2003 is time information when the image is shot. An image 2004 is an image to be searched. Note that the image 2004 may be pointer information to the image instead of the image itself (for example, JPEG data).

  FIG. 20 is an explanatory diagram illustrating the feature amount management information 2100 according to the first embodiment.

  The feature amount management information 2100 includes a feature amount ID 2101, an image ID 2102, and block positions 2103 and 2104. The feature amount ID 2101 is an identifier for identifying the feature amount. The image ID 2102 is an identifier for identifying the image stored in the image 2004 of FIG.

  The block position 2103 is information indicating the position of a block (rectangular area) included in the image specified by the image ID 2102. For example, the block position is indicated by the X and Y coordinates of the upper left point of the rectangular area and the X and Y coordinates of the lower right point. Further, it may be indicated by the X and Y coordinates of the upper left point of the rectangular area and the width and height of the rectangular area.

  The feature vector 2104 is a feature vector extracted from the area specified by the block position 2103 in the image specified by the image ID 2102. The feature quantity vector is a multidimensional vector created by, for example, a feature quantity extraction unit 204 in FIG.

  Note that the image database 140 and the feature database 150 may be stored in the HD 114 included in the server computer 110 or may be stored in another hard disk different from the HD 114.

  Cameras 160A to 160N are cameras installed in the tracking target area. Hereinafter, when the description is common to all of the cameras 160A to 160N, the cameras 160A to 160N are collectively referred to as the camera 160 in the description.

  The image registration process will be described below.

  FIG. 2 is a block diagram illustrating image registration processing according to the first embodiment.

  The image search system according to the present embodiment executes image registration processing by the video input unit 201, the still image extraction unit 202, the block division unit 203, and the feature amount extraction unit 204, and stores information on the search target image as a feature amount database. 150 and the image database 140. The CPU 112 shown in FIG. 1 loads various programs stored in the HD 114 into the memory 113, reads the various programs loaded in the memory 113, and executes the read various programs, thereby executing a video input unit. 201, the still image extraction unit 202, the block division unit 203, and the feature amount extraction unit 204 are realized.

  First, the video input unit 201 acquires video information captured by the camera 160. Note that the video input unit 201 may acquire video information stored in the HD 114 or an arbitrary recording medium.

  Next, the still image extraction unit 202 creates a still image frame (image) from the video information acquired by the video input unit 201. Note that the still image extraction unit 202 can set how many images are to be created from video data per second as specified by the user.

  Next, the block division unit 203 executes block division processing to determine a block that is a target of feature amount extraction. Specifically, the block division unit 203 determines whether or not the image created by the still image extraction unit 202 includes a foreground (an area where a person or an object is reflected), and determines that the image includes a foreground. The image is divided into blocks. Details of the block division processing will be described later with reference to FIG.

  The block dividing unit 203 selects an image including a foreground (an image showing a person), creates image management information 2000 indicating information on the selected image, and uses the created image management information 2000 as an image database. Stored in 140.

  Specifically, the block division unit 203 is assigned to the image including the foreground area in the image ID 2001, the camera ID 2002, the shooting time 2003, and the image 2004 of each entry of the image management information 2000 illustrated in FIG. The identifier of the image, the identifier of the camera that captured the image, the shooting time, and the image data are stored. Here, the image ID 2001 is an arbitrary number for identifying an image, for example. The image 2004 may store the image data itself, or may store a thumbnail created from the image. Further, link information for referring to an image may be stored. The block dividing unit 203 may not store all images in the image database 140.

  Next, the feature quantity extraction unit 204 extracts the feature quantity of each block divided by the block division unit 203 and determined as a feature quantity extraction target based on, for example, color information. Specifically, the feature amount extraction unit 204 divides a block, which is a target of feature amount extraction determined to include the foreground, out of the blocks divided by the block division unit 203 into finer grid regions. To do. Then, the feature amount extraction unit 204 calculates a color histogram of each divided region, and creates a multidimensional vector (feature amount vector) based on the calculated color histogram.

  Note that the feature quantity extraction unit 204 may create a feature quantity vector indicating the feature of each block based on color information, information such as a luminance value and a brightness value, and information obtained by combining them. Further, the feature quantity extraction unit 204 may create a feature quantity vector of a partial area of the block.

  The feature quantity extraction unit 204 creates feature quantity management information 2100 and stores the created feature quantity management information 2100 in the feature quantity database 150. Specifically, the feature quantity extraction unit 204 identifies the feature quantities in the feature quantity ID 2101, the image ID 2102, the block position 2103, and the feature quantity vector 2104 in the entry of the feature quantity management information 2100 illustrated in FIG. Therefore, the identifier, the identifier of the original image, the position information of the block from which the feature amount is extracted, and the feature amount vector extracted from the block are stored.

  With the above processing, the image registration processing is completed. The targets of search processing shown in FIG. 5 described later are image management information 2000 and feature amount management information 2100 stored in each database. Note that the image registration process may not be executed in advance, but may be executed when the user executes the search process.

  Details of the processing of the block dividing unit 203 will be described below.

  FIG. 3 is a flowchart illustrating block division processing according to the first embodiment.

  First, the block dividing unit 203 acquires an image created by the still image extracting unit 202 (S101). In step S101, a part of the image acquired by the block dividing unit 203 is, for example, (A) shown in FIG.

  Next, the block dividing unit 203 calculates a background difference value (S102). Specifically, first, the block dividing unit 203 converts a background image prepared in advance into a gray scale. Here, the background image is, for example, an image in which the same range as the acquired image is captured, and is an image of only the background in which no person or object is reflected.

  Next, the block dividing unit 203 converts the acquired image into a gray scale. Then, the difference value (background difference value) of the pixels of each image is calculated by comparing the background image converted to gray scale with the image converted to gray scale. Of the acquired image areas, an area with a large background difference value has a high possibility that a person or an object different from the “background” is reflected. Therefore, the block dividing unit 203 selects an area with a large background difference value as “ It is determined as “foreground”. In step S102, the background difference image represented by the background difference value is, for example, FIG.

  Next, the block dividing unit 203 divides the background difference image obtained in step S102 into blocks of a predetermined size (S103). In step S103, the background difference image divided into grid-like blocks by the block dividing unit 203 is, for example, FIG. The block dividing unit 203 may change the size of the block or the number of divisions according to the size of the person or object shown in the acquired image. The block sizes may all be the same, or may be changed according to the area of the image. For example, the upper part of the image, that is, the block in the depth direction of the shooting range may be made smaller than the lower part of the image. Further, since only a small object is often shown on the upper part of the still image, the upper part of the image does not have to be a target for feature amount extraction.

  Next, the block dividing unit 203 calculates the foreground level of each block (S104). Specifically, first, the block dividing unit 203 calculates the sum of the background difference values of each block of the background difference image corresponding to the acquired image. Next, the calculated background difference value of each block is set as the foreground level of each block. Note that, when the sizes of the divided blocks are different, the block dividing unit 203 normalizes the total value of the background difference values according to the block area.

  Finally, the block division unit 203 determines a block that is a target of feature amount extraction (S105).

  Here, for example, when the foreground level of the block exceeds a predetermined threshold, the block dividing unit 203 is likely to have a person or object that is different from the background in the block with the high foreground level. May be the target of feature quantity extraction. In addition, a predetermined number of blocks having a large foreground degree may be selected from the blocks sorted in order of the magnitude of the foreground degree, and the selected blocks may be targeted for feature amount extraction. In step S105, the feature value extraction target block determined by the block dividing unit 203 is, for example, a shaded block shown in FIG.

  Note that the block dividing unit 203 may omit the processes of steps S101 to S104. In this case, the block dividing unit 203 may divide the acquired image into grid-like blocks and determine each divided block as a feature amount extraction target.

  FIG. 4 is an explanatory diagram illustrating an image acquired by the block division processing according to the first embodiment.

  The upper left figure (A) is a part of an image first acquired by the block dividing unit 203, for example, an image showing a person. The upper right figure (B) is a background difference image created by the block dividing unit 203, and an area where a person or an object (foreground) is likely to be reflected is indicated by the background difference value.

  The lower left figure (C) is a background difference image divided into grid-like blocks by the block dividing unit 203. The lower right diagram (D) is an image showing a block that is a target of feature amount extraction determined by the block dividing unit 203. Block areas that are the target of feature quantity extraction are shaded.

  Details of the search process will be described below.

  FIG. 5 is a block diagram illustrating search processing according to the first embodiment.

  The image search system of this embodiment includes an image acquisition unit 301, a block division unit 302, an image display unit 303, a block selection unit 304, a search unit 305, a search result integration unit 306, a search result display unit 307, an image database 140, and features. A search process is executed by the quantity database 150. The CPU 112 shown in FIG. 1 loads various programs stored in the HD 114 into the memory 113, reads the various programs loaded in the memory 113, and executes the read various programs, thereby obtaining an image acquisition unit. 301, the block division unit 302, the image display unit 303, the block selection unit 304, the search unit 305, the search result integration unit 306, and the search result display unit 307 are realized.

  The image acquisition unit 301 acquires an image including a person or object that is selected by the user and serves as a search key from an image extracted from video captured by the camera 160.

  Alternatively, the image acquisition unit 301 acquires, from the image management information 2000 of the image database 140, an image including a person or object that is selected by the user and serves as a search key. In this case, the image search system displays an interface screen on which the user can select an image including a block serving as a search key by referring to a camera number (camera ID), an image shooting time, and the like. May be displayed.

  When the image acquisition unit 301 acquires an image from the camera 160, the block dividing unit 302 divides the acquired image into blocks by the same processing as the block dividing unit 203 shown in FIG. A block that is a target of feature amount extraction is determined.

  On the other hand, when the image acquisition unit 301 has acquired an image from the image database 140, the data registration process has already been completed, so the block dividing unit 302 adds the acquired image to the acquired image from the feature amount management information 2100 of the feature amount database 150. The corresponding image ID and block position are acquired.

  Next, the image display unit 303 displays the acquired image on the display of the client computer 130. In this case, the block position information acquired by the block dividing unit 302 is also displayed.

  FIG. 6 is an explanatory diagram illustrating an image displayed when a block is selected according to the first embodiment.

  The right figure (A) shows a block of search key candidates. A block 401 in the left figure (B) is a block selected by the user as a search key.

  In addition, when displaying the search key candidate block, the image display unit 303 may display each block of the search key candidate of the acquired image as a rectangle, as shown in FIG. Further, each block of search key candidates may be displayed side by side next to the acquired image.

  Next, the block selection unit 304 acquires a block serving as a search key selected by the user. The user can select a block to be a search key from each block included in the image displayed on the display of the client computer 130 using an input device such as a mouse.

  Further, the image display unit 303 may color the rectangle of the selected block or make the rectangle line thick, for example, in order to clearly indicate the block of the selected search key. Further, the inside of the rectangle of the selected block may be colored.

  Next, the search unit 305 searches the feature amount management information 2100 of the feature amount database 150 for a block having a feature amount similar to the feature amount of the block of the selected search key, and acquires a search result. Details of the search processing executed by the search unit 305 will be described later with reference to FIG.

  In the present embodiment, the search result display unit 307 displays an image (search result image) including the search result block instead of the search result block. The image ID of the image including the search result block is acquired from the management information 2100.

  Next, the search result integration unit 306 integrates the search results acquired by the search unit 305 and creates a search result display list. Details of the search result integration process executed by the search result integration unit 306 will be described later with reference to FIG.

  The search result display unit 307 displays an image having an image ID included in the search result display list created by the search result integration unit 306 on the display of the client computer 130. In this case, the search result display unit 307 may display only an image, or may display an image including a search result block indicated by a rectangle.

  Details of the search processing executed by the search unit 305 will be described below.

FIG. 21 is a flowchart showing the search process of the first embodiment. The search unit 305 first acquires the feature amount of the selected block (S601). When the image acquisition unit 301 acquires an image from the camera 160, the search unit 305 performs the same processing as the feature amount extraction unit 204 illustrated in FIG. 2 and extracts the feature amount of the selected block. On the other hand, when the image acquisition unit 301 acquires an image from the image database 140, the feature amount of each block of the acquired image has already been calculated. Therefore, the search unit 305 uses the feature amount management information 2100 of the feature amount database 150. The feature amount of the selected block is acquired.

  Next, the search unit 305 calculates the similarity between the feature amount of the selected search key block and the feature amount of each entry of the feature amount management information 2100 (S602). In this embodiment, the Euclidean distance between each feature vector is used to calculate the similarity. Note that the similarity between the feature amounts may be calculated using, for example, a Manhattan distance in addition to the Euclidean distance.

  Finally, the search unit 305 acquires, from the feature amount management information 2100 of the feature amount database 150, N blocks having high similarity to the search key block as search results (S603). Here, N may be a number sufficiently larger than the number of cases that the search result display unit 307 can display at a time. For example, if the number of cases that can be displayed is 20, the number N of search results acquired in step S603 is preferably about 1000.

  Details of the search result integration process executed by the search result integration unit 306 will be described below.

  FIG. 7 is a flowchart illustrating search result integration processing according to the first embodiment.

  First, the search result integration unit 306 acquires N search results having a high degree of similarity with the search key block output from the search unit 305, and sets the counter i to 0 (S201). Further, the search result display list is initialized. Here, the search result display list is an image ID and its image ID created by the search result integration unit 306 in order for the search result display unit 307 to display an image (search result image) including the search result block. This is a list in which the similarity set to an image having is associated with each other.

  Next, the search result integration unit 306 acquires, from the feature amount management information 2100 of the feature amount database 150, the image ID of an image including the i-th block having the i-th similarity among the N search result blocks (S202). ).

  Thereafter, the search result integration unit 306 determines whether or not the acquired image ID is included in the search result display list (S203). When it is determined in step S203 that the acquired image ID is not included in the search result display list, the search result integration unit 306 similar to the image ID acquired in the search result display list and the image having the image ID. The degree is added (S204). On the other hand, when it is determined in step S203 that the acquired image ID is included in the search result display list, the search result integration unit 306 moves to step S205 without adding the acquired image ID to the search result display list.

  Next, the search result integration unit 306 determines whether all the search results output from the search unit 305 have been processed (S205). If it is determined in step S205 that all search results have not been processed, that is, if the counter i is less than the number N of search results, the search result integration unit 306 adds 1 to the counter i (S206). , S202 and subsequent processes are repeated.

  On the other hand, if it is determined in step S205 that all search results have been processed, that is, if the counter i is greater than or equal to the number N of search results, the search result integration unit 306 adds the image ID added to the search result display list. Are sorted in descending order of similarity (S207), and the process ends.

  In the present embodiment, when a plurality of search result blocks are included in an image having an image ID added as a sort target, the similarity to the search key block is the largest among the plurality of search result blocks. The block of the search result is selected, and the similarity of the selected block is set to the similarity corresponding to the image having the added image ID. Note that an average value of similarities of a plurality of search result blocks may be calculated, and the calculated average value of similarities may be set to a similarity corresponding to an image having an added image ID.

  As described above, according to the image search system of the first embodiment, an image divided into grid-like blocks is searched in units of blocks, so that a search key can be used without strictly extracting a person or object region. A person or object similar to can be searched. Furthermore, since a search is possible if a portion similar to the search key block appears in a part of the image, the posture and occlusion of the person appearing in the search target image (the object in front of the camera is behind The effect of the phenomenon of shielding the object is reduced.

[Embodiment 2]
The image search system according to the second embodiment of the present invention will be described below with reference to the drawings.

  FIG. 8 is a block diagram illustrating a configuration of an image search system according to the second embodiment.

  The image search system according to the second embodiment includes a server computer 110, a client computer 130, a feature amount database 150, a camera image database 410, and a thumbnail image database 420. Each device is connected to each other by a communication infrastructure 120.

  The server computer 110, the client computer 130, the feature amount database 150, and the communication infrastructure 120 are the same as the devices to which the same reference numerals of the image search system of the first embodiment shown in FIG. To simplify.

  The camera image database 410 is a database that stores camera images (still images extracted from video data) previously captured by the camera. The camera image database 410 may store the image management information 2000 shown in FIG. 19 in the same manner as the image database 140 shown in FIG. In this case, a camera image (for example, a JPEG image) captured in advance is stored in the image 2004 of the image management information 2000. Further, the image ID, camera ID, and shooting time of an image shot in advance are stored in the image ID 2001, camera ID 2002, and shooting time 2003 of the image management information 2000, respectively.

  The thumbnail image database 420 is a database that stores thumbnails of camera images. The thumbnail image database 420 may store the image management information 2000 shown in FIG. 19 in the same manner as the image database 140 shown in FIG. However, in this case, the image 2004 of the image management information 2000 stores the thumbnail image or link information for referring to the original camera image of the thumbnail image.

  Note that the image retrieval system may not include the thumbnail image database 420 when the thumbnail image is not used.

  The image registration process will be described below.

  FIG. 9 is a block diagram illustrating image registration processing according to the second embodiment.

  In the image search system according to the present embodiment, the image input unit 501, the block division unit 502, the feature amount extraction unit 204, and the camera image database 410 execute an image registration process, and information on an image to be searched is stored in the feature amount database 150. And stored in the thumbnail image database 420. Note that the CPU 112 shown in FIG. 8 loads various programs stored in the HD 114 into the memory 113, reads the various programs loaded in the memory 113, and executes the read various programs, thereby executing an image input unit. 501, the block division unit 502, and the feature amount extraction unit 204 are realized.

  The feature quantity extraction unit 204 and the feature quantity database 150 are respectively the same as the feature quantity extraction unit 204 and the feature quantity database 150 of the first embodiment illustrated in FIG.

  First, the image input unit 501 acquires an image to be searched. Here, the image to be searched is an image stored in the camera image database 410.

  Next, the block division unit 502 executes block division processing. Specifically, the block division unit 502 divides the image output from the image input unit 501 into blocks, and determines a block that is a target for feature amount extraction among the divided blocks. Details of the block division processing executed by the block division unit 502 will be described later with reference to FIG.

  The block dividing unit 502 selects an image including a block determined as a feature amount extraction target, and creates a thumbnail of the selected image. The block division unit 502 stores the thumbnail image created in the image 2004 of the image management information 2000 in the thumbnail image database 420. Further, link information to the original image may be given to the thumbnail image.

  Further, the block division unit 502 stores the image ID, camera ID, and shooting time of the thumbnail image in each item of the image management information 2000 of the thumbnail image database 420.

  When the image search system does not include the thumbnail image database 420, the block division unit 502 is determined as a feature amount extraction target among images stored in the image management information 2000 of the camera image database 410, for example. The image ID of the image including the block may be rewritten to an image ID indicating that the target block for feature amount extraction is included.

  The feature quantity extraction unit 204 extracts a feature quantity from the feature quantity extraction target block determined by the block division unit 502, and extracts the extracted feature quantity ID, the original image image ID, the block position information, and the extracted feature quantity. The feature amount (feature amount vector) is stored in each item of the feature amount management information 2100 of the feature amount database 150.

  Details of the block division unit 502 process will be described below.

  FIG. 10 is a flowchart illustrating block division processing according to the second embodiment.

  First, the block dividing unit 502 acquires the image output from the image input unit 501 (S301). Next, the block dividing unit 502 divides the acquired image into lattice blocks (S302).

  The block dividing unit 203 may change the size of the block or the number of divisions according to the size of the person or object shown in the acquired image. The block sizes may all be the same, or may be changed depending on the area of the image. For example, the upper part of the image, that is, the block in the depth direction of the shooting range may be made smaller than the lower part of the image. Further, since only a small object is often shown on the upper part of the still image, the upper part of the image does not have to be a target for feature amount extraction.

  Next, the block dividing unit 502 calculates a color histogram of each divided block (S303). Specifically, first, RGB values are divided into a plurality of value ranges. For example, when each value of RGB is in a range between 0 and 255, this range is divided into four stages of 0 to 63, 64 to 127, 128 to 191, and 192 to 255. If the pixel values included in one stage are regarded as the same value, there are a total of 4 × 4 × 4 = 64 combinations of RGB. The block dividing unit 502 does not have to divide each of RGB into the same stage. For example, the resolution may be differentiated by dividing R into 12 levels, G as 21 levels, and B as 6 levels according to human visual characteristics.

  Next, the block dividing unit 502 calculates an RGB value for each pixel included in each divided block, and the combination of the calculated RGB values of each pixel (for example, 64 combinations). Judged to fall under. Then, the total number of pixels corresponding to each combination is calculated. As a result, a color histogram of the target block is created.

  Next, the block dividing unit 502 divides the background image prepared in advance into grid-like blocks, similarly to step S302. Then, similarly to step S303, a color histogram of each block of the background image is calculated. Then, the block dividing unit 502 compares the acquired image with the background image, and calculates the difference between the color histograms for each corresponding block (S304). In this case, for example, the difference between the color histograms may be calculated using the Euclidean distance.

  Finally, the block division unit 502 determines a block from which a feature amount is to be extracted (S305). Here, for example, when the difference value of a block exceeds a predetermined threshold, the block dividing unit 502 may set the block as a feature amount extraction target. In addition, a predetermined number of blocks having a large difference value may be selected from the blocks sorted in order of the difference value, and the selected block may be a target for feature amount extraction.

  Hereinafter, a search process for a person or an object will be described.

  FIG. 11 is a block diagram illustrating search processing according to the second embodiment.

  The image search system according to the second embodiment includes an image acquisition unit 601, an image display unit 602, a block selection unit 304, a search unit 603, a search result integration unit 306, a search result display unit 307, a feature amount database 150, and a thumbnail image database. By 420, a search process is executed.

  Note that the CPU 112 illustrated in FIG. 8 loads various programs stored in the HD 114 into the memory 113, reads the various programs loaded into the memory 113, and executes the read various programs, thereby obtaining an image acquisition unit. 601, an image display unit 602, a block selection unit 304, a search unit 603, a search result integration unit 306, and a search result display unit 307 are realized.

  The feature quantity database 150, the block selection unit 304, the search result integration unit 306, and the search result display unit 307 are the same as the processing units with the same reference numerals shown in FIG.

  Note that when the image search system does not include the thumbnail image database 420, the image search system executes a search process using the camera image database 410 instead of the thumbnail image database 420.

  First, the image acquisition unit 601 acquires, from the thumbnail image database 420, an image that includes a person or an object selected as a search key by the user. In the thumbnail image database 420, the image registration process has already been completed, and an image to be searched is stored.

  In this case, the image search system displays an interface screen on which the user can select an image including a block as a search key by referring to the camera number (camera ID), the image shooting time, and the like. You may display on a display. The image acquisition unit 601 acquires an image including a block serving as a search key selected from the interface screen when the user operates an input device such as a mouse or a keyboard.

  The image display unit 602 displays on the display of the client computer 130 the image that is output from the image acquisition unit 601 and includes a block that serves as a search key. In this case, the image display unit 602 acquires image block position information from the feature amount management information 2100 of the feature amount database 150, and displays the acquired image and block position information together.

  As shown in FIG. 6A, the image display unit 602 may display each block of search key candidates included in the acquired image in a rectangle, and the search key beside the acquired image. The candidate blocks may be displayed side by side.

  Next, the block selection unit 304 selects a block to be a search key (first search key in the search process of FIG. 12 described later) by the same processing as the block selection unit 304 shown in FIG.

  Next, the search unit 603 searches the image management information 2000 of the feature amount database 150 for a block having a feature amount similar to the feature amount of the block of the selected search key. Details of the processing of the search unit 603 will be described later with reference to FIG.

  Next, the search result integration unit 306 acquires the search results output from the search unit 603, and integrates the acquired search results by the search result integration process illustrated in FIG.

  Next, the search result display unit 307 performs the same processing as the search result display unit 307 illustrated in FIG. 5 to display an image having an image ID included in the search result display list created by the search result integration unit 306 as the client computer 130. On the display.

  Details of the search processing executed by the search unit 603 will be described below.

  FIG. 12 is a flowchart illustrating search processing according to the second embodiment.

  First, the search unit 603 acquires the camera ID and shooting time of the image including the block of the first search key selected by the block selection unit 304 from the image management information 2000 of the thumbnail image database 420 (S401).

  Next, the search unit 603 selects M second blocks similar to the selected first search key block among the image blocks of the previous and subsequent frames of the image including the selected first search key block. A search key block is acquired (S402).

  Here, first, the search unit 603 extracts and extracts an image shot in a range of t seconds before and after the acquired shooting time (for example, about 10 seconds) from the image shot with the acquired camera ID. Each block (information on each block position) included in the preceding and following images is acquired.

  Next, the search unit 603 refers to the image management information 2000 of the feature amount database 150, and resembles the feature amount of the block of the first search key and the feature amount of the block included in the acquired images before and after. Calculate the degree. Then, M blocks having a high similarity with the block of the first search key are acquired.

  Here, in order to calculate the similarity, the Euclidean distance between the feature vectors is used. However, in addition to the Euclidean distance, for example, a Manhattan distance may be used. Moreover, the search part 603 may acquire the block, when the calculated similarity exceeds a predetermined threshold value.

  Next, the search unit 603 determines whether or not the acquired number M of blocks satisfies M> 0 (S403).

  If it is determined in step S403 that the search result number M does not satisfy M> 0, that is, if there is no other image at t seconds before and after the shooting time of the block of the selected first search key, or When the similarity between the block of the first search key and the blocks included in the preceding and following images is equal to or less than a predetermined threshold, the search unit 603 uses the selected first search key to calculate the feature amount. All blocks of the management information 2100 are searched. Then, the search unit 603 acquires N blocks having a high similarity with the block of the first search key (S406). The process of step S406 is the same as the search process of steps S601 to S603 shown in FIG.

  On the other hand, if it is determined in step S403 that the number M of search results satisfies M> 0, the search unit 603 sets the acquired block as the second search key, and sets the first search key and the second search key. Using the key, all blocks of the feature amount management information 2100 are searched (S404).

  Then, the search unit 603 acquires a predetermined number of first blocks having a high degree of similarity with the block of the first search key. Also, a predetermined number of second blocks having a high degree of similarity with the second search key block are acquired. The number of acquired first blocks and the number of second blocks may be set to be N in total, for example. The process of step S404 is the same as the search process of steps S601 to S603 shown in FIG.

  However, since the search result (second block) acquired by the process of step S404 is searched using a plurality of second search keys, it may overlap. For this reason, the duplicated second blocks are integrated (S405). In this case, the similarity between the plurality of second search key blocks and the second block must be reset.

  Here, the second block having the highest similarity with the second search key is selected from the plurality of second blocks, and the similarity is set to the similarity corresponding to the second block. Note that an average value of each degree of similarity corresponding to a plurality of second blocks may be calculated, and the calculated degree of similarity may be set as the degree of similarity of the second block. If the first block searched using the selected first search key block and the second block searched using the second search key block overlap, the first block May be set to a similarity corresponding to an integrated block.

  Next, the search result integration unit 306 integrates the search results acquired in step S405 or S406. The search result integration process is the same as the search result integration process shown in FIG.

  Finally, the search result display unit 307 displays an image having an image ID included in the search result display list created by the search result integration unit 306 on the display of the client computer 130.

  As described above, according to the image search system of the second embodiment, in addition to the effects of the first embodiment, the search process is executed using a search key having slightly different feature amounts. And the search accuracy can be improved.

[Embodiment 3]
The image search system according to the third embodiment of the present invention will be described below with reference to the drawings.

  Since the configuration of the image search system and the image data registration process of the present embodiment are the same as those in the first embodiment or the second embodiment, description thereof will be omitted. Below, the search process of this embodiment is demonstrated.

  FIG. 13 is a block diagram illustrating search processing according to the third embodiment.

  The image search system of this embodiment includes an image acquisition unit 701, an image display unit 702, a block selection unit 703, a search unit 704, a search result integration unit 705, a search result display unit 706, an image database 710, and a feature amount database 150. Execute the search process.

  The CPU 112 shown in FIG. 1 or FIG. 8 loads various programs stored in the HD 114 into the memory 113, reads the various programs loaded in the memory 113, and executes the read various programs. The processing of each processing unit is realized.

  The feature amount database 150 is the same as the feature amount database 150 shown in FIG. When the image search system of the present embodiment is the same as the configuration of the image search system of the first embodiment, the image database 710 is the same as the image database 140 shown in FIG. Further, when the image search system of the present embodiment is the same as the configuration of the image search system of the second embodiment, the image database 710 is the same as the thumbnail image database 420.

  First, the image acquisition unit 701 acquires, from the image database 710, an image including a person or object that is selected by the user and serves as a search key. Here, the image database 710 has already completed the image registration process, and stores an image that is the target of the search key.

  In this case, the image search system displays an interface screen on which the user can select an image including a block as a search key by referring to the camera number (camera ID), the image shooting time, and the like. You may display on a display. The image acquisition unit 701 acquires an image selected from the interface screen by the user operating an input device such as a mouse or a keyboard.

  Next, the image display unit 702 displays the acquired image on the display of the client computer 130. In this case, the image display unit 702 acquires block position information from the feature amount management information 2100 of the feature amount database 150, and displays the acquired block position information and the acquired image. For example, the image display unit 702 may display each block included in the acquired image as a rectangle as illustrated in FIG.

  Next, the block selection unit 703 acquires a block serving as a search key selected by the user. In the present embodiment, the block selection unit 703 can acquire a plurality of blocks selected by the user. Note that the user can select a plurality of blocks serving as search keys from each block included in the image displayed on the display of the client computer 130 using an input device such as a mouse.

  Whether the image search system acquires search results corresponding to all selected search keys, or acquires a search result corresponding to any one of a plurality of selected search keys. An interface screen for designating the user may be displayed.

  FIG. 14 is an explanatory diagram illustrating an image displayed when a plurality of blocks are selected according to the third embodiment.

  FIG. 14 shows blocks 801 and 802 that are search keys selected by the user. Note that the image display unit 602 may color each rectangle surrounding the block in order to clearly indicate the selected block. Further, the rectangles surrounding each block may be colored with different colors.

  Returning to the description of the search process in FIG.

  Next, the search unit 704 searches the feature amount database 150 for a block having a feature amount similar to the block of the selected search key. When a plurality of blocks are selected by the block selection unit 703, the search unit 704 repeats the search process by the number of blocks of the selected search key.

  Next, the search result integration unit 705 acquires the search results output from the search unit 704 and integrates the acquired search results. Details of the processing of the search result integration unit 705 will be described later with reference to FIG.

  Next, the search result display unit 706 displays an image having an image ID included in the search result display list created by the search result integration unit 705 on the display of the client computer 130. In this case, the search result display unit 706 may display only an image, or may display a search result block surrounded by a rectangle in the image. When the search result display unit 706 surrounds the search result block with a rectangle, the search result block 706 indicates which search key block the search result block corresponds to. Therefore, the search result block corresponding to each search key block is different. You may color.

  Details of the processing of the search result integration unit 705 will be described below.

  FIG. 15 is a flowchart illustrating search result integration processing according to the third embodiment.

  First, the search result integration unit 705 initializes the contents of the search result display list and sets the counter i to 0 (S501).

  Next, the search result integration unit 705 acquires the Ni search results acquired by the search unit 704 using the i-th search key of the selected L search keys. Further, the search result integration unit 705 sets the counter j to 0 and initializes the search result list (search result list by search key) corresponding to the i-th search key (S502). Here, the search result display list for each search key is a list in which the image ID of the image including the search result block for each search key is associated with the similarity set for the image. Note that a search result list by search key L that is the same as the number of search keys is created.

  Next, the search result integration unit 705 acquires, from the feature amount management information 2100 of the feature amount database 150, the image ID of an image including the j-th block having a similarity degree among the acquired Ni search results (S503). ). This image ID is an image ID stored in the feature amount management information 2100 of the feature amount database 150 together with the feature amount extracted at the time of image data registration.

  Next, the search result integration unit 705 determines whether or not the acquired image ID is included in the search result list by search key (S504). If it is determined in step S504 that the acquired image ID is not included in the search result list by search key, the search result integration unit 705 has the acquired image ID and the image ID in the search result list by search key. The similarity corresponding to the image is added (S505). On the other hand, if it is determined in step S504 that the acquired image ID is included in the search result list by search key, the search result integration unit 705 does not add the acquired image ID to the search result list by search key, The process moves to step S506.

  Next, the search result integration unit 705 determines whether or not all the acquired Ni search results have been processed (S506). If it is determined in step S506 that all search results have not been processed, that is, if the counter j is less than the number of search results Ni, the search result integration unit 705 adds 1 to the counter j (S507). The processes after step S503 are repeated. On the other hand, if it is determined in step S506 that all search results have been processed, that is, if the counter j is greater than or equal to the number of search results Ni, the search result integration unit 705 is acquired using the i-th search key. Since all the search results have been processed, the process proceeds to step S508.

  Next, the search result integration unit 705 determines whether all search results acquired using the L search keys selected by the block selection unit 703 have been processed (S508). If it is determined in step S508 that all search results have not been processed, that is, if the counter i is less than the selected search key number L, the search result integration unit 705 adds 1 to the counter i ( (S509), the processing after step S502 is repeated.

  On the other hand, if it is determined in step S508 that all the search results have been processed, that is, if the counter i is greater than or equal to the number of search keys L, the search result integration unit 705 has processed all the search results. The process moves to S510.

  The search result integration unit 705 adds the image IDs included in any search key-specific search result list obtained by the processing of steps S501 to S509 to the search result display list (S510). As a result, the search result display unit 706 displays an image including all search result blocks corresponding to each of the L selected search keys.

  In this case, since the similarity corresponding to the image having the image ID is different for each search result list for each search key, the search result integration unit 705 includes the image with the image ID added to the search result display list. Set the highest similarity. Note that an average value of similarities may be set for the image with the image ID added to the search result display list.

  In step S510, the search result integration unit 705 may add at least one image ID included in any search result list for each search key to the search result display list. In this case, the search result display unit 706 displays an image including a search result corresponding to any one of the selected L search keys.

  Note that the search result integration unit 705 may use the positional relationship of the search result blocks corresponding to each search key for the processing of step S510. Specifically, the search result integration unit 705 stores the relative positional relationship of each search key block included in the image, and the relative position of each stored search key block and the relative relationship between each search result block. Compare with a specific position. If the relative positions of the images match, the image ID of the image including the search result is added to the search result display list.

  Finally, the search result integration unit 705 sorts the image IDs included in the search result display list according to the degree of similarity, and ends the process (S511).

  As described above, according to the image search system of the third embodiment, in addition to the effects of the first and second embodiments, a plurality of search keys are used, so that a person and an object having a plurality of features are searched. be able to.

[Embodiment 4]
The image search system according to the fourth embodiment of the present invention will be described below with reference to the drawings. The tracking map display unit 904 of the present embodiment displays the movement order of people and objects on the map.

  The configuration of the image search system and the image data registration process of the present embodiment are the same as the configuration of the image search system and the image data registration process of the first or second embodiment, respectively.

  FIG. 16 is a block diagram illustrating search processing according to the fourth embodiment.

  The image search system of the present embodiment includes an image acquisition unit 701, an image display unit 602, a search key area designation unit 901, a search unit 902, a search result integration unit 903, a tracking map display unit 904, an image database 710, and a feature amount database. The search process is executed by 150. The CPU 112 shown in FIG. 1 or FIG. 8 loads various programs stored in the HD 114 into the memory 113, reads the various programs loaded in the memory 113, and executes the read various programs. The processing of each processing unit is realized.

  The feature quantity database 150 is the same as the feature quantity database 150 shown in FIG. The image acquisition unit 701 is the same as the image acquisition unit 701 illustrated in FIG. The image database 710 is the same as the image database 710 shown in FIG. The image display unit 602 is the same as the image display unit 602 shown in FIG.

  First, the image acquisition unit 701 acquires an image including a person or an object selected by the user and serving as a search key.

  Next, the image display unit 602 displays the acquired image on the display of the client computer 130.

  Next, the search key area specifying unit 901 acquires an area that is specified by the user and serves as a search key. The user operates an input device such as a mouse to specify an area to be a search key in the image displayed by the image display unit 602.

  FIG. 17 is an explanatory diagram illustrating an image displayed when a search key area is designated according to the fourth embodiment.

  FIG. 17 shows a region 1001 having an arbitrary shape which is designated by the user and serves as a search key.

  Next, the search key area designating unit 901 extracts a feature quantity from the designated area in the same manner as the process executed by the feature quantity extracting unit 204. However, the extracted feature value is normalized according to the size of the designated area.

  Next, the search unit 902 calculates the similarity between the feature quantity of the designated area and the feature quantity of each entry from the feature quantity management information 2100 of the feature quantity database 150, and the similarity with the designated area. A block with a high degree is acquired as a search result. Here, the search unit 902 calculates the similarity between the feature amount of the designated area and the feature amount of each block using the Euclidean distance.

  Next, the search result integration unit 903 groups the search result blocks by camera ID. Details of the search result integration process will be described below.

  FIG. 22 is a flowchart illustrating search result integration processing according to the fourth embodiment.

  First, the search result integration unit 903 acquires the search result output from the search unit 902, and initializes the counter i to 0 (S701). Next, the search result integration unit 903 acquires the image ID of the image including the block of the search result having the i-th similarity from the feature amount management information 2100 of the feature amount database 150. Next, the search result integration unit 903 acquires a camera ID corresponding to the acquired image ID from the image management information 2000 of the image database 710 (S702).

  Thereafter, the search result integration unit 903 adds the image ID of the image including the block of the i-th search result to the group having the acquired camera ID (S703). If the image ID is already included in the group, the image having the image ID includes a plurality of search result blocks. Therefore, the image having the image ID is similar to the plurality of search result blocks. Set the largest similarity among degrees.

  If there is no group having the acquired camera ID, the search result integration unit 903 creates a new group and adds the image ID of the image including the block of the i-th search result to the created group.

  Next, the search result integration unit 903 determines whether the counter i exceeds the number N of search results (S704). If it is determined in step S704 that the counter i exceeds the number N of search results, the search result integration unit 903 adds 1 to i (S705), and repeats the processing from step S702 onward. Note that the search result integration unit 903 may exclude search results that are not included in the preset shooting time range from the search result integration processing targets.

  The tracking map display unit 904 displays an image (search result image) including a search result block for each camera ID created by the search result integration unit 903 on the display of the client computer 130. In the fourth embodiment, the tracking map display unit 904 creates a simple map of a space where a camera is installed, and displays an image including a search result block around the created simple map.

  FIG. 18 is an explanatory diagram illustrating a search result display screen according to the fourth embodiment.

  The search result display screen of the fourth embodiment includes a simple map 1101 and a search result image 1102 (or 1103). The search result image 1102 displayed around the simple map 1101 corresponds to each camera, and is connected to the camera installation position (camera shooting range) and the leader line. If the correspondence between the camera and the search result image 1102 is clearly indicated, the search result image 1102 may be displayed by another method. The shooting number 1104 assigned to the tip of the lead line is a number assigned in the order of the shooting times of the search result image 1102. That is, the shooting number 1104 indicates the order of movement of the person indicated as the search result.

  Returning to the description of FIG.

  For example, the tracking map display unit 904 creates a simple map 1101 of the space where the camera is installed based on the map information stored in the HD 114 and displays a search result image 1102 around the map.

  Also, the tracking map display unit 904 corresponds to the camera ID the image having the image ID of the search result having the highest similarity among the search results included in the group by camera ID created by the search result integration unit 903. The search result image 1102 is displayed. If there is no corresponding group, nothing is displayed. Note that the tracking map display unit 904 may not display an image having a similarity equal to or lower than a predetermined threshold. In addition, when there are images with significantly different shooting times in the same group, superimposed search result images 1103 may be displayed.

  Note that the image search system may present to the user an interface screen for displaying the clicked image among a plurality of superimposed search result images.

  Further, the tracking map display unit 904 may attach a shooting number 1104 to the tip of the lead line of the search result image 1102.

  As described above, according to the image search system of the fourth embodiment, in addition to the effects of the first and second embodiments, an arbitrary region designated by the user, that is, a characteristic of the search person or object Since the portion can be set as a search key, the search accuracy can be improved. Furthermore, since the search results are displayed on the simple map in time series, it is possible to track the movement route of a specific monitoring target.

  Although the first to fourth embodiments have been described, the scope of application of the present invention is not limited to each embodiment. For example, a plurality of embodiments can be combined.

110 Server computer 111 External interface 112 Central processing unit (CPU)
113 Main memory (memory)
114 Mass external storage (HD)
120 Communication Infrastructure 130 Client Computer 140 Image Database 150 Feature Data Database 160A to 160N Camera 201 Video Input Unit 202 Still Image Extraction Unit 203 Block Division Unit 204 Feature Value Extraction Unit 301 Image Acquisition Unit 302 Block Division Unit 303 Image Display Unit 304 Block Selection Unit 305 search unit 306 search result integration unit 307 search result display unit 401 selected block area 410 camera image database 420 thumbnail database 501 image input unit 502 block division unit 601 image acquisition unit 602 image display unit 603 search unit 701 image acquisition unit 702 image Display unit 703 Block selection unit 704 Search unit 705 Search result integration unit 706 Search result display unit 710 Image database 901 Search result integration unit 902 Search result display unit

Claims (16)

An image search system for searching for an image similar to an image specified by a user,
Comprising at least one computer comprising a processor, a memory connected to the processor, and a storage device;
The processor is
Obtain video captured by the camera or video stored in the storage device,
Extracting a predetermined number of images per unit time from the acquired video,
Dividing the extracted image into blocks of a predetermined size;
Based on the color histogram, the feature amount of the divided block is extracted,
A first relationship indicating a correspondence relationship between the identifier of the extracted image, information indicating where the block is located in the extracted image region, and the feature amount of the extracted block Create information,
Based on the first information, among the divided blocks, at least one block selected by the user is used as a first search key, and the feature amount of the first search key and the feature of another block A predetermined number of blocks similar to the first search key is detected as a search result by comparing the amount,
An image search system, wherein an image including the detected search result is displayed to the user. Creating second information indicating a correspondence relationship between the identifier of the extracted image, the identifier of the camera that has captured the extracted image, and the shooting time of the extracted image;
Based on the second information, the camera that has captured the image including the first search key and the shooting time are identified,
Of images taken by the specified camera, obtain images taken within a predetermined time before and after the specified shooting time,
Based on the first information, the feature amount of the first search key is compared with the feature amount of the block included in the acquired image within the predetermined time, thereby obtaining the first feature. Detecting a predetermined number of blocks similar to the search key of the second search key,
Further, a predetermined number of blocks similar to the second search key are detected as a search result by comparing the feature quantity of the detected second search key with the feature quantity of the other block,
The image search system according to claim 1, wherein an image including the detected search result is displayed to the user. Of the extracted images, a difference in pixel value between a first image in which a person is reflected and a second image only in a background in which no person is reflected is calculated,
Based on the difference between the calculated pixel values, a background difference image between the first image and the second image is created,
Dividing the created background difference image into blocks of a predetermined size;
Calculating the sum of the pixel values of the divided blocks as foreground,
The image search system according to claim 1, wherein when the foreground level of the divided block exceeds a predetermined threshold, the feature amount of the block is extracted based on a color histogram.   The image search system according to claim 1, wherein the first search key is a plurality of blocks selected by the user among the divided blocks.   2. The image search system according to claim 1, wherein the first search key is a region having a size and a shape designated by the user among regions of the divided blocks. Based on the distance between the feature quantity of the first search key and the feature quantity of the other block, the similarity indicating the similarity of the feature between the first search key and the other block is calculated. And
Detecting a predetermined number of other blocks having a high degree of similarity with the first search key as a search result;
Creating second information indicating a correspondence relationship between the identifier of the extracted image, the identifier of the camera that has captured the extracted image, and the shooting time of the extracted image;
Based on the created second information, the camera that captured the image including the detected search result is identified,
Classifying an image including the detected search result according to the identified camera identifier;
Among the classified images, an image including a block having the largest feature amount similarity with the first search key and including a block having the similarity greater than a predetermined threshold is displayed. The image search system according to claim 1. Create a map of the space where the camera is installed,
Of the images classified according to the camera identifier, obtain at least one image including a block having a large similarity between the first search key and the feature amount;
The correspondence between the acquired at least one image and the shooting range of the camera is displayed on the created map. Further, based on the second information, the shooting time of the image classified according to the identifier of the camera is displayed. Identify,
The image search system according to claim 6, wherein information indicating a shooting order of the images is displayed on the created map according to the specified shooting time.   The image search system according to claim 1, wherein the image is divided into blocks having different sizes according to the size and position of a person shown in the extracted image. An image search method executed in an image search system for searching for an image similar to an image designated by a user,
The image search system includes at least one computer including a processor, a memory connected to the processor, and a storage device,
The image search method includes:
The processor is
Obtain video captured by the camera or video stored in the storage device,
Extracting a predetermined number of images per unit time from the acquired video,
Dividing the extracted image into blocks of a predetermined size;
Based on the color histogram, the feature amount of the divided block is extracted,
A first relationship indicating a correspondence relationship between the identifier of the extracted image, information indicating where the block is located in the extracted image region, and the feature amount of the extracted block Create information,
Based on the first information, among the divided blocks, at least one block selected by the user is used as a first search key, and the feature amount of the first search key and the feature of another block A predetermined number of blocks similar to the first search key is detected as a search result by comparing the amount,
An image search method, comprising: displaying an image including the detected search result to the user. The processor is
Creating second information indicating a correspondence relationship between the identifier of the extracted image, the identifier of the camera that has captured the extracted image, and the shooting time of the extracted image;
Based on the second information, the camera that has captured the image including the first search key and the shooting time are identified,
Of images taken by the specified camera, obtain images taken within a predetermined time before and after the specified shooting time,
Based on the first information, the feature amount of the first search key is compared with the feature amount of the block included in the acquired image within the predetermined time, thereby obtaining the first feature. Detecting a predetermined number of blocks similar to the search key of the second search key,
Further, a predetermined number of blocks similar to the second search key are detected as a search result by comparing the feature quantity of the detected second search key with the feature quantity of the other block,
The image search method according to claim 9, wherein an image including the detected search result is displayed to the user. The processor is
Of the extracted images, a difference in pixel value between a first image in which a person is reflected and a second image only in a background in which no person is reflected is calculated,
Based on the difference between the calculated pixel values, a background difference image between the first image and the second image is created,
Dividing the created background difference image into blocks of a predetermined size;
Calculating the sum of the pixel values of the divided blocks as foreground,
The image search method according to claim 9, wherein when the foreground level of the divided block exceeds a predetermined threshold, a feature amount of the block is extracted based on a color histogram.   The image search method according to claim 9, wherein the first search key is a plurality of blocks selected by the user among the divided blocks.   The image search method according to claim 9, wherein the first search key is an area having a size and a shape designated by the user among the divided block areas. The processor is
Based on the distance between the feature quantity of the first search key and the feature quantity of the other block, the similarity indicating the similarity of the feature between the first search key and the other block is calculated. And
Detecting a predetermined number of other blocks having a high degree of similarity with the first search key as a search result;
Creating second information indicating a correspondence relationship between the identifier of the extracted image, the identifier of the camera that has captured the extracted image, and the shooting time of the extracted image;
Based on the created second information, the camera that captured the image including the detected search result is identified,
Classifying an image including the detected search result according to the identified camera identifier;
Among the classified images, an image including a block having the largest feature amount similarity with the first search key and including a block having the similarity greater than a predetermined threshold is displayed. The image search method according to claim 9. The processor is
Create a map of the space where the camera is installed,
Of the images classified according to the camera identifier, obtain at least one image including a block having a large similarity between the first search key and the feature amount;
The correspondence between the acquired at least one image and the shooting range of the camera is displayed on the created map. Further, based on the second information, the shooting time of the image classified according to the identifier of the camera is displayed. Identify,
The image search method according to claim 14, wherein information indicating a shooting order of the images is displayed on the created map according to the specified shooting time. The processor is
The image search method according to claim 9, wherein the image is divided into blocks having different sizes according to the size and position of a person shown in the extracted image.