JPWO2014065033A1 - Similar image search device - Google Patents

Abstract

Simplify the repetitive operation when specifying a search key image (face) from a video. When a face is clicked, the face detection processing is automatically performed on the image advanced by one frame, so that the person can be selected one after another, and scenes where the person appears continuously in time can be selected without operating the viewer. I did it. When the person is not detected (for example, overlaps with the previous person), the “next button” is pressed. As another example, when one face is specified, all face detection results in the vicinity of the specified image (for example, 10 seconds from 5 seconds before to 5 seconds) are displayed, and the user specifies the face of the person from among them. By doing so, you can specify multiple images of a series of people without operating the viewer. As another example, the result of automatically tracking the person is displayed on the operation terminal, so that the user can determine whether the result of the automatically tracked person is correct and correct it if there is an error. .

Description

  The present invention relates to a similar image search apparatus, and more particularly to a similar image search apparatus that performs a search by designating a plurality of desired search keys from a moving image.

2. Description of the Related Art Conventionally, there has been known a person search system that allows a computer to search for a desired person using video recognition technology or the like from video (moving images) shot or recorded by a surveillance camera or the like (for example, Patent Documents 1 to 4). reference.). Such a technique for searching based on the characteristics of the image itself without relying on external information such as tagging is generally called CBIR (Content-Based Image Retrieval).
The image recognition techniques used in Patent Documents 1 to 4 cut out a portion where a person (a face) is reflected from an image, extract a color histogram or the like as a feature quantity for individually identifying the person, and this feature quantity is desired. If it is similar to that of a person, it is estimated that they are the same person.

In such a similar face image search system, the accuracy of search is affected by key image conditions, such as facial expressions, face orientation, lighting conditions, and face detection (cutout) misalignment). Often appear as search results.
On the other hand, the user desires to be able to search for the person specified by the search key without being affected by such conditions. In such a case, the user can use a multiple key search function for searching for a plurality of designated key images using one key image at a time and combining and displaying the results. That is, first, a desired person image (one image is sufficient) is searched as a key image. Check the search results, and if images of the same person are included, then designate each frame of a moving image consisting of images before and after that image as a key image and use the multiple key search function again. Search for. This improves the possibility of finding images with various conditions.

JP 2009-027393 A JP2011-237879A JP 2011-107995 A JP 2011-048668 A JP 2010-140425 A

Michael J. Swain and Dana H. Ballard, “Color indexing”, International Journal of Computer Vision, USA, Springer, January 1991, Vol. 7, No. 1, p. 11-32

As described above, a user who expects the best search specifies as many images of a person who wants to search as possible from the already obtained moving images, designates all of them as key images, and tries to search for unspecified images. At this time, if it is guaranteed that the search target person is shown only once (one scene or one scene in which a plurality of frames are consecutive) in the acquired video (for example, synchronized with passing in a lane or the like in which only one person can pass). Operation such as shooting) can easily select all the images of the person in the video automatically.
On the other hand, if there is a possibility that the search target person appears multiple times as in normal security camera video, and there is a possibility that other persons will also be shown together in those images, the user can view each frame of the video. After operating the playback device for frame advance by mouse click, press the image designation button to detect the face, and specify the face of the person you want to search from the multiple faces that contain the face detection result by clicking with the mouse It is necessary to repeat the procedure as many times as the number of images, which is very troublesome.

This problem will be described in detail with reference to FIGS. FIG. 8 is a diagram showing a search screen 400 displayed on the conventional terminal device 103.
8 includes a reproduction image display area 301, an image reproduction operation area 303, a search key image designation area 304, a search refinement parameter designation area 308, a search execution area 317, and a search result display area 320.
An image reproduction operation area 303 is an area for performing an operation for reproducing an image recorded in the recording apparatus. Each button constituting the image playback operation area 303 is assigned with a unique playback type. For example, from the left, playback types of rewind, frame reverse, reverse playback, playback stop, forward playback, frame advance, and fast forward are provided. Is assigned. When the user appropriately presses each button with the mouse 282, the moving image 302 is reproduced in the reproduction image display area 301 with the reproduction type assigned to the button.

A search key image designation area 304 is an area for designating and displaying a search key image. This area includes a key original image 305, an image designation button 306, and a file read button 307.
The key original image display unit 305 is an area for displaying a key image for similarity search or an image based on the key image (referred to as a key original image). In the initial state, the search key original image is not designated, so that no image is displayed.
The image designation button 306 is a button for designating the moving image 302 currently displayed in the image reproduction operation area 303 as a key original image. For example, when the moving image 302 is stopped and the image designation button 306 is pressed, the image at that time is designated as the key original image and is also displayed on the key original image display unit 305. Each time the key original image is newly displayed, face detection processing is executed as necessary, and a frame for cutting out the detected face is automatically added. The frame is initially in an unselected state (if there are multiple frames). When one of the frames is selected, the designation of the key image (key face) is completed.
The file reading button 307 is a button for displaying an image other than the image recorded in the recording apparatus 102, for example, an image captured by a digital camera, an image captured by a scanner, or a moving image in the image playback operation area 303. . When the file reading button 307 is pressed, a file open dialog box is displayed, and the designated file is read there, and can be played back in the image playback operation area 303, or playback starts automatically. If the file is a still image, it is designated as the key original image as it is, and is displayed on the key original image display unit 305.

The search refinement parameter designation area 308 is an area for designating the type and value (range) of the refinement parameter at the time of search. This area is composed of imaging device designation check boxes 309, 310, 311 and 312, time designation check boxes 313 and 314, and time designation columns 315 and 316.
The imaging device designation check boxes 309, 310, 311 and 312 are buttons for designating imaging devices to be searched at the time of searching. When this button is pressed, a check mark indicating that it has been selected is displayed. This mark disappears when pressed again, and is repeatedly displayed and hidden when pressed. In the initial state, all imaging devices are targeted for search, and all imaging device designation check boxes are selected.
The time designation check boxes 313 and 314 are buttons for designating a time range to be searched in the search. As for the display mode, this button is the same as other check boxes. When the time designation check box 313 is selected, the start time is given to the time range. In the non-selected state, it means that the start time is not given to the time range, that is, the image with the oldest time recorded in the recording device is set as the search target range. The same applies to the time designation check box 314, and when this is set to the non-selected state, it means that the search target range is the image up to the latest time recorded in the recording apparatus.
The time designation columns 315 and 316 are input columns for designating the above-described start time and end time values.
In the initial state, since all time zones are to be searched, all the time specification check boxes are not selected and the time specification column is blank.

The search execution area 317 is an area for instructing search execution. This area includes a similar person search button 318 and an appearance event search button 319.
The similar person search button 318 is a button for instructing execution of a similar person search by the key original image display unit 305. When a parameter is specified in the search refinement parameter specification area 308, an instruction is given to execute the search according to the specified parameter.
The appearance event search button 319 is a button for instructing execution event search execution. Usually, in surveillance camera systems, etc., events such as motion detection, human sensor notifications, notifications from other systems such as entry / exit management, etc. are recorded in association with the video indirectly, and the video recording itself is also recorded. It is only performed when there is a report, and these are called event records. In the appearance event search, only the video associated with an event that can be expected to photograph a human face from the front direction among various recorded events is set as a search target. When a parameter is specified in the search refinement parameter specification area 308, an instruction is given to execute the search according to the specified parameter.

FIG. 9 is a diagram showing operations of each procedure for searching for each frame of a moving image as a plurality of key images on the operation screen 400. When searching for each frame of a moving image as a plurality of key images, the user first displays the first image of the moving image to be used as a key image in the replay image display region 301 using the image replay operation region 303 (procedure). 1). Next, the image designation button 306 is pressed (referred to as procedure 2). Next, from the rectangle indicating the face image displayed on the key original image display unit 305, the face to be used as a search key is designated with the mouse (referred to as procedure 3). Then, the image (its ID) at this time and information on the designated face area are held inside the terminal device 103. The stored information is accumulated until the person search button 318 is pressed, and then the user uses the frame playback function in the image playback operation area 303 to send one frame of the moving picture 302 (procedure 4). Next, procedure 2 and procedure 3 are performed in the same manner as before. Thereafter, Step 2, Step 3, and Step 4 are repeated until the last image for searching each frame of the moving image as a plurality of key images is obtained.
There is a problem that the repetition of the procedure 2, the procedure 3, and the procedure 4 is a very troublesome work for the user.

  The present invention has been made in view of such a problem, and an object thereof is to provide a user interface that is easy to use for general users by eliminating monotonous repetitive work by automation.

  Briefly describing the present invention, in one aspect of the present invention, when a face is clicked, a face detection process is performed on an image advanced by one frame, whereby the person can be selected one after another. The scenes that appear continuously can be selected without operating the viewer. When the person is not detected (for example, overlaps with the previous person), the “next button” is pressed.

  In another aspect of the present invention, when one face is specified, all face detection results in the vicinity of the specified image (for example, 10 seconds from 5 seconds before to 5 seconds after) are displayed, and The user can now specify multiple images of a series of people without operating the viewer.

  In another aspect of the present invention, the result of automatically tracking the person is displayed on the operation terminal, so that the user can determine whether or not the result of the automatically tracked person is correct. I made it. For example, a series of automatically determined human faces are displayed in a bold frame and others in a broken line.

In a more specific aspect of the present invention, a partial image showing a predetermined subject is specified as a key from a moving image photographed and recorded by a camera, and has a feature amount close to the specified feature amount of the partial image. In a similar image search device for searching for an image,
Means for providing a preview for designating a desired frame from a moving image;
Means for accepting an operation of designating one previewed frame as a key original image;
Means for automatically specifying a plurality of frames that are temporally adjacent to one frame specified by the receiving means to be added to the key original image;
One or more frames designated as the key original image by the accepting means or the additional designation means are displayed with a figure indicating an area corresponding to the subject detected from the one or more frames by a predetermined algorithm. Means,
A selection means for accepting an operation of setting one of the graphics to a selected state for one frame, with the initial state of the added figure being a non-selected state;
Means for requesting a search using a plurality of subjects corresponding to the figure in the selected state as a key.

  In the similar image search device, the subject is a human face, and the feature amount is extracted in advance for each subject automatically detected from the moving image, and the automatically detected original image In response to a request from the requesting means, the plurality of subjects corresponding to the figure in the selected state are regarded as the same person, and the plurality of subjects There may be provided search execution means for searching a plurality of feature amounts corresponding to the subject one by one as a key and combining and responding to the results.

In the similar image search device, the display means adds the figure to one frame designated by the receiving means and displays the figure in the key original image display area.
The selection means accepts an operation for setting the selected state for the graphic displayed in the key original image display area, or the additional designation means in response to a single press of a predetermined button, By specifying the next frame following the one frame displayed in the key original image display area to be added to the key original image, the next frame is automatically displayed in the key original image display area, and the selection is again made. The means for repeatedly requesting the operation of the means may be configured so that a plurality of subjects corresponding to the plurality of figures selected during the repetition are used as the key.

  In the above similar image search device, the additional designating means may be in response to an operation of designating the one frame as a key original image by the accepting means, automatically or only by pressing a predetermined button once. The additional means designates a plurality of frames in which at least a subject is detected among a plurality of frames that are temporally before or after the one frame designated by the accepting means, and the means for displaying the key images further comprises: A plurality of key original images designated by the designating means are displayed in the designated key display area with the figure added thereto, respectively, and the selection means displays a plurality of key images displayed in the designated key display area. A plurality of operations for setting the graphic to the selected state with respect to the key original image of the key, and the requesting means selects a plurality of subjects corresponding to the plurality of the graphic in the selected state as the key. It may be configured to be.

In the similar image search device, the display means adds the figure to one frame designated by the receiving means and displays the figure in the key original image display area.
The automatic adding means responds only to a single press of a predetermined button after the selection means accepts an operation for making the selected state for the graphic displayed in the key original image display area, and the selection means Starting from the subject corresponding to the figure in the state, the subject that satisfies the spatio-temporal continuity from the recorded moving image is tracked, and a plurality of frames in the range in which the tracking is successful are recorded as key sources. The display means is configured to be additionally specified, and the display means is configured to display a plurality of key original images specified by the additional specification means in the specified key display area with the figure added thereto, respectively. May be.

  According to the present invention, since a plurality of search keys are specified automatically or semi-automatically from a moving image, a highly accurate search can be performed with a simple operation.

The block diagram of the similar image search system (Examples 1-3). 2 is a hardware configuration diagram of the imaging apparatus 201. The hardware block diagram of the video recording apparatus. The hardware block diagram of the terminal device 103. FIG. FIG. 10 is a diagram illustrating a search screen 300 for performing a multiple key image search (Example 1). FIG. 10 is a diagram illustrating a search screen 330 for performing a multiple key image search (Example 2). FIG. 10 is a diagram illustrating a search screen 340 for performing a multiple key image search (Example 3). FIG. 10 is a diagram illustrating a key image correction operation on a search screen 340 (third embodiment). The figure which shows the search screen 400 displayed on the conventional terminal device 103. FIG. The figure which shows the procedure of performing a multiple key image search with the conventional search screen.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of each figure, components having substantially the same function are denoted by the same reference numerals and description thereof is omitted.

First, the configuration of a similar image search system according to an embodiment of the present invention will be described with reference to FIGS. FIG. 1 illustrates a system configuration of a similar image search system according to an embodiment of the present invention.
As shown in FIG. 1, the similar image search system is configured such that an imaging apparatus 201, a recording apparatus 102, and a terminal apparatus 103 are connected to a network 200 and can communicate with each other.

The network 200 is a communication unit that performs communication by mutually connecting devices such as a dedicated network that performs data communication, an intranet, the Internet, and a wireless local area network (LAN).
The imaging device 201 performs a digital conversion process on an image captured by a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) element, or the like, and outputs the converted image data to the recording device via the network 200. Devices such as cameras and surveillance cameras.

The recording apparatus 102 is an apparatus such as a network video recorder that records image data input from the imaging apparatus 201 via the network 200 on a recording medium such as an HDD. It also has most of the functions for searching people.
The recording apparatus 102 has an image transmission / reception unit 210, an image recording unit 211, a reproduction control unit 212, a person area detection unit 213, a person feature amount extraction unit 214, a person feature amount recording unit 215, an attribute information recording unit 216, as a functional configuration. A request receiving unit 217, a similar person search unit 218, an appearance event search unit 219, and a search result transmission unit 220 are included.

The image transmission / reception unit 210 is a processing unit that inputs and outputs images from the outside of the apparatus, and performs reception of image data from the imaging apparatus 201, transmission of image data to the terminal apparatus 103, and the like.
The image recording unit 211 writes and reads image data to and from a recording medium. At the time of writing, in addition to the image data, an image ID (image identification information) serving as information for reading the image data is also recorded.
The playback control unit 212 controls video (stream) playback to the terminal device 103.

The person area detection unit 213 performs person detection using image recognition technology on the image data received from the imaging device 201, determines the presence of a person in the image, and if a person exists, the person is used as a reference. Thus, the coordinates of a rectangular area having a predetermined aspect ratio including the periphery of the face under predetermined conditions are calculated.
The person feature amount extraction unit 214 performs feature amount calculation on the region detected by the person region detection unit 213 using an image recognition technique. The human feature amount calculated here is one that can be extracted from a still image. For example, the detected area is scaled to a certain size, and the color and brightness, or the gradient and texture of each pixel block are uniformly divided. This is a multidimensional vector in which the histograms are obtained and the results are aggregated. In addition to the recognition of appearance-base, the one that reconstructs and recognizes the three-dimensional shape, for example, the contour that strongly depends on the skeleton of the face and the relative arrangement of the feature points corresponding to the eyes, nose, and mouth. Although what is judged on a dimension is mentioned, in this embodiment, the kind and number of the feature-values to be used may be any. In order to reduce the dimension of the feature amount, a vector quantization method such as the Linde-Buzo-Gray method can be used. In the case of a histogram-based feature amount, Block Truncation Coding may be used in which one pixel block is finally approximated with two colors by Otsu's binarization method.

The person feature amount recording unit 215 writes and reads the feature amount calculated by the person feature amount extraction unit 214 to and from the recording medium. In order to perform a high-speed search from a large amount of feature amounts, the feature amounts are classified into clusters, and the recording structure (a tree, dictionary configuration, sector arrangement on the HDD, etc.) is optimized accordingly. For simplicity, a method of classifying with a simple rule hash function, creating a dictionary that associates hash values with recording destinations, and when the number of feature quantities belonging to one category has increased excessively, a method of further classifying by further increasing the hierarchy is a method. is there. In addition, various optimization algorithms such as the EM method are known.
The attribute information recording unit 216 writes and reads attribute information related to individual image data to and from the recording medium. The attribute information is, for example, an image shooting time, an imaging device number, flags of various events, coordinates of an area detected by the person area detection unit 213, and the like.

The request receiving unit 217 receives a search request from the terminal device 103. The search request includes a similar image search request and an appearance event search request.
The similar person search unit 218 performs a similar image search when the request received by the request reception unit 217 is a similar person search request. Basically, it is determined that the smaller the difference (norm) between the two feature vectors, the more similar. In the case of a histogram-based feature quantity, the similarity for one to a plurality of blocks can be obtained by Histogram Intersection described in Non-Patent Document 1, and the weighted addition of these can be used as the similarity. Output more than the value as search results.
The appearance event search unit 219 performs an appearance event search when the request received by the request reception unit is an appearance event search request.
The search result transmission unit 220 transmits the similar person search result and the appearance event search result obtained from the similar person search unit 218 and the appearance event search unit 219 to the terminal device 103.

The terminal device 103 is a device serving as an interface between the image recording unit 211 that actually performs the search and the user, and may be realized by a general PC (personal computer) having a network function.
The terminal device 103 includes processing units such as a search request transmission unit 221, a search result reception unit 222, a search result display unit 223, a reproduction image display unit 224, and a screen operation detection unit 225 as functional configurations.

The search request transmission unit 221 transmits a search request to the recording device 102. In the case of a similar person search, a key image (key face) is stored every time it is specified, and when the similar person search button 318 or the like is subsequently pressed, the one or more search key images stored and the narrowing parameters are stored. A search request (query) including the above is transmitted, and the accumulated key image is cleared. The key image is stored and transmitted not using the image data of the key image itself but also using a feature amount or a combination of the ID of the original image from which the key image is extracted and position information in the original image. it can. The transmitted key image may be stored separately as a search history.
The search result receiving unit 222 receives the search result from the recording device 102. The data received as the search result includes a set of images obtained by performing similar person search or appearance event search in the recording apparatus 102. Individual images constituting the set are generated by performing image size reduction processing or the like from the video recorded in the recording device 102. Hereinafter, each individual image is referred to as a “search result image”, and data transmitted and received as a search result is referred to as “search result data”.
The search result display unit 223 displays a screen of the search result received by the search result receiving unit 222. An example of the displayed screen will be described later.
The reproduced image display unit 224 uses DirectShow (trademark) or the like to decode the image data received from the recording apparatus 102 and reproduce it on the screen (moving image display).
The screen operation detection unit 225 detects and acquires operation content performed by the user.

FIG. 2 illustrates a hardware configuration of the imaging apparatus 201 used in the similar image search system according to the embodiment of the present invention.
As shown in FIG. 2, the hardware configuration of the imaging apparatus 201 is a form in which an imaging unit 241, a main storage unit 242, an encoding unit 243, and a network interface (I / F) 245 are coupled via a bus 240.

  The imaging unit 241 converts the optical signal captured by the lens into digital data. The encoding unit 243 encodes the digital data output from the imaging unit 241 and converts it into image data such as JPEG (Joint Photographic Experts Group). The main storage unit 242 stores captured digital data and encoded image data. The network I / F 245 is an interface for transmitting image data on the main storage unit 242 to the recording apparatus 102 via the network 200.

FIG. 3 illustrates a hardware configuration of the recording apparatus 102 used in the similar image search system according to the embodiment of the present invention.
As shown in FIG. 3, the hardware configuration of the recording apparatus 102 is a form in which a CPU (Central Processing Unit) 251, a main storage unit 252, an auxiliary storage unit 253, and a network I / F 254 are coupled via a bus 250.

  The CPU 251 performs control of each unit of the recording device 102 and execution of a program for realizing the function. The main storage unit 252 is realized by a semiconductor device such as a DRAM (Dynamic Random Access Memory), and is an intermediate memory for loading and storing image data for search and a program executed by the CPU 251. The auxiliary storage unit 253 is realized by an HDD, a flash memory, or the like, and has a larger capacity than the main storage unit 252 and stores image data and programs. The network I / F 254 is an interface for receiving image data from the imaging apparatus 201, receiving a search keyword from the terminal apparatus 103, and transmitting image data to the terminal apparatus 103 via the network 200.

FIG. 4 illustrates a hardware configuration of the terminal device 103 used in the similar image search system according to the embodiment of the present invention.
As shown in FIG. 4, the hardware configuration of the terminal device 103 includes a CPU 261, a main storage unit 262, an auxiliary storage unit 263, a display I / F 264, an input / output I / F 265, and a network I / F 266 that are coupled via a bus 260. It is a form. The display I / F 264 is connected to the display device 270, and the input / output I / F 265 is connected to input / output devices such as a keyboard 280 and a mouse 282.

  The CPU 261 performs control of each unit of the terminal device 103 and execution of a program for realizing the function. The main storage unit 262 is realized by a semiconductor device such as a DRAM, and is an intermediate memory for loading and storing image data for display and a program executed by the CPU 261. The auxiliary storage unit 263 is realized by an HDD, a flash memory, or the like, and has a larger capacity than the main storage unit 262, and stores search keywords, image data, and programs. The display I / F 264 is an interface for connecting to the display device 270. The input / output I / F 265 is an interface for connecting to input / output devices such as a keyboard 280 and a mouse 282. The network I / F 266 is an interface for receiving image data from the recording apparatus 102 and transmitting a search keyword to the recording apparatus 102 via the network 200. The display device 270 is a device such as an LCD (Liquid Crystal Display), for example, and is a device that displays an image or a moving image on its display unit. The user operates the input / output device such as the keyboard 280 and the mouse 282 on the image displayed on the display unit of the display device 270, for example, the GUI (Graphical User Interface), thereby performing the terminal device 103 and the like. Operate the image search system.

  Next, multiple key image search in the similar image search system according to the embodiment of the present invention will be described with reference to Examples 1 to 3.

Example 1 of the similar image search system according to the embodiment of the present invention will be described with reference to FIG.
FIG. 5 is a diagram illustrating a search screen 300 displayed on the terminal device 103 according to the first embodiment. The search screen 300 of this example is different from the conventional one in that a next button 321 is newly provided.

Using the search screen 300 of this example, the procedure for the user to search for each frame of a moving image as a plurality of key images is roughly as follows.
First, procedure 1 is performed as in the conventional case. That is, using the image reproduction operation area 303, the first image (or an image in an arbitrary moving image) of the moving image that is desired to be used as the key image is displayed in the reproduced image display region 301. In order to designate and reproduce the camera and time from the moving image recorded in the recording device 201, a general technique in the field of the surveillance camera system may be used.
Next, procedure 2 is performed as in the conventional case. That is, when the image designation button 306 is pressed, the moving image 302 (the first image) displayed at that time is captured as a key original image and displayed as the key original image 305 in the search key image designation area 304. .

Next, as step 3, if there is a face to be used as a search key in the displayed key original image 305, it is designated with the mouse 226 as in the conventional procedure 3, and if there is no face to be used as a search key, Press the next button 321.
At the moment when the face is designated, information on the image at this time and the designated face area is held in the terminal device 103. Or you may hold | maintain the feature-value of a face image (calculating suitably in the terminal device 103). After the holding, the terminal device 103 updates the key original image 305 to the one advanced by one frame and displays it in the search key image designation area 304. When the next button 321 is pressed, the key original image 305 is updated to one advanced by one frame without holding.

  Thereafter, the user only has to repeat the procedure 3 while the desired person appears in the image, and the procedures 2 and 4 can be omitted. The simplest method for implementing this example is a method (emulation) for generating the same key code or mouse event as the user's operation.

  Hereinafter, an example of the processing of this example will be described in detail. In order to realize this example, some optimal implementations are conceivable depending on when and where the face detection and facial image feature quantity calculation is performed, and whether the image during frame advance is transmitted and decoded. The implementation is a method (emulation) for generating the same key code and mouse event as the user's operation. In the following description, it is assumed that face detection and feature amounts are stream-transmitted to the terminal device 103 in a state where they are embedded in an image (moving image).

The process in Procedure 1 as in the prior art is as follows. In the case of a moving picture such as MPEG, a session is started between the recording apparatus 102 and the terminal apparatus 103 using a protocol such as RTSP or Media Resource Control Protocol (MRCP), and a reproduction position is designated from the reproduction image display unit 224 of the terminal apparatus 103. It is assumed that an image of a desired frame is displayed in the reproduction image display area 301 by transmitting the PLAY message or the like. At this time, the frame after the desired frame is also stored in the reception buffer of the reproduced image display unit 224. Even in the method of transmitting still images one by one, pre-fetching is performed to similarly store the images.
In the case of a moving image, the DirectShow input filter (splitter) of the reproduction image display unit 224 extracts the camera name, shooting time, image ID, serial number and coordinates (and feature amount) of the face area in the image, and draws the drawing filter. And pass to application software. In the case of a still image, the image embedded in the header of the image file is similarly extracted. The input filter passes the MPEG video elementary stream to the decoding filter.
The drawing filter (renderer) converts the camera name and shooting time into characters and images, superimposes them on the moving image (still image) decoded by the decoding filter, and performs a drawing process on the display I / F 264. As a result, the moving image 302 is displayed in the reproduced image display area 301. The writing filter (grabber) passes the decoded moving image (still image) to the application software in a predetermined format in units of frames. Playback control such as rewinding is performed using the IMediaControl and IMediaSeeking interfaces. For rewinding, it is sufficient to give a negative value with the Set_Rate method, but if it is not supported, playback frames are specified one by one with the SetPositions method. The same function as the export filter can be realized by using the GetCurrentImage method of the IBasicVideo interface or the Multimedia Streaming API.

  The process in the procedure 2 as in the prior art is as follows. That is, when the application software is notified from the screen operation detection unit 225 that the image designation button 306 has been pressed, the image for one frame received from the writing filter or the like (the image displayed as the moving image 302). Then, a frame image corresponding to the received coordinates of the face area is superimposed on the display I / F 264 to perform drawing processing. As a result, the same image as that displayed on the moving image 302 is displayed as the key original image 305.

The processing in procedure 3 of this example is as follows. That is, when the application software is notified from the screen operation detection unit 225 that the mouse operation has been performed in the search key image designation area 304, the application software compares the coordinates of the mouse operation with the coordinates displaying the frame of the face area. Determine whether there is a corresponding frame. If there is a corresponding frame, the serial number and coordinates (and feature amount) of the face area corresponding to the frame are stored in the array together with the image ID, assuming that the frame has been selected. Thereby, the partial image in the selected frame becomes the designated key image.
Next, one frame is advanced by the Step method of the IVideoFrameStep interface, and the image and the serial number and coordinates (and feature amount) of each face area in the image are received from the input filter or the like, and the key original image 305 is obtained in the same manner as in the above procedure 2. indicate.

  In this example, in order to simplify the operation as much as possible, an interface such as a list of specified key images, correction (reselecting a frame in the original image), deletion, etc. is not explicitly provided. It does not prevent you from providing.

Example 2 of the similar image search system according to the embodiment of the present invention will be described with reference to FIG. The description of the same parts as those in the first embodiment will be omitted.
FIG. 6 is a diagram illustrating a search screen 330 when the terminal device 103 according to the second embodiment performs a multiple key image search. The search screen 330 in this example includes a near all display button 331 instead of the next button 321, and further includes a designated key display area 332 for selecting a face for the automatically added key original image. Thus, the first embodiment is different from the first embodiment.

Using the search screen 330 of this example, the procedure for the user to designate and search one frame at a time as a plurality of key images is roughly as follows.
First, similar to the first embodiment, the conventional procedure 1 and procedure 2 are performed. That is, the image designation button 306 is pressed in a state where a desired image (however, it may not be the first one in the continuous images) displayed in the reproduced image display area 301 is displayed.

  Next, as procedure 3, the all near display button 331 is pressed. Then, an image having one or more face detection results in the moving image 302 in the vicinity of the image when the image designation button 306 is pressed in step 2 (for example, 10 seconds from 5 seconds before to 5 seconds after) is designated key. All are displayed in the display area 332. FIG. 6 shows an example in which three key original images 333, 334, and 335 are displayed. The terminal device 103 superimposes and displays a frame corresponding to the boundary of the face area on the key original images 333 to 335 based on the serial number and coordinates of the face area in the image.

Next, as step 4, the face of the target person is selected for each image displayed in the designated key display area 332 as in step 3 of the first embodiment. When a face is selected, the frame attached to the face is redrawn in a frame indicating the selected state (for example, a thick frame), and the image ID and information on the designated face area are held in the terminal device 103. .
An image without a target person's face may be left without selecting any face. In addition, when another face is clicked in the face-selected image, the newly clicked face is selected, the original face is not selected, and the information of the new face area is overwritten with the stored content. Thus, the face selected in the designated key display area 332 when the similar person search button 318 is pressed becomes a key image (key face).

According to this example, the repetition of the conventional procedure 2 and procedure 4 can be omitted.
The details of the process of the procedure 3 of this example will be supplemented below.
When the application software is notified from the screen operation detection unit 225 that the all near display button 331 has been pressed, the application software returns the playback position to, for example, 5 seconds ago by the SetPositions method. Manipulate the filter graph and connect the output pin to the Null renderer. Also, maximize the playback rate as much as possible. As a result, the frame image (GDI bitmap object) and the serial number and coordinates of the face area in the image are sequentially received, held in the memory, and displayed side by side in the designated key display area 332. The image is held in the memory because redrawing is required by reselecting the frame or scrolling the designated key display area 332. If the processing is faster in this case, the display size may be reduced to the display size in the designated key display area 332 instead of the original size. The image ID, the serial number of the face area, and the coordinates are stored in an array or the like.
When 10 seconds worth of image is captured, playback is stopped and the output pin is returned to the original renderer.
By connecting to the Null renderer, the display of the moving image 301 in the reproduction image display area 301 is not updated during this period. In addition, since the normal DirectShow filter operates in the thread of the upstream filter, it is easy to cause a deadlock with the thread of the application. Therefore, it is desirable to extract the image data from the filter graph using the Multimedia Streaming API.

  In this example, a large number of key original images (candidates) are displayed in the designated key display area 332, and the user selects one face at a time while scrolling appropriately. For this reason, the face is displayed in the same size as that displayed in the search key image designation area 304 so that the face can be easily confirmed. Note that the key original image may be displayed in the original size by reducing the display (icon) as a default and matching the mouse 226 to that, or may be sequentially hidden from the image after the face has been selected.

Example 3 of the similar image search system according to the embodiment of the present invention will be described with reference to FIGS. 7A and 7B. The description of the same parts as those in the first and second embodiments will be omitted.
FIG. 7A is a diagram illustrating a search screen 340 when a multiple key image search is performed on the terminal device according to the third embodiment. This example is different from Example 2 in that the designation of the key image in the designated key display area 332 is further automated.

Using the search screen 340 of this example, the procedure for the user to designate each frame of a moving image as a plurality of key images is roughly as follows.
First, similarly to the second embodiment, the conventional procedure 1 and procedure 2 are performed. That is, the image designation button 306 is pressed while a desired image showing the target person is displayed in the reproduction image display area 301.

Next, as a procedure 3, as in the first embodiment, a face to be used as a search key is designated with the mouse 226 from the rectangle indicating the face in the original image displayed on the key original image display unit 305.
Next, as procedure 4, the all near display button 331 is pressed. Then, the person tracking process is performed on the face person specified in step 3 in the temporal vicinity of the image specified in step 3, and as a result, all the images including the face made the same person are displayed in order of time. Further, among the frames indicating the face detection results, the face frames that are the same person are automatically selected.

  Next, as step 5, in the key original image displayed in the designated key display area 332, it is confirmed whether another person's face has been selected by mistake. If there is an error, the correct face is selected again. . As a result, as shown in FIG. 7B, the frame attached to the face is redrawn into a frame indicating the selected state (for example, a thick frame), and the information on the new face area is overwritten on the held content. Thus, the face selected in the designated key display area 332 when the similar person search button 318 is pressed becomes a key image (key face).

Here, the detail of the procedure 4 of this example is supplemented. When the application software receives a notification that the all near display button 331 has been pressed, it checks whether there is a frame selected in the key original image 305. If there is a selected frame, the information (image ID, serial number, coordinates, feature amount) is stored at the top of the array. Then, the previous frame is designated using the SetPositions method or the like, and the image and the serial number and coordinates (and feature amount) of the face area in the image are received from the input filter. If there is no face area, the data is discarded and the previous frame is designated. If there is more than one face area, find the face area whose coordinates are closest to the coordinates (center of gravity coordinates) of the face area already held in the array (the last), and the position (center of gravity coordinates) and size of the area are It is inspected whether a predetermined continuity condition is satisfied (for example, whether the difference from the output of the Kalman filter is within a predetermined value). If the continuity condition is satisfied, it is presumed that the person is the same person, so that it is stored in the array, and the previous frame is designated, and the same process is repeated until the same person is lost (lost). In addition, it is determined that the frame has been lost when the continuity condition cannot be satisfied continuously for a plurality of frames. In addition, the same processing is repeated for the image temporally after the key original image 305 displayed when the all near display button 331 is pressed.
Since the feature quantity is also acquired, the similarity between the feature quantity held at the beginning of the array (the face feature quantity specified by the user in step 3) and the feature quantity of each face area of the current frame is also obtained. The accuracy can be expected by using the inspection.

  In this way, the person tracking processing interruption in this example can be performed based on the spatiotemporal continuity of the face area on the terminal device 103 side, and for example, the method described in Patent Document 5 can be used. Furthermore, tracking by means other than the result of face detection can be used together. The simplest method is template matching using the partial facial image specified in step 3. For the initial template, it is recommended to perform motion detection by the difference method near the specified image in time and select an image of the entire person corresponding to the specified face (a plurality of frames having a size close to their median value) ) May be used. Further, a body part other than the face may be cut out from the image of the whole person, and the similarity of the feature amount extracted from the body image may be used together with the similarity of the face. Various moving object tracking methods using image processing are known and can be applied to this example. Even if multiple tracking methods are used together, it is expected to be supplemented again by continuing to give the tracking results of other methods to the lost method even if it is lost by one method, and it can be tracked until it is lost by all methods, and it is robust Improves. Alternatively, a means for directly measuring a distance or a three-dimensional shape such as a laser sensor may be provided in the camera, and an object may be detected and tracked from the measurement result. The space in spatiotemporal continuity may be two-dimensional, and is not limited to an image space such as pixel coordinates, but may be a geographical space such as longitude and latitude. By handling the measurement results of position detection means such as a plurality of cameras in a common coordinate system, tracking can be performed without being conscious of each camera.

As described above, the embodiment of the present invention is suitable when a desired person who can be expected to be continuously reflected in a time-series image (moving image) is designated as a search key image collectively. is there. Note that the search target is not limited to a recorded image, but can be applied to a system that searches (collates) a captured image in real time. In addition, the moving image is not limited to the image designated from the reproduction image display area, and by selecting a plurality of still image files (those photographed in time series) in the dialog box when the file reading button 307 is pressed, The key original image may be designated collectively.
Here, the configuration of the system and apparatus according to the present invention is not necessarily limited to the configuration described above, and various configurations may be used. The present invention also provides, for example, a method or method for executing the processing according to the present invention, a program for causing a computer to implement such a method or method, a tangible medium for recording the program, and the like. You can also.

  It can be used for CCTV (Closed-Circuit Television) system, face authentication system, criminal database, etc., as well as TV program production system, personal electronic album, and other systems that handle video images of people and vehicles. .

102: Recording device, 103: Terminal device, 113: Multiple search key selection unit,
200: Network 201: Imaging device 210: Image transmission / reception unit 211: Image recording unit 212: Playback control unit 213: Person area detection unit 214: Person feature amount extraction unit 215: Person feature amount recording unit 216: Attribute information recording unit, 217: Request receiving unit, 218: Character search unit, 219: Appearance event search unit, 220: Search result transmission unit,
240: Bus, 241: Imaging unit, 242: Main storage unit, 243: Encoding unit, 245: Network I / F,
250: Bus, 251: CPU, 252: Main memory, 253: Auxiliary memory, 254: Network I / F,
260: Bus, 261: CPU, 262: Main memory, 263: Auxiliary memory, 264: Display I / F, 266: Network I / F, 270: Display device, 280: Keyboard, 282: Mouse,
300, 330, 340: Search screen, 301: Reproduction image display area, 302: Movie, 303: Image reproduction operation area, 304: Search key image designation area, 305: Search key image, 306: Image designation button, 307: File Read button, 308: Search refinement parameter designation area, 309-312: Imaging apparatus designation check box, 313, 314: Time designation check box, 315, 316: Time designation column, 317: Search execution area, 318: Similar person search Button, 319: appearance event search button, 320: search result display area,
331: All neighborhood display button, 332: Designated key display area, 333-335: Key original image.

Claims (5)

In a similar image search device that specifies a partial image in which a predetermined subject is captured as a key from a moving image photographed and recorded by a camera, and searches for an image having a feature amount close to the feature amount of the specified partial image.
Means for providing a preview for designating a desired frame from a moving image;
Means for accepting an operation of designating one previewed frame as a key original image;
Means for automatically specifying a plurality of frames that are temporally adjacent to one frame specified by the receiving means to be added to the key original image;
One or more frames designated as the key original image by the accepting means or the additional designation means are displayed with a figure indicating an area corresponding to the subject detected from the one or more frames by a predetermined algorithm. Means,
A selection means for accepting an operation of setting one of the graphics to a selected state for one frame, with the initial state of the added figure being a non-selected state;
And a means for requesting a search using a plurality of subjects corresponding to the graphic in the selected state as a key. The subject is a human face;
The feature amount is extracted in advance for each subject automatically detected from the moving image and recorded in association with the automatically detected original image,
In response to the request from the requesting means, the plurality of subjects corresponding to the figure in the selected state are regarded as the same person, and a plurality of feature amounts corresponding to the plurality of subjects are set as one. 2. The similar image retrieval system according to claim 1, further comprising retrieval execution means for retrieving each key as a key and combining and responding to the results. The display means adds the figure to one frame designated by the receiving means and displays it in the key original image display area.
The selection means accepts an operation for setting the selected state for the graphic displayed in the key original image display area, or the additional designation means in response to a single press of a predetermined button, By specifying the next frame following the one frame displayed in the key original image display area to be added to the key original image, the next frame is automatically displayed in the key original image display area, and the selection is again made. The means repeats the action of accepting the operation,
The similar image retrieval apparatus according to claim 1, wherein the requesting unit uses a plurality of subjects corresponding to the plurality of figures selected during the repetition as the keys. The additional designating means is designated by the accepting means automatically after the operation for designating the one frame as the key original image is accepted by the accepting means or only by pressing the predetermined button once. In addition, a plurality of frames in which at least a subject is detected are additionally specified as a key original image among a plurality of frames that are temporally before or after one frame,
The display means is to display a plurality of key original images designated by the additional designation means in the designated key display area with the figure added thereto,
The selection means accepts a plurality of operations for setting the graphic in the selected state for a plurality of key original images displayed in the designated key display area,
The similar image search apparatus according to claim 1, wherein the requesting unit uses a plurality of subjects corresponding to the plurality of graphics in the selected state as the key. The display means adds the figure to one frame designated by the receiving means and displays it in the key original image display area.
The automatic adding means responds only to a single press of a predetermined button after the selection means accepts an operation for making the selected state for the graphic displayed in the key original image display area, and the selection means Starting from the subject corresponding to the figure in the state, the subject that satisfies the spatio-temporal continuity from the recorded moving image is tracked, and a plurality of frames in the range in which the tracking is successful are recorded as key sources. It is to be added to the image,
5. The display unit according to claim 2, wherein the display unit displays a plurality of key original images designated by the additional designation unit in the designated key display area with the figure added thereto. Similar image search device.

Images