JP5826513B2 - Similar image search system - Google Patents

Description

  The present invention relates to a similar image search system and a similar image search method, and in particular, a similar image search system suitable for use in making a user interface for searching for a person in a video surveillance system easy to use, The present invention also relates to a similar image search method.

  Conventionally, video surveillance systems are installed in facilities visited by an unspecified number of people such as hotels, buildings, convenience stores, financial institutions, dams and roads for the purpose of crime prevention and accident prevention. This is because a person to be monitored is photographed by an imaging device such as a camera, and the video is transmitted to a monitoring center such as a management office or a security room, and the resident supervisor monitors it, and the purpose and necessity Be careful depending on the situation, or record video.

In video surveillance systems, a random access medium represented by a hard disk drive (HDD) is increasingly used as a recording medium for recording video from a conventional videotape medium. In recent years, the capacity of such recording media has been increasing.
Increasing the capacity of recording media has dramatically increased the amount of video that can be recorded, making it possible to record at many locations and for a long time, while increasing the burden of visually checking recorded images has become a problem. It's getting on.

Against this background, video surveillance systems having a search function for finding a desired video more easily are becoming widespread. In particular, in recent years, the occurrence of specific events (events) in video can be automatically detected in real time using image recognition technology and recorded along with the video, and these events can be searched afterwards. A system with a simple search function is emerging. One of them is a person search function.
The person search function is a function that records in real time the appearance of a person in a video as an object of automatic detection, and can search for a person appearance image from recorded images after the fact.

From the functional aspect, the person search function is roughly divided into the following two types.
The first is an appearance event search function. The appearance event search function is a function for simply searching for the presence or absence of a person (event) in the video. In the search result, in addition to the presence or absence of an event, if it is determined that there is an event, the number of events, the occurrence time of each event, the imaging device number that captured the event, the captured image (person appearance image), etc. are presented . In many cases, this search query is given as information for narrowing the search target range such as the event occurrence time and the imaging device number. Hereinafter, the information for narrowing down the search target range is referred to as “narrowing parameter”.
The second is a similar person search function. While the above-mentioned appearance event search function is a search that does not specify a character, this is whether or not a specific person specified by the user has been taken by an imaging device at another time or at another point. This is a function for searching for recorded images. In the search result, in addition to the presence / absence of other images showing a specific person, the number and shooting time, imaging device number, captured image (person appearing image), similarity degree to be described later, and the like are returned if present. .
The designation of the specific person is performed by the user designating one image (hereinafter referred to as a search key image) showing the person to be searched. The search key image is designated from a recorded image or an arbitrary image from an external device. In the search, the image feature quantity of the person in this search key image is extracted by image recognition technology, collated with the image feature quantity of the person in the recorded image, the similarity (similarity) is obtained, and the same person determination is performed. It is realized by doing. The extraction and recording of the human feature amount in the recorded image is performed in advance at another timing such as during video recording. Even in this search query, it is often possible to give a refinement parameter.

In both search functions, linkage information for taking out a recorded image is given to the search result, and it is possible to cue and reproduce the recorded image from the search result.
In Patent Document 1, a search key image is designated in this way, selected from search result images, displayed in another display area, and used as the next search key image. An image search apparatus that improves the performance is disclosed.
Patent Document 2 describes a similar image search system that further enhances user convenience. In this patent document 2, the user gives keywords such as a person name and features to the result of similar image search. There are two types of search methods available for image search: similar image search that excludes images with keywords from search targets, and appearance event search that searches only images with keywords. is there. The user performs similar event search a plurality of times, and performs appearance event search after determining that keywords have been assigned to a sufficiently large number of images.

JP 2009-123196 A JP 2011-090476 A

The keyword assignment function as shown in Patent Document 2 is very convenient when performing a similar image search. However, the keyword assignment function in the current similar person search has a problem that the user's input load is large.
That is, the user needs to select a number of correct images from the search results and assign keywords to the similar image search results.
Further, when it is desired to assign a keyword to the result of the appearance event search, it is necessary to take time and effort to input the keyword from the keyboard each time.
Further, in the operation of the system, for example, when it is stipulated that a full name combining a surname and a name is input as a keyword, the user needs to know the full name completely. In other words, if you have entered it before, but you have remembered your last name but forgot your first name, or if you forgot your kanji or spelling, you have to investigate and get complete information There is a problem that cannot be entered.
In view of the above problems, an object of the present invention is to provide a similar image search system with an improved keyword assignment function and improved search ease.

  In order to achieve the above object, a similar image search system according to the present invention includes an image pickup device, a recording device that records image data input from the image pickup device on a recording medium, and a terminal device. The recording device extracts the feature amount of the image data, evaluates the similarity of the images, designates a search key image, and outputs image data having a high similarity to the designated search key image. It has a means for searching and assigning a keyword to each image data, and a means for dynamically presenting the keyword assignment candidate.

  In the similar image search system of the present invention, the terminal device automatically selects and displays an image having a high degree of similarity based on a search result of the recording device.

  In order to achieve the above object, a similar image retrieval system of the present invention includes an imaging device, a recording device that records image data input from the imaging device on a recording medium, and a terminal device. The terminal device displays a result of the recording device executing the appearance event search means, and the recording device searches for an image selected by the user from a list of image data displayed on the terminal device. Search for image data with a high similarity as a key image, and if the similar image search result includes a keyword-added result, the keyword is displayed as a candidate for addition. A list of image data regardless of the image similarity It has a means to search for the appearance event which presents.

  According to the present invention, it is possible to select a large number of images from the results of similar image search and assign keywords at once.

It is a system configuration | structure figure of one Example of the similar image search system of this invention. It is a figure which shows one Example of the hardware constitutions of the imaging device used for the similar image search system of this invention. It is a figure which shows one Example of the hardware constitutions of the video recording apparatus used for the similar image search system of this invention. It is a figure which shows one Example of the hardware constitutions of the terminal device used for the similar image search system of this invention. It is a figure which shows one Example of the data structure used for the similar image search system of this invention. It is a figure which shows one Example of the data structure used for the similar image search system of this invention. It is a figure which shows one Example of the data structure used for the similar image search system of this invention. It is a process sequence diagram between the video recording apparatus 102 and the terminal device 103 in the keyword provision from the similar image search result of this invention. In the similar image search system of this invention, it shows one Example of the process sequence figure between the video recording apparatus and a terminal device in the keyword assignment from an appearance event search result. In the similar image search system of this invention, it is the figure which showed one Example of the search screen in the initial state before search execution. In the similar image search system of this invention, it is the figure which showed one Example of the search screen in the state immediately before similar person search execution. In the similar image search system of this invention, it is the figure which showed one Example of the search screen in the state immediately after similar person search execution. In the similar image search system of this invention, it is the figure which showed one Example of the search screen in the state immediately after keyword provision execution. In the similar image search system of this invention, it is the figure which showed one Example of the search screen in the state just before execution event search execution. It is the figure which showed one Example of the search screen in the state immediately after execution appearance search in the similar image search system of this invention. It is the figure which showed one Example of the search screen after the person selection operation by the user after an appearance event search in the similar image search system of this invention. In the similar image search system of this invention, it is the figure which showed one Example of the search screen in the state immediately after the keyword provision execution after appearance event search.

The similar image retrieval system of the present invention retrieves a desired image from, for example, an imaging device, a recording device that records image data transmitted from the imaging device, and image data recorded in the recording device. In a similar image retrieval system configured by requesting and displaying a terminal device, the recording device includes a first CPU that executes a first program for controlling each part of the recording device and realizing a predetermined function; A first main storage unit that loads and stores the image data for search and the first program executed by the first CPU, and a first main unit that stores the image data and the first program received from the imaging apparatus. And an auxiliary storage unit.
In addition, the terminal device includes, for example, a second CPU that executes a second program for realizing control of each unit of the terminal device and a predetermined function, image data to be displayed on the display device, and a second CPU 261. The second main storage unit for loading and storing the second program to be executed in (2), and the second auxiliary storage unit for storing the search keyword, the image data, and the second program.

Further, for example, the first CPU performs person detection using image recognition technology on the image data received from the imaging device, determines the presence of a person in the image, and if a person exists, A person area detection unit that performs coordinate calculation, a person feature amount extraction unit that performs image feature calculation on an area detected by the person area detection unit, and a feature amount calculated by the person feature amount extraction unit A personal feature amount recording unit to be recorded in the unit, a request receiving unit for receiving a search request from the terminal device, a search unit for performing a search according to the search request received by the request receiving unit, and a search result obtained from the search unit A search result transmission unit that transmits to the terminal device, a keyword recording unit that records a keyword in the main storage unit based on a keyword assignment request received by the request reception unit from the terminal device, and a keyword in the search request data received by the request reception unit. If that contained the de includes a keyword search part which performs a search on the keyword for the image data recorded in the auxiliary storage unit.
Also, for example, the second CPU receives the search request transmission unit that transmits the search request to the recording device, the search result reception unit that receives the search result image and the search result data from the recording device, and the search result reception unit. A search result display unit that displays a search result image on a screen, a playback image display unit that continuously displays image data input from a recording device, a screen operation detection unit that detects and obtains operation contents by a user, and a keyword assignment request A keyword assignment request transmitter for transmitting to the recording device is provided.

  Furthermore, the search request of the similar image search system of the present invention is a similar image search request and an appearance event search request.

  Hereinafter, an embodiment according to the present invention will be described with reference to FIGS. 1 to 7 and FIGS. 8A to 8H. In the description of each figure, the same reference numerals are assigned to components having a common function, and the description is omitted to avoid duplication as much as possible.

  First, the configuration of a similar image search system according to an embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a system configuration diagram of an embodiment of a similar image search system of the present invention. FIG. 2 is a hardware configuration diagram of the imaging apparatus. FIG. 3 is a hardware configuration diagram of the recording apparatus. FIG. 4 is a hardware configuration diagram of the terminal device.

As shown in FIG. 1, the similar image search system is configured such that an imaging apparatus 201, a recording apparatus 102, and a terminal apparatus 103 are connected to a network 200 and can communicate with each other.
The network 200 is a communication unit that performs communication by mutually connecting devices such as a dedicated network that performs data communication, an intranet, the Internet, and a wireless local area network (LAN).
The imaging device 201 performs digital conversion processing on an image captured by a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) element, or the like, and transmits the converted image data to the recording device via the network 200. Devices such as network cameras and surveillance cameras.
The recording device 102 is a device such as a network digital recorder that records image data input from the imaging device 201 via the network 200 on a recording medium such as an HDD. Further, this apparatus is equipped with a person search function including the similar image search system of the present invention.

  The recording apparatus 102 has an image transmission / reception unit 210, an image recording unit 211, a reproduction control unit 212, a person area detection unit 213, a person feature amount extraction unit 214, a person feature amount recording unit 215, an attribute information recording unit 216, as a functional configuration. The processing unit includes a request receiving unit 217, a search unit, a search result transmission unit 220, a keyword recording unit 110, and a keyword search unit 111. The search unit includes a similar person search unit 218 and an appearance event search unit 219.

In the recording device 102 of FIG. 1, the image transmission / reception unit 210 is a processing unit that inputs and outputs images from the outside of the device, receives input image data from the imaging device 201, and transmits output image data to the terminal device 103. I do.
The image recording unit 211 writes input image data to a recording medium and reads output image data from the recording medium. At the time of writing, in addition to the image data, an image ID (described later) serving as information for reading the image data is also recorded.
The playback control unit 212 controls video playback on the terminal device 103.

The person area detection unit 213 performs person detection using image recognition technology on the input image data, determines the presence of a person in the image, and, if a person exists, calculates the coordinates of that area.
The person feature amount extraction unit 214 performs feature amount calculation on the region detected by the person region detection unit 212 using an image recognition technique. The human feature value calculated here is, for example, the shape and direction of the person's outline, the skin color, the gait (how to roll the legs such as which leg is moved at what timing), or the person is specified The shape and direction of the contour of the face, which is a representative part, and the size, shape, arrangement relationship, etc. of main components such as eyes, nose, and mouth, etc., but in this embodiment, the type of feature amount to be used Any number can be used.
The person feature amount recording unit 215 writes and reads the feature amount calculated by the person feature amount extraction unit to and from the recording medium. The image data recording medium in the image recording unit and the human feature amount recording medium in the processing unit may be the same or different.
The attribute information recording unit 216 writes and reads attribute information related to image data to and from a recording medium. The attribute information is, for example, an image shooting time, an imaging device number, or the like.

The request receiving unit 217 receives a search request and a keyword assignment request from the terminal device. The search request includes a similar image search request and an appearance event search request.
The similar person search unit 218 searches for a similar image when the request received by the request reception unit is a similar person search request.
The appearance event search unit 219 searches for an appearance event when the request received by the request reception unit is an appearance event search request.
The search result transmission unit 220 transmits the similar person search result and the appearance event search result obtained from the similar person search unit 218 and the appearance event search unit 219 to the terminal device.
The keyword recording unit 110 writes and reads keywords to the recording medium based on the keyword assignment request received by the request receiving unit.
The keyword search unit 111 performs a keyword search when the keyword is included in the search request data received by the request receiving unit.

The terminal device 103 may be realized by a general PC (personal computer) having a network function, or may be a dedicated search terminal.
The terminal device 103 includes, as functional configurations, processing units such as a search request transmission unit 221, a search result reception unit 222, a search result display unit 223, a reproduction image display unit 224, a screen operation detection unit 225, and a keyword assignment request transmission unit 112. Configured. The terminal device 103 is also equipped with a person search function for realizing the present invention.

The search request transmission unit 221 transmits a search request to the recording device. In the case of the similar person search, the search request data includes a search key image. The search request data can also include a refinement parameter.
The search result receiving unit 222 receives the search result from the recording device. The data received as a search result includes a set of images obtained by performing similar person search or appearance event search in the recording apparatus. The individual images constituting the set are generated by performing image size reduction processing or the like from the video recorded in the recording device. Hereinafter, each individual image is referred to as a “search result image”, and data transmitted and received as a search result is referred to as “search result data”.
The search result display unit 223 displays a screen of the search result received by the search result receiving unit. An example of the displayed screen will be described later.
The reproduction image display unit 224 performs continuous moving image display on the screen of the image data input from the recording device.
The screen operation detection unit 225 detects and acquires the operation content by the user.
The keyword assignment request transmission unit 112 transmits a keyword assignment request to the recording device.

  With reference to FIG. 2, an embodiment of the hardware configuration of the imaging apparatus 201 used in the similar image search system of the present invention will be described. FIG. 2 is a diagram illustrating an example of a hardware configuration of an imaging apparatus used in the similar image search system of the present invention. The configuration of the imaging apparatus 201 is a configuration in which an imaging unit 241, a main storage unit 242, an encoding unit 243, and a network interface (I / F) 244 are coupled by a bus 240.

  In FIG. 2, the imaging unit 241 converts an optical signal captured by a lens into digital data. The encoding unit 243 encodes the digital data output from the imaging unit 241 and converts it into image data such as JPEG (Joint Photographic Experts Group). The main storage unit 242 stores captured digital data and encoded image data. The network I / F 244 is an interface for transmitting image data on the main storage unit 242 to the recording device 102 via the network 200.

  With reference to FIG. 3, an embodiment of the hardware configuration of the recording apparatus 102 used in the similar image search system of the present invention will be described. FIG. 3 is a diagram showing an embodiment of a hardware configuration of a recording apparatus used in the similar image search system of the present invention. As a hardware configuration of the recording apparatus 102, as shown in FIG. 3, a CPU (Central Processing Unit) 251, a main storage unit 252, an auxiliary storage unit 253, and a network I / F 254 are coupled via a bus 250. .

  The CPU 251 performs control of each unit of the recording device 102 and execution of a program for realizing the function. The main storage unit 252 is realized by a semiconductor device such as a DRAM, and is an intermediate memory for loading and storing image data for search and a program executed by the CPU 251. The auxiliary storage unit 253 is realized by an HDD, a flash memory, or the like, and has a larger capacity than the main storage unit 252 and stores image data and programs. The network I / F 254 is an interface for receiving image data from the imaging apparatus 201, receiving a search keyword from the terminal apparatus 103, and transmitting image data to the terminal apparatus 103 via the network 200.

  An example of the hardware configuration of the terminal device 103 used in the similar image search system of the present invention will be described with reference to FIG. FIG. 4 is a diagram showing an embodiment of a hardware configuration of a terminal device used in the similar image search system of the present invention. As shown in FIG. 4, the hardware configuration of the terminal device 103 includes a CPU 261, a main storage unit 262, an auxiliary storage unit 263, a display I / F 264, an input / output I / F 265, and a network I / F 266 coupled via a bus 260. It is a form made. The display I / F 264 is connected to the display device 270, and the input / output I / F 265 is connected to input / output devices such as a keyboard 280 and a mouse 282.

  In FIG. 4, the CPU 261 performs control of each unit of the terminal device 103 and execution of a program for realizing the function. The main storage unit 262 is realized by a semiconductor device such as a DRAM (Dynamic Random Access Memory), and is an intermediate memory for loading and storing image data for display and a program executed by the CPU 261. The auxiliary storage unit 263 is realized by an HDD, a flash memory, or the like, and has a larger capacity than the main storage unit 262, and stores search keywords, image data, and programs. The display I / F 264 is an interface for connecting to the display device 270. The input / output I / F 265 is an interface for connecting to input / output devices such as a keyboard 280 and a mouse 282. The network I / F 266 is an interface for receiving image data from the recording apparatus 102, transmitting a search keyword to the recording apparatus 102, and receiving image data from the recording apparatus 102 via the network 200. The display device 270 is a device such as an LCD (Liquid Crystal Display), for example, and is a device that displays an image or a moving image on its display unit. The user operates the input / output device such as the keyboard 280 and the mouse 282 on the image displayed on the display unit of the display device 270, for example, the GUI (Graphical User Interface), thereby performing the terminal device 103 and the like. Operate the image search system.

Next, a data structure used in the similar image search system according to the embodiment of the present invention will be described with reference to FIGS. 5A and 5B. 5A and 5B are diagrams showing an embodiment of a data structure used in the similar image search system of the present invention.
An example of the data structure used in the similar image search system of the present invention includes a frame table 300 as shown in FIG. 5A and an attribute information data table 310 as shown in FIG. 5B.

  The frame table 300 is a table for storing video data. An image ID 301 is assigned to the frame table 300, and frame data 302 associated with the assigned image ID is recorded. The frame data 302 is recorded, for example, as JPEG format data.

The attribute information data table 310 is a table for storing attribute information data for an image obtained as a result of analyzing image data. In the attribute information data table 310, a registration ID 311 is assigned for identification, and an image ID 312, a feature amount 313, a camera ID 314, time information 315, and a keyword 316 are recorded in association with the assigned registration ID 311.
For example, a part of the frame stored in the frame table 300 in association with the given registration ID 311 is designated by the image ID 312. In addition, the feature amount of the image is the feature amount 313, the ID of the imaging device 201 that has captured the image is the camera ID 314, the information related to the imaging time of the frame is the time information 315, and the keyword assigned to the frame is the keyword 316 And stored in each field.

  For example, when the recording frame rate is 30 fps (Frames Per Second), the image is analyzed because there is a part recognized as having a person, and the image is recorded at a maximum frame rate of about 3 fps. It is common to capture and analyze.

  Next, with reference to FIG. 6, FIG. 8A, FIG. 8B, FIG. 8C, and FIG. 8D, a selection method for automatically displaying a search result image having a high similarity in the similar image search of the present invention in a selected state will be described. . FIG. 6 is a processing sequence diagram between the recording device 102 and the terminal device 103 in assigning keywords from the similar image search result of one embodiment of the similar image search system of the present invention. 8A to 8D are diagrams showing screens of respective phases when a similar image search displayed on the display device 270 of the terminal device 103 is performed in the similar image search of the present invention.

  In FIG. 6, an axis 501 represents a time series that proceeds from the top to the bottom of the recording apparatus 102, and an axis 502 represents a time series that proceeds from the top to the bottom of the terminal apparatus 103. Timings 503 to 506 represent timings on the time series, respectively. An example of a screen displayed on the terminal device 103 at each timing and an example of user operation contents will be described later.

In FIG. 6, communications 510 to 512 indicate main communications between the recording device 102 and the terminal device 103.
Communication 510 and communication 511 are in a request-response relationship, communication 510 includes a similar person search request, and communication 511 includes a similar person search result. One similar person search is executed by these two communications.
Communication 512 includes a keyword assignment request for an image.
Here, the similar person search result included in the communication 511 includes not only the static information of the attribute information data table 310 such as the image ID 301 but also the similarity to the search key image calculated during the search process in the recording device 102. To include.
In this embodiment, the similarity is a numerical value in the range of 0 to 100, and the closer to 100, the more similar to the search key image.

  Next, a user operation image in the terminal device 103 of the similar image search system of the present invention will be described with reference to FIGS. 8A to 8D.

FIG. 8A shows an example of a search screen 801 displayed on the display unit of the terminal device 103 at the initial state before execution of the search, that is, at the timing 503 in FIG. The user starts a search from this search screen 801.
8A includes a reproduction image display area 3001, an image reproduction operation area 3003, a search key image designation area 3004, a search refinement parameter designation area 3008, a search execution area 4017, and a search result display area 4020.

In the search screen 801 of FIG. 8A, a playback image display area 3001 is an area for displaying an image recorded in the recording apparatus 102 as a moving image. A moving image 3002 in the reproduction image display area 3001 displays an image recorded in the recording device as a moving image.
An image reproduction operation area 3003 is an area for performing an operation for reproducing an image recorded in the recording apparatus.
Each button constituting this area 3003 is assigned a unique reproduction type. Each button in FIG. 8A is assigned, for example, from the left, playback types of rewind, reverse playback, playback stop, forward playback, and fast forward. When the user appropriately presses each button with the mouse 282, the moving image 3002 is switched to the reproduction type assigned to the button.

A search key image designation area 3004 is an area for designating and displaying a search key image. This area includes a search key image 3005, a video designation button 3006, and a file designation button 3007.
A search key image 3005 is an image used as a key for similarity search. In the initial state, the search key image is not specified, so that no image is displayed. In the case of non-designation, an indication of non-designation may be provided, such as displaying an image indicating a non-designated state prepared separately.

A video designation button 3006 is a button for designating an image displayed in the reproduction image display area 3001 as a search key image 3005 when pressed.
The file designation button 3007 is a button for designating an image other than the image recorded in the recording apparatus 102, for example, an image taken with a digital still camera or an image taken with a scanner as the search key image 3005. When this button 3007 is pressed, a dialog box for designating those images is displayed, and the user can designate a desired image there.

  The search refinement parameter designation area 3008 is an area for designating the type and value (range) of the refinement parameter at the time of search. This area 3008 includes imaging device (camera) designation check boxes 3009, 3010, 3011, and 3012, time designation check boxes 3013 and 3014, and time designation columns 3015 and 3016.

Imaging device designation check boxes 3009, 3010, 3011, and 3012 are buttons for designating imaging devices (cameras 1 to 4) to be searched at the time of searching. When this button is pressed, a check mark indicating that it has been selected is displayed. This mark disappears when pressed again, and is repeatedly displayed and hidden when pressed.
In the initial state, all imaging devices are targeted for search, and all imaging device designation check boxes are selected.

The time designation check boxes 3013 and 3014 are buttons for designating a time range to be searched in the search. As for the display mode, this button is the same as other check boxes. When the time designation check box 3013 is selected, the start time is given to the time range. In the non-selected state, it means that the start time is not given to the time range, that is, the search target range is up to the image of the oldest time recorded in the recording device.
Similarly, when the time designation check box 3014 is selected, the end time is given to the time range. In the non-selected state, it means that the end time is not given to the time range, that is, the image up to the latest time recorded in the recording device is set as the search target range.
The time designation fields 3015 and 3016 are input fields for designating the above-mentioned start time and end time values.
In the initial state, since all time zones are to be searched, all the time specification check boxes are not selected and the time specification column is blank.

The search execution area 4017 is an area for instructing search execution. This area includes a keyword specification check box 4021, a keyword specification field 4022, and a keyword assignment button 4023 in addition to a similar person search button 3018 and an appearance event search button 3019.
The similar person search button 3018 is a button for instructing execution of a similar person search by the search key image 3005. When a parameter is specified in the search refinement parameter specification area 3008, an instruction is given to execute the search according to the specified parameter.
The appearance event search button 3019 is a button for instructing execution event search execution. When a parameter is specified in the search refinement parameter specification area 3008, an instruction is given to execute the search according to the specified parameter.
The keyword designation check box 4021 is a button for designating whether the keyword designation field 4022 is valid or invalid. Regarding the display mode, this button is the same as the imaging device designation check box.
The keyword specification field 4022 is an input field for specifying a keyword value.
The keyword assignment button 4023 is a button for instructing execution of the keyword input in the keyword designation field 4022.
In the initial state, the keyword designation check box 4021 is not selected, and the keyword designation column 4022 is blank.
The function of the keyword and the relationship between the similar person search button 3018 and the appearance event search button 3019 and the keyword will be described later.

The search result display area 4020 is an area for displaying search results. Search results are displayed by displaying a list of search result images. In the initial state, nothing is displayed in the search result display area 4020.
Here, the user depresses the video designation button 3006, depresses the imaging apparatus designation check boxes 3009, 3010, and 3012, depresses the time designation check boxes 3013 and 3014, and enters the time designation columns 3015 and 3016. Enter "2009/6/26 15:30:20" and "2009/7/13 12:30:20" respectively.
By this operation, the search screen transits to the state immediately before the similar person search execution, that is, the state of the terminal device 103 at the timing 504 in FIG. FIG. 8B shows an example of the search screen 802 in this state.

In the search screen 802 of FIG. 8B, the search key image 3005 displays the person “Mr. A” displayed in the video 3002 as a search key image, and “camera 1, camera 2, The state that 3 imaging devices of “Camera 4” are specified and “From 6/26/15 15:30:20 to 2009/7/13 12:30:20” is specified as the time range to be searched. Show.
Here, the user presses the similar person search button 3018. By this pressing, the search screen transitions from the timing 504 to the state immediately after the similar person search execution, that is, the state of the terminal device 103 at the timing 505 in FIG. FIG. 8C shows an example of the search screen 803 in this state.

In the search screen 803 of FIG. 8C, the search result display area 4020 displays search results obtained by executing similar person search using the search key image 3005 as a key. Search results are displayed by displaying search result images 3031 to 3141 in a search result display area 4020 as a list.
The search result images 3031 to 3141 are displayed in order of similarity to the search key image, for example, from the top left to the right and then from the second left to the right. In this display example, the search result image 3031 has the highest similarity to the search key image 3005, and the search result image 3141 has the lowest similarity.
Here, what is displayed in the search result display area 4020 is “Images of Camera 1, Camera 2, and Camera 4 from 2009/6/26 15:30:20 to 2009/7/13 12:30:20. , “A similar image to Mr. A”.

In the notation shown in FIG. 8C, the circles and alphabets shown on the search result image in the search result display area 4020 are simplified representations of the person's face and person name.
For example, the search result image 3031 indicates that a person “Mr. A” appears. Of course, an actual image is displayed in this simplified display portion in the actual system display.
In the vicinity of the search result image 3031, a cue playback button 3032, a search key image designation button 3033, and a keyword target check box 4034 are provided. The same applies to other search result images.
The cue playback button 3032 is a button for instructing the start of continuous video playback starting from the search result image. For example, when the cue / play button 3032 is pressed, the moving image 3002 is switched to the search result image 3031 and the user can view the moving image starting from the search result image.

The search key image designation button 3033 is a button for designating the search result image 3031 as the search key image 3005. For example, when a search key image designation button 3033 is pressed, a search result image 3031 is displayed on the search key image 3005. As a result, the search can be performed again using the designated search result image 3005.
The similarity display column 8035 is an area for displaying the similarity to the search key image. For example, a value of 90 is displayed in the similarity display column 8035 of the search result image 3031, and a numerical value is also displayed in each of the similarity display columns of other search results.
The keyword assignment check box 4034 is a button for designating a search result image as a keyword assignment target. As for the display mode, this button is the same as other check boxes. For example, when the user presses the keyword assignment check box 4034, a check mark is displayed, and the search result image 3031 becomes a keyword assignment target.

The example shown in FIG. 8C is a search result when a similar person search is performed for the purpose of “Mr. A”. Therefore, the search result images 3031, 3041, 3051, 3061, 3081, 3091, 3121, 3141 are correct, and the search result images 3071, 3101, 3111, 3131 are incorrect.
Here, the keyword assignment check boxes of the search result images 3031, 3041, 3051, 3061, 3071, 3081, and 3091 in the similarity display column 8035 are a predetermined value or more in FIG. 8C. .

Immediately after the similar person search is executed (timing 505), the keyword target check box 4034 can be set to a non-selected state or all by providing a process for switching whether or not the keyword target check box 4034 is selected depending on the similarity. The correct image can be selected with a smaller number of selection operations compared to the conventional method of selecting.
For example, when all the search result images 3031, 3041, 3051, 3061, 3081, 3091, 3121, and 3141 that are correct answers are selected, the selection operation is performed eight times in the method in which all are selected immediately after the similar person search is executed. In the method of making all the selected states immediately after executing the similar person search, an operation of removing the selection four times is necessary. On the other hand, in the method of the present invention, a total of three operations, that is, an operation of removing one selection and an operation of selecting two times are sufficient.

  Further, the user presses the keyword specification check box 4021, inputs “Mr. A” in the keyword specification field 4022, and further presses the keyword assignment button 4023. As a result of the user's operation, the search screen 803 enters the “Mr. A” input state in the keyword designation field 4022 immediately after execution of keyword assignment. That is, the terminal state 103 at the timing 506 in FIG. FIG. 8D shows an example of the search screen 804 at the timing 506.

  Next, with reference to FIGS. 7, 8A, 8D, 8F, 8G, 8H, and 5C, a method of presenting keyword assignment candidates using similarity in appearance event search according to the present invention will be described. . FIG. 7 shows an example of a processing sequence diagram between the recording device and the terminal device in assigning keywords from the appearance event search result in the similar image search system of the present invention. FIG. 5C is a diagram showing an example of a data structure used in the similar image search system of the present invention, and shows data 320 included in the search result of the similar image search.

The appearance event search processing sequence between the recording device 102 and the terminal device 103 in the keyword assignment from the appearance event search result of the present invention will be described with reference to FIG. FIG. 7 is a processing sequence diagram between the recording device 102 and the terminal device 103 in assigning keywords from the appearance event search result in the similar image search system of the present invention.
Here, as in FIG. 6, the axes 501 and 502 in FIG. 7 represent time series that proceed from the top to the bottom of the recording device 102 and the terminal device 103. Timings 703 to 707 represent timings on the time series, respectively. An example of a screen displayed on the display unit of the terminal device 103 at each timing and an example of user operation content will be described later.

In FIG. 7, communications 711 to 718 indicate main communications between the recording device 102 and the terminal device 103.
The communication 711 and the communication 712 have a request-response relationship. The communication 711 includes an appearance event search request, and the communication 712 includes an appearance event search result. Will be executed. Further, the communication 713 includes a similar person search request, and the communication 714 includes a similar person search result. The same applies to the communications 716 and 717. The communication 715 includes a keyword assignment request for the image. The same applies to the communication 718.

Next, a user operation image in the terminal device 103 of the similar image search system of the present invention will be described with reference to FIGS. 8E, 8D, 8F, 8G, and 8H.
FIG. 8A, FIG. 8E, FIG. 8F, FIG. 8G, and FIG. 8H show screens for each phase when searching for appearance events displayed on the display device 270 of the terminal device 103 in the similar image search of the present invention. FIG.

FIG. 8A shows the initial state (search screen 801) before execution of search, as described above. This is an example of a search screen on the terminal device 103 at the timing 703 in FIG.
Here, the user depresses the imaging device designation check boxes 3009, 3010, and 3012, depresses the time designation check boxes 3013 and 3014, and displays “2009/6/26 15: Enter "30:20" and "2009/7/13 12:30:20".
By this user operation, the search screen transits to the state immediately before the appearance event search execution, that is, the state of the terminal device 103 at the timing 704 in FIG. FIG. 8E shows an example of the search screen 805 in this state.

“Camera 1, Camera 2, Camera 4” is designated as the imaging device to be searched, and the time range to be searched is from “2009/6/26 15:30:20 to 2009/7/13 12:30 : Up to 20 ”is indicated.
Compared to the search screen 802 in FIG. 8B, the search key image 3005 is not specified in the search screen 805 in FIG. 8E.
Here, the user presses the appearance event search button 3019. Then, the search screen 805 transitions to the state immediately after the appearance event search is executed, that is, the state of the terminal device 103 at the timing 705 in FIG. FIG. 8F shows an example of the search screen 806 in this state.

Here, reference numeral 8036 in FIG. 8F or the like denotes a person rectangle display, which is a person surrounded by a rectangle. Selection of the person rectangular display 8036 can be performed by a user operation such as clicking with a mouse. When the selected human face rectangle display 8036 is selected again, it can be brought into a non-selected state. The selected state and the non-selected state may be presented by changing the thickness of the rectangular frame, or may be presented by changing the color.
In the appearance event search, there may be a case where two or more persons to be selected appear in one image like a search result image 3081. For this reason, a selectable display such as a person rectangle display 8036 is provided instead of the keyword target check box 4034 in FIG. 8C.
The other items are the same as in FIG.

Here, it is assumed that the user wants to assign a keyword to the search result image 3031 which is one of the search result images, and the user or another user has previously performed an operation of giving the keyword. To do.
At this time, the user performs a selection operation by clicking the person rectangle display 8036 of the search result image 3031 with the mouse. As a result, the search screen transits to the person selected state, that is, the state of the terminal device 103 at the timing 706 in FIG.
At this time, inside the terminal device 103, the person image of Mr. A is used as a key image to make a similar person search request.

Here, in the embodiment of the previous operation, the search key image 3005 is not specified and the search result display area 4020 is not updated as compared to when the transition is made from FIG. 8A to FIG. 8B and FIG. 8B to FIG. Note that similar person search request processing is performed in the background.
In this embodiment, the similarity information for the attribute information and the search key image is stored in the similar person search result. That is, the keyword and similarity of an image similar to Mr. A can be acquired here.
Data 320 shown in FIG. 5C is data included in the search result of the similar image search, and is an essential part of the present invention in the return data of the search result.
In the data 302 included in the search result of the similar image search, here, data to which the keywords “Mr. A”, “Mr. B”, and “Mr. C” are assigned and blanks, that is, those to which no keyword is assigned are returned as search results. Represents that

In the state of the terminal device 103 at the timing 706 in FIG. 7, the search screen 807 in FIG. 8G is displayed. FIG. 8G shows an example of a search screen after a person selection operation by the user after appearance event search.
Here, the person rectangle display 8036 is a thick frame display indicating the selected state.
A keyword designation field 4022 is a keyword designation and keyword candidate selection field 8100.
In the keyword designation and keyword candidate selection field 8100, a keyword can be input, and can be selected from candidates displayed.

The terminal device 103 selects the same keyword having the highest similarity among the search result data, and displays it on the keyword candidate display field 8100.
The user sees this, confirms that “Mr. A” in the keyword candidate display field 8100 is the target “Mr. A”, and selects it by clicking with the mouse. Then, the state transitions to the state of the terminal device 103 at timing 707 in FIG. In the state of the terminal device 103 at the timing 707 in FIG. 7, the search image 808 in FIG. 8H is displayed.
Reference numeral 8037 in FIG. 8H denotes a keyword display field. In the keyword display field 8037, assigned keywords are displayed.

Thereafter, the user can select one or more of the search result images at the same time and perform the keyword assignment operation. When the user's keyword assignment operation in one appearance event search is completed, the user can change the conditions and perform the appearance event search again.
In this way, even if the user does not know the exact kanji or spelling of the name “Mr. A”, if the other user has already entered the correct kanji or spelling, it can be entered without error. Further, even if the user does not input the name “Mr. A” with the keyboard every time, the user can perform the task of assigning a keyword by simply clicking with the mouse.

  As described so far, the similar image search system shown in the present embodiment enables the user to assign keywords to the result of similar image search with a smaller number of selection of result images. In addition, when a user assigns a keyword in an appearance event search, it is possible to assign a keyword without having to input an accurate keyword with the keyboard every time.

  In the description so far, the imaging device, the recording device, and the terminal device are not limited to one each, and a plurality of imaging devices and terminal devices may be connected as shown in FIG. In FIG. 1, only one recording device is used, but a plurality of recording devices may be connected.

  In the present embodiment, as shown in FIG. 7, a similar person search request is made after selecting a search result after an appearance event search. However, processing equivalent to that requested for the similar person search in the recording apparatus 102 at the time of appearance event search request 711 is performed, and appearance event search results 712 including keyword assignment candidates based on similarities for the respective search result images are included. As another example, an embodiment may be adopted in which a similar person search request that is performed each time a search result is selected is omitted.

  Moreover, in this embodiment, as shown to FIG. 8G, although the keyword and the similarity were displayed on 8100 which is a keyword candidate display column, you may include the number of cases. In the data 302 included in the search result of the similar image search in FIG. 5C, “Mr. A” has three results, “Mr. B” and “Mr. C” each have one result, and this is displayed as a keyword candidate display field. It may be an embodiment in which the reference information of the user's judgment is increased by displaying it in 8100.

Further, in the present embodiment, as shown in FIG. 5C, the similarity is also transmitted with the keyword non-assigned search result, but the embodiment in which the keyword non-assigned is excluded from the communication may be used.
In the present embodiment, the person detection process related to person search, the person feature amount extraction process, and the like have been described as being performed on the recording apparatus. However, the apparatus is implemented on a separate apparatus from the networked recording apparatus. You may make it do.
In the present embodiment, the keyword is shown as a character string. However, the keyword may be executed with a specific number or symbol string.
Furthermore, in the present embodiment, the person search is shown, but the present invention can be applied not only to a person but also to a general image search.

  102: Recording device, 103: Terminal device, 110: Keyword recording unit, 111: Keyword search unit, 112: Keyword assignment request transmission unit, 200: Network, 201: Imaging device, 202: Recording device, 203: Terminal device, 210 : Image transmission / reception unit 211: Image recording unit 212: Playback control unit 213: Person area detection unit 214: Person feature amount extraction unit 215: Person feature amount recording unit 216: Attribute information recording unit 217: Request Receiving unit, 218: similar person searching unit, 219: appearance event searching unit, 220: search result transmitting unit, 221: search request transmitting unit, 222: search result receiving unit, 223: search result displaying unit, 224: reproduction image display Unit, 225: screen operation detection unit, 240: bus, 241: imaging unit, 242: main storage unit, 24 : Encoding unit, 244: Network I / F, 250: Bus, 251: CPU, 252: Main storage unit, 253: Auxiliary storage unit, 254: Network I / F, 260: Bus, 261: CPU, 262: Main Storage unit 263: Auxiliary storage unit 264: Display I / F, 265: Input / output I / F, 266: Network I / F, 270: Display device, 280: Keyboard, 282: Mouse, 300: Frame table, 301 : Image ID, 302: Frame data, 310: Attribute information data table, 311: Registration ID, 312: Image ID, 313: Feature quantity, 314: Camera ID, 315: Time information, 316: Keyword, 320: Similar image search 801, 802, 803, 804, 805, 8 6, 807, 808: search screen, 3001: playback image display area, 3002: moving image, 3003: image playback operation area, 3004: search key image designation area, 3005: search key image, 3006: video designation button, 3007: file Designation button, 3008: Search refinement parameter designation area, 3009-3012: Imaging device designation check box, 3013-3014: Time designation check box, 3015-3016: Time designation column, 3017: Search execution area, 3018: Similar person search Button, 3019: Appearance event search button, 3020: Search result display area, 3031, 3041, 3051, 3061, 3071, 3081, 3091, 3101, 3111, 3121, 3131, 3141: Search result image, 3032: Head Play button, 3033: Search key image designation button, 4017: Search execution area, 4020: Search result display area, 4021: Keyword designation check box, 4022: Keyword designation column, 4023: Keyword assignment button, 4034: Keyword assignment check box 8035: Similarity display field 8036: Person rectangle display 8037: Keyword display field 8100: Keyword specification and keyword candidate selection field

Claims (3)

A similar image search system for storing image data of a plurality of captured images, extracting a feature amount of the image, and searching for an image having a high similarity with a feature amount of a designated key image,
A first user interface displaying a list of search result images;
A second user interface that accepts an input for designating each of the search result images as either a selected state or a non-selected state;
A third user interface that accepts an input of an arbitrary keyword given to the image in the selected state,
The second user interface automatically selects at least one of the images that are similar to the feature amount of the key image among the search result images, and automatically selects the image.
The third user interface presents keywords assigned to an image similar to the image in the selected state in descending order of similarity as keyword candidates to be assigned to the image in the selected state. Similar image search system. An image similar to the image in the selected state is a similar image obtained in the background using the image in the selected state as a key image when an input for selecting any image is received by the second user interface. As a result of the search,
The similar image search system according to claim 1, wherein the keyword candidates are those having the highest similarity among keywords assigned to images similar to the image in the selected state . The similar image search system has an appearance event search function that is an image search method that does not depend on the similarity of images,
When the second user interface accepts an operation for selecting an image as a result of the appearance event search, a similar image search is performed using the selected image as a key image, and a keyword has been assigned to the result of the similar image search The similar image search system according to claim 1, wherein, if an image is included, the keyword is presented as an assignment candidate.

Images