JP2010160733A - Search result displaying method and monitoring system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for displaying search results with which correlation among each image is easily understood. <P>SOLUTION: The method for displaying search results is implemented in a monitoring system which includes an imaging device, a recording device, and a terminal device, and searches similar images. Firstly, three axes are provided to define the searched images. The three axes being used are, a similarity axis for designating similarity differences, a distance axis for designating distance differences among the searched images, and a time difference axis for designating time differences among the searched images. Then, the searched result images are displayed in a space configured by the three axes. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a search result display method and a monitoring system, and more particularly to a search result display method and a monitoring system in a search using an image feature amount.

Conventionally, video surveillance systems have been installed in facilities visited by an unspecified number of people such as hotels, buildings, convenience stores, financial institutions, dams, and roads for the purpose of crime prevention and accident prevention.
In such a video monitoring system, a person or the like to be monitored is photographed by an imaging device such as a camera, and the video is transmitted to a monitoring center such as a management office or a security room.
A monitor resident in the monitoring center monitors the video, and pays attention or records / stores the video according to the purpose and necessity.

As a recording medium for recording and storing video in this video surveillance system, a sequential access storage medium such as a video tape has been used in the past.
However, in recent years, an example in which a randomly accessible storage medium represented by a hard disk drive (HDD) is used in a video surveillance system is increasing.
Randomly accessible storage media are becoming popular because they are easy to organize videos and are resistant to repeated use.

Furthermore, the capacity of this randomly accessible recording medium is increasing year by year.
With this increase in capacity, the amount of video data that can be recorded has increased dramatically. This made it possible to record for a long time.
However, an increase in the burden on the observer when visually checking a large amount of recorded video is becoming a problem.

Some conventional monitoring systems have a text search function for searching text data of time and external sensor values embedded in image data.
Furthermore, in recent years, systems equipped with a search function that uses image information as search input information, that is, a so-called similar image search function, have become widespread. Among such image search functions, a monitoring system having a “person search function” has been frequently used.

The person search function is a function that, when a user designates a person shown in a certain image, searches for other videos showing the same person as that person in the recording apparatus and presents the list to the user.
In this case, a person's “face” image is often used as a clue for determining the same person.

Here, an example of the configuration of a conventional monitoring system Y having a person search function for searching for a person from a face image will be described with reference to FIG.
The monitoring system Y is configured by connecting an imaging device 201, a recording device 202, and a terminal device 203 via a network 200.

The network 200 is a dedicated line for interconnecting each device to perform data communication, or a communication line such as an intranet, the Internet, or a wireless LAN.
The imaging device 201 is a device such as a network camera or a surveillance camera that performs digital conversion processing on an image captured by a CCD or CMOS element, and outputs the converted image data to the recording device 202 via the network 200. The imaging device 201 includes a control unit, a storage unit that stores captured images, and a network connection unit.
The recording device 202 is a device such as a network digital recorder that records image data input from the imaging device 201 via the network 200 on a recording medium such as an HDD. The recording device 202 includes a control unit, a storage unit that stores image data captured by the imaging device 201, and a network connection unit that acquires the image data and transmits it to the terminal device 203. The recording device 202 is also equipped with a person search function, such as recording an image feature amount calculated using an image recognition technique on a recording medium, searching an image having a similar image feature amount from the recording medium, and the like. Also do.
The terminal device 203 is a device such as a PC that displays the image data acquired from the recording device via the network 200 on a monitor such as a liquid crystal display or a CRT. The terminal device 203 includes a user input unit such as a keyboard and a mouse, and provides a user interface for performing a reproduction operation, a moving image display, a person search execution operation, a result display, and the like recorded in the recording device 202.

The terminal device 203 will be described in more detail. The terminal device 203 includes a search request transmission unit 211, a search result reception unit 212, a search result display unit 213, and a video reproduction display unit 214.
The search request transmission unit 211 is a part that transmits a search request to the recording apparatus 202 via the network 200. The data transmitted as a search request includes an image showing a person to be searched. Hereinafter, this image is referred to as a search image.
The search result receiving unit 212 is a part that receives a search result from the recording device 202 via the network 200. The data received as a search result includes a set of images obtained by executing an image search in the recording device 202. Individual images constituting the set of images are generated by performing reduction processing or the like from the video recorded in the recording device 202. Hereinafter, the individual images are referred to as search result images.
The search result display unit 213 is a part that displays the search result received by the search result receiving unit 212 on a display unit such as a liquid crystal display. A screen example displayed by the search result display unit 213 will be described later.
The video playback display unit 214 is a processing unit that plays back video recorded in the recording device 202 and displays the screen. The video playback display unit 214 detects a playback operation by the user from an input unit such as a pointing device or a keyboard, converts the playback command into a playback command, transmits the playback command to the recording device 202 via the network 200, and the network. The screen display of the video data received from the recording apparatus 202 via 200 is performed.

FIG. 11 shows an example of a screen displayed on the terminal device 203 in a system having a conventional person search function.
The reproduced video display area 301 is an area for displaying a video recorded in the recording device 202 as a moving image.
The video playback operation area 302 is an area where a video recorded in the recording device 202 is played back. Each button constituting the video playback operation area 302 is assigned a unique playback type. In FIG. 11, for example, playback types of rewinding from the left, reverse playback, playback stop, forward playback, and fast forward are assigned. By appropriately pressing each button, the moving image display on the playback video display area 301 is switched to the playback type assigned to the button and displayed.
A search image display area 303 is an area for designating and displaying a search image. A search image designation button constituting the search image display area 303 designates an image displayed in the playback video display area 301 as a search image when pressed. The image designated as the search image is displayed on the screen in the search image display area 303.
The search image 311 is a search image designated in the conventional monitoring system Y.
The search execution area 304 is an area where search execution is performed. The search execution button constituting the search execution area 304 executes a search using the search image specified in the search image display area 303 when pressed. Although not shown in the conventional monitoring system Y, the search execution area 304 may be provided with means for limiting the search target, for example, a means for specifying a date / time range, a search target camera, and the like.
A search result display area 305 is an area for displaying a search result acquired from the recording apparatus 202. In the conventional monitoring system Y, the search result displayed in the search result display area 305 is a result obtained by executing a search using the search image 311. The display is performed by displaying a list of search result images.
Search result images 321 to 325 are each one of the search result images. Each displayed search result image is a button type, and when one of them is pressed, the display video in the playback video display area 301 is switched to the pressed image. This switching is called cueing.
The search result display order designation area 306 is an area for designating “display order”, which is the display order of the search result list display in the search result display area 305. In the conventional monitoring system Y, the display order can be specified from three types: similarity order, time order, and camera order.

FIG. 11 shows an example in which the search results displayed in the search result display area 305 are displayed based on the order of similarity.
In the case of the display in the order of similarity, the uppermost left of the search result display area 305 has the highest similarity, the similarity becomes lower as it goes to the right of the same level, and the lower the level, the more similar to the upper level. The degree is low. In the conventional monitoring system Y, the search result image 321 is the image with the highest similarity to the search image 311, followed by the search result images 322, 323, and 324, and the search result image 325 is among the search result images. Then, the image with the lowest similarity is obtained.

  Similarly, in the conventional monitoring system Y, the search result display order designation area 306 can be designated to be performed based on the time order which is the designation of the order by time. In this way, when the display order is specified in the time order, the uppermost left of the search result display area 305 is an image taken at the oldest time, and the image becomes a newer image as it goes to the right of the same stage. As the stage progresses to the lower part, an image having a newer time than the upper part is displayed.

  Similarly, in the conventional monitoring system Y, the search result display order designation area 306 can be designated to be performed based on the camera order, which is the order designation by the camera number. When the camera order is set in this way, the upper left of the search result display area 305 is an image taken with a camera having a smaller camera number, and the image is taken with a camera with a larger camera number toward the right in the same row. In addition, as the stage advances to the lower part, an image shot with a camera having a camera number higher than that of the upper part is displayed.

The person search function is very convenient because it provides easy cueing to a target person image from a large number of search target images recorded in a recording device.
When the same person can be found by the person search function in this way, it is important to tell the user “when and where” the same person who was found was shot.
Furthermore, it is desirable that the temporal relative relationship and the positional relative relationship between the shooting time and shooting location of the search image used for the search and that of the search result image obtained as the search result are expressed in an easy-to-understand manner for the user.

However, as shown in FIG. 10, in the current person search function, the search result is a method in which the display order is determined based on one scale.
In the above example, the search results are displayed based on the order specified by any one of the similarity order, the time order, and the camera order. In FIG. 11, since a wrap is provided for the convenience of screen display, the display is two-dimensional, but it can be said that the display is essentially a one-dimensional display based on each order.

Therefore, the relative relationship between the dimensions selected as the display order can be visually expressed. However, for other dimensions, there is a problem that it is difficult to obtain information between the search image and the search result image.
For this reason, for example, it was possible to display the shooting time next to the search result image displayed in the order of similarity, but in such an expression that only displays the character, the correlation with each dimension is shown. It was difficult to express.
Therefore, two-dimensional or more correlations such as similarity and time, similarity and camera, time and camera, similarity and time and camera cannot be visually expressed simultaneously.

Therefore, as a similar image search method for displaying the conventional similar image search three-dimensionally, for example, referring to Patent Document 1, an image with a high similarity is large according to the value of the similarity with the image desired by the user, A similar image search method for displaying an image with a low degree of similarity so as to be small is described (hereinafter referred to as prior art 1).
Further, in the image search method of the prior art 1, referring to paragraph [0016] and the like, the coordinate axis of the visualization space 210 is selected from the image feature amount, and the axis is changed in a certain time to perform “fluffing motion”. The search result according to the search condition can be shown to the user as a continuous change.
According to the similar image search method of the prior art 1, an effect that a large amount of search results can be presented to the user in an orderly form like a scatter diagram is obtained.

JP 2000-29885 A

However, in the prior art 1, only the similarity and the image feature amount are used for displaying the result of the search image, and the search result display for displaying based on the axis of the series necessary for the monitoring system such as the camera number order. The method was not disclosed.
For this reason, when the display method disclosed in the prior art 1 is used for the search result display method of the monitoring system, there is a problem that it is difficult to grasp the correlation between the images and the usability is poor.

  This invention is made | formed in view of such a condition, and makes it a subject to eliminate the above-mentioned subject.

The search result display method of the present invention includes an imaging device, a recording device, and a terminal device, and a search result display method in a monitoring system that searches for similar images. The search result image is displayed in a space constituted by a distance axis having a series of distance differences with respect to the search image and a time difference axis having a series of time differences with respect to the search image.
The monitoring system of the present invention is a similar image search system including an imaging device, a recording device, and a terminal device, wherein the recording device or the terminal device has a similarity axis based on a similarity difference with respect to a search image. And a display means for displaying the search result image in a space constituted by a distance axis having a series of distance differences with respect to the search image and a time difference axis having a series of time differences with respect to the search image.

  According to the present invention, in the person search function of the video surveillance system, each sequence of similarity, distance, and time between the search image and the search result image is expressed simultaneously on a three-dimensional axis so that two or more dimensions can be obtained. It is possible to provide an easy-to-use result display method capable of visually grasping the correlation at the same time.

It is a system configuration figure of surveillance system X provided with a person search function including a result display system concerning an embodiment of the invention. It is a figure which shows the example of a screen displayed on the terminal device 103 in the system provided with the person search function including the result display system which concerns on embodiment of this invention. It is the figure which expanded and displayed the search result display area 405 of the screen displayed on the terminal device 103 in the system provided with the person search function including the result display system which concerns on embodiment of this invention. It is a conceptual diagram explaining the relationship between the three-dimensional space comprised by 3 axes | shafts of the time difference axis | shaft 421, the distance axis | shaft 422, and the similarity degree axis | shaft 423 which concerns on embodiment of this invention, and a screen display. It is a figure which shows the example of a screen displayed on the terminal device 103 which concerns on embodiment of this invention. It is a figure which shows the example of a screen displayed on the terminal device 103 which concerns on embodiment of this invention. It is a flowchart which shows the flow of the video recording process of the system provided with the person search function including the search result display method which concerns on embodiment of this invention. It is a flowchart which shows the flow of the video reproduction process of the system provided with the person search function including the search result display method which concerns on embodiment of this invention. It is a flowchart which shows the flow of a person search process of the system provided with the person search function including the result display system which concerns on embodiment of this invention. It is a system block diagram of the monitoring system Y provided with the conventional person search function. It is a figure which shows the example of a screen displayed on the terminal device in the monitoring system Y provided with the conventional person search function.

<Embodiment>
(Control configuration)
With reference to FIG. 1, the control configuration of the monitoring system X according to the embodiment of the present invention will be described. In the monitoring system X, an imaging device 201, a recording device 202, and a terminal device 103 are connected via a network 200. In FIG. 1, the same reference numerals as those in FIG. 10 indicate parts that provide the same functions as those in the prior art. That is, the imaging device 201 and the recording device 202 show the same components as the conventional monitoring system Y in FIG.
The network 200 includes a LAN, an optical fiber, c. A line capable of data communication such as a link, a wireless LAN, and a mesh network. The network 200 may be an IP network such as a dedicated line, an intranet, or the Internet.
The recording device 202 is a recording device that records image data stored in an HDD or the like, and is a recording device similar to the recording device 202 of FIG.
The terminal device 103 is a PC / AT compatible machine that displays the image data acquired from the recording device 202 via the network 200 on a display unit such as a liquid crystal display or a CRT, or a PC (personal computer) such as a MAC. It is a dedicated monitoring terminal device. The terminal device 103 includes a control unit such as a CPU that controls processing described below, a program that describes processing executed by the control unit, image data for displaying results, attributes of image data, and temporary data. And a storage unit such as a RAM, a ROM, an HDD, and a flash memory. The terminal device 103 includes a user input unit such as a keyboard and a mouse, and provides a user interface for performing a reproduction operation, a moving image display, a person search execution operation, a result display, and the like recorded in the recording device 202.
The terminal device 103 may be realized by a program stored in a storage unit of a PC in which a normal OS (operating system) is installed.
Furthermore, the program executed by the control unit may be used to implement each unit using hardware resources, and may be stored in a storage unit or a ROM or flash memory in the control unit. It may be.

More specifically, the terminal device 103 includes a search result 3D display unit 113 (display unit) in addition to the search request transmission unit 211, the search result reception unit 212, and the video reproduction display unit 214 which are the same as those in the conventional art. .
The search result 3D display unit 113 is a processing unit that displays the search result on the screen. In the search result display processing, the search result display method according to the embodiment of the present invention is used. The search result 3D display unit 113 performs various calculation (calculation) processes that require high-speed responsiveness such as feature quantity extraction and a large calculation performance in the SIMD (single instruction multiple data) format. Processing can be performed with a chip or board equipped with a processing unit. The search result 3D display unit 113 may be realized as a program executed by the control unit of the terminal device 103.

[Search result display screen example]
Next, an example of a screen displayed on the terminal device 103 by the search result display method according to the embodiment of the present invention in the monitoring system X having the person search function will be described with reference to FIG. Also in FIG. 2, the same reference numerals as those in FIG. 11 indicate regions and buttons that provide similar functions.
In the screen example of FIG. 2, a search result display area 405 is provided in addition to the conventional playback video display area 301, video playback operation area 302, search image display area 303, and search execution area 304.

The search result display area 405 will be described in detail. The search result display area 405 is an area to be displayed while associating the search result acquired from the recording device 202 with the search result searched in the past.
Here, the search result displayed in the search result display area 405 shows a display example of the result obtained by executing the search with the search image 311. This display example is obtained by displaying a list of search result images by the search result display method according to the embodiment of the present invention.

Search result images 411 to 415 are respective search result images.
As with the conventional monitoring system Y, each displayed search result image is a button. When the search result 3D display unit 113 detects that the user has pressed any of the search result images, the search result 3D display unit 113 switches the display video in the playback video display area 301 to the pressed image and performs cueing.

Here, a screen example in which the display of the search result display area 405 is enlarged will be described with reference to FIG.
Search result images 411 to 415 are search result images similar to those in FIG.
The search result 3D display unit 113 displays the search result images 411 to 415 in a three-dimensional space configured by three axes of a time difference axis 421, a distance axis 422, and a similarity axis 423, which will be described later. As shown in FIGS. 2 and 3, it is preferable that the three-dimensional space is represented by simple oblique projection transformation without actually performing affine transformation or the like as in the prior art 1. As a result, the sense of distance with respect to each series (scale) of each axis becomes easy to understand, and the effect that the relationship between each result image becomes easy to understand in the monitoring system is obtained.

  The time difference axis 421 is an axis indicated by a time difference series of the photographing time between the search image and the search result image. In the monitoring system X according to the embodiment of the present invention, the time difference increases from left to right in the figure.

The distance axis 422 is an axis indicating a series of distances between the shooting positions of the search image and the search result image. In the monitoring system X according to the embodiment of the present invention, the distance increases from the bottom to the top of the figure.
Here, the distance between the shooting positions is synonymous with the distance between the cameras. The distance between the cameras is obtained in advance by a linear distance that can be calculated based on latitude and longitude obtained from GPS, a map, or the like, or by surveying. Use the distance. The physical value based on the metric system is generally used as the distance value, but an appropriate reference value, for example, a logical value with the minimum distance between all cameras as the reference value 1 is used. May be. In addition, considering the top and bottom of the terrain, a distance such as a distance that can be moved on foot may be specified.

  The similarity axis 423 is an axis indicating the similarity between the search image and the search result image. In the monitoring system X according to the embodiment of the present invention, the degree of similarity decreases from the lower left to the upper right of the figure, that is, the search image and the search result image are not similar.

The coordinate points 431 to 435 are coordinate points plotted on a coordinate plane constituted by the time difference axis 421 and the similarity axis 423.
The coordinate points 431 to 435 are in correspondence with the search result images 411 to 415, respectively.
That is, in the monitoring system X according to the embodiment of the present invention, the search result image 411 corresponds to the coordinate point 431, and represents that the time difference with respect to the search image is ΔT3 and the similarity is S1.
Similarly, the search result image 412 corresponds to the coordinate point 432, and represents that the time difference with respect to the search image has ΔT5 and the similarity has a value of S2.
Further, the search result image 413 corresponds to the coordinate point 433, and represents that the time difference with respect to the search image has ΔT1 and the similarity has a value of S3.
The search result image 414 corresponds to the coordinate point 434, and represents that the time difference with respect to the search image is ΔT2 and the similarity is S4.
Further, the search result image 415 corresponds to the coordinate point 435, and represents that the time difference with respect to the search image has a value of ΔT4 and the similarity has a value of S5.

The coordinate points 441 to 445 are coordinate points plotted on a coordinate plane constituted by the distance axis 422 and the similarity axis 423.
The coordinate points 441 to 445 are in correspondence with the search result images 411 to 415, respectively. That is, in the monitoring system X according to the embodiment of the present invention, the search result image 411 corresponds to the coordinate point 441, and has a value of ΔD1 as the distance to the search image and a value of S1 as the similarity.
Similarly, the search result image 412 corresponds to the coordinate point 442, and represents that the distance from the search image is ΔD5 and the similarity is S2.
The search result image 413 corresponds to the coordinate point 443, and represents that the distance from the search image is ΔD2 and the similarity is S3.
The search result image 414 corresponds to the coordinate point 444, and represents that the distance to the search image is ΔD4 and the similarity is S4.
Further, the search result image 415 corresponds to the coordinate point 445, and represents that the distance from the search image is ΔD3 and the similarity is S5.

In the monitoring system X according to the embodiment of the present invention, the coordinate points plotted on the coordinate plane composed of the time difference axis 421 and the distance axis 422 are not shown for the sake of simplification, but the same applies. Needless to say.
In addition, although these coordinate points are provided for ease of explanation in the monitoring system X according to the embodiment of the present invention, they may not actually be present.

In addition, since each image disappears when it is hidden behind, the scaling of the image displayed in the three-dimensional space can be freely changed. Furthermore, instead of simple oblique projection (parallel projection), it is possible to perform display with “rotating oblique projection” in which the relationship between the distances can be easily grasped even if rotated because the axis hidden behind is seen. . Details of the display in the rotating oblique projection will be described later.
The time difference axis rotation button 451 is an operation button for rotating the time difference axis 421. When this button is operated, the three-dimensional space is rotated about the time difference axis as a central axis, and the display can be changed. Although not shown here, the amount of rotation can also be indicated near the button by character display, graphic expression, or the like.
The distance axis rotation button 452 is an operation button for rotating the distance axis 422. When this button is operated, the three-dimensional space rotates around the distance axis as a central axis, and the display changes. Although not shown in the monitoring system X according to the embodiment of the present invention, the rotation amount may be indicated by character display near the button, graphic expression, or the like.
The similarity axis rotation button 453 is an operation button for rotating the similarity axis 423. When this button is operated, the three-dimensional space is rotated about the similarity axis as a central axis, and the display changes. Although not shown in the monitoring system X according to the embodiment of the present invention, the rotation amount may be indicated by character display near the button, graphic expression, or the like.
Changes in display when these axis rotation buttons are operated will be described later. Instead of rotating oblique projection, conversion may be performed so that three dimensions can be displayed stereoscopically.

The display range designation buttons 461 and 462 are operation buttons for giving a maximum value and a minimum value of a range to be displayed as a search result. As the shape of the button, a button with a graphical expression centered on the axis can be used. FIG. 3 shows an example of expressing as a circle with the axis as the center point.
In the monitoring system X according to the embodiment of the present invention, the display range designation button 461 is displayed at coordinates having a value larger than the similarity value S1, and the display range designation button 462 is displayed at a location having a value smaller than the similarity value S5. .
Therefore, for example, when the display range designation button 462 is moved between the similarity values S4 and S3 by a drag operation, the search result images to be displayed among the search result images 411 to 415 are the search result images 411 to 413. Become.

The display range designation button can be displayed with the button shape changed depending on the installation position.
For example, the search result 3D display unit 113 obtains the radius of the display circle based on the distance from the origin of the installation position, and can be displayed, for example, as a circle with a large radius because it is less visible as it moves away from the origin.
For simplification of explanation, the display range designation button is shown only for the similarity axis 423, but it can also be displayed on the time difference axis and the distance axis. The change in search result image display due to the operation method and operation in that case is the same as that of the display range designation buttons 461 and 462 described above.

By such a search result display method, it is possible to visually grasp the relationship between the similarity, the time difference, and the distance between the search image and the search result image at the same time.
For example, when focusing on the search result image 411, it is possible to grasp at a glance that the similarity to the search image 311 is extremely high, the time difference is large, and the distance is very small, and the visibility of the monitoring image is greatly increased. Can be enhanced.

(Screen display with rotating oblique projection)
Next, regarding the relationship between the three-dimensional space composed of the three axes of the time difference axis 421, the distance axis 422, and the similarity axis 423, and the screen display in oblique projection that enables rotation, FIG. This will be described with reference to (c).

First, referring to FIG. 4A, the display surface 601 schematically represents the screen of the terminal device.
The three-dimensional space 602 is a space composed of a time difference axis 421, a distance axis 422, and a similarity axis 423. In the monitoring system X according to the embodiment of the present invention, all the axial lengths can be shown as cubes. .
The search results displayed in the screen search result display area 405 shown in FIGS. 2 and 3 are obtained by projecting the three-dimensional space 602 on the display surface 601 and displaying it on the screen.

Therefore, for example, referring to FIG. 4B, an axis rotation button of each axis is shown for a case where the display surface 601 and the surface 603 constituted by the time difference axis 421 and the distance axis 422 are operated in parallel. Yes.
With reference to FIG. 5, the example of a screen displayed on the terminal device 103 by this operation is demonstrated. FIG. 5 is a schematic diagram, and some of the component parts and reference signs shown in FIGS. 2 and 3 are omitted.
When the screen display shown in FIG. 5 is displayed, the relationship between the search result images with respect to the time difference and the distance can be visually grasped. In this state, the display range designation buttons 461 and 462 for the similarity are displayed as concentric circles. In this case, when the display range is designated, it is executed by dragging on the circle line in the circle diameter direction and changing the radius.

Similarly, for example, referring to FIG. 4C, a case where the axis rotation button of each axis is operated so that the surface 604 constituted by the time difference axis 421 and the similarity axis 423 and the display surface 601 are parallel to each other. Show.
An example of a screen displayed on the terminal device 103 will be described with reference to FIG. FIG. 6 is a schematic diagram, and a part of component display and symbol display shown in FIGS. 2 and 3 are omitted.
When the screen display shown in FIG. 6 is displayed, the relationship between the time difference and the search result image regarding the similarity can be visually grasped. In this state, the display range designation button for the distance is displayed as a concentric circle. In this case, when the display range is designated, it is executed by dragging on the circle line in the circle diameter direction and changing the radius.

  As shown in FIGS. 5 and 6, the search result image is preferably displayed so as to be always parallel to the display surface 601 with the image center point as the origin. By this “rotating oblique projection”, it is possible to grasp the relationship between the distances of the respective axes while maintaining a sense of distance.

[Processing of Search Result Display Method According to Embodiment of Present Invention]
Next, with reference to FIGS. 7 to 9, the flow of processing when the search result display method according to the embodiment of the present invention is executed in the monitoring system X having the person search function will be described.
The processing of the search result display method mainly includes three processes: (1) recording processing, (2) video reproduction processing, and (3) person search processing.

(Recording process)
First, the recording process (1) will be described in detail with reference to FIG.
The recording process is executed by a process including communication between the imaging device 201 and the recording device 202, and the image data captured by the imaging device 201 is recorded in the recording device 202.

Here, first, the flow of processing on the recording apparatus 202 side will be described.
In step S1000, the control unit of the recording device 202 performs an image reception standby process.
When the control unit of the recording device 202 detects the arrival of image data, the control unit proceeds to step S1001.

  In step S <b> 1001, the control unit of the recording device 202 performs reception processing for receiving data including image data from the imaging device 201. The received data includes image attribute data such as shooting time information and shooting position information in addition to the image data.

In step S1002, the control unit of the recording apparatus 202 performs processing for recording the received data and the image ID on a recording medium such as an HDD.
The image ID (identification) is information serving as an index for later retrieving image data.
As the image ID, for example, a frame number assigned in order in units of frames from the start of recording so as to be unique within the recording apparatus 202 can be used.

  In step S1003, the control unit of the recording device 202 performs face detection processing on the received image. The face detection process is performed by a known image recognition technique that uses facial features such as the arrangement of the main components of the face such as the eyes, nose, and mouth, and the difference between the forehead and the eye.

In step S1004, the control unit of the recording apparatus 202 performs a face detection result determination process in step S1003.
If Yes, that is, if a face is detected, the control unit of the recording apparatus 202 advances the process to step S1005.
No, that is, if not detected, the control unit of the recording apparatus 202 returns the process to step S1000.

In step S1005, the control unit of the recording apparatus 202 calculates a face image area based on the face detection result in step S1003.
This face image area data is hereinafter referred to as face image data.

In step S1006, the control unit of the recording device 202 calculates the image feature amount of the calculated face image data.
The image feature amount is a value indicating the tendency of an image calculated using an image recognition technique. As the image feature amount, for example, an image color distribution, an edge pattern composition distribution, a combination thereof, or the like can be calculated and used.

In step S1007, the control unit of the recording apparatus 202 records the calculated image feature amount on a recording medium such as an HDD in accordance with the corresponding image ID. Using this image feature amount, similarity information described later is calculated.
After completing the recording of the image feature amount, the control unit of the recording device 202 returns the process to step S1000.

Next, a processing flow of the imaging apparatus 201 will be described.
In step S <b> 1010, the control unit of the imaging apparatus 201 waits for a captured image output from an imaging element such as a CCD or CMOS provided in the imaging apparatus 201. This captured image output can be performed every predetermined time using an interval timer or the like, or the captured image output can always be performed at 1/60 frames, for example.
When the control unit of the imaging apparatus 201 detects the output of the captured image, the process proceeds to step S1011.

  In step S <b> 1011, the control unit of the image capturing apparatus 201 digitally converts the captured image output from the image sensor.

In step S1012, the control unit of the imaging apparatus 201 transmits the image data of the digitally converted image to the recording apparatus 202 via the network 200.
At this time, as described above, data of various attributes such as image capturing time information and image capturing position information are also transmitted as data including image data.

  A timing T1020 represents communication between the imaging device 201 and the recording device 202, and here, data including image data is transmitted.

(Video playback processing)
Next, the flow of video playback processing will be described with reference to FIG.
The video reproduction process is executed by a process including communication between the recording device 202 and the terminal device 103. Then, the video recorded in the recording device 202 is played back by the terminal device 103.

First, the flow of processing on the terminal device 103 side will be described.
In step S1100, the video reproduction display unit 214 of the terminal device 103 performs a user reproduction operation standby process. When the user's playback operation is detected, the video playback / display unit 214 advances the process to step S1101.

In step S1101, the video playback display unit 214 of the terminal device 103 determines a “video playback request” command according to the playback operation content of the user.
The video playback request includes, for example, a playback type such as forward playback and fast forward, a playback time direction such as forward playback and reverse playback, playback speed, and the like.

  In step S1102, the video playback display unit 214 of the terminal device 103 performs processing for transmitting the determined video playback request command.

  At timing T <b> 1120, the video playback display unit 214 communicates a specific video playback request command to the recording device 202 via the network 200.

  In step S1103, the video reproduction display unit 214 of the terminal device 103 performs a data reception standby process. When the incoming data is detected, the video reproduction display unit 214 advances the process to step S1104.

In step S <b> 1104, the video reproduction display unit 214 of the terminal device 103 receives data from the recording device 202.
As will be described later, this data includes distribution image data of an image that is actually played back as a moving image and the like, and a video playback completion notification indicating that video playback has been completed.

In step S1105, the video reproduction display unit 214 of the terminal device 103 determines the content of the received data. Specifically, the received data. It is determined whether the image data is distribution image data.
If Yes, that is, if the received data is distribution image data, the video reproduction display unit 214 proceeds with the process to step S1106.
If No, that is, if the received data is a video reproduction completion notification, the process returns to step 1100.

In step S1106, the video reproduction display unit 214 of the terminal device 103 images (decodes) the received distribution image data and displays it on the screen.
After this display is completed, the control unit of the video reproduction display unit 214 returns the process to step 703.

Next, the flow of processing on the recording device 202 side will be described.
In step S1110, the control unit of the recording apparatus 202 performs a video reproduction request reception standby process.
When the incoming of a video playback request command from the network 200 is detected, the control unit of the recording apparatus 202 advances the process to step S1111.

  In step S <b> 1111, the control unit of the recording device 202 receives a video playback request command from the terminal device 103.

In step S1112, the control unit of the recording apparatus 202 determines the content of video playback based on the video playback request.
The content of the video reproduction includes, for example, the image ID of the image to be distributed first, the number of images to be distributed, the image ID of the image to be distributed next, the distribution timing, and the like.

  In step S1113, the control unit of the recording device 202 extracts image data from the recording medium. For extraction, the image ID of the image to be distributed next in the video reproduction content is used.

In step S1114, the control unit of the recording device 202 waits for transmission time. The transmission time uses a video re-request command and image data distribution timing.
When the transmission time is reached, the control unit of the recording device 202 advances the process to step S1115.

  In step S1115, the control unit of the recording device 202 transmits the extracted image data as distribution image data to the terminal device 103 via the network 200.

  At timing T <b> 1121, the control unit of the recording device 202 communicates distribution image data with the terminal device 203.

In step S1116, the control unit of the recording device 202 determines completion of video reproduction.
In the completion determination, it is determined whether or not the image distribution satisfying the previously determined video reproduction content has been completed.
If it is Yes, that is, if it is determined that the image distribution is completed, the control unit of the recording device 202 advances the process to step S1117.
No, that is, if it is determined that the image distribution has not been completed, the control unit of the recording device 202 advances the process to step S1118.

In step S <b> 1117, the control unit of the recording device 202 transmits a video reproduction completion notification to the terminal device 103 via the network 200.
After the transmission is completed, the control unit of the recording device 202 returns the process to step S1110.

  At timing T <b> 1122, the control unit of the recording device 202 performs communication of a video reproduction completion notification.

In step S1118, the control unit of the recording apparatus 202 updates the video playback content.
At this time, the image ID and distribution timing of the image to be distributed next are updated.
After completion of the update, the control unit of the recording device 202 returns the process to step S1113.

(Person search processing)
Next, the flow of person search processing will be described in detail with reference to FIG.
The person search process is executed in a process including communication between the recording device 202 and the terminal device 103, and the terminal device 103 searches the set of search result images found from the video recorded in the recording device 202. Display as a result.
The search result display method according to the embodiment of the present invention is also included in the processing flow described in FIG.

First, the flow of processing on the terminal device 103 side will be described.
In step S900, the search result 3D display unit 113 of the terminal device 103 waits for a user operation. When it is detected that the user has operated the mouse or keyboard of the input unit, the search result 3D display unit 113 advances the process to step S901.

In step S901, the search result 3D display unit 113 of the terminal device 103 determines the content of the detected user operation.
Here, it is determined whether the operation content is a search execution operation or not. The user can perform operations such as an axis rotation operation and a display range designation operation as operations other than the search execution operation.
If it is determined Yes, that is, if the search execution operation is determined, the search result 3D display unit 113 advances the process to step S902.
No, that is, if it is determined that the operation is a shaft rotation operation or a display range designation operation, the search result 3D display unit 113 advances the process to step S908.

In step S <b> 902, the search request transmission unit 211 of the terminal device 103 transmits a search request command to the recording device 202 via the network 200.
The search request command includes the search image 311 displayed in the search image display area 303.
For the sake of simplification of explanation, the monitoring system X according to the embodiment of the present invention will not be described in detail, but the search request includes a date and time range, a search condition such as a search target camera, and the maximum display number of search result images. It is also possible to provide information such as display conditions including the above.

  At timing T940, the search request transmission unit 211 communicates a search request command.

In step S903, the search result receiving unit 212 of the terminal device 103 waits for reception of search result data.
When detecting the arrival of the search result data, the search result receiving unit 212 advances the process to step S904.

In step S <b> 904, the search result receiving unit 212 of the terminal device 103 receives the search result data transmitted from the recording device 202.
The search result data includes one or more search result images, shooting time information of the images, shooting position information, and similarity information described later.

In step S905, the search result receiving unit 212 of the terminal device 103 determines the content of the received data. Specifically, it is determined whether the received data is a search result or a search impossible notification.
If Yes, that is, if the content of the received data is determined as the search result, the search result receiving unit 212 advances the process to step S906.
If it is determined as No, that is, a search impossible notification, the search result receiving unit 212 advances the process to step S913.

  In step S906, the search result 3D display unit 113 of the terminal device 103 calculates a time difference for all the search result images included in the received search result from the shooting time information of the search image 311 and the shooting time information of the search image 311 and stores them. Are stored in a temporary storage medium such as a variable on a RAM, HDD, or flash memory, a temp file, or the like. The time difference can be calculated by simple subtraction between times, the absolute value of the subtraction, or the difference in the number of frames of the image ID.

In step S907, the search result 3D display unit 113 of the terminal device 103 calculates the distance for all the search result images included in the received search result from the shooting position information and the shooting position information of the search image 311, and temporarily Store in a storage medium.
For example, when the shooting position information is the longitude and latitude provided by the GPS, two plane rectangular coordinates are obtained from the latitude and longitude by a general survey calculation, and the distance between the two points is obtained. It is possible. For example, when given as the coordinates of a plane map, the distance between two points can be obtained from the coordinates of the two points.

In step S908, the search result 3D display unit 113 of the terminal device 103 calculates the display position in the three-axis search result display area 305 and the display range obtained from these, and displays them on the screen.
The display position is calculated based on the rotation amount stored in the storage unit for the time difference axis rotation button 451, the distance axis rotation button 452, and the similarity axis rotation button 453. As described above, the amount of rotation can be changed by the user using the user interface.

In step S909, the search result 3D display unit 113 of the terminal device 103 determines whether or not the processing after step S910 has been completed for all the search result images received in step S904.
If it is determined Yes, that is, it is determined that the search is completed, the search result 3D display unit 113 returns the process to step S900 and waits for an operation.
If it is determined No, that is, it is determined that the search is not completed, the search result 3D display unit 113 proceeds with the process to step S910.

In step S910, the search result 3D display unit 113 of the terminal device 103 performs display determination of the search result image.
In the display determination, the time difference, the distance, and the similarity information are each compared with the specified value given by the display range specifying button, and it is determined whether or not the display range is displayed in the search result display area 305. To do. As described above, the time difference is calculated in step S906 and stored in the temporary storage medium. The distance is calculated in step S907 and stored in the temporary storage medium. The similarity information is created by the recording apparatus 202 in step S925 described later, received in step S904, and stored in the temporary storage medium.
Yes, that is, for example, when the time difference, the distance, and the similarity of the search result image of the search image 311 of interest are all within the display range, it is determined to be displayed, and the search result 3D display unit 113 performs the process in step S911. Proceed to
In No, ie, other than that, it determines with non-display, and the search result 3D display part 113 returns a process to step S909.

In step S911, the search result 3D display unit 113 of the terminal device 103 calculates the display position of the search result image in the search result display area 305.
The display position is based on the time difference calculated in step S906 and stored in the temporary storage medium, the distance calculated in step S907 and stored in the temporary storage medium, and the similarity information included in the search result received in step S904. calculate.

In step S912, the search result 3D display unit 113 of the terminal device 103 displays the search result image on the screen with the display position obtained in step S911 as the center coordinates.
After the display is completed, the search result image of interest is advanced by one, and then the search result 3D display unit 113 returns the process to step S909.

In step S913, the search result 3D display unit 113 of the terminal device 103 displays a screen indicating that the search is not possible.
After the display is completed, the search result 3D display unit 113 returns the process to step S900.

Next, the flow of processing on the recording device 202 side will be described.
In step S920, the control unit of the recording device 202 waits for reception of a search request. When the incoming search request is detected, the control unit of the recording apparatus 202 advances the process to step S921.

  In step S <b> 921, the control unit of the recording device 202 receives the search request transmitted from the terminal device 103.

  In step S922, the control unit of the recording device 202 performs face detection processing on the search image included in the received search request.

In step S923, the control unit of the recording apparatus 202 determines the face detection result in step S922. Face detection is performed by a known technique.
If Yes, that is, if a face is detected, the control unit of the recording apparatus 202 advances the process to step S924.
If No, that is, no face is detected, the control unit of the recording apparatus 202 advances the process to step S928.

  In step S924, the control unit of the recording apparatus 202 calculates a face image area from the face detection result obtained in step S922, and obtains face image data.

  In step S925, the control unit of the recording apparatus 202 calculates the image feature amount of the obtained face image data. The type and calculation method of the image feature amount to be calculated are the same as the conventional one.

In step S926, the control unit of the recording device 202 performs a search based on the image feature amount of the search image obtained in step S925.
At the time of this search, the control unit of the recording device 202 calculates a search result including image capturing time information, image capturing position information, similarity information, and the like in addition to image data.
For calculating the similarity information, for example, the image feature amount of the search image described above is compared with the image feature amount of each image recorded in the recording device 202 by using a predetermined weight. The image feature amount information between them is calculated as a vector amount or a scalar amount.
And the control part of the video recording apparatus 202 produces | generates the collection of an image provided with the similarity more than a predetermined value as a search result.
The set of images included in the search result created by the control unit of the recording device 202 uses an image (thumbnail) obtained by reducing each image recorded in the recording device 202.

  In step S <b> 927, the control unit of the recording device 202 transmits the search result to the terminal device 103 via the network 200.

At timing T941, the control unit of the recording apparatus 202 communicates search results.
After the transmission is completed, the control unit of the recording device 202 returns the process to step S920.

In step S <b> 928, the control unit of the recording device 202 transmits a search impossible notification to the terminal device 103 via the network 200.
This non-searchable notification indicates that an image having a similarity greater than or equal to a predetermined value could not be found in the searched image.

At timing T <b> 941, the control unit of the recording device 202 performs communication of a search impossible notification.
After the transmission is completed, the control unit of the recording device 202 returns the process to step S920.

With the configuration described above, the following effects can be obtained.
First, the image search method of the prior art 1 can display the search result by performing “fluffy” movement in the visualization space in the form of a scatter diagram according to the similarity of the images.
However, in the prior art 1, only the similarity and the image feature amount are used for displaying the result of the search image, and the series necessary for the monitoring system, such as a camera number order series, an image time series, and a similarity series. The search result display method for displaying based on the axis of was not shown.
For this reason, when the display method disclosed in the related art 1 is used for the search result display method of the monitoring system, there is a problem that it is difficult to grasp the correlation between the images and the usability is poor.

On the other hand, in the search result display method in the monitoring system X according to the embodiment of the present invention, the similarity axis based on the similarity difference for the search image and the distance axis based on the distance difference for the search image And a search result image can be displayed in a space composed of a time difference axis having a time difference with respect to the search image as a series.
Thereby, the correlation of the image of a search result can be easily grasped from a two-dimensional or higher viewpoint.

In addition, the search result display method in the monitoring system X according to the embodiment of the present invention displays an oblique projection (parallel projection) in which the sense of distance can be easily grasped even if the axis is rotated. Furthermore, each axis can be rotated, and even if it is rotated, it is possible to display with “rotating oblique projection” that maintains the distance ratio.
Thereby, even if the correlation of each image performs a three-dimensional display, meaning between each image can be performed easily. Therefore, for example, it is possible to easily grasp how much time has passed a person from this place to the place.

Furthermore, in the search result display method in the monitoring system X according to the embodiment of the present invention, it is possible to set a range of where to display an image on each axis while rotating each axis.
This makes it possible to easily narrow down the images of the target person in the vicinity of each series of certain images, and easily find the images necessary for searching from a large number of images.

  For simplification of explanation, FIG. 1 shows the imaging apparatus 201, the recording apparatus 202, and the terminal apparatus 103 as one unit each, but a plurality of these units can be connected to the network 200. .

  In the monitoring system X according to the embodiment of the present invention, the face detection process related to person search, the image feature amount calculation process, and the like are shown to be executed on the recording device 202. It may be executed by a device separate from the recording device 202.

  Further, in the monitoring system X according to the embodiment of the present invention, the time difference calculation and the distance calculation are shown as processing elements in steps S906 and 907 in the terminal device 103, but it is also possible to use the processing elements on the recording device 202 side. is there.

  Further, in the monitoring system X according to the embodiment of the present invention, the recording device 202 has shown the flow of processing for transmitting the search result to the terminal device 103 after the search result is completely generated. However, the search result is partially Each time it is generated, it may be a flow of processing for sequentially transmitting to the terminal device 103 side.

  In the monitoring system X according to the embodiment of the present invention, the person search is shown. However, it goes without saying that the present invention can be applied not only to a person but also to a general image search.

  It should be noted that the configuration and operation of the above-described embodiment are examples, and it is needless to say that the configuration and operation can be appropriately changed and executed without departing from the gist of the present invention.

103 terminal device 113 search result 3D display unit 200 network 201 imaging device 202 recording device 203 terminal device 211 search request transmission unit 212 search result reception unit 213 search result display unit 214 video playback display unit 301 playback video display region 302 video playback operation region 303 Search image display area 304 Search execution area 305 Search result display area 306 Search result display order designation area 311 Search image 321 to 325 Search result image 405 Search result display area 411 to 415 Search result image 421 Time difference axis 422 Distance axis 423 Similarity Axis 431-435 Coordinate point 441-445 Coordinate point 451 Time difference axis rotation button 452 Distance axis rotation button 453 Similarity axis rotation button 461-462 Display range designation button 601 Display surface 602 3D space 603-604 Surface X, Y Monitoring system

Claims (2)

In a search result display method in a monitoring system that includes an imaging device, a recording device, and a terminal device to search for similar images,
A similarity axis based on the similarity difference for the search image,
A distance axis having a series of distance differences with respect to the search image;
A search result display method, comprising: displaying a search result image in a space composed of a time difference axis having a time difference with respect to the search image as a series. In a similar image search system including an imaging device, a recording device, and a terminal device,
The recording device or the terminal device is
A similarity axis based on the similarity difference for the search image,
A distance axis having a series of distance differences with respect to the search image;
A monitoring system comprising: display means for displaying a search result image in a space constituted by a time difference axis having a time difference as a series with respect to the search image.

Images