JP2009301501A - Search result display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve a problem wherein a level relation and a sequence relation are not got visually at the same time, as to binary information of a time and a similarity or the like, in a conventional similar image search system. <P>SOLUTION: This search result display method for an image search system has an imaging device, an image recorder for recording an image picked up by the imaging device, and a monitoring terminal, searches a similar image similar to a specific image 303, from the image recorded in the image recorder, and displays the searched image on the monitoring terminal. This search result display method sorts the searched result images, based on a first reference 510, and image-deforms the searched result image 517, based on a second reference 513, to be displayed on a screen in the monitoring terminal. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a search result display method of an image search system, and more particularly to a technique for displaying a search result of similar images.

  With the increase in crime occurrence today, it is required to store surveillance images from a plurality of imaging devices over a long period of time. Until now, it was common to store only the latest monitoring video with a single imaging device in a small store, etc. Monitoring became possible. While the burden on the video supervisor increases, searching for a specific person or specific scene from the recorded surveillance video for the purpose of reducing the burden on the supervisor is similar using image composition, color distribution, etc. It has become possible with image search technology. In such a similar image search system for monitoring videos, various search result display methods have been proposed.

  FIG. 9 shows an example of a configuration of a similar image search system for a conventional surveillance video. The network 200 is a communication network such as a dedicated line, an intranet, the Internet, or a wireless LAN that connects devices to each other and performs data communication. The imaging device 201 is a device such as a network camera that performs digital conversion processing on the captured video and outputs the converted video data to the recording device 202 via the network 200. The recording device 202 includes a microcomputer and the like, and is a device such as a network digital recorder that records image data input from the imaging device 201 via the network 200 on a recording medium such as an HDD (hard disk device). The monitoring terminal 210 is a device such as a PC (personal computer) that displays the image data acquired from the recording device 202 via the network 200 on a monitor such as a liquid crystal display or a CRT. The monitoring terminal 210 includes a monitor that displays a recorded image, a search result image, a similar image search menu, and a user input unit such as a keyboard and a mouse. The monitoring terminal 210 reproduces a video recorded in the recording device 202, searches for a similar image, and the like. Provides a user interface.

  The recording device 202 includes functional units such as a network transmission / reception unit 203, a video recording unit 204, a video distribution unit 205, an image feature amount recording unit 206, an image feature amount extraction unit 207, and an image similarity determination unit 208.

  The network transmission / reception unit 203 is a processing unit that transmits / receives data to / from devices connected via the network 200. The data transmitted / received here is, for example, a captured image transmitted from the imaging device 201, a video reproduction request or search request from the monitoring terminal 210, a distribution image to the monitoring terminal 210, or a search result. Here, the captured image received by the network transmission / reception unit 203 from the imaging device 201 via the network 200 is output from the network transmission / reception unit 203 to the video recording unit 204 and the image feature amount recording unit 206. The video reproduction request signal received from the monitoring terminal 210 via the network 200 is output to the video distribution unit 205, and the search request signal is output to the image similarity determination unit 208. The distribution image is input from the video distribution unit 205 and the search result is input from the image similarity determination unit 208 to the network transmission / reception unit 203 and transmitted to the monitoring terminal 210.

  The video recording unit 204 is a processing unit that records image data input from the imaging apparatus 201 via the network 200 onto a recording medium such as an HDD (not shown). When recording image data, information for later retrieving image data, for example, an image ID is recorded at the same time. In addition, information for identifying when the image was taken, for example, the image time is also recorded. Here, the image ID is, for example, a frame number assigned in order in units of frames from the start of recording so as to be unique within the apparatus. The image time is, for example, a device time output from a clock function unit built in the recording device 202 or a device time output from a clock function unit built in the imaging device 201.

  The video distribution unit 205 is a processing unit that distributes image data recorded in a recording medium such as the video recording unit 204 and the image feature amount recording unit 206. The video distribution unit 205 determines a distribution image according to a video reproduction request signal input from the monitoring terminal 210 via the network 200, reads out image data from the recording medium using the image ID and the image time, and transmits and receives via the network. Input to the unit 203.

  The image feature amount recording unit 206 is a processing unit that records an image feature amount on a recording medium. The image feature amount is acquired by outputting the input image data to the image feature amount extraction unit 207. When recording the image feature amount, the image ID corresponding to the input image data is also recorded. The list data composed of the image ID and the image feature amount generated by this recording is hereinafter referred to as image feature amount list data.

  The image feature amount extraction unit 207 is a processing unit that calculates the feature amount of the image data input from the network transmission / reception unit 203 using an image recognition technique. Here, as the image feature amount, for example, an image color distribution, an edge pattern composition distribution, a combination thereof, or the like is used.

  The image similarity determination unit 208 is a processing unit that performs an image search and outputs a search result. The image similarity determination unit 208 includes an image feature amount of a search image designated as a template of an image to be searched for determining similarity, and an image feature of image data recorded on a recording medium of the video recording unit 204. The similarity is calculated from the quantity, and the search result is generated from the calculated similarity. The search image is included in the search request signal input from the monitoring terminal 210 via the network 200. By outputting this search image to the image feature amount extraction unit 207, the image feature amount of the search image data is obtained. get. The image feature amount of the image data recorded on the recording medium is acquired from the image feature amount list data recorded in the image feature amount recording unit 206 using an image ID corresponding to the image data.

  Regarding the calculation method of similarity, refer to the following non-patent document 1 ("Representation model for large-scale image collection" Kaoru Tsujiike et al., Japanese Society of Photography, Vol. 66, No. 1, P93-P101) can do.

  The monitoring terminal 210 includes functional units such as a network transmission / reception unit 211, a search request unit 212, a search result display unit 213, and a video playback unit 214.

  The network transmission / reception unit 211 is a processing unit that transmits / receives data to / from the recording device 202 connected via the network 200. The data transmitted / received here is, for example, a video reproduction request signal or a search request signal to the recording device 202, a distribution image from the recording device 202, or a search result. Here, the distribution image received from the recording device 202 is output to the video reproduction unit 214, and the search result is output to the search result display unit 213. The video playback request signal is input from the video playback unit 214, and the search request signal is input from the search request unit 212 to the network transmission / reception unit 211, and transmitted from the network transmission / reception unit 211 to the recording apparatus 202 via the network 200.

  The search request unit 212 is a processing unit that converts a search operation performed by the user from a screen displayed on the monitor of the monitoring terminal 210 into a search request signal and outputs the search request signal.

  The search result display unit 213 is a processing unit that draws the input search result from the recording device 202 on a screen displayed on the monitor of the monitoring terminal 210.

  The video playback unit 214 is a processing unit that converts a playback operation by the user given from a screen displayed on the monitor of the monitoring terminal 210 into a video playback request signal and outputs the video playback request signal. Also, a process of rendering the input distribution image from the recording device 202 on a screen displayed on the monitor of the monitoring terminal 210 is performed.

  In FIG. 10, an example of the screen displayed on the monitor of the conventional monitoring terminal 210 is shown.

  The recorded video display area 300 is an area for reproducing and displaying the monitoring video recorded in the recording device 202.

  The video playback operation area 301 is an area in which a button for playing back video recorded on the recording device 202 is displayed. Each button constituting the video playback operation area 301 is assigned a unique playback type. In FIG. 10, for example, playback types of rewinding from the left, reverse playback, playback stop, forward playback, and fast forward are assigned. By appropriately pressing each button, the moving image display on the playback video display area 300 is switched according to the playback type assigned to the button.

  The search image designation button 302 is a button for designating an image displayed in the recorded video display area 300 as a search image as a model of an image to be searched for determining the similarity.

  The search image display area 303 is an area for displaying the search image designated by the search image designation button 302.

  The search condition setting area 304 is an area composed of buttons and input fields that give conditions related to search refinement. The search condition setting area 304 displays input fields for a search start time specification field 305, a search end time specification field 306, and a camera selection button 307.

  The search start time specification field 305 is an input field for specifying the start time as a search range among the search narrowing conditions. The search end time designation field 306 is an input field for designating the end time as a search range among the search refinement conditions. The camera selection button 307 is a button for designating an imaging device among the search narrowing conditions.

  The display condition setting area 308 is an area composed of buttons and input fields for giving conditions relating to the display of search results. In the display condition setting area 308, input fields for a display image number designation field 309 and a display order condition setting area 310 are displayed.

  The display image number designation column 309 is an input column for designating the number of images to be displayed in the search result display area 316 as a search result. The display order condition setting area 310 is an area for setting the display order of images to be displayed in the search result display area 316 as a search result, and includes a time series button 311 and a similarity order button 312. The time series button 311 is a button that is selected when the display order is time series. The similarity order button 312 is a button that is selected when the display order is the similarity. The selection of 311 and 312 is an alternative.

  The search execution button 313 is a button for executing a search under the search conditions set in the search condition setting area 304 and the display conditions set in the display condition setting area 308 using the image displayed in the search image display area 303 as a search image. It is.

The search result display area 316 is an area for displaying the search result, and the display content includes a reduced image 317, a similarity display 318, and a time display 319. These are displayed for the number specified in the display image number specification field 309, and the display order follows the conditions specified in the display order condition setting area 310. The reduced image 317 is an image obtained by reducing and displaying the search result image. The similarity display 318 is a text display of the similarity of the image indicated by the reduced image 317 with respect to the search image. The time display 319 is a text display of the image time of the image indicated by the reduced image 317.
JP 2003-15626 A "Representation model for large-scale image collection" Satoshi Tsunoike et al. Photographic Society of Japan, Vol. 66, No. 1, P93-P101

  In a conventional similar image search system, when time series display is performed, the similarity of each displayed image is only performed by displaying characters in the vicinity of the image, and the similarity of each image It was difficult to visually grasp the magnitude relationship. Similarly, when the order of similarity is displayed, the time of each displayed image is only displayed in the vicinity of the image, and the time sequence of each image is visually displayed. It was difficult to grasp. This invention is made | formed in view of such a condition, and it aims at solving the above-mentioned subject.

  A search result display method of an image search system according to the present invention includes an imaging device, a recording device that records an image captured by the imaging device, and a monitoring terminal, and records a similar image for a specific image in the recording device. In a search result display method of an image search system for searching from searched images and displaying the searched images on the monitoring terminal, the search result images are sorted based on a first criterion, and the search is performed based on a second criterion The result image is deformed and displayed on the monitor terminal.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the search result display method which can grasp | ascertain simultaneously the magnitude relationship and front-back relationship about binary information like time and similarity.

  Hereinafter, the present invention will be illustrated by specific embodiments.

  FIG. 1 shows an example of the configuration of a similar image search system according to the present invention. In this figure, the same reference numerals as those in FIG. 9 denote an apparatus and a processing unit that provide the same functions as those in the prior art.

  The recording device 402 includes a microcomputer and the like, and is a device such as a network digital recorder that records image data input from the imaging device 201 via the network 200 on a recording medium such as an HDD. The recording apparatus 402 includes a new image display displacement amount calculation unit 409 in addition to the processing units shown in the conventional example.

  The image display displacement amount calculation unit 409 is a processing unit that calculates the image display displacement amount from the input search result. The image display displacement amount is information used for displaying the search result on the search result display unit 413 in the monitoring terminal 410. The amount of image display displacement and its calculation method will be described later.

  The monitoring terminal 410 is a device such as a PC that displays the image data acquired from the recording device 402 via the network 200 on a monitor such as a liquid crystal display or a CRT. The monitoring terminal 410 includes a monitor that displays a recorded image, a search result image, a similar image search menu, and a user input unit such as a keyboard and a mouse. The monitoring terminal 410 reproduces a video recorded in the recording device 402, searches for a similar image, and the like. Provides a user interface.

  The monitoring terminal 410 includes a search result display unit 413 having a new process in addition to the conventional network transmission / reception unit 211, search request unit 212, and video playback unit 214.

  The search result display unit 413 is a processing unit that draws an input search result on a screen displayed on the monitor of the monitoring terminal. At that time, a screen display using the image display displacement amount is performed. This screen display method will be described later.

  FIG. 2 shows an example of a screen displayed on the monitoring terminal in the present invention. In FIG. 2, the same reference numerals as those in FIG. 10 indicate areas, buttons, and input fields that provide functions similar to those in the related art.

  The display condition setting area 508 is an area composed of buttons and input fields that give conditions relating to the display of search results. In the display condition setting area 508, input fields for a display image number designation field 509, a display order condition setting area 510, and an image enhancement condition setting area 513 are displayed.

  The display image number designation column 509 is an input column for designating the number of images to be displayed in the search result display area 516 as a search result. The display order condition setting area 510 is an area for setting the display order of images to be displayed in the search result display area 516 as a search result, and includes a time series button 511 and a similarity order button 512. The time series button 511 is a button that is selected when the display order is time series. The similarity order button 512 is a button that is selected when the display order is the similarity. The selection of 511 and 512 is an alternative.

  The image enhancement condition setting area 513 is an area for setting an enhancement type of an image to be displayed in the search result display area 516 as a search result, and includes a rotation button 514 and a size button 515. The rotation button 514 is a button that is selected when the enhancement type is the rotation angle of the image. The size button 515 is a button that is selected when the enhancement type is the image size. The selection of 514 and 515 is an alternative. The emphasized reduced image 517 is an image obtained by reducing and displaying the search result image.

  Here, the difference between the conventional search result display in the similar image search system and the search result display of the present invention will be described with reference to FIGS.

  FIG. 3A schematically shows the conventional technique for displaying the search result display only in the order of similarity, and the figure in the figure is a reduced image 317 displayed in the conventional search result display area 316. Represents. The reduced image examples 103 and 104 are two arbitrary images among the reduced images used in this description.

  FIG. 3B schematically shows the search result display of the present invention, and the figure in the figure emphasizes “similarity” as the display order in the display conditions applied to the search result image. An example of the case where “rotation” according to the image time is set as the type is shown, and represents an enhanced reduced image 517 displayed in the search result display area 516. The emphasized reduced image examples 100 to 102 are arbitrary three images among the emphasized reduced images used in this description.

  The magnitude relationship between the similarities in the reduced image examples 103 and 104 in FIG. 3A can be understood in the display order. In this example, it can be understood from the visual display, that is, the display position of the image, that the reduced image example 103 has a higher degree of similarity than the reduced image example 104. However, there is no visual display of the relationship between the image times of the reduced image example 103 and the reduced image example 104.

  On the other hand, in FIG. 3B, the magnitude relationship between the similarities in the emphasized reduced image examples 100, 101, and 102 can be understood in the display order as in FIG. In this example, it can be understood from the visual display, that is, the display position of the image, that the degree of similarity increases in the order of the emphasized reduced image examples 101, 100, and 102. Further, the front-rear relationship of the image times in the emphasized reduced image examples 100, 101, and 102 can be understood from the inclination of the display. In this example, when it is assumed that the image time of the enhanced reduced image example 100 is the time of interest, for example, the time closest to the time when the search execution button 313 is pressed, the enhanced reduced image example 100 facing the front is on the left. It can be understood that the tilted emphasized reduced image example 101 is an image captured before the emphasized reduced image 100. Similarly, it can be understood that the emphasized reduced image 102 tilted to the right with respect to the emphasized reduced image example 100 facing the front is an image captured after the emphasized reduced image 100.

  Next, FIG. 4, FIG. 5, and FIG. 6 show an example of the processing flow of the similar image search system for the surveillance video in the present invention. The process flow is roughly divided into three processes: a recording process, a video reproduction process, and a similar image search process.

  FIG. 4 shows an example of the flow of recording processing. The recording process is performed by a process including communication between the imaging device 201 and the recording device 402, and the video from the imaging device 201 is recorded in the recording device 402. The processing flow shown in this figure is the same in the conventional similar image search system.

In step 600, the recording apparatus 402 waits for image reception. When the network transmitting / receiving unit 203 detects an incoming image, the process proceeds to step 601.
In step 601, the recording apparatus 402 receives an image from the imaging apparatus 201 by the network transmission / reception unit 203 via the network 200.
In step 602, the recording device 402 records the received image data, image ID data, and image time data on a recording medium. As the image ID, for example, a frame number assigned in order from the start of recording to each frame so as to be unique within the recording apparatus 402 is used. Further, as the image time, for example, the device time output from the clock function unit built in the recording device 402 or the device time output from the clock function unit built in the imaging device 201 is used. .
In step 603, the recording apparatus 402 extracts the feature amount of the image by the image feature amount extraction unit 207.
In step 604, the recording apparatus 402 records the image feature amount extracted by the image feature amount extraction unit 207 on the recording medium according to the image ID by the image feature amount recording unit 206.

In step 610, the imaging apparatus 201 waits for a captured image output from an imaging element such as a CCD or CMOS mounted in the imaging apparatus 201. If image output is detected, the process proceeds to 611.
In step 611, the imaging apparatus 201 digitally converts the output captured image.
In step 612, the imaging apparatus 201 transmits the digitally converted image to the recording apparatus 402 via the network 200.
Timing 620 represents communication between the imaging device 201 and the recording device 402.

  FIG. 5 shows an example of the flow of video reproduction processing. The video playback process is performed by a process including communication between the recording device 402 and the monitoring terminal 410, and the video recorded in the recording device 402 is played back by the monitoring terminal 410. The processing flow shown in this figure is the same in the conventional similar image search system.

In step 700, the monitoring terminal 410 waits for the user's reproduction operation. If the monitoring terminal 410 detects a reproduction operation by the user operating the user input unit, the process proceeds to step 701.
In step 701, the video playback unit 214 of the monitoring terminal 410 determines the content of the video playback request according to the playback operation content of the user. The content of the video playback request is composed of the type of playback such as forward playback and fast forward, the time direction of playback, the speed of playback, and the like.
In step 702, the monitoring terminal 410 transmits the content of the determined video playback request as a video playback request signal from the network transmission / reception unit 211 to the recording device 402 via the network 200 (timing 720).
In step 703, the monitoring terminal 410 waits for reception of data from the recording device 402. When the arrival of data from the recording device 402 is detected, the process proceeds to step 704.
In step 704, the monitoring terminal 410 receives the transmission data from the recording device 402 by the network transmission / reception unit 211 via the network 200 (timing 721, 722).
In step 705, the monitoring terminal 410 verifies the received content in the video reproduction unit 214. If the received content is a distribution image, the process proceeds to step 706. If the received content is a video reproduction completion notification, the process proceeds to step 700.
In step 706, the monitoring terminal 410 displays the received distribution image on the monitor screen by the video reproduction unit 214. After the display is completed, the process proceeds to step 703.

In step 710, the recording apparatus 402 waits for reception of a video reproduction request. If an incoming video playback request is detected, the process proceeds to step 711.
In step 711, the recording apparatus 402 receives the video reproduction request signal from the monitoring terminal 410 via the network 200 by the network transmission / reception unit 203 (timing 720).
In step 712, the recording apparatus 402 determines the content of video playback based on the video playback request signal by the video distribution unit 205. The content of the video reproduction is composed of, for example, the image ID of the image to be distributed first, the number of images to be distributed, the image ID of the image to be distributed next, the distribution timing, and the like.
In step 713, the recording device 402 extracts an image from the recording medium. At the time of extraction, the image ID of the image to be distributed first in the video reproduction content is used.
In step 714, the recording apparatus 402 waits for a transmission time. As the transmission time, the delivery timing in the video playback content is used. When the transmission time is reached, the process proceeds to step 715.
In step 715, the recording device 402 transmits the extracted image from the network transmission / reception unit 203 to the monitoring terminal 410 via the network 200 (timing 721).
In step 716, the recording apparatus 402 determines completion of video reproduction. Completion determination is performed based on whether or not image distribution satisfying the previously determined video reproduction content has been completed. If it is determined to be complete, the process proceeds to step 717, and if it is determined to be incomplete, the process proceeds to step 718.
In step 717, the recording apparatus 402 transmits a video reproduction completion notification from the network transmission / reception unit 203 to the monitoring terminal 410 (timing 722).
In step 718, the recording apparatus 402 updates the video playback content. At this time, the image ID and distribution timing of the image to be distributed next are updated. When the update is completed, the process proceeds to step 713.
Timings 720 to 722 represent communication timings between the recording device 402 and the monitoring terminal 410.

  FIG. 6 shows an example of the flow of similar image search processing. The similar image search process is performed by a process including communication between the recording device 402 and the monitoring terminal 410. An image similar to the search image is searched from the video recorded in the recording device 402 and transmitted to the monitoring terminal 410. And display it as a search result. The search result display process is also included in the process flow shown in the figure.

In step 800, the monitoring terminal 410 waits for a user search operation. If a user search operation is detected, the process proceeds to step 801.
In step 801, the monitoring terminal 410 verifies the content of the detected user operation. If the operation content is pressing of the search image designation button 302, the process proceeds to step 802. If the search execution button 313 has been pressed, the process proceeds to step 804. Otherwise, the process proceeds to step 803.
In step 802, the monitoring terminal 410 sets the image displayed in the recorded video display area 300 as a search image, and displays the searched image in the recorded video display area 303. At this time, the image time associated with the search image is taken out and stored in a temporary storage area.
In step 803, the monitoring terminal 410 stores the content set by the user operation, that is, the search condition or the display condition in the temporary storage area.
In step 804, the monitoring terminal 410 transmits a search request signal from the network transmission / reception unit 211 to the recording device 402 via the network 200 (timing 820). The search request signal includes a search image, a search condition, a display condition, an image time of the search image, and the like.
In step 805, the monitoring terminal 410 waits for reception of a search result from the recording device 402. If an incoming search result is detected, the process proceeds to step 806.
In step 806, the monitoring terminal 410 receives the search result transmitted from the recording device 402 by the network transmission / reception unit 211 (timing 821).
In step 807, the monitoring terminal 410 displays the received search result on the search result display area 316 on the monitor by the search result display unit 413. This process includes a search result display process using the image display displacement amount calculated in step 814 to be described later. The detailed processing flow and the data structure of the search result will be described later.

In step 810, the recording apparatus 402 waits for reception of a search request signal. When the incoming search request signal is detected, the process proceeds to step 811.
In step 811, the recording apparatus 402 receives the search request signal transmitted from the monitoring terminal 410 by the network transmission / reception unit 203 via the network 200 and outputs it to the image similarity determination unit 208 (timing 820).
In step 812, the image similarity determination unit 208 calculates the image feature amount from the search image included in the received search request signal by the image feature amount extraction unit 207.
In step 813, the image similarity determination unit 208 determines the similarity between the image feature amount of the search image calculated in step 812 and the image feature amount of each image recorded in the image feature amount recording unit 206. And a set of images having a certain degree of similarity and satisfying the search condition is extracted and generated as a search result. In addition to the extracted image, the search result is composed of image time and similarity corresponding to each of these images. The data structure of this search result will be described later. Further, the set of extracted images included in the search result may be a set of reduced images obtained by reducing each image recorded in the video recording unit 204.
In step 814, the recording apparatus 402 applies display conditions to the extracted individual images (hereinafter referred to as search result images) included in the generated search results by the image display displacement amount calculation unit 409, respectively. Then, the image display displacement amount is calculated, and the result is also added to the search result. This processing includes calculation processing of an image display displacement amount used in the search result display processing in step 807. The image display displacement amount and its calculation process will be described later.
In step 815, the recording apparatus 402 transmits the search result from the network transmission / reception unit 203 to the monitoring terminal 410 via the network 200 (timing 821).
Timings 820 and 821 represent communication timings between the recording device 402 and the monitoring terminal 410.

  Next, the image display displacement amount and the calculation process in the search result display process in step 807 will be described.

In the above-described step 814, the case where “similarity” is set as the display order and “rotation” corresponding to the image time is set as the enhancement type in the display condition applied to the search result image will be described as an example. This corresponds to the case where the search execution button 313 is pressed in a state where the similarity order button 512 and the rotation button 514 are selected in FIG. In this case, since the rotation button 514 is selected, the image display displacement amount is a rotation angle. A process for calculating the rotation angle when the image display displacement amount is a rotation angle will be described below.
The time when the search execution button 313 is pressed is assumed to be t _c . If the search result image obtained in step 813 is an image v, the search result is composed of the number of images v designated in the display image number designation column 509. If the image time of the image v is t _v , the rotation angle Rv [rad] of the image v is (π / 2)
× {1 / (| Tmax-t _c |)} × (t _v -t _c ) Here, Tmax is the farthest from time t _c of the image v. In addition, the time t _c is a reference time for rotation, and in this description, the search execution button is pressed. However, the reference time may be used as the image time of the search image. Any given time may be used.

In the above-described step 814, an example will be described in which “similarity” is set as the display order and “size” corresponding to the image time is set as the enhancement type in the display condition applied to the search result image. This corresponds to the case where the search execution button 313 is pressed in a state where the similarity order button 512 and the size button 515 are selected in FIG. In this case, since the size button 515 is selected, the image display displacement amount is the size. The size calculation process when the image display displacement amount is a size will be described below.
The time when the search execution button 313 is pressed is assumed to be t _c . If the search result image obtained in step 813 is an image v, the search result is composed of the number of images v designated in the display image number designation column 509. When the image time of the image v and t _v, the vertical size of the image v is, (Hmin + Hmax)
/ 2+ [{(Hmax-Hmin) / 2} / (| Tmax-t _c |)] × (t _v -t _c ), horizontal size is (Wmin + Wmax)
/ 2+ determined by [{(Wmax-Wmin) / 2} / (| | Tmax-t c)] × (t v -t c). Here, Hmin is the minimum display vertical size, Hmax is the maximum display vertical size, Wmin is the minimum display horizontal size, and Wmax is the maximum display horizontal size. In addition, Tmax is the most distant time to time t _c in the recorded video. In addition, the time t _c is a time used as a size reference, and in this description, the search execution button is pressed. However, the reference time may be used as the image time of the search image. Any given time may be used.

In the above-described step 814, the case where “time series” is set as the display order and “rotation” corresponding to the similarity is set as the display type in the display condition applied to the search result image will be described as an example. This corresponds to the case where the search execution button 313 is pressed in a state where the time series button 511 and the rotation button 514 are selected in FIG. In this case, since the rotation button 514 is selected, the image display displacement amount is a rotation angle. The calculation process of the rotation angle when the image display displacement amount is the rotation angle will be described below.
If the search result image obtained in step 813 is an image v, the search result is composed of the number of images v designated in the display image number designation column 509. If the similarity of image v is S _v , the rotation angle Rv [rad] of image v is (π / 2)
× Calculated with {S _v / Smax}. Here, Smax is the similarity of the most dissimilar images. However, it is assumed that the similarity Sv is 0 or more, and the similarity is lower as the value is larger.

In the above-described step 814, the case where “time series” is set as the display order and “size” corresponding to the similarity is set as the display type in the display condition applied to the search result image will be described as an example. This corresponds to the case where the search execution button 313 is pressed in a state where the time series button 511 and the size button 515 are selected in FIG. In the above case, since the size button 515 is selected, the image display displacement amount is the size. A method for calculating the size when the image display displacement amount is a size will be described below.
If the search result image obtained in step 813 is an image v, the search result is composed of the number of images v designated in the display image number designation column 509. If the similarity of image v is S _v , the vertical size of image v is Hmax
- (Hmax-Hmin) × ( S v / Smax), horizontal size, Wmax - (Wmax-Wmin) × (S v / Smax), obtained by. Here, Hmin is the minimum display vertical size, Hmax is the maximum display vertical size, Wmin is the minimum display horizontal size, Wmax is the maximum display horizontal size, and Smax is the similarity of the least similar images. However, it is assumed that the similarity Sv is 0 or more, and the similarity is lower as the value is larger.

  FIG. 7 shows an example of the data structure of the search result in the search result display method of the present invention. In FIG. 7, as an example, in the above-described step 814, “similarity” is set as the display order and “rotation” corresponding to the image time is set as the enhancement type in the display condition applied to the search result image. explain. In FIG. 7, for simplicity of explanation, the data structure is described using a table format composed of rows and columns. However, in the implementation, a structure such as a tree format or a list format may be used.

The row number column 1000 is a column for storing numbers used when reading data from this data structure. Numbering starts from the top row of the table. Each data stored in the same row is data associated with the same search result image.
An image data column 1001 is a column for storing image data of a search result image. In FIG. 7, an image is displayed in a table cell as an image for storing image data. However, in implementation, a separate image storage area may be provided to store a reference address and an image ID for the area. Good.
The similarity column 1002 is a column that stores the similarity of search result images. In FIG. 7, since “similarity” is set in the display order, the stored values of this column are stored in ascending order.
The time information column 1003 is a column that stores the image time of the search result image. When “time series” is set in the display order, the stored values of this column are stored in ascending order.
The image ID column 1004 is a column that stores an image ID.
The displacement amount column 1005 is a column that stores the image display displacement amount of the search result image. In the example of FIG. 7, since “rotation” is set as the enhancement type, the stored value of this column stores the rotation angle, but when “size” is set, the vertical size and The horizontal size is stored.

FIG. 8 shows an example of a detailed processing flow in step 807 shown in FIG. 6 including the search result display processing of the present invention.
Step 901 is performed after step 806. The steps described in FIG. 8 are performed by the search result display unit 413 of the monitoring terminal 410. In step 901, an image display area is secured in the search result display area 516. If the previous search result is already displayed, delete it.
In step 902, the setting state of the emphasis type, that is, the selection state of the rotation button 514 and the size button 515 is acquired.
In step 903, the reference row counter is initialized to 1. Here, the reference row counter is used for reading from the data structure of the search result (see FIG. 7), and the row having the row number of the value indicated by the reference row counter becomes the reference row.
In step 904, it is determined whether the reference row counter has exceeded the last row. The determination is performed by comparing the number of rows in the data structure of the reference row counter and the search result. If the reference row counter does not exceed the last row, the process proceeds to step 905. If the reference row counter exceeds the last row, the process proceeds to step 800.
In step 905, the image data, the similarity, the time information that is the image time of the search result image, and the displacement amount of the image display are read from the reference row. In this example, an example in which image data is directly read has been described. However, when an image ID is stored in the image data string, communication processing for acquiring an image from the recording device based on the image ID is performed here. You may make it implement.
In step 906, the display position of the image data is calculated. The display position is calculated according to the rule from top to bottom and from left to right in ascending order based on the counter value of the reference row counter. In this example, since the “similarity” is stored in ascending order in the similarity column 1002, display positions are determined from top to bottom and from left to right in descending order of similarity values. .
In step 907, the set value is determined for the enhancement type acquired in step 902. If the set value is “rotation”, the process proceeds to step 908. If it is “size”, the process proceeds to step 909.
If the set value is “rotation”, in step 908, rotation conversion processing is performed on the image data using the displacement amount of the image display, that is, the rotation angle. For example, in an image having a horizontal size W, when the rotation angle is calculated as Rv [rad] by the above-described calculation method, the horizontal size is set to W × cos (Rv). As a result, in the coordinate system in which the horizontal direction of the image is the x axis, the vertical direction is the y axis, and the depth direction is the z axis, an image obtained by rotating the image around the y axis by Rv [rad] is an image viewed from the front. . Further, the image may be expressed three-dimensionally by changing the viewpoint before and after image conversion. Japanese Patent Laid-Open No. 2003-15626 disclosed in Patent Document 1 is a conventional technique for rotating and displaying a plurality of images three-dimensionally on a display. In this conventional technique, an image is rotated and displayed in a virtual three-dimensional space.
If the set value is “size”, in step 909, a size conversion process is performed on the image data using the displacement amount of the image display, that is, the size. For example, in an image with the vertical size H and the horizontal size W, if the converted vertical size and horizontal size are determined as Hv and Wv by the above calculation method, the vertical size of the image is Hv / H and the horizontal size is Wv. Convert to / W times.
In step 910, the converted image data is displayed as the enhanced reduced image 517 at the display position calculated in step 906.
In step 911, the similarity and image time are displayed as characters in the vicinity of the lower part of the emphasized reduced image 517 as a similarity display 518 and a time display 519, respectively.
In step 912, 1 is added to the reference row counter.

As described above, the present invention has been described with the specific embodiments. However, the present invention is not limited to the above-described embodiments. For example, the search function is described as being implemented in the recording device 402. The structure mounted in another apparatus may be sufficient.
Further, in this example, the image display displacement amount calculation unit 409 has been described as being mounted on the recording device 402, but the image display displacement amount calculation unit 409 may be mounted on the monitoring terminal 410.

  In this example, the case where there is one person in the search image is shown. However, when there are a plurality of persons in the search image, the search results for each person are added together. The result may be displayed.

  Also, in this case, the monitoring terminal 410 notifies the user that there are a plurality of people, and designates and selects one person to be searched for with a pointing device such as a mouse provided in the monitoring terminal 410, and the selection is made. You may make it carry out with respect to one person.

  Furthermore, in this example, the case where the search image is selected from images captured by the imaging device 201 and recorded in the recording device 402 has been described. For example, an image captured by a device different from the imaging device 201 is used. It is also possible.

  In this example, the highlighted graphic deformation is shown by the case of rotation or size change. For example, the corner of a reduced image displayed in a rectangle may be rounded and the emphasis may be expressed by the roundness. Alternatively, for example, the resolution of an image may be changed, and an image that is desired to be emphasized may have a higher resolution and a clearer image.

  Further, the highlighting may be based on a graphic expression in the vicinity of the image. For example, a frame may be attached to the image and the emphasis may be expressed by the thickness thereof. Emphasis may be expressed by type or shading, and for example, an axis indicating emphasis may be added for each image, and emphasis may be expressed by the scale. The imaging device and the recording device may be integrally formed.

  Also, in this example, if the display condition applied to the search result image is “similarity” as the display order and “rotation” according to the image time as the enhancement type, “similarity” is set as the display order. If “size” according to the image time is selected as the highlighting type, “time series” is selected as the display order, and “rotation” is selected as the highlighting type according to the similarity, and “time series” is selected as the display order and emphasized. Example of “size” corresponding to the similarity as the type, ie when “similarity” is selected as the display order, highlighting is performed according to the image time, and “time series” is selected as the display order Explained the example of performing highlighting according to the similarity, but it is also possible to implement “similarity” as the display order and “rotation” according to similarity as “highlighting type” or “size” according to similarity Can do. In this case, a selection button for changing the rotation and size of the image may be provided in the image enhancement condition setting area 513. In this modified example, although it is not possible to display an image with respect to time, the degree of similarity is displayed as a graphic deformation, so that the degree of similarity can be easily grasped visually. In this case as well, the type of emphasis is not limited to “rotation” or “size”, but the corners of the above-mentioned reduced image are rounded according to the degree of similarity, and the image resolution, graphic representation near the image, frame, axis, etc. are transformed. It can be displayed.

  In addition, images having a similarity with the search image higher than a predetermined criterion are detected in advance, and these detected images are displayed in “time series” as the display order and “rotation” corresponding to the time series as the enhancement type. Or it can implement also as "size" according to time series. Also in this case, a selection button for changing the rotation and size of the image depending on what is preferably provided in the image enhancement condition setting area 513. In this modified example, although it is not possible to highlight the similarity, an image with high similarity can be detected in advance, and then the image time is highlighted, so that the similarity close to the time of the search image is obtained. It becomes easy to grasp a high image visually. In this case as well, the type of emphasis is not limited to “Rotate” or “Size”, but the rounded corners of the above-mentioned reduced images are rounded according to the time series, and the image resolution, graphic representation near the image, frame, axis, etc. are transformed. It can be displayed.

In summary, the search result display method of the image search system of the present invention includes an imaging device that captures an image, a recording device that records the image captured by the imaging device, and a monitoring terminal. In the search result display method of the image search system for searching similar images for the video from the video recorded in the recording device and displaying the search result images on the monitoring terminal, the search result images are sorted based on a first criterion, The search result image is deformed based on a second criterion and displayed on the monitor terminal.
In the search result display method of the image search system according to the present invention, the first reference is a similarity relationship or a context of image time, and the second reference is an image time context or similarity. It is characterized by a large and small relationship.
In the search result display method of the image search system according to the present invention, the search result images based on the first reference may be sorted in time-series order of captured time or similarity order with respect to a specific video. Features.
The search result display method of the image search system according to the present invention is characterized in that the image deformation of the search result image based on the second reference is an image rotation deformation or an image size deformation.
The image search system of the present invention includes an imaging device that captures a video, a recording device that records the video captured by the imaging device, and a monitoring terminal, and a similar image for a specific video is stored in the recording device. In an image search system for searching from recorded video and displaying a search result image on the monitoring terminal, an image display position calculating means for sorting the search result images based on a first criterion, and a second criterion The image processing apparatus includes image deformation means for deforming the search result image and image display means for displaying the search result image on the monitoring terminal.
In the image search system of the present invention, the first criterion is a similarity relationship or a context relationship of image times, and the second criterion is a context relationship of image times or a similarity relationship. It is characterized by.
In addition, the sorting of the search result images based on the first criterion in the image search system of the present invention is a time-series order of captured times or a similarity order with respect to a specific video.
Further, the image deformation of the search result image based on the second criterion in the image search system of the present invention is an image rotation deformation or an image size deformation.

  While the present invention has been specifically described in the above embodiment, it is needless to say that the above embodiment is an exemplification of the present invention and the present invention is not limited to the above embodiment.

  The present invention can be widely used for a display device that highlights a specific image when displaying a plurality of images.

It is a figure which shows an example of a structure of the similar image search system for the surveillance video of embodiment in this invention. It is a figure which shows an example of the screen displayed on the monitoring terminal of embodiment in this invention. It is a figure explaining the image display of embodiment of this invention. It is a figure which shows an example of the flow of the video recording process of the similar image search system which made object the monitoring image | video of embodiment in this invention. It is a figure which shows an example of the flow of the image | video reproduction | regeneration processing of the similar image search system for the monitoring image | video of embodiment in this invention. It is a figure which shows an example of the flow of the similar image search process of the similar image search system for the surveillance video | video of embodiment in this invention. It is a figure which shows an example of the data structure of the search result in the search result display method of this invention. It is a figure which shows an example of the flow of a detailed process in step 807 including the search result display method of this invention. It is a figure which shows an example of the structure of the similar image search system for the conventional monitoring image | video. It is a figure which shows an example of the screen displayed on the conventional monitoring terminal.

Explanation of symbols

100 to 102: Enhanced reduced image example, 103 to 104: Reduced image example, 200: Network, 201: Imaging device, 202: Recording device, 203: Network unit, 204: Video recording unit, 205: Video distribution unit, 206: Image feature amount recording unit, 207: Image feature amount extraction unit, 208: Image similarity determination unit, 210: Monitoring terminal, 211: Network unit, 212: Search request unit, 213: Search result display unit, 214: Video playback unit 300: Recorded video display area 301: Video playback operation area 302: Search image designation button 303: Search image display area 304: Search condition setting area 305: Search start time designation field 306: Search end time designation Column, 307: camera selection button, 308: display condition setting area, 309: display image number designation field, 310: display order condition setting area, 311 Time series button, 312: Button in order of similarity, 313: Search execution button, 316: Search result display area, 317: Reduced image, 318: Display of similarity, 319: Time display, 402: Recording device, 409: Image display displacement Quantity calculation unit, 410: monitoring terminal, 413: search result display unit, 513: image enhancement condition setting area, 514: rotation button, 515: size button, 516: search result display area, 517: enhanced reduced image, 518: similar Degree display, 519: Time display, 600-604: Step, 610-612: Step, 620: Timing, 700-706: Step, 710-718: Step, 720-722: Timing, 800-807: Step, 810 815: Step, 820-821: Timing, 901-912: Step, 1000: Line number Column, 1001: image data string 1002: similarity column, 1003: time information sequence, 1004: Image ID column 1005: displacement train

Claims (1)

An imaging device, a recording device that records an image captured by the imaging device, and a monitoring terminal, search for images similar to a specific image from images recorded in the recording device, and search for the searched image In the search result display method of the image search system to be displayed on the monitoring terminal,
A search result display method for an image search system, wherein search result images are sorted based on a first criterion, and the search result image is deformed based on a second criterion and displayed on the monitor terminal.

Images