JP2010233185A - Apparatus, system and program for processing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To store only a partial image-of-interest obtained by surely tracking a mobile without any processing burden, and to efficiently retrieve a desired image from among stored images. <P>SOLUTION: A gaze area is set in a panoramic image of a monitoring area, an image in the set gaze area is analyzed by an image analyzing means to detect a mobile, and a zoom camera of a camera unit 21 is operated on the basis of the detection to photograph the mobile. In that case, information of a spatial relationship with the panoramic image for the partial image-of-interest is created together with information of a temporal relationship indicating the transition of the partial image-of-interest on a time base in the panoramic image, and the partial image-of-interest is stored in an image storage section 13 after giving the relational information thereto, When retrieving, each piece of the relational information is used to retrieve the panoramic image and the partial image-of-interest and both the images are overlapped and displayed on a display section 14. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus, an image processing system, and a program that can store a wide-angle image and a zoom image together with related information and can search for an image stored using the related information as a clue. .

For example, when monitoring with an image of a surveillance camera in a security system or the like, an image is captured so that an omnidirectional camera and an image taken using a zoom camera, that is, an omnidirectional image and a zoom image can be confirmed within the same field of view. An image displayed in an overlapping manner is known (see Patent Document 1).
This conventional monitoring system, as shown in FIG. 6, displays an omnidirectional image, an omnidirectional camera 2 that captures an omnidirectional image having a maximum field of view of 360 ° via an optical system 2a. Display unit 3e, a predetermined range of an omnidirectional image is set as a monitoring region, a detection unit 35a that detects the movement of a subject existing in the monitoring region, and one monitoring region is detected by a detection result of detection unit 35a or manually. The designation unit 32a to be designated and the zoom camera control unit 4 for controlling the zoom camera 1 to perform zoom photographing based on the control information in which the position / magnification of the zoom camera 1 is set in advance for the designated monitoring area. ing.

As a result, omnidirectional images captured over a wide range with an omnidirectional camera are set as a plurality of monitoring areas, and one monitoring area is designated automatically or manually from the plurality of monitoring areas. It is possible to zoom with high accuracy.
By the way, the image information captured in this way is stored in a storage device so that it can be used later. However, since the amount of data is enormous, it is an urgent task to perform the search efficiently. .

  As a conventional search method, it is not described in patent literature, but (1) picking up event information in real time using a simple motion detect (motion detection) function, and adding the meta information to a recorded image (2) Search by the meta information, (2) After recording the image, analyze the image, estimate the part where the scene of the image changes greatly, pick up the event information, add the meta information to the recorded image, What performs a search based on the meta information is known.

However, in the method (1), as a result of picking up a noise-like event, noise increases during a search. In addition, the method (2) for analyzing an image after image recording has not reached a level that can be used on a daily basis at a monitoring site.
In addition, in the conventional method, the gaze target is captured only small, and therefore, sufficient information is not obtained, or the gaze target is buried in another target, which may be difficult to track. Therefore, it is conceivable to increase the resolution of the image of the object being watched in order to compensate for the above drawbacks, but in that case, the amount of data processing becomes large, so that the processing burden becomes too heavy and is not practical. is there.

JP 2003-259350 A

  The present invention has been made in view of the above-described conventional problems in image search means, and its purpose is to provide a wide-angle camera image full view image and a gaze area in the full view image without increasing the information processing burden. The gaze partial image, which is the captured zoom camera image, is saved with a correspondence relationship, and the desired image is surely obtained from the saved image based on the relationship information that defines the correspondence relationship between the panoramic image input at the time of saving and the gaze partial image. And to be able to search efficiently.

According to the first aspect of the present invention, a gaze area setting unit that sets a gaze area in the entire view image of the monitoring area, an image analysis unit that detects an object by analyzing the image of the set gaze area, and detects the object A camera control unit that controls the operation of a gaze partial image capturing camera that captures the target object, a gaze partial image acquisition unit that acquires the captured gaze partial image, and a gaze partial image acquired by the gaze partial image acquisition unit A spatial relationship information creation unit that creates spatial relationship information between the image and the full-view image, a temporal relationship information creation unit that creates time-related information indicating a transition on the time axis of the gaze partial image in the full-view image, and the full view An image processing apparatus characterized by having an image storage unit that records an image and adds the related information to a gaze partial image.
According to a second aspect of the present invention, in the image processing apparatus according to the first aspect of the present invention, the image processing apparatus further includes a condition setting unit that sets a determination condition for the target, and the gaze partial image acquisition unit detects the target when the target is detected. The image processing apparatus is characterized in that the gaze partial image and the related information are recorded in the image storage unit only when it is determined that the object matches the set determination condition.
According to a third aspect of the present invention, in the image processing apparatus according to the second aspect, the determination condition is an operation different from a route, a flow line, or a preset operation of the moving body. Device.
According to a fourth aspect of the present invention, in the image processing apparatus according to any one of the first to third aspects of the present invention, an image search is performed for searching the stored full-view image and gaze partial image based on the spatial and temporal relationship information. And an image display control unit that displays the retrieved images in a superimposed manner.
The invention according to claim 5 is a program for realizing the functions of the respective units of the image processing apparatus according to any one of claims 1 to 4 by a computer.
A sixth aspect of the present invention includes the image processing device according to any one of the first to fourth aspects, and a photographing device including a wide-angle image photographing camera and a gaze partial image photographing camera for panoramic photographing. Is an image processing system.

According to the present invention, it is possible to reduce an information processing burden for monitoring by setting a gaze area in a full-view image. In addition, by setting a determination condition for the gaze partial image, it is possible to record and save the entire view image and the gaze partial image while reducing noise.
At the time of search, it is possible to efficiently and reliably search for a desired image from the stored images using the correspondence information of both images used when saving the full-view image and the gaze partial area image as a search key (search clue). it can.

1 is a block diagram showing a configuration of an image processing system according to an embodiment of the present invention. It is a block diagram which shows the structure of the control part of the image processing apparatus which concerns on embodiment of this invention. It is a figure for demonstrating the relationship between a full view image and a gaze partial image. It is a flowchart explaining the storage procedure of image data. It is a flowchart for demonstrating the search procedure of the accumulate | stored image data. It is a block diagram which shows the conventional monitoring system.

FIG. 1 is a block diagram showing a configuration of an image processing system according to an embodiment of the present invention.
The image processing system according to the present embodiment includes an image processing apparatus 10 and a photographing apparatus 20 that are connected to be communicable with each other.
The image processing apparatus 10 includes a control unit 11 that controls the entire image processing apparatus 10, a communication unit 12 that performs wired or wireless communication with the photographing apparatus 20, a DVD-ROM (Digital Versatile Disk Read Only Memory), for example, BD (Blue-ray Disc) -Display comprising an image storage unit 13 composed of an arbitrary storage medium for recording images such as a ROM, and a display device such as liquid crystal (LC) or EL (electroluminescence) capable of displaying images And an input unit 15 for operations such as setting.
The photographing device 20 is a photographing composed of a wide-angle image camera and a high-resolution camera or a PTZ (pan / tilt / zoom) camera (zoom image camera) having a resolution that does not lack the gaze partial information in the full-scale image (wide-angle image). And a communication unit 22 for transmitting image data and commands between the image processing apparatus 10 and the image processing apparatus 10.

FIG. 2 is a block diagram showing the control unit 11 of the image processing apparatus 10 according to the embodiment of the present invention.
The image processing apparatus 10 does not acquire a full-view image (wide-angle image) and a gaze partial image (zoom image) with the photographing unit 21 of the photographing apparatus 20 and record them as they are. ) And a combination of high resolution but limited gaze partial images, and spatial relationship information between the two images, as well as change information on the time axis of the gaze partial images in the full-view images, that is, Temporal relationship information is added and recorded in the image storage unit 13.
The recorded image information is searched using the related information as a search key.

  The control unit 11 of the image processing apparatus 10 controls the processing described above, and includes a camera control unit 111 that controls the camera of the photographing unit 21, a gaze area setting unit 112 that sets a gaze area in the full-scale image, and a gaze. In order to detect an object (moving body) to be watched in the area, for example, an image analysis unit 113 that performs processing such as creating a difference image by pattern matching for each frame before and after the whole-view image, and the detected moving body is watching A condition setting unit 114 for setting a condition for determining whether or not the image is a partial image, a gaze partial image acquisition unit 115 for acquiring the image as a gaze partial image when the set condition is met, and a panoramic image (wide-angle image) ) And the spatial relationship information creating unit 116 that records the spatial relationship of the frames of the gaze partial image within the frame, and each event of the gaze partial image on the time axis, that is, movement A time relationship information creation unit 117 that records time point information by recording an image generation point, a change time point that changes more than a predetermined level (referred to as a large change point), and a disappearance time point, and an image search unit that searches the recorded image information 118 and an image display control unit 119 for controlling the display unit 14.

  The control unit 11 includes a RAM 120 serving as a temporary storage unit for the operation of the control unit 11 and a ROM 121 that stores a program for performing image processing in the image processing apparatus 10.

The gaze area is set by, for example, an operator who monitors the image on the display unit 14 at the time of image acquisition specifying the gaze area on the monitor screen with a pointing device or the like.
Here, when the gaze partial image acquisition unit 115 that acquires the gaze partial image detects a moving object such as an intruder in the gaze area in the image set as described above, the camera control unit In 111, the gaze partial image acquisition camera (zoom camera) of the photographing unit 21 of the photographing apparatus 20 is operated.

Further, when the gaze partial image acquisition unit 115 detects a moving body in the gaze partial image, the gaze partial image acquisition unit 115 sets this as a gaze partial image candidate, which is also set by the condition setting unit 114, and the moving body path, flow line, and gaze area The image analysis unit 113 automatically discriminates the movement of the gaze and determines whether or not the gaze partial image candidate is a gaze partial image. When it is determined as a gaze partial image, it is a recording target from the acquisition start (generation of the gaze partial image) to a specific acquisition stop condition set by the condition setting unit 114 (for example, exit from the gaze area).
On the other hand, when it is determined that the image is not the gaze partial image, the content acquired as the gaze partial image candidate is removed from the recording target (or the priority is lowered if priority is given).

The temporal relationship information creation unit 117 that creates temporal relationship information between the gaze partial image and the full view image, for example, assumes that each frame of the gaze partial image corresponds to which frame of the full view image when recording at 15 frames / second. To create information (corresponding relationship information on the time axis). A spatial relationship information creating unit 116 that creates spatial relationship information between the gaze partial image and the full view image creates spatial relationship information indicating which region in the full view image frame the frame of the gaze partial image corresponds to. The data set is recorded (saved) in the image storage unit 13. Alternatively, it may be recorded as a characteristic expression or a data table for interpolating between sparse data sets.
In this embodiment, this correspondence information is used later as a search clue (search key) at the time of search.

  Here, a dense data set means that correspondence information is prepared and incorporated as data for each frame of 15 frames / second, for example, and a sparse data set is 1 second for an image of 15 frames / second, for example. The data set is stored in only one frame, and the parameters which are data constituting the interpolation table or the parameters for calculation by the interpolation characteristic formula are incorporated in the remaining 14 frames.

The time-related information creating unit 117 records each time point when the gaze partial image is generated, a large change time, the disappearance time of the gaze partial image, and the recording gaze time point, and how much deviation from a certain time point on the reference time axis. Record in a data set format that expresses whether
For example, if the reference time (reference time) is 0:00:00 and the gaze partial image candidate occurrence time is 0: 5: 20: 50, the event “gaze partial image candidate occurrence time” In the accompanying data set, “deviation from reference time: 5 minutes 20 seconds 50” is included.

FIG. 3 is a diagram for explaining the setting of the gaze area and the determination condition.
The figure shows a gaze partial image (moving body) 32 in the whole view screen 30. Here, for example, a case will be described in which a gaze partial image is acquired by tracking a moving object when a moving object is detected in a gaze region R such as a passage set in advance in an indoor panoramic image.
That is, when the moving body 32 is detected at the time t1, the gaze partial image acquisition unit 115 starts acquiring images using the mobile body 32 as the gaze partial image candidate, and the spatial relationship information creation unit 116 detects the gaze partial image. The spatial relationship correspondence information between the image and the full-view image (wide-angle image) is created and recording thereof is started, and the time relationship information creation unit 117 generates the gaze partial image candidate on the time axis, the large change point, Recording to the image storage unit 13 starts at the time of disappearance.

  In FIG. 3, while the moving body 32 is moving in the gaze region R, the gaze partial image acquisition unit 115 continues to acquire an image using the mobile body 32 as the gaze partial image candidate, and the spatial relationship information creation unit 116. And the time relation information creating unit 117 continues to create and record the correspondence information on the time axis and the spatial position correspondence information, and when the moving body 32 disappears from the full-view image 30 at time t3, the gaze is received. The partial image acquisition unit 115 determines that the moving body 32 is a gaze partial image and records (saves) it in the image storage unit 13.

On the other hand, when the moving body 32 that is the gaze partial image candidate is displayed and disappears in the middle of the outward viewing region R, the gaze partial image acquisition unit 115 determines that the moving body 32 is the gaze portion. When judging that it is not an image and performing the process of excluding all the gaze partial image data and related information that have been continuously acquired and recorded from the recording target, or prioritizing and storing the data, Save at a lower priority.
On the other hand, when the moving body 32 passes through the gaze area R as it is and disappears from the full-view image, it is determined that the gaze partial image candidate is not a gaze partial image, and deviates in the middle of the gaze area R. When it disappears at time t2, it may be determined as a gaze partial image and registered in the image storage unit 13 together with related information.

Next, the operation of the image processing apparatus having the above configuration will be described.
FIG. 4 is a flowchart showing a processing procedure of image data of the image processing apparatus according to the present embodiment.
In carrying out this processing procedure, first, a gaze area and determination conditions (such as a moving path, a flow line, and an operation of a moving body) are set in advance.
In this state, video acquisition processing is started (S101), and recording by a wide-angle image camera is first started (S102). Here, if the moving object 32 is in the set gaze area (S103, Yes), recording of the zoom image camera is started under the control of the camera control unit 111 (S104), and the moving object 32 is tracked by the zoom image camera ( S105). During the tracking of the moving body 32, the spatial relationship information creation unit 116 and the temporal relationship information creation unit 117 create spatial relationship information between the wide-angle (full view) image and the zoom image, and change point information on the time axis. (S106). After that, when an event that should divide the recording according to the determination condition set by the gaze area setting unit 112 occurs, for example, the moving body 32 goes out of the gaze area or the operation image of the mobile body 32 changes significantly. (S107, Yes), it is determined whether or not the event or the moving body 32 matches the determination condition (preliminary setting condition) set in advance by the condition setting unit 114 (S108). Yes), search cues (association information between the gaze partial image and the panoramic image (wide-angle image), the occurrence time of the gaze partial image candidate on the time axis, the large change time, the disappearance time occurrence information) and the gaze partial image ( The zoom image is stored in the image storage unit 13, and the process returns to step S103 to start monitoring the next moving object.

In step S108, when the route, flow line, movement in the gaze area, etc. do not match the preset condition (S108, No), the recording of the zoom image camera is stopped (S109), and the zoom image camera image is deleted or deleted. The priority is downgraded (S110).
As described above, the moving object in the preset gaze area is monitored, and the gaze partial image that matches the judgment condition set in advance is stored in the image storage unit 13 together with the search clue.

Next, the search procedure for the image stored in the image storage unit 13 as described above will be described with reference to the flowchart of FIG.
In the search, first, search conditions such as the order, time point, and location to be read in the search are set (S201). When the search is started (S202), first, a search clue (search key) in the specified order in the recorded image. Or the latest search clue time of the past example at the designated time point, or the search clue time related to the vicinity of the designated location (S203). Therefore, the reproduction of the wide-angle image camera recording image is started (S204), the zoom image camera recording image is reproduced using the search clue information, and the spatial relationship between the wide-angle image camera recording image and the image camera recording image is displayed on the display unit 14. It is displayed (S205). Here, when the user performs an operation such as search, rewind, and stop from the input unit 15 (S206, Yes), and when the operation is not a stop operation (S207, No), an operation designated by the user Accordingly, the time-related information is used as a search clue (search key) to move the time (S208), and the process returns to step (S203). If the operation in step S207 is a stop operation (S207, Yes), playback of the recorded image of the zoom image camera is stopped (S209), playback of the wide-angle image camera recorded image is stopped (S210), and the search process is terminated. To do.

  According to the present embodiment, based on the stored zoom camera recording image and wide-angle image camera recording image, and search clue information, a search that gives an image of a moving body at an arbitrary time and an arbitrary place at the time of each image storage It is possible to search efficiently based on clue information without waste.

  DESCRIPTION OF SYMBOLS 10 ... Image processing apparatus, 12 ... Communication part, 20 ... Imaging device, 21 ... Imaging part, 22 ... Communication part, 111 ... Camera control part, 112 ... Gaze area Setting unit 113 ... Image analysis unit 114 ... Condition setting unit 115 ... Gaze partial image acquisition unit 117 ... Time relation information creation unit 118 ... Image search unit 119 Image display control unit, 120... RAM, 121.

Claims (6)

A gaze area setting unit for setting a gaze area in the whole view image of the monitoring area;
An image analysis unit that detects an object by analyzing an image of a set gaze area;
A camera control unit that controls the operation of a gaze partial image capturing camera that captures the object when the object is detected;
A gaze partial image obtaining unit for obtaining the photographed gaze partial image;
A spatial relationship information creating unit that creates spatial relationship information between the gaze partial image acquired by the gaze partial image acquisition unit and the panoramic image;
A time-related information creating unit that creates time-related information indicating a transition on the time axis of the gaze partial image in the whole view image;
An image processing apparatus, comprising: an image storage unit that records the full-view image and adds the related information to a gaze partial image. The image processing apparatus according to claim 1,
It has a condition setting part that sets the judgment condition of the object,
The gaze partial image acquisition unit records the gaze partial image and the related information in the image storage unit only when it is determined that the target matches the set determination condition when the target is detected. An image processing apparatus. The image processing apparatus according to claim 2,
The image processing apparatus according to claim 1, wherein the determination condition is a path different from the path of the moving object, a flow line, or a preset action. The image processing device according to any one of claims 1 to 3,
An image having an image search unit for searching for the accumulated whole-view image and gaze partial image based on the spatial and temporal relationship information, and an image display control unit for displaying the searched images in an overlapping manner. Processing equipment.   The program for implement | achieving the function of each part of the image processing apparatus as described in any one of Claim 1 thru | or 4 with a computer. An image processing device according to any one of claims 1 to 4,
An image processing system comprising: a wide-angle image capturing camera for panoramic image capturing, and an image capturing apparatus including a gaze partial image capturing camera.

Images