JP2014222825A - Video processing apparatus and video processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform motion image photography in which a moving body is extracted from within wide-angle video and an object to be monitored is automatically selected.SOLUTION: A video processing apparatus includes: image pickup means of picking up wide-angle video; moving body extraction means of extracting a moving body from the wide-angle video picked up by the image pickup means; priority imparting means of imparting priority to the moving body extracted by the moving body extraction means; video segmentation means of segmenting the wide-angle video into video of a partial area; and video conversion means of converting the video into first video data and second video data. The video segmentation means achieves segmentation such that an area including a moving body with a top priority level among priority levels set by the priority imparting means has an aspect ratio for moving image display, and the video conversion means converts the area segmented by the video segmentation means and including the moving body with the top priority into the first video data and the remaining area into the second video data.

Description

  The present invention relates to a video processing apparatus and a video processing method, and in particular, automatically selects a size suitable for a playback video for a region of interest including moving objects in a region captured by a wide-area surveillance camera using a wide-angle lens or an omnidirectional lens. The present invention relates to a technique suitable for use in selection.

  Conventionally, when shooting with a wide-angle lens or a fish-eye lens, there may be multiple objects in the wide-angle video, and it is considered to install a function to cut out a part of the video to narrow down a specific target. . A method that can display a cut-out video by compressing and distributing a video cut out from a wide-angle video and applying a conversion in accordance with the display size on the receiving side has been studied.

  For example, in Patent Document 1, when an image is cut out from a wide-angle video, the cut-out video and the cut-out position in the wide-angle video are associated with each other, and the cut-out image is temporarily held and used as a reference frame. It discloses that the compression ratio of an image is increased.

  For example, in Patent Document 2, an area with a large movement is cut out as a main area. Also disclosed is a method for efficiently transmitting a moving part in a wide-angle video by lowering the image quality of a small internal movement and superimposing a large movement from the auxiliary area on the reduced quality part. ing.

JP 2009-218851 A JP 2004-363974 A

  However, in the conventional technique disclosed in Patent Document 1 described above, when a target object is cut out from a wide-angle video, a moving object portion is cut out by a monitoring camera or the like together with a method for specifying a range to be cut out while a user looks at the screen. Is presented. Therefore, the video other than the cut-out video is displayed as a part of the wide-angle video and needs to be viewed on the receiving side based on the entire wide-angle image.

  In the prior art disclosed in Patent Document 2, the important area and the non-important area are determined in the moving body portion in the wide-angle video, but the size of the main area to be cut out is not mentioned, and the image quality reduction area And the sub screen area do not match, the sub screen area is deformed to correspond. If the video size considering the display image is not set for the main area and the transmission is performed considering the compression rate and the data size, the load for converting the video data will increase, and the distribution video size will be suppressed. However, there is a problem that the processing on the video distribution side is limited.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to extract moving objects from a wide-angle video and automatically select a monitoring target to perform moving image shooting.

  The video processing apparatus according to the present invention includes an imaging unit that captures a wide-angle video, a moving object extraction unit that extracts a moving object from the wide-angle image captured by the imaging unit, and a priority that gives priority to the moving object extracted by the moving object extraction unit. And a video conversion means for converting a video into a first video data and a second video data. The means cuts out an area including the moving body having the highest priority among the priorities set by the priority giving means into an aspect ratio for displaying a moving image, and the video converting means is the highest cut out by the video cutting means. A region including a moving object having a high priority is converted into first video data, and the remaining region is converted into second video data.

  According to the present invention, a part with many moving objects can be preferentially extracted from a wide-angle image, and a moving image can be shot with a video size corresponding to the display device, and a part with low priority can be distributed separately.

It is a figure which shows embodiment of this invention and demonstrates the extraction process of the area | region containing a moving body. It is a block diagram which shows embodiment and shows the structural example of an omnidirectional monitoring camera. It is a figure which shows embodiment and shows an example of the image | video of an omnidirectional monitoring camera. It is a flowchart which shows embodiment and demonstrates the process sequence which implement | achieves the moving body extraction method. It is a flowchart which shows embodiment and demonstrates the procedure of the highest priority determination. It is a flowchart which shows embodiment and demonstrates the procedure which determines area | region size. It is a flowchart which shows embodiment and demonstrates the procedure of a video conversion process.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
First, the configuration of an omnidirectional monitoring camera that is a video processing apparatus according to an embodiment of the present invention will be described with reference to FIG.

  The video processing apparatus of the present embodiment can be applied to a wide range of video processing apparatuses such as a video camera, a digital camera, a surveillance camera, and a fixed point camera. Further, the video processing apparatus according to the present embodiment may be an apparatus that inputs and processes video captured by an external device in a system such as a surveillance camera.

FIG. 2 shows an example of a configuration diagram of the omnidirectional monitoring camera.
In the omnidirectional monitoring camera, an image input through the lens 201 is captured by the image capturing unit 203 and processed as digital image data.

  The lens 201 is controlled by the lens control unit 202 so that the subject can be photographed with an optimum value by focus, zoom control, iris adjustment, or the like. The digital video data captured by the imaging unit 203 is output to the video correction unit 204, and the video correction unit 204 removes distortion of the video and corrects the video for easy monitoring.

  The video corrected by the video correction unit 204 is output to the moving object extraction unit 205, and the moving object extraction unit 205 extracts the moving object by taking a difference from the background portion in the video. The extracted position, size, period during which the moving object is detected, direction, and moving speed are held. The video cutout unit 206 cuts out a video at an arbitrary position based on the video corrected by the video correction unit 204 and the moving body information extracted by the moving body extraction unit 205.

  The video cut out by the video cutout unit 206 is output to the video conversion processing unit 207, subjected to an encoding process optimal for communication, and converted into video data. The video data encoded and converted by the video conversion processing unit 207 is transmitted to an external network by the communication unit 208. The communication unit 208 controls communication according to a control request or a distribution request from a client.

  Each block is connected by an internal bus 209, and operates by receiving a control signal from the CPU 210. The video data is temporarily saved in the RAM 211 and used to wait for a processing sequence, and setting information, a program, and the like from the user are loaded. In the ROM 212, a program is stored, and after being activated, the program is loaded into the RAM 211, and operates by reading from the CPU 210. Further, setting information from the user may be temporarily stored in the RAM 211, or may be stored in the ROM 212 and become effective after the apparatus is restarted.

  The moving object extraction unit 205 can change the conditions of moving objects extracted by the CPU 210. The extracted moving body information is temporarily stored in the RAM 211, calculated from the extracted information by the CPU 210, and transmitted to the video cutout unit 206 as the cutout position and size when cutting out a partial area video from the wide-angle video. Note that the CPU 210 controls transmission of moving body information and setting values in the RAM 211 and ROM 212 to the client on the network via the communication unit 208.

Here, FIG. 3 shows an example of an image captured by the omnidirectional monitoring camera.
FIG. 3 is a diagram schematically showing an omnidirectional image (FIG. 3A) representing an optical system photographed using a fisheye lens and a developed image (FIG. 3B).

  An omnidirectional image 301 shown in FIG. 3A is a state taken directly from the ceiling, and subject information for each azimuth is obtained from the image center 303 in an annular region as a whole. Holds radially in the radial direction. The azimuth angles Θ1 and Θ2 are represented as a line L1 and a line L2 from a position where the center line is 304, respectively.

  On the other hand, the developed image shown in FIG. 3B has a rectangular shape as a whole, and the vertical direction corresponding to the vertical direction of the real space corresponds to the radial direction in the omnidirectional image and the horizontal direction corresponding to the horizontal direction of the real space. The direction corresponds to the circumferential direction in the omnidirectional image.

  When the center 303 of the circular image in FIG. 3A, a line 304 perpendicular to the center and a line 305 extending vertically from the center are expanded on the basis of the perpendicular 305, the perpendicular 304 is obtained as shown in FIG. 3B. It becomes a rectangle with the center.

Here, the object information for each azimuth angle is formed in parallel along the horizontal direction, the line L1 corresponding to the azimuth angle Θ1 is represented as L1 ′, and the line L2 corresponding to the azimuth angle Θ2 is L2 ′. Expanded as.
When the areas shown in FIG. 3A are expanded as 306a, 307a, and 308a through the video correction unit 204, the horizontal direction is about the vertical line 304 as shown in 306b, 307b, and 308b in FIG. 3B. Is expanded and corrected.

  In this embodiment, the example which uses the omnidirectional image which used the fisheye lens as an omnidirectional camera was shown. In addition to this, not only fish-eye lenses, but also wide-angle images can be taken, such as optical 360-degree lenses that use mirrors and lenses that can shoot ultra-wide-angle images. However, the object of the present embodiment can be satisfied.

Here, a processing procedure for realizing the moving object extraction method in the moving object extraction unit 205 will be described using the flowchart of FIG.
The moving object extraction unit 205 learns the background image of the input image, and detects the moving object by comparing the difference between the input image data and the background image.

  When the process is started, video data is input in S401, and the background difference between the background image learned in advance and the input video data is compared (S402). The video data in S401 can learn the background by inputting the required number of video data at the time of initial startup. Here, a sufficient amount of time has passed since the startup, and the background video already learned is It shall exist.

In S402, if there is a background difference, it is checked whether a change in the minimum size that can be recognized as a moving object has occurred (S403).
The background difference in S402 and the error not exceeding the minimum size in S403 are ignored, and the process returns to the video data input in S401. When the minimum moving object size is exceeded in S403, the information on the position and size of the moving object is updated (S404).

Further, in S405, it is determined whether or not there is a correlation with past moving object information. If it is determined that they are the same object, the process proceeds to S406, and the appearance time information of the moving object is updated.
If it is determined in S405 that the correlation is not obtained and the objects are not the same, the process proceeds to S407 and is registered as the appearance of a new object.

The object whose time information has been updated in S406 determines whether or not the target object has moved by comparing the time information with the position and size (S408).
If it is a moving object, the speed information is updated from the correlation between the position and the size (S409), and if it is not moving, the speed information is reset to 0 as a stationary object (S410).

  The moving object information is updated for each of the speed information and the new object (S411), and it is confirmed whether all moving objects have been extracted (S412). If all the moving objects have not been updated, the process returns to S405 to update the information about each moving object, and when the information update for all moving objects is completed, the moving object extraction process is completed.

  If there is no moving object, video data input is repeated. In this case, since video data is directly output from the video correction unit 204 to the video cutout unit 206, video conversion processing and communication with an external client are performed. It will not interfere.

  In the present embodiment, the description has been made using the moving object detection method using the background difference, but the object can be achieved as long as the moving object can be detected. For example, a method may be used in which moving objects are extracted by processing input video data such as inter-frame differences in units of frames of a certain period.

Processing for determining the highest priority among the moving objects extracted by the moving object extraction unit 205 will be described with reference to the flowchart of FIG.
Among the moving objects that have finished the moving object extraction process, the moving objects within the specified distance are merged so as to be handled as one object (S501).
After moving object merge in S501, all moving object information is sorted in the order of size, existing time, and moving speed (S502).
It is confirmed whether or not the maximum size is the same among all moving objects (S503), and if the maximum size is different, priority giving processing is performed in which the maximum size is the highest priority of the moving object (S509).

  Next, if the maximum sizes match, the existence time after the moving object appears is compared (S504). If there is a match, the moving speeds of the moving objects are compared (S505). If they do not match, the moving object has the longest moving time and the moving object having the fastest moving speed is the highest priority. Priority giving processing is performed (S509).

  If there are a plurality of moving objects having the same size, existing time, and moving speed, it is determined whether or not a cutout area already exists (S506), and if it exists, it is closest to the center position of the current area. A moving object is selected (S507).

  When there is no cutout area and there are a plurality of candidate moving objects, the moving object closest to the predetermined default center position is selected (S508). The default area is the center position of the image developed in FIG. By this processing, priority assignment processing is performed for setting the highest priority among moving objects that has the largest size, the longest existence time, and the high moving speed (S509).

  Since the moving object extraction unit 205 can calculate the moving object having the highest priority and the position in the video, the position information and the size information of the moving object can be stored in the RAM 211. The CPU 210 can instruct the lens control unit 202 to set the shutter speed and the brightness according to the moving object having the highest priority based on the position information of the moving object. Accordingly, it is possible to perform lens control in accordance with the moving body and continue shooting with shooting parameters suitable for the moving body.

Next, an extraction process of a region including a moving object having the highest priority in the video cutout unit 206 is shown in FIGS.
FIG. 1A is an example of the developed image shown in FIG.
There are four moving objects M101, M102, M103, and M104 in order of size, extracted by the moving object extraction unit 205.

  In accordance with the method for determining the highest priority described above, priority is determined by the distance and size between moving objects for each moving object as shown in FIG. 1B, and expressed by a circumscribed rectangle surrounding each moving object. The areas are R101, R102, and R103. Here, when a plurality of moving bodies are densely packed, the region R101 including the moving body M102 and the moving body M103 is targeted.

  The region R101 is represented by a circumscribed rectangle having a height Hr and a width Wr. When the height of the target region R101 is less than the height of the developed image, the region R104 is set so as to extend to the maximum height H1 of the developed image as shown in FIG.

  Here, in consideration of the distribution size to the client displaying the video of the surveillance camera, the region R104 is selected so as to select a region suitable for the video image size of 3: 4 or 9:16 so that the aspect ratio becomes constant. The video size in the video conversion process is determined as a reference.

  Using the height H1 of the region R104 as a reference, when setting the width, the width is determined so that the height vs. width is 3: 4. As shown in FIG. 1D, a region R105 having a width W1 of 3: 4 with respect to the height H1 is determined to be a cut-out position for moving body imaging. The region R105 is a rectangular region having a height H1 and a width W1, and the position and size to be cut out by the video cutout unit 206 are determined using the center position, width, and height of the region R105.

  In moving object extraction, when the size of a moving object is large, when a plurality of moving objects are merged and extracted as a moving object region, a region having a width that does not fit in the aspect ratio 3: 4 for moving image display is assumed. The

An example when the moving object size is large is shown in FIG. 1E, and a flowchart for determining the region size is shown in FIG.
As shown in FIG. 1E, the region R106 having the moving body M105 and the moving body M106 has a height Hr and a width Wr. The height of the developed image is H1, the width when the aspect ratio is 3: 4 with respect to H1, the width is W1, the width when the aspect ratio is 9:16 is W2, and the region where the height is H1 and the width W2 is region R107.

First, processing for setting the priority of moving object extraction is performed, and the region R106 having the moving object M105 and the moving object M106 is selected as the highest priority (S601).
When the width Wr of the target region R106 is larger than the width W1 of the 3: 4 size based on the height H1 of the developed image (S602), when W1 is wide among Wr and W1, the width Wr is 3: 4. The area is selected as a cut-out area (S603).

If it is determined in step S602 that the width Wr of the region R106 is wider, the region R107 having a width W2 having an aspect ratio of 9:16 is selected as a cutout region (S604).
Next, the width Wr of the region R106 is compared with the width W2 of 9:16 size (S605). If the width W2 is wide, the center position of the region R106 that is a circumscribed rectangle is set as the center of extraction (S607). ). Similarly, the center position of the circumscribed rectangle is set as the center of the cutout when cutting into the 3: 4 area (S608).

  When the width Wr of the region R106 including the moving object is wider than W2, the size that can be expressed as the display video size is exceeded. Therefore, the center position for the moving body having the largest size among the moving bodies included is used as the center of the cutout position while maintaining the region size (S606).

  As described above, the size of the cutout position is set according to the size of the aspect ratio 3: 4 or 9:16 for moving image display, and the center position of the cutout is the center position of the area including the moving object or the center of the moving object of the maximum size. Use to set.

A video conversion process in the video conversion processing unit 207 based on the cutout information determined by the video cutout unit 206 will be described with reference to the flowchart of FIG.
Detailed information on the area, position, and size is acquired as the cutout information (S701).
Next, the time when the cut-out area information is acquired is measured (S702), and the cut-out area is converted into an image at a generation timing specified in advance (S703).

Time measurement is performed in order to measure input time intervals between video data by obtaining time information such as a frame rate, a cycle, and timing, and to perform video conversion at regular intervals.
In the video conversion in S703, continuous generation of still images (second video data) in JPEG format or the like, Using a coding method such as H.264 or MPEG4, conversion that can be distributed as a moving image (first video data) to the client is performed.

When the video conversion of the cutout region is completed, it is determined whether or not the time interval such as the frame rate, cycle, timing, etc., set in advance as a parameter by the client is exceeded (S704).
If the specified time has elapsed, all the images except the cut-out area are converted (S705). In the video conversion in S705, video conversion for handling as a still image such as JPEG format is performed, and the video is stored in the RAM 211 so that the video can be acquired at a constant cycle. Information on the moving image obtained by converting the cutout area and the still image obtained by converting the video other than the cutout area is updated, and the video conversion process ends (S706).

The updated video information is stored in the RAM 211, and when there is a client request via the communication unit 208, the latest video information can be transmitted to distribute the current video information.
In addition, when a video acquisition request is received from a client, it is distributed at regular intervals via the communication unit 208. The client can acquire a moving image at the video cutout position and a still image excluding the cutout region from the whole by specifying a time interval such as a frame rate and period information.

  In the description of the present embodiment, JPEG and H.264 video conversion processing for moving images and still images is performed. The description has been given using an encoding method such as H.264 or MPEG4. However, since the encoding method and the compression method included in the category of moving image and still image can satisfy the purpose of the present embodiment, the encoding method is not limited.

  As described above, in the present embodiment, a moving object with a motion is extracted from a video using an omnidirectional image, and a region including a moving object with the highest priority among the extracted moving objects can be always captured with a moving image. The cutout video size is determined. The video size to be cut out is set in accordance with the aspect ratio of the moving image display size suitable for moving image display.

As a result, only a moving image can be cut out from an omnidirectional image, an image other than the cut-out image can be saved as a still image, and a target portion can be easily extracted and the state of the entire captured image can be grasped. Can do.
As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  Each means constituting the video processing apparatus and each step of the control method in the embodiment of the present invention described above can be realized by operating a program stored in a RAM, a ROM, or the like of a computer. This program and a computer-readable storage medium storing the program are included in the present invention.

Further, the present invention can be implemented as, for example, a system, apparatus, method, program, or recording medium, and may be applied to an apparatus composed of a single device.
Note that the present invention supplies a software program that implements the functions of the above-described embodiments directly or remotely to the apparatus. This includes the case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

M101, M102, M103, M104, M105, M106 Moving objects R101, R102, R103, R104, R105, R106, R107 , 208 communication unit, 301 omnidirectional image, 302 toric imaging region, 303 center of omnidirectional image, 304 center line of developed image, 305 dividing line of developed image, 306, 307, 308 moving object region

Claims (16)

An imaging means for imaging a wide-angle image;
Moving object extraction means for extracting a moving object from a wide-angle video imaged by the imaging means;
Priority giving means for giving priority to the moving object extracted by the moving object extracting means;
Video cutout means for cutting out a video of a part of the area from the wide-angle video;
Video conversion means for converting video into first video data and second video data;
Have
The video cutout unit cuts out an area including a moving object having the highest priority among the priorities set by the priority giving unit into an aspect ratio for displaying a moving image,
The video conversion means converts the area including the moving body having the highest priority cut out by the video cut-out means into first video data, and converts the remaining area into second video data. Video processing device.   The video processing apparatus according to claim 1, wherein the first video data and the second video data are different in at least a timing, a cycle, and a frame rate that are generated.   The video processing apparatus according to claim 2, wherein the first video data is generated earlier than the first video data and the second video data.   The video processing apparatus according to claim 1, wherein when the aspect ratio for moving image display is cut out, the video processing apparatus includes at least an image size having an aspect ratio of 3: 4 or 9:16.   The video processing according to claim 1, wherein, when the aspect ratio for moving image display is cut out, the height of the area converted by the video conversion means is set to a video size that matches the vertical length of the aspect ratio. apparatus.   6. The video processing apparatus according to claim 1, wherein the priority assigning unit sets a plurality of moving objects as a high priority when the moving objects are dense.   The video processing apparatus according to claim 1, wherein the priority assigning unit sets a moving object having a large maximum size as a high priority.   The video processing apparatus according to claim 1, wherein the priority assigning unit assigns a moving object having a long existence time to a high priority when the maximum sizes coincide with each other. .   6. The priority giving means according to claim 1, wherein when the maximum size and the existence time coincide with each other, the moving object having a high moving speed is given a high priority. Video processing device.   The priority assigning means determines a high priority according to whether or not a cutout area exists when the maximum size, the existing time, and the moving speed match. The video processing device according to any one of?   11. The video processing apparatus according to claim 10, wherein, when a cutout area exists, the priority assigning unit selects a moving object closest to the current area and sets the priority to a high priority.   The video processing apparatus according to claim 10, wherein when the cutout area does not exist, the priority assigning unit selects a moving object closest to a default position and sets the priority to a high priority. The imaging means includes, as a lens for capturing the wide-angle image, a lens capable of photographing all directions, a lens capable of photographing 360 degrees, and a lens capable of photographing all directions, or a lens capable of photographing 360 degrees. Image correction means for correcting distortion from the captured circular image,
The video processing according to claim 1, wherein the moving object extraction unit extracts a moving object by taking a difference from a background portion in the video corrected by the video correction unit. apparatus. An imaging process for imaging a wide-angle image;
A moving object extracting step of extracting a moving object from the wide-angle video imaged in the imaging step;
A priority providing step of setting priority to the moving object extracted in the moving object extraction step;
A video cutout step of cutting out an image of a partial area from the wide-angle video;
A video conversion step of converting the video into first video data and second video data;
Have
The video cutout step cuts out an area including the moving body having the highest priority among the priorities set in the priority assignment step into an aspect ratio for displaying a video,
In the video conversion step, the region including the moving body having the highest priority cut out in the video cutout step is converted into first video data, and the remaining region is converted into second video data. Video processing method. An imaging procedure for imaging a wide-angle image;
A moving object extraction procedure for extracting a moving object from the wide-angle video imaged in the imaging procedure;
A priority assignment procedure for providing priority to the moving object extracted in the moving object extraction procedure;
A video cutout procedure for cutting out a part of the video from the wide-angle video;
A video conversion procedure for converting the video into first video data and second video data;
A program for causing a computer to execute
The video cutout procedure cuts out an area including a moving object having the highest priority among the priorities set in the priority assignment procedure into an aspect ratio for displaying a moving image,
The video conversion procedure controls the computer to convert the area including the moving body having the highest priority extracted in the video cutout procedure into first video data and convert the remaining area into second video data. A program characterized by that.   A computer-readable storage medium storing the program according to claim 15.

Images