WO2013080245A1

WO2013080245A1 - Video processing device and video processing method

Info

Publication number: WO2013080245A1
Application number: PCT/JP2011/006619
Authority: WO
Inventors: 諭鈴木
Original assignee: パナソニック株式会社
Priority date: 2011-11-29
Filing date: 2011-11-29
Publication date: 2013-06-06

Abstract

A video processing device is provided with: a video acquisition unit which acquires video from a camera and outputs video signals; a video analysis unit which calculates, from the video signal, an area in which the difference between successive frames exceeds a predetermined threshold value, as a crop region; a video adjustment unit which calculates and outputs zoomed video signals, on the basis of the video signals, in accordance with the crop regions and a zoom percentage; and a video output unit which outputs output video signals on the basis of the video signals and the zoomed video signals. The video analysis unit calculates an inclusive crop region which encompasses all of the plurality of crop regions for a plurality of predetermined successive frames.

Description

Video processing apparatus and video processing method

The present invention relates to a video processing apparatus and a video processing method for detecting and removing a background portion from a video acquired from a camera connected to a television or a PC and automatically displaying a subject in an appropriate size at the center of the screen. . In particular, the video analysis unit that detects the background from the acquired video, the video adjustment unit that adjusts the angle of view, such as cutting out the acquired video from the analysis result, and the camera position and zoom (zoom) are controlled automatically. The angle is adjusted.

In a video device such as a camera-integrated VCR (Video Cassette Recorder) or a stationary VCR, by specifying a moving subject within the screen, the subject is always projected at a certain position on the screen, for example, the center position. An image processing apparatus for obtaining a video signal is disclosed (see, for example, Patent Document 1). This conventional image processing device detects a moving vector of a subject based on a CCD (Charge Coupled Device) image signal in order to detect a moving subject, and is set so that the subject is always located at the center of the cutout frame. To do. As shown in FIG. 16, the conventional image processing apparatus always projects the subject at a fixed position on the screen, for example, the center position.

As shown in FIG. 16, the conventional image processing apparatus detects the motion vector of the subject as the subject changes at times t1, t2, and t3. The conventional image processing apparatus determines a certain area including the subject as a cutout frame from the screen configured by the video signal at time t1 on the screen 1202. In FIG. 16, the conventional image processing apparatus moves the cutout frame according to the motion vector, and shows that the subject is always displayed at the center position of the screen of the image processing apparatus.

By the way, when the amount of movement of the subject is relatively small, the change in the cutout frame may be small due to image processing. In this case, as the change in the cutout frame is small, the background variation is small. In such a situation, a minute fluctuation of the background occurs in a short time. For a user, it may be accompanied by considerable discomfort when gazing at a screen in which minute background fluctuations are continued.

Japanese Patent Laid-Open No. 06-014323

The video processing apparatus of the present invention includes a camera, a video acquisition unit, a video analysis unit, a video adjustment unit, and a video output unit. The camera captures an image including a subject. The video acquisition unit acquires video from the camera and outputs a video signal. The video analysis unit analyzes the video in the time direction with respect to the video signal. The video adjustment unit calculates an enlarged video signal based on the video signal according to the cutout frame calculated by the video analysis unit based on the video signal and the enlargement ratio calculated based on the display screen size signal. Output. The video output unit outputs an output video signal based on the video signal and the enlarged video signal output from the video adjustment unit. The video analysis unit detects, for each successive frame, an area where the difference between successive frames exceeds a predetermined threshold. In addition, the video analysis unit calculates an included cutout frame so as to include all of the plurality of cutout frames in a plurality of predetermined consecutive frames.

With such a configuration, the video processing apparatus of the present invention captures a video including a subject from a camera, detects a background portion, and substantially fixes the background portion in terms of time. The video processing apparatus can cut out the subject and display it as large as possible on the entire screen. Therefore, even when the user gazes at the screen, discomfort can be less likely to occur.

The video processing method of the present invention is a video processing method having the following steps. In other words, the video including the subject is shot, the video is acquired, the video is converted into a video signal and output, the video is analyzed in the time direction with respect to the video signal, the cutout frame is calculated based on the video signal, An area where the difference between successive frames exceeds a predetermined threshold is detected for each successive frame as a cutout frame, and the included cutout frame is calculated by including all of the plurality of cutout frames in a plurality of predetermined continuous frames. In accordance with the included cutout frame and the enlargement ratio calculated based on the display screen size signal, an enlarged video signal is calculated based on the video signal, and based on the video signal and the enlarged video signal, Output video signal.

FIG. 1 is a diagram showing an example of setting cut frames for continuous frames in the embodiment of the present invention. FIG. 2 is a diagram showing an example in which cutout frames of continuous frames are enlarged and displayed as an enlarged display image on the entire screen in the embodiment of the present invention. FIG. 3 is a block diagram showing the configuration of the video processing apparatus according to the embodiment of the present invention. FIG. 4 is a block diagram showing a detailed configuration of the video analysis unit in the embodiment of the present invention. FIG. 5 is a block diagram showing a detailed configuration of the position information determination unit in the present embodiment. FIG. 6 is a diagram for explaining a method of calculating the boundary between the moving pixel region and the still pixel region in the present embodiment. FIG. 7 is a diagram showing the relationship between the cutout frame and the included cutout frame in the present embodiment. FIG. 8 is a block diagram showing a detailed configuration of the video adjustment unit in the present embodiment. FIG. 9A shows an example in which an enlarged cutout frame is calculated from an included cutout frame in the present embodiment, and the width of the included cutout frame on the screen in the vertical direction of the screen included in the display screen size signal. It is a figure which shows the case where it is larger than the width | variety decided by an aspect ratio. FIG. 9B shows an example in which an enlarged cutout frame is calculated from the cutout frame included in the present embodiment, and the horizontal width on the screen of the cutout frame included in the screen includes the display screen size signal. It is a figure which shows the case where it is larger than the width | variety decided by an aspect ratio. FIG. 10 is a diagram showing a predetermined frame of the enlarged video signal displayed on the screen in the present embodiment. FIG. 11 is a diagram illustrating an example of a display screen during image processing when the video output unit according to the present embodiment does not use an output flag. FIG. 12 is a diagram illustrating an example of a display screen during image processing when the video output unit according to the present embodiment uses an output flag. FIG. 13 is a diagram illustrating an example in which the video output unit according to the present embodiment uses a frame memory to output a stable display screen even during image processing. FIG. 14A is a conceptual diagram for explaining the zoom control parameters output by the video analysis unit in the present embodiment. FIG. 14B is a conceptual diagram for explaining the zoom control parameters output by the video analysis unit in the present embodiment. FIG. 15 is a flowchart showing the procedure of the video processing method in the present embodiment. FIG. 16 is a diagram conceptually showing image processing of a conventional image processing apparatus.

(Embodiment)
Hereinafter, a video processing apparatus and a video processing method according to an embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram showing an example of setting continuous

frame cutout frames

1301, 1302, and 1303 in the embodiment of the present invention. As shown in FIG. 1, when a subject included in a video signal acquired from a camera or the like is moving, a method of setting a moving portion of the subject as

cut frames

1301, 1302, and 1303 for each frame can be considered. In this case, the

clipping frames

1301, 1302, and 1303 including the moving part of the subject can be determined using the image difference between frames. That is, based on the detection result of the image difference between frames, the moving pixel region and the still pixel region of each frame are determined. A rectangular shape including a moving pixel region is determined as the

cutout frames

1301, 1302, and 1303.

The horizontal axis in FIG. 1 is a frame, and shows only a part of a continuous frame. Here, N and m are arbitrary integers. As shown in FIG. 1, the positions and sizes of the cutout frames 1301, 1302, and 1303 in the frame N, the frame (N + m), and the frame (N + 2m) are the latest frame preceding each frame. And the amount of movement of the subject in each frame. Note that the number of frames of the most recent frame preceding each frame for determining the amount of movement of the subject can be set as appropriate based on the specifications of the system to be applied.

In FIG. 1, it is assumed that the amount of movement of the subject is relatively small. As shown in FIG. 1, the positions and sizes of the cutting frames 1301, 1302, and 1303 are slightly different.

FIG. 2 is a diagram showing an example in which cut-

out frames

1301, 1302, and 1303 of continuous frames in the embodiment of the present invention are enlarged by image processing and displayed as an enlarged display image on the entire screen.

Screens

1401, 1402, and 1403 correspond to cut

frames

1301, 1302, and 1303, respectively. In FIG. 2, the cutout frames 1301, 1302, and 1303 are enlarged as they are, but the

screens

1401, 1402, and 1403 make it easy to determine the subject. In this embodiment, the rectangular shape including the moving pixel region is determined as the cutout frames 1301, 1302, and 1303. However, the rectangular shape including the moving pixel region is not enlarged and displayed as a cutout frame as it is. This is because the background tends to fluctuate between frames as shown in FIG. Details of the method for determining the cutout frames 1301, 1302, and 1303 and the method for determining the enlarged display image will be described below.

FIG. 3 is a block diagram showing the configuration of the video processing apparatus 100 according to the embodiment of the present invention. The video processing apparatus 100 according to the present embodiment includes a camera 101, a video acquisition unit 102, a video analysis unit 103, a video adjustment unit 104, a camera control unit 106, and a video output unit 107.

The camera 101 captures a video including the subject and outputs the captured signal to the video acquisition unit 102. The video acquisition unit 102 converts a signal input from the camera 101 into a video signal 301. The video analysis unit 103 analyzes the video in the time direction with respect to the video signal 301. Specifically, the video analysis unit 103 receives the video signal 301 and detects a difference between successive frames of the video signal 301 to separate the background portion from the moving subject. Then, the video analysis unit 103 calculates a cutting frame based on the video signal 301 in accordance with the detected background portion and the moving subject. That is, the video analysis unit 103 detects, for each successive frame, an area where the difference between successive frames exceeds a predetermined threshold as a cutout frame. Then, the video analysis unit 103 calculates the included cutout frame 306 so as to include all of the plurality of cutout frames in a predetermined plurality of frames.

Further, when the subject protrudes from the video area acquired by the video acquisition unit 102, the video analysis unit 103 can adjust the zoom of the camera 101 to the wide side (that is, the zoom lens to the wide angle side). Judge as necessary. Here, zoom is performed using a zoom lens, but the image received by the image sensor may be calculated electronically. Then, the video analysis unit 103 outputs the zoom control parameter 307 of the camera 101 to the camera control unit 106. Details of the operation of the video analysis unit 103 will be described later.

The camera control unit 106 transmits a zoom control signal to the camera 101. The camera control unit 106 adjusts the zoom value of the camera 101 based on the zoom control parameter 307 output from the video analysis unit 103. The zoom control parameter 307 is used to control the zoom value of the camera 101 when the ratio of the background portion in the video area is small in the video analysis unit 103 and the subject is outside the shooting range. Further, when the moving subject is extremely small with respect to the shooting range and is difficult for the user to visually recognize, the position of the camera 101 may be controlled so that the subject moves to the center of the screen. Then, the zoom may be adjusted to the telephoto side (that is, the zoom lens is set to the telephoto side), and control may be performed so that the subject is displayed larger on the screen. Such control makes it easier for the user to recognize the subject.

The video adjustment unit 104 inputs the video signal 301, the included cutout frame 306, and the display screen size signal 105. Then, the video adjustment unit 104 outputs an enlarged video signal 309 based on the video signal 301 according to the included cutout frame 306 and the enlargement ratio calculated based on the display screen size signal 105. That is, the video adjustment unit 104 outputs an enlarged video signal 309 including a subject cut out except for the background portion to the video output unit 107 according to the subject and background portion separated by the video analysis unit 103. Further, the video output unit 107 receives the video signal 301, the enlarged video signal 309, and the output flag 308 output from the video analysis unit 103, and outputs the output video signal 108.

The display screen size signal 105 is, for example, a signal input from a control unit (not shown), and the aspect ratio of the screen displayed by the output video signal 108 of the video processing apparatus 100 and the screen display size of the output video signal 108. It contains information indicating.

Here, it is assumed that the control unit communicates and exchanges information with the device on which the output video signal 108 of the video processing device 100 is displayed, and acquires the screen aspect ratio and screen display size of the device. ing. The apparatus on which the output video signal 108 is displayed is an apparatus connected to the video processing apparatus 100 according to the present embodiment by a communication system or the like, and is used by, for example, another user who uses the video conference system. It may be a device that performs. Further, for example, it may be a monitor device directly connected to the video processing device 100 by a video signal connection cable or the like. Furthermore, the present invention can also be applied to a configuration in which the output video signal 108 is displayed on a video processing device in which an image display unit is incorporated.

Next, the detailed configuration and operation of the video analysis unit 103 will be described. FIG. 4 is a block diagram showing a detailed configuration of the video analysis unit 103 in the embodiment of the present invention. The video analysis unit 103 includes a video storage unit 302, a difference calculation unit 303, a background detection unit 304, a position information determination unit 305, and a cut-out position storage unit 310.

The video storage unit 302 and the difference calculation unit 303 each input the video signal 301 output from the video acquisition unit 102. The video storage unit 302 stores one frame of the video signal 301. Then, the video storage unit 302 delays the input video signal 301 by one frame and outputs it to the difference calculation unit 303. The video storage unit 302 is a frame memory, and may be configured by, for example, a semiconductor memory.

The difference calculation unit 303 detects a difference between the video signal 301a delayed from the video storage unit 302 by one frame and the original video signal 301 that has not been delayed. Here, if the current frame of the video signal 301 is a frame N (N is an integer), the video storage unit 302 outputs a frame (N−1) preceding the current frame N. The difference calculation unit 303 detects the difference between the video signals of the frame N and the frame (N−1) for each pixel. Then, the difference calculation unit 303 outputs the detected difference result to the background detection unit 304.

The background detection unit 304 determines that each pixel is a moving pixel or a still pixel by comparing the difference for each pixel with a predetermined threshold. The predetermined threshold is, for example, about 10 levels. Here, it is assumed that each pixel is expressed by 8 bits (when a video signal is handled with 256 levels of accuracy). At this time, if the difference for each pixel exceeds a predetermined threshold, the pixel is a moving pixel. That is, the moving pixels in the frame N and the frame (N−1) are regarded as moving pixels. Further, when a pixel has a difference equal to or less than a predetermined threshold, it is regarded as a still pixel that has not moved. Then, the background detection unit 304 divides the moving pixel region and the still pixel region in the current frame N, and outputs them to the position information determination unit 305.

Next, the detailed configuration and operation of the position information determination unit 305 will be described with reference to FIG. FIG. 5 is a block diagram illustrating a detailed configuration of the position information determination unit 305 in the present embodiment. As illustrated in FIG. 5, the position information determination unit 305 includes a coordinate calculation unit 351 and a change amount calculation unit 352. The position information determination unit 305 outputs the included cutout frame 306 to the video adjustment unit 104 using the cutout position storage unit 310.

FIG. 6 is a diagram for explaining a method of calculating the boundary 632 between the moving pixel region and the stationary pixel region in the present embodiment. The coordinate calculation unit 351 calculates a boundary 632 between the background portion and the subject portion from the moving pixel region and the still pixel region in the current frame N. 6 indicates a boundary 632 between the moving pixel region and the stationary pixel region. A boundary 632 has a rectangular shape indicating the boundary between the background portion and the subject portion, and indicates a cutout frame.

It should be noted that the extraction start coordinate 630 and the extraction end coordinate 631 use the upper left end coordinate and the lower right end coordinate of the boundary 632 so that the boundary 632 having a rectangular shape can be specified on the screen. This is an example for reducing the memory amount of the cutout position storage unit 310 as much as possible, and the present embodiment is not limited to using these coordinates. For example, the coordinates of the upper right end and the lower left end of the boundary 632 having a rectangular shape may be used. Further, the number of pixels in the vertical direction and the horizontal direction of the boundary 632 which is a rectangular shape, and the center coordinates may be used. In this way, a cutout frame having a rectangular shape is calculated from the boundary 632 between the background portion and the subject portion.

As shown in FIG. 5, the coordinate calculation unit 351 outputs the cut coordinates 312 including the cut start coordinates 630 and the cut end coordinates 631 to the cut position storage unit 310 and the change amount calculation unit 352.

The cut-out position storage unit 310 stores cut-out coordinates 313 for nine frames from the frame (N-1) preceding the current frame N to the frame (N-9). When the cutout coordinates 312 of the current frame N are input, the cutout position storage unit 310 calculates the amount of change of the cutout coordinates 313 for nine frames from the stored frame (N-1) to the frame (N-9). To the unit 352. Then, the cut-out position storage unit 310 updates the stored content. That is, the cutout position storage unit 310 stores the cutout coordinates 313 for nine frames from the current frame N to the frame (N-8).

The change amount calculation unit 352 applies the cut-out coordinates 312 of each of the current frame N to the frame (N-9) and the latest frame preceding each of the current frame N to the frame (N-9), for example, the current frame N. Compares the cut-out coordinates 313 of the frame (N−1) and verifies the movement amount of both. Then, when the amount of movement of both is equal to or less than a predetermined amount of movement threshold, the change amount calculation unit 352 extracts 10 frames of cutout coordinates 312 for 10 frames from the current frame N to a frame (N-9) preceding the current frame N. 313 is output to the coordinate calculation unit 351 as post-determination cut-out coordinates 314. Here, in the video processing apparatus 100 according to the present embodiment, if a video conference system is assumed, the predetermined movement amount threshold is preferably about 200 pixels, for example.

The coordinate calculation unit 351 inputs the post-judgment cut-out coordinates 314, and calculates an included cut-out frame 306 so as to include all of the plurality of cut-out frames in a plurality of predetermined consecutive frames. Here, the predetermined number of consecutive frames is, for example, 10 frames. In this case, the coordinate calculation unit 351 inputs post-determination cut-out coordinates 314 for 10 consecutive frames. Then, the coordinate calculation unit 351 includes all of the post-determination cut-out coordinates 314 for 10 frames, and calculates an included cut-out frame 306 that is a wider cut-out frame. Here, the shape of the included cutout frame 306 is a rectangle.

In this way, since the shape of the included cutout frame 306 is simple, the subsequent signal processing can be performed more easily. That is, the shape of the included cutout frame 306 should not be a simple composition of the post-judgment cutout coordinates 314. However, if the signal processing capability of each part of the video processing apparatus 100 in the present embodiment is improved and there is a margin in the signal processing capability thereafter, the shape of the included extraction frame 306 is simply extracted after the determination coordinates. It is good also as composition of 314.

On the other hand, the change amount calculation unit 352 includes the cut-out coordinates 312 of each of the current frame N to the frame (N-9) and the latest frame preceding each of the current frame N to the frame (N-9), for example, the current frame N. If the movement amount of both of the frames N exceeds the predetermined movement amount threshold value by comparing the cut coordinates 313 of the frame (N-1), for example, the frame N-1 to the current frame N When the movement amount to exceeds the predetermined movement amount threshold, the cut-out coordinates 312 of the current frame N are excluded from the cut-out coordinates 314 after determination. Then, the change amount calculation unit 352 outputs the cut coordinates 313 of the frame (N−1) preceding the current frame N as the cut coordinates 314 after determination instead of the cut coordinates 312 of the current frame N. Further, the change amount calculation unit 352 determines to the coordinate calculation unit 351 the extracted coordinates 312 for nine frames from the frame (N−1) preceding the current frame N to the frame (N-9) preceding the current frame N. Output as post-cutout coordinates 314. That is, data is output twice for the cut-out coordinates 313 of the frame (N−1).

As described above, when the movement amount of the cut frames of the continuous frames exceeds the predetermined movement amount threshold, the video analysis unit 103 sets the cut frames of the continuous frames to all of a plurality of predetermined continuous frames. Remove from the cropping frame. Even in this case, the coordinate calculation unit 351 inputs post-determination cut-out coordinates 314 corresponding to 10 frames. Then, the coordinate calculation unit 351 includes all of the post-judgment cutout coordinates 314 corresponding to 10 frames, and calculates an included cutout frame 306 that is a wider cutout frame.

FIG. 7 is a diagram showing the relationship between the cut frames 601, 602, and 603 and the included cut frame 306 in the present embodiment. FIG. 7 illustrates the case of three frames as an example for the sake of simplicity. The coordinate calculation unit 351 includes inclusion frames 601, 602, and 603 included when the included extraction frame 306 is calculated. The relationship with the cut out frame 306 is shown. As shown in FIG. 7, the coordinate calculation unit 351 calculates the minimum rectangular shape that includes all the cut frames 601, 602, and 603 as the included cut frame 306.

Next, the detailed configuration and operation of the video adjustment unit 104 will be described with reference to FIG. FIG. 8 is a block diagram showing a detailed configuration of the video adjustment unit 104 in the present embodiment. The video adjustment unit 104 includes an enlargement ratio calculation unit 403 and a video enlargement unit 405.

The enlargement ratio calculation unit 403 inputs the included cutout frame 306 and the display screen size signal 105, and outputs the enlargement ratio E of the included cutout frame 306 and the enlargement start positions 360a and 360b. Then, the video enlargement unit 405 inputs the enlargement ratio E of the included cutout frame 306, the enlargement start positions 360a and 360b, and the video signal 301, and outputs the post-enlargement video signal 309. Here, the enlargement ratio E is the ratio of the enlarged cutout frame that becomes the display screen size in the post-enlargement video signal 309 and the included cutout frame 306. Further, the enlargement start positions 360a and 360b may be, for example, the upper left end of the enlargement cutout frame.

Specific operations for outputting the enlarged video signal 309 with the above-described configuration will be described with reference to FIGS. 9A and 9B. FIG. 9A shows an example in which the enlarged cutout frame 350a is calculated from the cutout frame 306a included in the present embodiment, and the vertical width of the included cutout frame 306a on the screen is included in the display screen size signal. It is a figure which shows the case where it is larger than the width | variety determined by the aspect-ratio of the screen to be displayed. FIG. 9B shows an example in which the enlarged cutout frame 350b is calculated from the cutout frame 306b included in the present embodiment, and the horizontal width on the screen of the cutout frame 306b included is included in the display screen size signal. It is a figure which shows the case where it is larger than the width | variety determined by the aspect-ratio of the screen to be displayed. 9A and 9B, the included

cutout frames

306a and 306b and the enlarged cutout frame for outputting the enlarged video signal are indicated by broken lines. 9A and 9B also show enlargement start positions 360a and 360b.

Here, the coordinates of the upper left end (Xs, Ys) and the coordinates of the lower right end (Xe, Ye) are used to specify the included

cutout frames

306a, 306b on the screen. An operation for obtaining each of the enlargement ratio E and the enlargement start positions 360a and 360b in the above two cases will be described.

Specifically, the included cutout frame 306a is expanded in the horizontal or vertical direction of the screen so as to match the aspect ratio of the screen on which the enlarged video signal 309 is displayed. Here, the number of pixels on the screen is 1920 × 1080, and the coordinates of the end of the screen are represented by (Dx, Dy) = (1920, 1080). Then, the enlargement ratio E for enlarging the image signal 309 after enlargement so as to match the aspect ratio of the screen on which the image signal 309 is displayed can be expressed as follows. Here, min (A, B) is a function for selecting the smaller numerical value of the argument A and the argument B.

E = min (Dx × 1 / | Xe−Xs |, Dy × 1 / | Ye−Ys |)
First, in the case of FIG. 9A, the enlargement ratio E is obtained using the above equation assuming specific numerical values. Here, for example, if the upper left end coordinates (Xs, Ys) = (800, 540) and the lower right end coordinates (Xe, Ye) = (992, 1080) of the enlarged cutout frame 350a,
E = min (1920 × 1 / | 992-800 |, 1080 × 1 / | 1080-540 |)
It is. Therefore,
E = min (10, 2) = 2 (times)
It can be asked. That is, if the included cutout frame 306a is expanded by twice in the horizontal direction, the enlarged cutout frame 350a can be calculated. In this case, the included cutout frame 306a is expanded by an equal width on both sides in the horizontal direction. In this way, the enlargement start position 360a can be obtained.

Next, in the case of FIG. 9B, the enlargement ratio E is obtained assuming specific numerical values. Here, for example, if the upper left end coordinates (Xs, Ys) = (100, 540) and the lower right end coordinates (Xe, Ye) = (1900, 1080) of the enlarged cutout frame 350b,
E = min (1920 × 1 / | 1920−100 |, 1080 × 1 / | 1080−540 |)
It is. Therefore,
E = min (1.06, 2) = 1.06 (times)
It can be asked. That is, if the included cutout frame 306b is expanded by 1.06 times in the vertical direction, the enlarged cutout frame 350b can be calculated. In this case, the included cutout frame 306a is expanded upward in the vertical direction. This is because, in the present embodiment, the included

cutout frames

306a and 306b are along the lower end of the display screen. In this way, the enlargement start position 360b can be obtained.

Note that the included

cutout frames

306a and 306b do not necessarily have to be along the lower end of the display screen. In that case, the included cutout frame 306b is expanded by an equal width on both sides in the vertical direction.

As described above, the video enlargement unit 405 can calculate the enlargement cut frames 350a and 350b, that is, the enlargement ratio E and the enlargement start positions 360a and 360b, and output the enlarged image signal 309. That is, the video adjustment unit 104 expands the cutout frame 306 included so as to have an aspect ratio equivalent to the aspect ratio of the display screen, and outputs the expanded cutout video signal 309 to the video output unit 107.

FIG. 10 is a diagram in which predetermined frames of the enlarged video signal 309 are displayed on the

screens

601a, 602a, and 603a. FIG. 10

shows screens

601a, 602a, and 603a in a frame N, a frame (N + m), and a frame (N + 2m) (N and m are arbitrary integers). As shown in FIG. 10, according to the video processing apparatus 100 in the present embodiment, a constant region is always displayed as a background even if the frame changes. As a result, the user is less likely to experience discomfort even if he / she gazes at the

continuous screens

601a, 602a, and 603a. In addition, it can be enlarged and displayed almost at the center of the

screens

601a, 602a, and 603a for displaying the subject. As a result, the user can accurately capture the subject.

Next, the output flag 308 shown in FIG. 3 will be described. The video analysis unit 103 outputs an output flag 308 based on the included enlarged video signal 309 output from the video adjustment unit 104. Specifically, the video analysis unit 103 generates an enlarged video signal from a predetermined continuous frame including the current frame N of the enlarged video signal 309 output from the video adjustment unit 104 and a plurality of frames preceding the current frame N. After calculating 309, the output flag 308 is output. That is, for example, using a predetermined continuous frame, which is 10 frames, the video adjustment unit 104 extends the included cutout frame 306 corresponding to the current frame N and a plurality of frames preceding the current frame N. Then, the video adjusting unit 104 outputs an enlarged video signal 309 based on the video signal 301 in accordance with the plurality of included cutout frames 306 described above.

Such operation will be described with reference to FIGS. FIG. 11 is a diagram illustrating an example of a display screen during image processing when the video output unit 107 in the present embodiment does not use the output flag 308. In FIG. 11, the enlarged video signal 309 is also displayed from the video enlargement unit 405 during the period t1 to t2 (during the enlargement process) during signal processing for 10 frames including the current frame N. During this period, since the video adjustment unit 104 and the video enlargement unit 405 perform the expansion processing of the plurality of included cutout frames 306, the size of the subject changes. After that, in the period t3 (enlargement process completed), when the extension process is completed, the size of the subject is fixed, and the enlarged video signal 309 can be displayed stably. When the output flag 308 is not used in this way, the enlarged video signal 309 is output while the included extraction frame 306 is being expanded, and thus the user feels uncomfortable when watching such a screen. There is a possibility to give.

FIG. 12 is a diagram illustrating an example of a display screen during image processing when the video output unit 107 according to the present embodiment uses the output flag 308. As shown in FIG. 12, during the period t <b> 1 to the period t <b> 2 (during the expansion process), the video enlargement unit 405 outputs the video signal 301. After that, in the period t3 (enlargement process completion), when the expansion process of the included cut frame 306 corresponding to a plurality of predetermined consecutive frames including the current frame N is completed, the expanded video signal after the expansion process 309 is output. That is, the video output unit 107 calculates the enlarged video signal 309 from a predetermined continuous frame including the current frame N of the video signal 301 and a plurality of frames preceding the current frame N. Thereafter, the output video signal 108 may be switched from the video signal 301 based on the video from the camera 101 to the enlarged video signal 309 based on the output flag 308 output from the video analysis unit 103.

In addition, when the movement amount of the cut frame of the frame subsequent to the current frame N exceeds a predetermined movement amount threshold, the cut frame of the subsequent frame (N−1) following the preceding frame N of the continuous frame is changed by the change amount. Excluded from the cut out frames of a plurality of predetermined consecutive frames in the post-judgment cut out coordinates 314 output by the calculation unit 352. In such a case, the video output unit 107 includes, from the video signal 301 based on the video from the camera 101, the included clipping corresponding to a plurality of predetermined consecutive frames that do not include the current frame N and precede the current frame N. The output video signal 108 may be switched to the post-enlargement video signal 309 including the information of the frame 306. In this way, an image in which the size of the subject being enlarged is changed is not displayed.

FIG. 13 is a diagram illustrating an example in which the video output unit 107 according to the present embodiment uses the frame memory to output a stable display screen during image processing. When the video enlargement unit 405 has a frame memory, the frame memory includes a frame (N−1) preceding the current frame N and included clipping frames 306 included in a plurality of predetermined consecutive frames. A corresponding enlarged video signal 309 may be stored. As shown in FIG. 13, during the period t1 to the period t2 (during the expansion process), the video enlargement unit 405 corresponds to a plurality of predetermined consecutive frames before the frame (N−1) preceding the current frame N. An enlarged video signal 309 based on the information of the included cutout frame 306 is output from the frame memory. After that, in the period t3 (enlargement process complete), when the expansion process of the included cutout frame 306 corresponding to a plurality of predetermined consecutive frames including the current frame is completed, the expanded video signal 309 after the expansion process is completed. Is output. In other words, the video output unit 107 has a frame memory for one frame, and the frame memory includes information on the included cutout frame 306 corresponding to a plurality of predetermined consecutive frames preceding the current frame N. The post video signal 309 is stored. The video output unit 107 calculates the current frame from the enlarged video signal 309 from the frame memory based on the output flag 308 output from the video analysis unit 103 after the video adjustment unit 104 calculates the enlarged video signal 309. The output video signal 108 may be switched to a post-enlarged video signal 309 including information of the included cutout frame 306 corresponding to a predetermined plurality of consecutive frames including N.

In this way, a frame memory for one frame is required for the video enlargement unit 405, but the enlarged video signal 309 can be displayed almost continuously and stably with respect to the size of the subject. If no data is stored in the frame memory, the video enlargement unit 405 may output the enlarged video signal 309.

Next, the zoom control parameter 307 output from the video analysis unit 103 will be described with reference to FIGS. 14A and 14B. In FIG. 14A, the included cutout frame 306c is indicated by a broken line. In addition, in order to specify the included cutout frame 306c on the screen, upper left end coordinates (Xs, Ys) and lower right end coordinates (Xe, Ye) are shown. In this case, the upper left end coordinates (Xs, Ys) and the lower right end coordinates (Xe, Ye) are located at the upper and lower ends of the screen. Therefore, the camera control unit 106 adjusts the zoom value of the camera 101 based on the zoom control parameter 307 output from the video analysis unit 103. Specifically, the zoom control parameter 307 is when the ratio of the background portion in the video area is small in the video analysis unit 103 and the subject is outside the shooting range. Is larger than a predetermined background threshold value, it is used to control the zoom value of the camera 101. Here, for example, the ratio of the background portion in the video area may be ¼ of the video area.

That is, the video analysis unit 103 calculates the position of the background in the shooting area of the camera 101 based on the calculated cutout frame, and if the range occupied by the background is larger than a predetermined background threshold, the included cutout frame 306c is displayed in the video. Output to the adjustment unit 104. On the other hand, when the range occupied by the background is smaller than the predetermined background threshold, the video analysis unit 103 outputs a control parameter for adjusting the zoom of the camera 101 to the wide side to the camera control unit 106.

In FIG. 14B, an included cutout frame 306d is shown. The coordinates (Xe, Ye) of the lower right end of the included cutout frame 306d are located at the right end of the screen. Thus, even when the coordinates of the upper left end (Xs, Ys) or the coordinates of the lower right end (Xe, Ye) are located at the left end or the right end of the screen, the camera 101 controls the zoom value of the camera 101. A control parameter for adjusting the zoom to the wide side is output to the camera control unit 106.

When the subject is displayed small on the screen and the movement of the subject is small, the included cutout frame is displayed small on the screen. In such a case, the position of the camera 101 may be controlled so that the subject moves to the center of the screen. Alternatively, the zoom may be adjusted to the telephoto side so that the subject is displayed larger on the screen. Such control makes it easier for the user to recognize the subject.

Next, a video processing method of the video processing apparatus 100 according to the present embodiment will be described with reference to FIG. The configuration of the video processing apparatus 100 is shown in FIG. 3, and its operation has already been described. FIG. 15 is a flowchart showing the procedure of the video processing method in the present embodiment.

First, the camera 101 captures an image including a subject (step S601). The video acquisition unit 102 acquires the video captured by the camera 101 (step S602), converts the video into the video signal 301, and outputs the video signal 301 to the video analysis unit 103, the video adjustment unit 104, and the video output unit 107 ( Step S603).

The video analysis unit 103 analyzes the video in the time direction with respect to the video signal 301 (step S604). Specifically, by inputting the video signal 301 and detecting a difference between frames of the video signal 301, the background portion and the moving subject are separated. That is, the background detection unit 304 determines that each pixel in the current frame N is a moving pixel or a still pixel by comparing the difference for each pixel between successive frames of the video signal 301 with a predetermined threshold.

Here, the predetermined threshold is, for example, about 10 levels. Here, it is assumed that each pixel is expressed by 8 bits (when a video signal is handled with 256 levels of accuracy). At this time, if the difference for each pixel exceeds a predetermined threshold, the pixel is a moving pixel. Based on the determination result, the coordinate calculation unit 351 calculates the boundary 632 between the background portion and the subject portion from the moving pixel region and the still pixel region in the current frame N. The video analysis unit 103 uses the boundary 632 as a cutting frame.

That is, the video analysis unit 103 calculates a cutout frame based on the video signal 301 (step S605). Here, the video analysis unit 103 calculates the cutout frame as a rectangular shape in order to simplify the signal processing. Then, the video analysis unit 103 detects, for each successive frame, an area where the difference between successive frames exceeds a predetermined threshold as a cutout frame (step S606). Then, the video analysis unit 103 calculates a cut frame 306 that is included in a plurality of predetermined frames so as to include all of the plurality of cut frames (step S607).

Here, the predetermined number of frames is, for example, 10 frames. The shape of the included cutout frame 306 is a rectangle so that the subsequent signal processing can be performed more easily. Note that the video analysis unit 103 may remove the cut frame from all of the plurality of cut frames in the predetermined continuous frames when the movement amount of the cut frames of the continuous frames exceeds a predetermined movement amount threshold value. Good. Here, in the video processing apparatus 100 according to the present embodiment, if a video conference system is assumed, the predetermined movement amount threshold is preferably about 200 pixels, for example.

The video adjustment unit 104 inputs the video signal 301, the included cutout frame 306, and the display screen size signal 105. Then, the video adjustment unit 104 calculates an enlarged video signal 309 based on the video signal according to the included cutout frame 306 and the enlargement ratio calculated based on the display screen size signal 105 (step S608). ). Here, the display screen size signal 105 is a signal input from a control unit (not shown), for example, and the screen aspect ratio displayed by the output video signal 108 of the video processing apparatus 100 and the screen of the output video signal 108 are displayed. Information indicating the display size is included. The enlargement ratio E is the ratio of the enlarged cutout frame that is the display screen size in the post-enlargement video signal 309 and the included cutout frame 306.

Then, the video adjustment unit 104 expands the included cutout frame 306 so as to have an aspect ratio equivalent to the aspect ratio of the display screen, and outputs the expanded video signal 309 to the video output unit 107. At this time, the expanded cut-out frame 306b is enlarged so as to be arranged at substantially the center of the screen on which the moving subject is displayed, and the enlarged video signal 309 is calculated.

The video adjustment unit 104 outputs the enlarged video signal 309 including the subject cut out from the background part to the video output unit 107 based on the background part separated by the video analysis part 103 (step S609).

As described above, the video adjustment unit 104 calculates the enlarged video signal 309 from the included cutout frame 306, and the video output unit 107 outputs the enlarged video signal 309 as the output video signal 108. As a result, according to the video processing method of the present embodiment, a constant area is always displayed as the background even if the frame changes. As a result, the user is less likely to feel discomfort even if such a screen is watched.

When the video processing apparatus according to the present invention transmits / receives video between different users, the video transmitted to the other party can be transmitted to the other party without moving the user according to the camera. Can be sent. In addition, by detecting the background part from the video difference at different times and cutting out the part other than the background part, it is possible to detect it uniformly regardless of the type, color, or number of subjects. This is useful for a video processing apparatus when performing the above.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Camera 102 Image | video acquisition part 103 Image | video analysis part 104 Image | video adjustment part 105 Display screen size signal 106 Camera control part 107 Image | video output part 108 Output image |

video signal

301, 301a Image | video signal 302 Image | video storage part 303 Difference calculation part 304 Background Detection unit 305 Position

information determination unit

306, 306a, 306b, 306c, 306d Included extraction frame 307 Zoom control parameter 308 Output flag 309 Image signal after enlargement 310 Extraction

position storage unit

312, 313 Extraction coordinate 314 Post-determination extraction coordinate 350a, 350b Enlarged cutout frame 352 Change

amount calculation unit

360a, 360b Enlargement start position 405

Video enlargement unit

601, 602, 603, 1301, 1302, 1303

Cutout frame

601a, 602a, 03a, 1401,1402,1403 screen 632 boundary E expansion rate

Claims

A camera that shoots images including the subject;
A video acquisition unit that acquires the video from the camera and outputs a video signal;
A video analysis unit that analyzes video in a time direction with respect to the video signal;
Based on the cutout frame calculated by the video analysis unit based on the video signal and the enlargement ratio calculated based on the display screen size signal, an enlarged video signal is calculated based on the video signal and output. A video adjustment unit;
Based on the video signal and the enlarged video signal output by the video adjustment unit, a video output unit that outputs an output video signal, the video analysis unit,
An area where the difference between successive frames exceeds a predetermined threshold is detected for each successive frame as the cutout frame,
A video processing apparatus that calculates an included cutout frame so as to include a plurality of all the cutout frames in a predetermined plurality of consecutive frames.
The video output unit
After the video adjustment unit calculates the enlarged video signal, based on the output flag output by the video analysis unit,
The video processing apparatus according to claim 1, wherein the output video signal is switched from the video signal based on the video from the camera to the enlarged video signal.
The video analysis unit
When the movement amount of the cut frame of the continuous frames exceeds a predetermined movement amount threshold,
The video processing apparatus according to claim 1, wherein the cutout frame is excluded from all of the plurality of cutout frames in the predetermined plurality of consecutive frames.
The video processing apparatus further includes a camera control unit that transmits a zoom control signal to the camera,
The position information determination unit calculates the position of the background in the shooting area of the camera based on the calculated cutout frame,
If the range occupied by the background is greater than a predetermined background threshold,
Outputting the included cutout frame to the video adjustment unit;
When the range occupied by the background is smaller than the predetermined background threshold,
The video processing apparatus according to claim 1, wherein a control parameter for adjusting the zoom of the camera to a wide side is output to the camera control unit.
The video output unit
After the video adjustment unit calculates the enlarged video signal, based on the output flag output by the video analysis unit,
The output video signal from the video signal based on the video from the camera to the post-enlarged video signal including information on included cutout frames corresponding to a plurality of predetermined consecutive frames preceding the current frame. The video processing apparatus according to claim 1 to be switched.
The video output unit
It has a frame memory for 1 frame,
In the frame memory, the enlarged video signal including information of included cutout frames corresponding to a plurality of predetermined consecutive frames preceding the current frame is stored,
After the video adjustment unit calculates the enlarged video signal, based on the output flag output by the video analysis unit,
The output video from the enlarged video signal from the frame memory to the enlarged video signal including information of the included cutout frames corresponding to the predetermined consecutive frames including the current frame The video processing apparatus according to claim 1, wherein the signal is switched.
The display screen size signal includes information indicating an aspect ratio of the display screen of the output video signal and a screen display size of the output video signal,
The shape of the included cutting frame is a rectangle,
The video adjustment unit extends the included clipping frame so as to have an aspect ratio equivalent to the aspect ratio of the display screen, and outputs the expanded cut-out video signal to the video output unit. Video processing equipment.
Take a picture of the subject,
Get the video,
Converting the video into a video signal and outputting it,
Analyzing the video in the time direction with respect to the video signal,
Calculate a cutting frame based on the video signal,
An area where the difference between successive frames exceeds a predetermined threshold is detected for each successive frame as the cutout frame,
Calculating a contained cutout frame so as to include all of the cutout frames in a plurality of predetermined consecutive frames;
According to the included cutout frame and the enlargement ratio calculated based on the display screen size signal, calculate an enlarged video signal based on the video signal,
A video processing method for outputting an output video signal based on the video signal and the enlarged video signal.