WO2014097465A1

WO2014097465A1 - Video processor and video p rocessing method

Info

Publication number: WO2014097465A1
Application number: PCT/JP2012/083190
Authority: WO
Inventors: 佐々木　昭; 恭一中熊; 溝添　博樹
Original assignee: 日立マクセル株式会社
Priority date: 2012-12-21
Filing date: 2012-12-21
Publication date: 2014-06-26

Abstract

As the processing amount of video shot by a camera increases, distortion of the video results, making it inconvenient for a user to handle the video. The present invention provides a video processor having a video input unit to which v ideo information shot by a camera is inputted, a video processing unit for correcting video of a human face included in the video information inputted to the video input unit, and an output unit for outputting video information proce ssed by the video processing unit, wherein the video processor is chara cterized in that the video processing unit determines according to a presc ribed condition whether the video of the human face included in the video information needs to be corrected.

Description

Video processing apparatus and video processing method

The technical field relates to a video processing apparatus and a video processing method.

Patent Document 1 states that “in a conventional system, it is possible to realize a dialogue between users who actually view the display with their line of sight matched, such as a half mirror, a hologram screen, or a projector. It is necessary to use a special device and a simple and inexpensive system cannot be configured "(see Patent Document 1 [0011]). Images are taken from different angles, the foreground image area including the subject to be photographed and the background image are separated from each photographed image, and each pixel position is associated with the subject to be photographed between the separated foreground image regions. To generate relative position information that indicates the relative positional relationship of the shooting target with respect to each camera and associates them with each other. From the pixel position and its luminance component, the pixel position and its luminance component constituting the virtual viewpoint image to be newly generated are obtained according to the generated relative position information, and the pixel position and its luminance component are obtained. “Transmission of virtual viewpoint image to outside” (see Patent Document 1 [0015]) and the like.

Japanese Patent Laid-Open No. 2005-065051

However, if the processing amount of the video shot with the camera increases, the processing for the shot image will not be in time, the video will be disturbed, and it will be unusable for the user.

In order to solve the above problems, for example, the configuration described in the claims is adopted.
The present application includes a plurality of means for solving the above-mentioned problems. For example, the present application includes a video input unit to which video information captured by a camera is input, and a video information input to the video input unit. In a video processing apparatus having a video processing unit that corrects a video of a face and an output unit that outputs video information processed by the video processing unit, the video processing unit includes a face included in the video information according to a predetermined condition It is characterized in that it is determined whether or not to correct the video.

According to the above means, it is possible to suppress image disturbance.

It is a system configuration figure showing one embodiment of a video conference communication system. It is a block block diagram which shows one specific example of the video processing apparatus in FIG. It is a figure which shows one specific example of the program read from the memory | storage part in FIG. 2, and expand | deployed to memory. It is a figure which shows one specific example of the determination list | wrist produced | generated when determining the necessity of the face correction process. It is a flowchart which shows one specific example of a series of main processes of a line-of-sight alignment. It is a figure which shows one specific example of the positional relationship of two cameras, a monitor, and a video conference participant. It is a flowchart which shows a specific example of the subroutine process of a line-of-sight alignment. It is a figure which shows an example of the image | video image | photographed with the video conference communication system. It is a figure which shows one specific example of the determination list | wrist produced | generated when determining the necessity of the face correction process on several conditions. It is a flowchart which shows a specific example of the subroutine process of a line-of-sight alignment which performs determination on multiple conditions. It is a figure which shows the structural example which combined the video processing apparatus and the other video conference apparatus. It is a figure which shows the structural example which a video processing apparatus connects with another video conference apparatus via a network. It is a figure which shows one specific example of the determination list | wrist produced | generated when determining the necessity of the face correction process using the area ratio of a face. It is a flowchart which shows a specific example of the subroutine process of a line-of-sight alignment using the area ratio of a face.

First, the outline of the embodiment will be described. In videophone and videoconferencing, even if the user is looking at the other party's eyes displayed on the monitor, the user's line of sight is far away from the camera's installation position. There is a problem that the lines of sight do not match each other.

テレビ Video conferencing systems that adjust the line of sight to correct individual faces taken with a camera are being studied, but when multiple people are within the shooting range, the amount of processing increases by the number of people. When the number of people is large, the processing time increases, and the processing time becomes long. Therefore, when a series of processing is performed for all faces, for example, 29.97 frames are normally processed per second, but the number of frames is less than that. Become. If it does so, smoothness will be impaired as a picture, and it will be in the state where a so-called frame was dropped.

On the other hand, according to the embodiment described below, correction processing can be performed without reducing the number of video frames processed per second even when there are a large number of subjects in the shooting range. . Embodiments will be described below with reference to the drawings.

In the following embodiment, a video conference terminal will be described as an example of a video processing device.

FIG. 1 is a system configuration diagram showing an embodiment of a video conference communication system. In FIG. 1, 1 and 2 are video processing devices, 3 is a network, 4 is a camera, 5 is a monitor, 6 is a microphone, 7 is a speaker, and 8 is a remote control. In the figure, a video processing apparatus 1 is connected to a camera 4, a monitor 5, a microphone 6, and a speaker 7, and can input and output video and audio. The video processing device 2 has the same device and configuration as the video processing device 1.

The video processing apparatus 1 can hold a video conference with the video processing apparatus 2 via the network 3. The video processing apparatus can be operated with the remote controller 8.

The camera 4 is connected to the video processing device, and outputs the captured video signal to the video processing device.

The monitor 5 is connected to the video processing device, and receives and displays the video signal output from the video processing device.

The microphone 6 is connected to the video processing device and outputs the collected sound as an audio signal to the video processing device.

Speaker 7 is connected to a video processing device, inputs an audio signal output from the video processing device, and outputs audio.

The remote controller 8 transmits a remote control signal to the video processing device and transmits an operation instruction from the user to the video processing device.

FIG. 2 is a block diagram showing a specific example of the internal configuration of the video processing apparatus in the video conference communication system shown in FIG. In FIG. 2, 101 is a control unit, 102 is a memory, 103-1 is a video encoder, 103-2 is an audio encoder, 104-1 is a video decoder, 104-2 is an audio decoder, 118 is a multiplexing unit, and 119 is separated. 105, a stream processing unit, 106 a storage unit, 107-1 a video processing unit for video output to the monitor 5, 107-2 a video processing unit for video input from the camera 4, and 108-1 Is a sound processing unit for outputting sound to the speaker 7, 108-2 is a sound processing unit for inputting sound from the

microphone

6, 109 is a remote control processing unit, 110 is a network connection unit, 112 is a video input terminal, 113 Is a video output terminal, 114 is an audio input terminal, 115 is an audio output terminal, 116 is a remote control input terminal, and 117 is a network connection terminal.

As described later, the control unit 101 develops a program stored in the storage unit 106 in the memory 102 and executes the developed program to realize functions corresponding to various programs. Further, the program is controlled in accordance with operation information input from the remote control processing unit 109.

The encoder 103-1 receives the video signal from the video processing unit 107-2, the encoder 103-2 receives the audio signal from the audio processing unit 108-2, and compresses and encodes the input signal information. The audio data is output to a multiplexing unit 118 or a stream processing unit 105 described later.

The decoder 104-1 and the decoder 104-2 receive the compression-coded video data and audio data output from the separation unit 119 or the stream processing unit 105, which will be described later, respectively. Each expands and expands to an audio signal.

The multiplexing unit 118 multiplexes the compression-encoded video data and the compression-encoded audio data, which are defined as Elementary Stream (ES) in the MPEG-2 system, input from the encoder 103, and transmits Transport Stream ( The packetized video / audio data called TS) is output.

The separation unit 119 separates the video / audio data output from the stream processing unit 105 into video data and audio data.

The stream processing unit 105 generates a network packet that is a packet of a network protocol to be transmitted to another video processing apparatus 2 from the input video and audio data, and outputs a video and audio stream that is a continuous network packet. Also, the video / audio stream received from another video processing apparatus is changed to video / audio data of TS which is a format for processing by the separation unit 119.

The video / audio stream is obtained by adding header data such as time information generated by the stream processing unit 105 and video / audio format information to the video / audio data.

The storage unit 106 is for storing a program to be executed by the control unit 101.

The video processing units 107-1 and 107-2 control the video input terminal 112 and the video output terminal 113, respectively, and output the video signal input from the video input terminal 112 to the encoder 103-1, or the decoder 104-1. The video signal input from is output to the video output terminal 113. The video processing unit 107-2 can simultaneously process a plurality of video signals from the video output terminal 112.

The audio processing units 108-1 and 108-2 control the audio input terminal 114 and the audio output terminal 115, respectively, and output the audio signal input from the audio input terminal 114 to the encoder 103-2 or the decoder 104-2. The audio signal input from is output to the audio output terminal 115.

The remote control processing unit 109 is for outputting a remote control signal input from the remote control input terminal 116 to the control unit 101 as operation information.

The network connection unit 110 transmits / receives a video / audio stream and connection information necessary for performing a video conference with the network connection terminal 117 and other video conference communication apparatuses connected via the network 3.

The video input terminal 112 is connected to the camera 4 and outputs the video signal input from the camera 4 to the video processing unit 107-2. The video input / output terminal 112 can also connect a plurality of cameras and simultaneously output a plurality of video signals to the video processing unit 107-2.

The video output terminal 113 is connected to the monitor 5 and outputs the video signal input from the video processing unit 107-1 to the monitor 5.

The audio input terminal 114 is connected to the microphone 6 and outputs an audio signal input from the microphone 6 to the audio processing unit 108-2.

The audio output terminal 115 is connected to the speaker 7 and outputs the audio signal input from the audio processing unit 108-1 to the speaker 7.

FIG. 3 is a diagram showing a specific example of a program read from the storage unit 106 of the video processing apparatus in FIG. 2 and developed in the memory 102, where 301 is a face detection unit, 302 is a virtual viewpoint calculation unit, and 303. Is an angle calculation unit, 304 is a face area calculation unit, 305 is a correction processing amount calculation unit, 306 is a correction necessity calculation unit, 307 is a correction processing unit, and 309 is a processing load management unit.

The face detection unit 301 controls each unit such as the video processing unit 107-2, detects a human face and facial organs from the video signal captured by the camera 4, and extracts the coordinates of facial feature points. Or a program for specifying the face of a person who is speaking.

The virtual viewpoint calculation unit 302 is a program for correcting the video signal of the camera 4 and calculating the coordinates of the virtual viewpoint for generating a video shot from a viewpoint different from the viewpoint of the camera 4. is there.

The angle calculation unit 303 calculates a line segment connecting the face and the camera 4, the face, and the virtual viewpoint from the coordinates of the person's face obtained by the face detection unit 301 and the coordinates of the virtual viewpoint obtained by the virtual viewpoint calculation unit 302. This is a program for calculating an angle formed by connecting line segments.

The face area calculation unit 304 is a program for calculating the area ratio of the face in the captured video from the coordinates of the feature points of the person's face obtained by the face detection unit 301.

The correction processing amount calculation unit 305 is a program for estimating and calculating a calculation processing amount necessary for correcting the face of the person obtained by the face detection unit 301 into a face image captured from a virtual viewpoint. is there.

The correction necessity degree calculation unit 306 calculates the necessity degree of face correction in order to obtain the effect of matching the line of sight from the information obtained by the face detection unit 301, the face angle calculation unit 303, and the face area calculation unit 304. It is a program.

The correction processing unit 307 corrects the face of a person in the video signal captured by the camera 4 from information obtained by the face detection unit 301, the virtual viewpoint calculation unit 302, the angle calculation unit 303, and the like, and captures the image from the virtual viewpoint. It is a program to make such a picture.

The processing load management unit 309 is a program for managing the processing load status of the video processing device 1 in real time and calculating the processing amount when the correction processing unit 307 performs processing.

Hereinafter, the line-of-sight processing method of the video processing apparatus 1 will be described with reference to FIGS. 4, 5, 6, and 7.

The reason why the line of sight does not match in a videophone or a video conference occurs because the position of the camera is different from the position of the participant's line of sight. As to what is acceptable in the camera installation position and the line of sight, “Study on the position of the image pickup tube in the video telephone, Sho 42 Rendai, No. 1998” announced by Sato, Miura and Nagata. (1967) "describes the allowable range of limit angles.

For this reason, it is determined whether the line-of-sight shift is within an allowable range, and if it is within the allowable range, face correction is not performed, thereby reducing the processing load without causing a sense of incongruity of the line-of-sight shift. Can do.

FIG. 4 is a diagram of a determination list for determining whether or not the line-of-sight shift is within an allowable range when the video processing device 1 performs the line-of-sight process. The determination list is generated on the memory 102 by the control unit 101. 700 is a determination list, 701 is an ID for identifying the face detected by the

face detection unit

301, 702 is a shift angle that expresses the shift of the line of sight as an angle, and 703 is a correction that quantifies the necessity of correcting the detected face. Necessity 704 is an estimated processing amount that is an estimated value of the amount of calculation processing required when executing the face correction processing.

ID 701 is a number assigned when the face detection unit 301 detects a face. The deviation angle 702 is a value calculated by the angle calculation unit 303. The correction necessity 703 is a value calculated by the correction necessity calculation unit 306. The estimated processing amount 704 is a value calculated by the correction processing amount calculation unit 305.

FIG. 5 is a flowchart showing a specific example of the main process of the series of line-of-sight processing. This process is executed by the control unit 101 every time the video processing apparatus 1 inputs a video signal. The subsequent programs handle the input video signal in units of video frames, which are one still image.

First, the face detection unit 301 detects a face included in the video frame (S901). The face detection unit 301 assigns a numerical value as an ID to the detected face and writes the ID in the determination list 700.

Next, the loop processing from S902 to S905 is performed for the number of detected faces. S902 and S905 indicate processing loop ends. In the loop process, one face that has not yet been processed is selected from the detected faces (S903). Next, a subroutine for determining the necessity of correcting the face is executed (S904).

Subroutine processing will be described later with reference to FIG. The deviation angle 702, the correction necessity 703, and the estimated processing amount 704 of the determination list 700 are calculated by a subroutine process.

When the processing for all faces is completed by the loop processing from S902 to S905, the estimated processing amount 704 of the determination list 700 is added together (S906). It is determined whether the combined estimated processing amount does not exceed the limit processing amount that can be processed in real time by the current control unit 101 (S907).

The limit processing amount is a value calculated by the processing load management unit 309 every time the flowchart of FIG. 5 is executed. This value is, for example, a processing amount necessary for performing the line-of-sight processing at a speed of 29.97 video frames per second. When processing exceeding this value is performed, the number of video frames that can be processed per second is reduced, resulting in dropped frames. In a state where the processing load of the control unit 101 is large, the limit processing amount becomes much smaller.

Note that the number of video frames to be processed per second, which is a reference for calculating the limit processing amount, is set to other values such as 30 and 60 according to the number of video frames to be displayed per second of the video processing device 1. can do. Alternatively, the processing amount obtained by subtracting a predetermined processing amount from the maximum processing amount of the control unit 101 may be used as the limit processing amount, or the limit processing amount may be set using another criterion.

When the estimated processing amount added in S906 exceeds the limit processing amount (Y in S907), the face to be corrected is selected using the determination list 700 (S908). If not exceeded (N in S907), correction processing is performed on all faces corresponding to IDs 701 whose correction necessity level is other than 0 in the determination list 700 (S909).

In S908, the following processing is performed. The estimated processing amounts are added together in descending order of necessity of correction in the determination list 700, and faces up to a point where the limit processing amount calculated by the processing load management unit 309 is not selected are selected. In the determination list 700 of FIG. 4, for example, when the limit processing amount is 25, faces whose IDs are 1 and 2 are selected so that the total value of the estimated processing amounts 704 does not exceed 25.

When S907 is N or when the processing of S908 is completed, the selected face is corrected (S909). The face correction process is performed by processing the face image so that the line of sight matches. There are several face correction processing methods, and the effects of the present embodiment can be obtained by any method. Therefore, not all of them will be described here. However, here, as an example, the cameras 4 are respectively installed at the upper and lower portions of the monitor 5. A method of correcting a face taken from an arbitrary position between two cameras using a face image taken from two different angles will be described.

In step S901, face detection is performed for each of the video frames input from the two cameras, and feature points of facial organs such as eyes, nose, and mouth are extracted. Using the extracted feature points, the faces in the two video frames are matched. Two matched faces are extracted from the video frame, and the two faces are synthesized by morphing processing. At this time, the morphing ratio is adjusted so that the line of sight meets the angle.

At this time, the morphing ratio may not be an angle at which the line of sight meets but a maximum angle within an allowable range of the line of sight deviation. In the face correction process, if the face photographed by the camera is greatly deformed, the sense of incongruity due to the correction may be greater than the sense of incongruity due to lack of line of sight. If it is possible to reduce the amount of deformation of the face by setting the maximum angle within the allowable range of line of sight rather than the angle at which the line of sight meets, use the maximum angle within the allowable range of line of sight. The ratio, that is, the correction amount of the face may be determined.

Finally, paste the corrected face on the face in the video frame taken with the upper camera.

As another correction processing method, a TOF (Time of Flight) camera is used to generate a 3D model of the subject's face and rotate the 3D model so that the face has an angle that matches the line of sight. There is a way to correct it.

The video frame on which the control unit 101 has performed line-of-sight processing is output to the video encoder 103-1, and is transmitted to the video processing device 2 as a video / audio stream by the multiplexing unit 118 and the stream processing unit 105.

Next, a method for obtaining the gaze shift angle will be described.

FIG. 6 is a diagram showing a specific example of the positional relationship between the camera, the monitor, and the video conference participants. 4-1 and 4-2 are cameras installed above and below the

monitor

5, 401, 403, and 406 are participants of the video conference, 407 is a conference table, 501 is a virtual viewpoint where the line of sight is eliminated, and 502 is a camera 4-1, a line segment connecting the center of the lens 4-1 and the

virtual viewpoint

501, 503 is a line segment connecting the center of the lens of the camera 4-1 and the eyes of the

participant

401, and 504 is the center of the lens of the camera 4-2 and the participant. A line segment connecting the

eyes

401, 505 is an angle formed by the line segment 502 and the

line segment

503, 506 is formed by a line segment connecting the center of the lens of the camera 4-1 and the center of the lens of the camera 4-2, and the line segment 504. An angle, 507 is a line segment connecting the virtual viewpoint 501 and the eyes of the

participant

401, and 508 is an angle formed by the line segment 505 and the line segment 507, which is an angle indicating a line-of-sight shift.

Whether the line-of-sight shift is within the allowable range can be determined by calculating the angle 508. There are several methods for calculating this angle, and any method can be used to obtain the effect of this embodiment. Therefore, all the methods will not be described, but here two cameras are used as an example. A method to be used will be described. Subsequent processing for obtaining the angle 508 is performed by the angle calculation unit 303.

In order to obtain the angle 508, it is only necessary to know the triangle formed by the line segment 502, the line segment 503, and the line segment 507. This triangle can be obtained if the numerical values of the line segment 502, the line segment 503, and the angle 505 are known. The coordinate system of FIG. 6 is a two-dimensional plane perpendicular to the ground and the screen of the monitor 5, and the origin is the center on the screen of the monitor 5.

The length of the line segment 502 can be obtained if the distance between the virtual viewpoint 501 and the center of the camera 4-1 lens is known.

The position of the virtual viewpoint 501 can be rephrased as the position that the participant should see in order to match the line of sight. The position to be viewed may be, for example, a position where the eyes of the participant on the other side of the video conference are displayed. In a situation where one of the participants on the other side is speaking, the position where the participant's eyes are to be viewed may be set.

On the other hand, when the participant on the other side is a listener, the position to be viewed may not be one place. The position to be viewed here may be simplified and may be the center on the screen of the monitor 5, but the display position of the participant's eyes is calculated by the virtual viewpoint calculation unit 302 and the position to be viewed can be set by this. good.

If the position of the lens of the camera 4-1 does not change during use of the video processing apparatus, a value measured in advance may be used. The value is stored in the storage unit 106. When the position of the camera 4-1 is variable, the value may be changed according to the position.

As described above, the length of the line segment 502 is determined from the distance between the virtual viewpoint 501 and the center of the lens of the camera 4-1.

The length of the line segment 503 can be obtained from the distance between the center of the lens of the camera 4-1 and the center of the lens of the camera 4-2, and the angle 505 and the angle 506 in the manner of triangulation. If the distance between the lens centers of the camera 4-1 and the camera 4-2 does not change during use of the video processing device, a value measured in advance may be stored in the storage unit of the video processing device. When the positions of the cameras 4-1 and 4-2 are variable, the values may be changed according to the positions.

The angle 505 can be obtained from the depression angle and angle of view of the camera 4-1, and the position of the participant's eyes on the video frame. The coordinate system of the eye position is a two-dimensional plane on the video frame. The depression angle of the camera is 0 when the camera orientation is horizontal and −90 degrees when the camera orientation is directly below.

If the position of the camera 4-1 does not change during use of the video processing device, the values measured in advance may be stored in the storage unit of the video processing device 1. The position of the participant's eyes on the video frame can be obtained by the face detection unit 301.

As an example, when the depression angle of the camera 4-1 is −30 degrees and the field angle is 46 degrees, the value of the angle 505 is as follows. If the participant's eyes are at the center of the video frame, the angle 505 is 60 degrees. If the participant's eyes are at the top of the video frame, the angle 505 is 83 degrees.

Since the angle 506 can be obtained from the elevation angle and angle of view of the camera 4-2 and the position of the participant's face, as with the angle 505, the description is omitted. Thus, the length of the line segment 503 is obtained.

The angle 508 is obtained from the triangle formed by the line segment 502, the line segment 503, and the angle 505 obtained so far.

Another method for obtaining the angle 508 is to use a TOF camera for distance measurement in addition to the camera for shooting. The installation position and the depression angle of the TOF camera are set to be as identical as possible with the camera 4-1. The method for obtaining the line segment 502 and the method for obtaining the angle 505 are the same as those already described. The line segment 503 is obtained by measuring the distance to the participant 401 with a TOF camera. Since the distance measurement method using the TOF camera is a well-known technique, the description thereof is omitted. The angle 508 is obtained from the triangle formed by the line segment 502, the line segment 503, and the angle 505 obtained so far.

FIG. 7 is a flowchart of the subroutine processing executed in S904 of FIG. In the subroutine, processing for calculating the degree of necessity for face correction is executed.

When the subroutine is executed from the main process, a face shift angle is first calculated (S1001). The deviation angle is determined by the angle detection unit 303 calculating the angle 508 of FIG.

Next, it is determined whether the calculated shift angle is equal to or greater than the specified angle (S1002). The designated angle is stored in the storage unit 106 and is information set in advance in the video processing apparatus 1 for the determination process in S1002. The value of the designated angle is, for example, the maximum angle in the allowable range of gaze deviation already described. Here, as an example, the designated angle is 9 degrees.

If the deviation angle is greater than or equal to the specified angle (Y in S1002), a correction necessity degree is calculated by quantifying whether or not to correct (S1003). The degree of necessity for correction is a value calculated by the degree of necessity correction calculator 306, and the calculated value is written in the degree of necessity for correction 703 of the determination list 700. It shows that it should correct | amend so that the numerical value of a correction necessity is large. The calculation method of the degree of necessity of correction is 0 when the deviation angle is less than the threshold value, and the deviation angle value when the deviation angle is greater than or equal to the threshold value.

Next, an estimated processing amount, which is a value obtained by estimating the processing amount when performing face correction processing, is calculated (S1004). The estimated processing amount is a value calculated by the correction processing amount calculation unit 305, and the calculated value is written in the estimated processing amount 704 of the determination list 700. The estimated processing amount is calculated by multiplying the face area calculated by the face area calculation unit 304 by a constant.

Here, the area of the face will be described. FIG. 8 is a diagram illustrating a specific example of a video frame shot by the camera 4 connected to the video processing device. 400 is a video frame, 401 to 406 are participants of a video conference, and 407 is a conference table. Of the participants, only the participant 402 is speaking.

As shown in FIG. 8, the camera 4 is installed at a position to capture the entire meeting scene, and the larger the participant in the front of the video, the smaller the participant in the back. The larger the apparent size, the larger the processing amount required for image processing, and the smaller the apparent size, the smaller the processing amount required for image processing.

When the processing of the subroutine of FIG. 7 ends, the processing returns to the processing of S905 of FIG. 5 already described, and the main processing for line-of-sight alignment is continued.

As described above, face correction that does not require correction is performed without selecting correction, and selecting and correcting faces in descending order of necessity of correction, thereby reducing processing load and eliminating frame dropping. Can be processed.

In the first embodiment, the necessity of correction is determined based on the shift angle, but in the second embodiment, the necessity of correction is determined based on the face area without using the shift angle.

The problem that the line of sight does not match is also related to the apparent size of the participant's face shown on the monitor 5. If the apparent size of the participant's face photographed by the camera 4 connected to the video processing device displayed on the monitor 5 connected to the video processing device at another base is small, whether or not the line of sight is present Since it becomes difficult to understand, the problem that the line of sight does not match does not occur.

For this reason, in the video processing apparatus, in order to reduce the load of the line-of-sight alignment process, participants with a small apparent size are excluded from the face correction process.
The content of the line-of-sight process is almost the same as that of the first embodiment, but a part of the determination list and the flowchart are different, and therefore will be described with reference to FIGS.

FIG. 13 is a determination list in the second embodiment. 1300 is a judgment list, and 1301 is an area ratio. The other items are the same as those in FIG. The area ratio 1301 is a ratio of the face area to the entire screen, and is calculated by the face area calculation unit 304.

The correction necessity degree 703 of the determination list 1300 is calculated based on the area ratio 1301 by the correction necessity degree calculation unit 306. Here, when the area ratio of the face is less than the predetermined area ratio, the necessity of correction is set to zero. If the area ratio is greater than or equal to the area ratio, the area ratio value is used. Note that, instead of the face area ratio, it may be determined whether or not the correction necessity is set to 0 according to the face area.

FIG. 14 is a flowchart of the subroutine processing in the second embodiment, which is executed from S904 in FIG. In the subroutine, the degree of necessity for face correction is calculated.

When the subroutine is executed from the main process, the face area ratio is first calculated (S1401). Next, it is determined whether or not the calculated area ratio is greater than or equal to the specified threshold area ratio (S1402). The area ratio of the threshold is stored in the storage unit 106 and is information set in advance in the video processing device 1 for the determination process in S1402. Here, as an example, the threshold value is 1.0%.

If the area ratio is equal to or greater than the threshold value (Y in S1402), the degree of necessity for correction is calculated by quantifying whether to correct (S1403). In S1403, as already described, the correction necessity is calculated using the area ratio. The calculated value is written in the determination list 1400.

Since the processing after S1403 and the processing after N in S1402 are the same as in the first embodiment, description thereof will be omitted.

As described above, by using the area of the face in the video frame, the face is not corrected for the face with low necessity for face correction, and the face is selected and corrected in the order of the necessity of face correction. It is possible to perform line-of-sight processing that suppresses the load and does not cause frame dropping.

In addition, as a method of determining whether or not face correction is necessary, a method of correcting only the face of the speaker who is speaking detected by the face detection unit can be used. Thereby, only the speaker's face needs to be corrected, and the line-of-sight processing can be further reduced.

In the first embodiment, the degree of necessity of correction is calculated using the deviation angle. However, when the values of the deviation angles of two faces are close or the same, the degree of correction is close or the same. Therefore, in the third embodiment, the area ratio and the like in addition to the deviation angle are used for calculating the degree of correction. As a result, even when there are a plurality of faces having the same shift angle, a difference occurs in the necessity of correction, and it is possible to select a face that needs more correction with high accuracy.

Therefore, as a method for calculating the degree of necessity for correction, a method using a combination of information on the angle of deviation, the area ratio, and whether or not the speaker is used will be described.
FIG. 9 is a determination list in the case where the correction necessity is calculated by combining a plurality of conditions. Reference numeral 800 denotes a determination list, reference numeral 801 denotes a speaker flag indicating whether the speaker is a speaker, and reference numeral 802 denotes an area ratio. The other items are the same as those in FIG. In FIG. 4, the correction necessity degree is calculated only from the deviation angle, but here, the necessity degree of correction is calculated from the speaker flag 801, the area ratio 802, and the deviation angle 702.

The calculation method for the necessity of correction can be changed depending on which condition is important. As an example, if expressed simply by mathematical formulas, if the speaker flag x, the area ratio y, and the shift angle z are set to a, b, and c, respectively, the correction necessity degree is r = ax + by + cz. The mathematical formula and the value of the constant can be changed depending on which condition is emphasized.

The selection of the correction target and the correction process are realized by the flowchart of FIG. 5 and the flowchart of FIG. FIG. 5 has already been described in the first embodiment, and the processing content is the same, so that description thereof is omitted.

FIG. 10 is obtained by adding some processes to the flowchart of FIG. The processing to be added is S1101 and S1102. First, the area ratio is calculated (S1101), and the calculated value is written in the area ratio 802 of the determination list 800. It is determined whether the calculated area ratio is greater than or equal to the specified area ratio (S1102). If the determination is true (Y in S1102), a shift angle detection process (S1001) is performed. If the determination in S1102 is false (N in S1102), the correction necessity level is set to 0 (S1005). The other processes in FIG. 10 are the same as those described in FIG.

In the third embodiment, the degree of necessity for correction is calculated based on three factors: whether the face to be corrected is a speaker, the area ratio of the face to be corrected, and the angle of gaze shift of the correction target. Or you may calculate based on four or more elements. In addition, the degree of necessity for correction may be calculated using elements other than whether the face to be corrected is a speaker, the area ratio of the face to be corrected, or the angle of gaze deviation of the correction target.

As described above, the face that needs to be corrected can be selected with higher accuracy.

In the first to third embodiments, after calculating the necessity level of correction, the processes of S906 to S908 are further performed to correct the face. However, the processes of S906 to S908 are omitted and the degree of correction requirement is other than 0. You may comprise so that a correction process may be performed with respect to a face.

As a result, it is possible to prevent the correction process from being performed on a face that is determined not to require correction by a simpler method, thereby reducing processing in the video processing apparatus and reducing video disturbance. can do.

In the first to fourth embodiments, the video processing apparatus has been described as a video conference apparatus. However, as another form, the video processing apparatus may have a system configuration in which a video conference is performed in combination with another video conference apparatus.

FIG. 11 shows a configuration in which a video processing device and another video conference device are combined.

Reference numerals

1111 and 1112 denote video processing apparatuses, and 1201 and 1202 denote video conference apparatuses. In the figure, a video processing device 1111 is connected to the camera 4 and the video conference device 1201 and can input and output video. Note that the video processing apparatus 1112 has the same apparatus and configuration as the video processing apparatus 1111.

In this configuration, the video processing device 1111 inputs a video signal captured by the camera 4. The video processing device 1111 performs line-of-sight processing on the input video signal and outputs the processed video signal to the video conference device 1201. Since the video processing device 1111 only needs to perform line-of-sight processing, a connection function to the network 3 related to a video conference and a function of encoding and decoding a video signal are not necessary.

By adopting this configuration, the video conference device can be configured to connect the video processing device to the input terminal to which the camera is normally connected. Therefore, even the video conference device without a line-of-sight function is connected. Thus, the effect of line-of-sight alignment can be obtained.

In the first to fifth embodiments, the configuration in which one video processing device is required for each video conference base has been described. However, as another form, there is a method for performing line-of-sight processing with a video processing device on a network. Also good.

FIG. 12 shows a configuration in which the video processing apparatus is connected to another video conference apparatus via the network 3. 1200 is a video processing apparatus. The video processing apparatus 1200 in this embodiment receives a video / audio stream from the video conference apparatus 1201 via the network 3, converts the video / audio stream into video data and audio data, and decodes the video data. Then, the video signal is converted into a video signal, and the line-of-sight processing is performed on the video signal. Next, the video signal subjected to the line-of-sight processing is encoded into video data, and the video data and audio data are multiplexed and packetized, converted into a video stream, and output to the video conference device 1202.

Similarly, the video processing apparatus receives the video / audio stream from the video conference apparatus 1202 via the network 3, performs similar processing including line-of-sight processing, and outputs the video / audio stream to the video conference apparatus 1201. To do.

By adopting such a configuration, it is not necessary to install a video processing device at each base, and the effect of line-of-sight alignment can be obtained even at a base without a video processing device.

1, 2 Video conference communication device 3 Network 4, 4-1, 4-2 Camera 5 Monitor 6 Microphone 7 Speaker 8 Remote control 101 Control unit 102 Memory 103-1, 103-2 Encoder 104-1, 104-2 Decoder 118 Multiplex Conversion unit 119 separation unit 105 stream processing unit 106 storage unit 107-1, 107-2 video processing unit 108-1, 102-2 audio processing unit 109 remote control processing unit 110 network connection unit 112 video input terminal 113 video output terminal 114 audio Input terminal 115 Audio output terminal 116 Remote control input terminal 117 Network connection terminal 118 Multiplexing unit 119 Separation unit 301 Face detection unit 302 Virtual viewpoint calculation unit 303 Angle calculation unit 304 Face area calculation unit 305 Correction processing amount calculation unit 306 Correction necessity calculation Part 307 correction processing part 309 processing negative Management unit 400 image frames 401 and 402, ..., 406 person 407 conference table 501 the virtual viewpoint 700 determines list 701 ID
702 Deviation angle 703 Correction necessity 704 Estimated processing amount 800 Determination list 801 Speaker 802

Area ratio

1111, 1112 Video processing device 1200 Video processing device 1201 Video conference device 1202 Video conference device

Claims

A video input unit for inputting video information captured by the camera;
A video processing unit for correcting a video of a face included in the video information input to the video input unit;
An output unit for outputting video information processed by the video processing unit,
The video processing apparatus, wherein the video processing unit determines whether or not correction of a face video included in video information is necessary according to a predetermined condition.
2. The video processing apparatus according to claim 1, wherein the predetermined condition is a face gaze shift angle included in the video information.
2. The video processing apparatus according to claim 1, wherein the predetermined condition is a face area or an area ratio included in the video information.
2. The video processing apparatus according to claim 1, wherein the predetermined condition is whether or not a face included in the video information is a speaker.
The video processing device according to claims 1 to 4, comprising:
An image processing apparatus, wherein when a processing amount in the image processing unit is larger than a predetermined processing amount, a face image to be corrected is selected based on the predetermined condition.
A video processing method in a video processing apparatus for processing video information,
A video input step for inputting video information captured by the camera to the video processing device;
A video processing step for correcting a face image included in the input video information;
Outputting the video information processed in the video processing step,
A video processing method characterized in that, in the video processing step, it is determined whether correction of a video of a face included in video information is necessary according to a predetermined condition.
7. The video processing method according to claim 6, wherein the predetermined condition is a shift angle of a face line of sight included in the video information.
7. The video processing method according to claim 6, wherein the predetermined condition is a face area or an area ratio included in the video information.
7. The video processing method according to claim 6, wherein the predetermined condition is whether or not the face included in the video information is a speaker.
The video processing method according to claim 6-9,
An image processing method comprising: selecting a face image to be corrected based on the predetermined condition when a processing amount in the image processing step is larger than a predetermined processing amount.