JP4863937B2 - Encoding processing apparatus and encoding processing method - Google Patents

Description

  The present invention relates to an encoding processing apparatus and encoding processing method for encoding video.

  With the widespread use of broadband networks, it is becoming increasingly popular to distribute video and audio streams over the Internet. Stream distribution is used for Internet services such as Internet telephony, remote video conferencing, and chat. In a chat system that uses video and audio streams, the user's face image and sound at a remote location are transmitted to each other over the network, and the video and sound are played back on the user's terminal. You can chat as if you were.

  In video stream distribution, a video stream generated by compressing and encoding a frame image of a moving image using a moving image encoding method such as MPEG (Moving Picture Experts Group) is stored in an IP (Internet Protocol) packet and transferred over the Internet. This is realized by allowing the user's communication terminal to receive it. Since the Internet transfers packets on a best effort basis, when the network is congested, packets may be discarded or delayed, and data may be lost, and frame images may not be received correctly.

  Therefore, the bit rate of the encoded video or audio stream is adjusted according to the network bandwidth. In addition, by setting a region of interest (ROI) in the image, assigning enough bits to the region of interest, and encoding with fewer bits allocated to the non-region of interest, the network bandwidth is reduced, Ingenuity has been devised to avoid congestion or to ensure reproduction quality at least in the attention area even during congestion.

For example, Patent Document 1 discloses an image encoding method in which encoding is performed by changing the compression rate of the attention area and the compression ratio of the remaining area.
JP 2005-295379 A

  In an application such as a chat using a user's face image, it is an important factor that influences the satisfaction level of the user that the face images of communication partners are good in appearance. Therefore, by automatically detecting the user's face displayed in the image, setting the detected face area as the attention area, and generating a video stream that encodes the attention area with high image quality, the reproduction quality of the face area It is done to ensure. However, the face area may not be detected correctly because the performance of the camera used in the chat is low or the user moves too fast, and the face image may not be provided with sufficient quality to the receiver. In addition, an area that is not actually a face may be erroneously detected as a face area and set as an attention area, causing inconvenience.

  The present invention has been made in view of these problems, and an object thereof is to provide a moving image encoding technique for appropriately encoding a region of interest of a moving image.

  In order to solve the above problems, an encoding processing device according to an aspect of the present invention includes a detection unit that detects a face area in a moving image frame, and a recording unit that records a history of the detected face area in units of frames. The detection of the face area by the detection unit by tracking the face area detected by the detection unit over a plurality of consecutive frames with reference to the detection history of the face area in units of frames recorded in the recording unit A tracking unit that corrects a result, a region of interest determination unit that determines a region of interest based on a predetermined area based on the face region corrected by the tracking unit, and a code of the region of interest different from other regions in image quality And an encoding unit that generates an encoded stream of a moving image.

  Another aspect of the present invention is a program. This program refers to a detection function for detecting a face area in a frame of a moving image, a recording function for recording the history of the detected face area in units of frames, and a detection history of the recorded face area in units of frames. A tracking function that corrects the detection result of the face area by tracking the detected face area over a plurality of consecutive frames, and an attention area determination function that determines the attention area based on a predetermined standard based on the corrected face area And the encoding function for generating the encoded stream of the moving image by encoding the attention area with different image quality from the other areas.

  This program may be provided as a part of firmware incorporated in the device in order to perform basic control of hardware resources such as video and audio decoders. This firmware is stored, for example, in a semiconductor memory such as a ROM or a flash memory in the device. In order to provide the firmware or to update a part of the firmware, a computer-readable recording medium storing the program may be provided, and the program may be transmitted through a communication line. Good.

  It should be noted that any combination of the above-described constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a data structure, a recording medium, and the like are also effective as an aspect of the present invention.

  According to the present invention, it is possible to appropriately set and encode an attention area of a moving image.

  FIG. 1 is a configuration diagram of a chat system according to an embodiment. Microphones 230a to 230c, cameras 240a to 240c, speakers 250a to 250c, and displays 260a to 260c are connected to a plurality (three in this case) of information processing apparatuses 100a to 100c, respectively. The plurality of information processing apparatuses 100 a to 100 c are connected to the network 300. A plurality (three in this case) of users A to C use their respective information processing apparatuses 100a to 100c to transmit their face images and sounds to each other in real time via the network 300, and are input from the keyboard. By exchanging text data, communication using audio and video (so-called chat) is performed.

  Hereinafter, when generically referring to the configuration of the information processing apparatuses 100a to 100c of each user, the reference signs a to c are omitted, and are simply indicated by the reference numeral 100 or the like.

  FIG. 2 is a configuration diagram of the information processing apparatus 100. Here, the configuration related to chat is omitted, and the configuration related to encoding and decoding of audio and video is shown.

  The information processing apparatus 100 includes an encoding processing block 200, a decoding processing block 220, and a communication unit 270. As an example, the information processing apparatus 100 may be a personal computer, a portable terminal, or a multiprocessor system. When the information processing apparatus 100 is a personal computer, the encoding processing block 200 and the decoding processing block 220 may be realized by separately mounting a dedicated circuit having an image encoding / decoding function in the personal computer. Further, when the information processing apparatus 100 is a multiprocessor system, the high processing capability of the multiprocessor can be used, so the encoding processing block 200 and the decoding processing block 220 may be realized by software.

  The encoding processing block 200 compresses and encodes the audio input to the microphone 230 and the moving image shot by the camera 240 to generate an audio encoded stream and a video encoded stream. The audio encoded stream and the video encoded stream can be multiplexed into a single stream. The audio encoded stream and video encoded stream generated by the encoding processing block 200 are packetized by the communication unit 270 and transmitted to the chat partner via the network 300.

  The communication unit 270 receives audio encoded stream and video encoded stream packets from the chat partner via the network 300 and supplies the packets to the decoding processing block 220. The decoding processing block 220 decodes the received audio encoded stream and video encoded stream to reproduce audio and video, and outputs them to the speaker 250 and the display 260, respectively.

  FIG. 3 is a configuration diagram of the encoding processing block 200. This figure depicts a block diagram focusing on functions, and these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  Each frame of the moving image captured by the camera 240 is stored in the frame memory 10. The display control unit 14 reads the frame from the frame memory 10 in accordance with the vertical synchronization signal of the display 260 and causes the display 260 to display the frame.

  The face area detection unit 12 detects an area in which a human face is projected in a frame stored in the frame memory 10. Existing technology is used for this face area detection. Several human face feature patterns are prepared in advance, and a face region is detected by searching for a region having features similar to the feature pattern in the frame. The facial features can be obtained, for example, by extracting the contour of the face and the shape and position of the characteristic parts such as eyes, nose and mouth by edge extraction processing.

  A plurality of face regions may be detected. The face area detection unit 12 generates position information for each face area. If the face area is a rectangular area, the position information of the area is represented by, for example, the coordinate value of the left corner point, which is a representative point, and the vertical and horizontal sizes of the area. The face area detection unit 12 gives the number and position information of the face areas detected by the known face detection algorithm in this way to the ROI specifying unit 18.

  The ROI specifying unit 18 first verifies the validity of the face area detection result by the face area detection unit 12. Even if a face is shown in the frame, it is not always detected as a face area by the face detection algorithm, and even if the face is not shown, it may be erroneously determined to be a face area. is there. This is because the processing performance and shooting resolution of the camera 240 are limited, and it is impossible to follow the fast movement of the user, and the face feature cannot be picked up in the face detection process because the user faces sideways. Therefore, in order to improve the accuracy of face detection, a score that indicates the likelihood of a face is evaluated to reduce false detection, or a detection history of the face area is used to prevent omission of detection. A device for improving the accuracy of face detection will be described later.

  Next, the ROI specifying unit 18 specifies, as the attention area, an area for which visual quality is desired to be improved based on the face area whose validity has been verified. An area including the peripheral area centered on the face area may be set as the attention area. For example, an area including the face and the upper half of the detected face area is set as the attention area.

  The region of interest is not limited to a rectangular shape, and may be an arbitrary shape. The shape of the region of interest is specified by mask information. For example, when the heart-shaped shape is specified by the mask information, the ROI specifying unit 18 specifies the heart-shaped region as the attention region around the face region detected by the face region detecting unit 12.

  The ROI specifying unit 18 generates ROI information including the number of regions of interest and position information, and provides the ROI information to the non-ROI filter 22 and the video encoder 24. Further, the ROI specifying unit 18 gives the ROI information to the multiplexing unit 32 when it is desired to include the ROI information in the multiplexed stream. Including ROI information in the multiplexed stream is optional. For example, when it is desired to use the ROI information on the receiving side, the ROI information may be included in the multiplexed stream.

  At the time of video encoding by the video encoder 24, the non-target area is a low bit allocation area and the target area is a high bit allocation area. That is, the quality of the attention area is made higher than the quality of the non-attention area by assigning a larger number of bits to the attention area and encoding than the non-attention area. For this purpose, the ROI specifying unit 18 determines a bit allocation strength β indicating the ratio of the number of bits allocated to the attention region to the number of bits allocated to the non-target region, and gives the bit allocation strength β to the video encoder 24. The bit allocation strength β takes a value of 1 or more. When β is 1, the bit allocation amount is the same in the non-attention area and the attention area, and when β is larger than 1, the non-interest area depends on the size of β. By reducing the bit allocation amount of the attention area, the bit allocation amount of the attention area is relatively increased.

  Further, the ROI specifying unit 18 determines a blur strength γ indicating the degree of blurring of the image by comparing the non-target region with the target region, and provides the non-ROI filter 22 with the blur strength γ. The non-ROI filter 22 visually blurs the non-focused area by applying filtering that removes high-frequency components to the non-focused area based on the blur intensity γ. The blur intensity γ takes a value of 1 or more. When γ is 1, no blur process is performed, and when γ is 1 or more, the degree of the blur process is increased according to the magnitude of γ.

  The band information acquisition unit 20 acquires band information such as a bit rate and a congestion state of the communication path from the communication unit 270, and gives the acquired band information to the ROI specifying unit 18 and the video encoder 24. The ROI specifying unit 18 refers to the band information and adjusts the bit allocation strength β and the blur strength γ. The video encoder 24 adaptively adjusts the bit rate of the video stream with reference to the band information.

  The video encoder 24 receives the filtered image from the non-ROI filter 22 and, as an example, compresses and encodes video data according to the MPEG standard to generate an encoded video stream. The video encoder 24 refers to the ROI information received from the ROI identification unit 18 to identify the attention area, encodes the non-attention area and the attention area with quality based on the bit allocation strength β, and encodes the encoded video stream into the video packet. To the conversion unit 26.

  As an example, the audio encoder 28 compresses and encodes audio data input from the microphone 230 in accordance with a standard such as MPEG audio, generates an encoded audio stream, and supplies the encoded audio stream to the audio packetization unit 30.

  The stream encoded by the video encoder 24 and the audio encoder 28 is called an elementary stream (ES). For multiplexing, video and audio streams are packetized.

  For example, the video packetization unit 26 packetizes the encoded video stream output from the video encoder 24 into RTP (Real-time Transport Protocol) packets. Similarly, the audio packetization unit 30 packetizes the encoded audio stream output from the audio encoder 28 into RTP packets. RTP is a transmission protocol for streaming video and audio. Note that the encoded video / audio stream may be packetized into PES (Packetized Elementary Stream) packets.

  The multiplexing unit 32 multiplexes video and audio RTP packets to generate a multiplexed stream. The generated multiplexed stream is sent to the network 300 by the communication unit 270.

  FIG. 4 is a functional configuration diagram of the ROI specifying unit 18. The face verification unit 40 receives, from the face region detection unit 12, face position, face size, and facialness score information for the face region detected by the face region detection unit 12 in units of frames. The face-likeness score is a degree indicating the possibility that an image having facial features extracted by the face detection algorithm is really a face. The face verification unit 40 verifies the validity of the face area detected by the face area detection unit 12 based on the position and size of the face and the score of facialness. The face verification unit 40 records information on the face area determined to be valid in the face verification process as a history in the face area history storage unit 44 in units of frames. The face verification unit 40 provides the ROI determination processing unit 46 with information on the face area that has passed the face verification process.

  The tracking unit 42 is detected with reference to the detection history of the face area in units of frames recorded in the face area history storage unit 44 in order to eliminate erroneous detection and omission of detection of the face area by the face area detection unit 12. The detection result of the face area is corrected by tracking the face area over a plurality of continuous frames.

  Even when a face area is detected in a certain frame, the tracking unit 42 determines that the face area is not detected continuously in a predetermined number of consecutive frames after the frame. The detected face area detection history is determined to be invalid and is deleted from the face area history storage unit 44. Thus, the face area information is held in the face area history storage unit 44 as an effective detection history when the face area is continuously detected in a predetermined number or more of consecutive frames.

  On the other hand, when the face area in which the detection history exists in the face area history storage unit 44 continues to be detected in a predetermined number or more of consecutive frames, the tracking section 42 no longer has the face area. Since it is certain that it does not exist, it is determined that the detection history of the face area is unnecessary, and is deleted from the face area history storage unit 44.

  The tracking unit 42 determines that a face region that has passed the face verification process by the face verification unit 40 is a false detection if it does not match the face region detection history in the past frame. For example, when the position and size of the face area are significantly different from the position and size of the face area in the past frame, it is determined that there is no consistency.

  The tracking unit 42 does not determine whether a face area detected in a certain frame is erroneously detected based on only the detection history in the past frame, but also refers to information on the face area detected in frames subsequent to the determination target frame. Then, it may be determined whether it is a false detection. This is because it is more preferable to invalidate the detection history of the face region retroactively because the history of the erroneously detected face region will affect the determination result of the face region in the future frame.

  Specifically, the tracking unit 42 collates the detection result of the face area in a certain frame with the detection history of the face area in a predetermined number of frames before and after the frame, and the face area detected in the frame is erroneously detected. It is determined whether or not. For example, the tracking unit 42 collates a face area detected in a certain frame with face areas detected in frames before and after the frame with respect to the position and size of the face, and the position and size differ by a predetermined threshold or more. At this time, the face area detected in the frame is determined to be erroneous detection.

  The tracking unit 42 deletes the detection history of the face area determined to be erroneous detection from the face area history storage unit 44 and warns the face verification unit 40 of the erroneous determination. When receiving a false determination warning from the tracking unit 42, the face verification unit 40 discards even a face region that has passed the face verification process and does not give it to the ROI determination processing unit 46.

  In addition, the tracking unit 42 makes effective the detection history of the face area of the past frame recorded in the face area history storage unit 44 even when the face area detection unit 12 has not detected the face area in the current frame. If it exists and a face area has been detected in a past frame, it is determined that a face area detection failure has occurred in the current frame. The tracking unit 42 reuses the information on the position and size of the face area of the past frame as information on the position and size of the face area of the current frame, so that Interpolate face area information. The position and size of the face area of the current frame in which detection omission has occurred may be determined from the position and size of the face area for the past several frames. The tracking unit 42 records the face area information of the current frame interpolated in this way in the face area history storage unit 44 and warns the face verification unit 40 of a detection omission. When receiving the detection omission warning from the tracking unit 42, the face verification unit 40 reads out the information on the face area of the current frame interpolated from the face area history storage unit 44 and gives it to the ROI determination processing unit 46.

  For example, when the user performs an operation such as temporarily facing sideways or facing back, a frame in which a face area is not detected by an existing face detection algorithm may occur. Even in such a case, the tracking unit 42 reuses the detection result of the face area of the past frame for a frame in which the face area detection unit 12 misses the detection of the face region, so that the target region is not set correctly. Can be prevented.

  Furthermore, the imaging control unit may control pan / tilt of the camera 240 in accordance with the position of the face area tracked by the tracking unit 42. Further, the imaging control unit may control the zoom of the camera 240 in accordance with the size of the face area tracked by the tracking unit 42. Even if the user moves, the user's face can be captured by panning and tilting the camera 240. Even if the distance between the camera and the user changes, the user's face can be made a certain size in the screen by zooming in and out.

  The ROI determination processing unit 46 determines a final attention region from the information on the face region that has undergone the verification processing by the face verification unit 40 based on the determination criterion stored in the determination criterion storage unit 48. The attention area is determined according to the application and use case. Once the ROI determination processing unit 46 determines the attention area in a certain frame, the ROI determination processing unit 46 may continue to use the same attention area without determining and updating the attention area for a while after that. For example, instead of re-determining the attention area for each frame, the attention area may be reset at the change of GOP using the same attention area in units of GOP (group of picture). As a result, the load caused by the ROI determination process can be reduced, and it is only necessary to generate ROI information in units of GOPs.

  The ROI determination processing unit 46 provides the ROI information including the finally determined number of regions of interest and position information to the non-ROI filter 22, the video encoder 24, and the multiplexing unit 32. When an appropriate region of interest cannot be determined, filtering by the non-ROI filter 22 and ROI encoding by the video encoder 24 are not performed, and conventional video encoding is performed.

  The ROI parameter adjustment unit 50 determines ROI parameters such as the bit allocation strength β and the blur strength γ for the attention area finally determined by the ROI determination processing unit 46. When there are a plurality of attention areas, priority may be determined between the attention areas, and the bit amount to be allocated to the attention area may be determined according to the priority.

  The bit allocation strength β and the blur strength γ may be determined according to the size of the attention area. If the size of the region of interest is large, the bit rate of the video stream will increase if the bit allocation strength β is increased too much. Therefore, the bit rate of the video stream is optimized by reducing the bit allocation strength β for a large region of interest. In addition, since the effect of highlighting an extremely small face area or an extremely large face area may not be expected, the bit allocation intensity β and the blur intensity γ may be reduced in such a case.

  Further, the bit allocation strength β and the blur strength γ may be determined according to the position of the attention area. For example, if there is a region of interest at the edge of the image, the effect of highlighting may be small, so the bit allocation strength β and blur strength γ are reduced, and if there is a region of interest near the center of the image, it is highlighted Therefore, the bit allocation strength β and the blur strength γ are increased.

  Further, the bit allocation strength β and the blur strength γ may be determined according to the facialness score. If the face-like score is large, the effect of highlighting the face area can be expected. Therefore, the bit allocation strength β and the blur strength γ are increased. However, if the face-like score is low, the effect may be adversely affected. Therefore, the bit allocation strength β and the blur strength γ are reduced.

  The ROI parameter adjustment unit 50 can also adjust the bit allocation strength β and the blur strength γ based on the band information received from the band information acquisition unit 20. For example, if a sufficient bit rate is guaranteed for the frame size and frame rate of the video, such as there is sufficient available bandwidth because the network bandwidth is originally large or not congested, the non-target area It is not necessary to reduce the bit allocation, and the entire image may be encoded as a high bit allocation region regardless of the distinction between the attention region and the non-attention region. In this case, the bit allocation strength β is set to 1 and the blurring strength γ is set to 1.

  On the other hand, if you cannot guarantee a sufficient bit rate for the video frame size and frame rate, such as when the network bandwidth is limited or the bandwidth available due to congestion is low, the bit allocation strength β The bandwidth used is reduced by adjusting the blur intensity γ to a large value.

  The ROI parameter adjustment unit 50 provides the bit allocation strength β to the video encoder 24 and the blur strength γ to the non-ROI filter 22.

  Next, filter processing by the non-ROI filter 22 of FIG. 3 will be described in detail. The non-ROI filter 22 performs low-pass filtering on the non-attention region, thereby blurring the non-attention region and relatively highlighting the attention region. In general, when an image is compression-coded in the frequency domain, block noise increases as the bit rate decreases. In the video encoder 24, the non-attention area is encoded with a smaller number of allocated bits than the attention area, so that block noise is likely to occur. Therefore, for the non-target region, the non-ROI filter 22 performs filtering to remove high frequency components, thereby obtaining an effect of reducing block noise. The filtering by the non-ROI filter 22 has a secondary effect of reducing block noise in addition to visually blurring a region other than the region of interest.

  Further, since the high-frequency component is removed from the non-target region by non-ROI filtering, as a result, the effect of increasing the number of bits that can be allocated to the target region under a constant bit rate is also obtained.

  The attention area and the non-attention area are exclusive areas that do not overlap, and when the non-ROI filter 22 performs the process of blurring the non-attention area 440, the image quality is discontinuous at the boundary between the attention area 420 and the non-attention area 440. And the attention area 420 alone may stand out more than necessary, giving an unnatural impression. Therefore, a contrivance is made to eliminate discontinuity at the boundary between the attention area and the non-attention area.

  FIGS. 5A to 5C are diagrams illustrating a method for eliminating discontinuity at the boundary between the attention area and the non-attention area. As shown in FIG. 5A, the area surrounded by the thick line at the center of the image 400 is the attention area 420, and the remaining area is the non-attention area 440. A peripheral area 430 (a hatched area) is set on the outer edge of the attention area 420. The peripheral area 430 exists in the non-target area 440.

  The attention area 420 is ROI-encoded with high image quality by the video encoder 24 under the bit allocation strength β. On the other hand, in the non-target region 440, the high-frequency component is cut by the non-ROI filter 22 under the blur intensity γ. Since the peripheral area 430 provided at the outer edge of the attention area 420 exists in the non-focus area 440, the high-frequency component is cut by the non-ROI filter 22 under the blur intensity γ. Also performs ROI encoding under the bit allocation strength β. That is, in the peripheral area 430, the blurring process and the process for improving the image quality are overlapped. The peripheral area 430 is ROI encoded and the high frequency component is cut, so that the image quality is intermediate between the attention area and the non-attention area. Since the peripheral area 430 near the boundary between the attention area and the non-attention area has an intermediate image quality, unnaturalness at the transition between the attention area and the non-attention area can be reduced.

  As another method, the non-ROI filter 22 may change the image quality continuously by performing filtering while gradually increasing the blur intensity γ in the peripheral region 430. For this purpose, a non-uniform filter that takes a weighted average with a large weight applied to peripheral pixels located close to the target pixel for filtering and a small weight applied to peripheral pixels located far from the target pixel; As an example, a Gaussian filter may be used.

  As shown in FIG. 5B, a peripheral region 430 may be provided on the inner edge of the attention region 420. In this case, since the peripheral region 430 exists in the attention region 420, ROI encoding is performed under the bit allocation strength β, but the high frequency component is cut under the blur strength γ for the peripheral region 430. Processing is also performed. Alternatively, as shown in FIG. 5C, a peripheral area 430 is provided across both the outer edge and the inner edge of the attention area 420, and the process of increasing the image quality and the process of blurring are performed in the peripheral area 430. You may do it.

  FIG. 6 is a flowchart for explaining the processing procedure of ROI encoding by the encoding processing block 200.

  The face area detection unit 12 executes face area detection processing for the current frame (S10). If a face is detected by the face area detection unit 12 (Y in S12), the process proceeds to the face verification process in step S14. If no face is detected by the face area detection unit 12 (N in S12), the process proceeds to face interpolation processing by tracking in step S18.

  In step S14, the face verification unit 40 verifies whether the face area detected by the face area detection unit 12 is valid. The tracking unit 42 performs an erroneous detection determination process by tracking for the face area verified by the face verification unit 40 (S15). As a result, the erroneously detected face area is discarded. When the number of faces that have passed the verification process by the face verification unit 40 is 0 (N in S16), the process proceeds to the face interpolation process by tracking in step S18. If the number of faces that have passed the verification process is 1 or more (Y in S16), the process proceeds to the ROI determination process in step S24.

  In step S18, the tracking unit 42 refers to the face area information of the past frame, executes face interpolation processing by tracking, and interpolates the face area information missing in the current frame. If the number of interpolated faces is 1 or more (Y in S20), the process proceeds to ROI determination processing in step S24. If the number of interpolated faces is 0 (N in S20), the process proceeds to step S22. In this case, the attention area is not set.

  In step S24, the ROI determination processing unit 46 determines a final attention area based on the face area that has passed the verification process in step S14 or interpolated by tracking, and in step S26, ROI parameter adjustment is performed. The unit 50 adjusts ROI parameters for making the image quality of the attention area and the non-attention area different.

  When the next frame is input (Y in S28), the process returns to step S10, and a series of processes are repeated. When no frame is input (N in S28), the process is terminated.

  FIG. 7 is a flowchart for explaining the procedure of the face verification process in step S14 of FIG. Until all the face areas detected by the face area detecting unit 12 are verified (N in S30), the processes in steps S32 to S38 are repeated, and when all the detected face areas are verified (Y in S30), the face verification process is performed. End and proceed to step S15.

  The face verification unit 40 determines whether or not the size of the face included in the face area is appropriate (S32), whether the face position is appropriate (S34), and whether the score of the face likelihood is larger than the threshold (S36). ). If any test fails (N in S32, N in S34, or N in S36), the process returns to step S30. If all of these tests pass (Y in S32, Y in S34, and Y in S36), the information is stored in the face area history storage unit 44 as a face area that has passed verification (S38).

  In the face size test, it is determined whether the face size is within an assumed size range. For example, when the face size is too large or too small, it is not adopted as the face area. In the face position test, it is determined whether or not the face position is within the assumed position range. For example, when it is near the edge of the image, it is not adopted as the face area. In the face-likeness score test, it is determined whether or not the face-likeness score is within a range of assumed values. If the score is unexpectedly low, the face-likeness score is not adopted.

  FIG. 8 is a flowchart for explaining the procedure of erroneous detection determination processing by tracking in step S15 of FIG. Until all the face areas verified by the face verification unit 40 are checked (N in S50), the processes in steps S52 to S58 are repeated, and when all the verified face areas are checked (Y in S50), a false detection determination is made. The process ends, and the process proceeds to step S16.

  The tracking unit 42 checks whether or not the verified face area is continuously detected in a predetermined number of consecutive frames (S52). If the face area has been continuously detected (Y in S52), the process proceeds to step S56. When the face area is not continuously detected (N in S52), it is determined that the detection history of the face area is invalid, and is deleted from the face area history storage unit 44 (S54).

  Next, the tracking unit 42 checks whether or not the verified face area matches the detection history of the face area in the past frame (S56). If the face area matches the past frame detection history (Y in S56), the process returns to step S50. If the face area does not match the past frame detection history (N in S56), it is determined that the face area is erroneously detected, and the detection history of the face area determined to be erroneously detected is stored in the face area history storage unit 44. Delete (S58).

  FIG. 9 is a flowchart for explaining the procedure of face interpolation processing by tracking in step S18 of FIG. Until all the face information recorded as a history in the face area history storage unit 44 is checked (N in S40), the processes in steps S42 to S46 are repeated, and when all the face information in the history is checked (Y in S40). Then, the face interpolation process is terminated, and the process proceeds to step S20.

  The tracking unit 42 checks the number of frames in which no face has been detected (S42). If the number of frames in which no face has been detected is equal to or less than the threshold (N in S42), the process proceeds to face interpolation processing in step S46. This threshold is determined experimentally according to the frame rate, for example. For example, if the frame rate is 30 frames per second, the threshold is set to 1 to 3 frames with 1/10 of the frame rate as a guide.

  In step S46, if a face has been detected in the past frame, but no face has been detected in the current frame, it is determined that the face is not detected, and the current frame is located at the position detected in the past frame. Assuming that there is a face, information on the position and size of the face of the past frame is reused to interpolate the face information of the current frame and stored as face information of the current frame (S46). Using the motion vector information of the moving image, the face position of the current frame may be interpolated in the time direction from the face position of the past frame.

  If the number of frames in which no face has been detected is greater than the threshold (Y in S42), it is determined that no face area exists since the face has not been detected, and the face information detection history is deleted. (S44). For example, in the case of a frame rate of 30 frames per second, when a face is not detected for 3 frames or more with reference to 3 frames, which is 1/10 of the frame rate, information on the face area is deleted from the history. Since the threshold is set to 3 frames, the face area history information is not erroneously deleted even if a face area detection error occurs in one frame.

  Next, a method of determining a region of interest by the ROI determination processing unit 46 will be described in detail with an example.

  FIG. 10 is a diagram for explaining how attention areas are set in the image 400. Suppose a user is using video chat in his room. A first face area 420a and a second face area 410b are detected in the image 400. Since the first face area 420a includes the user's face 410a, it should be set as the attention area. The second face area 420b is the face of a person shown on a poster attached to the wall of the room. Should not be set as the attention area.

  Therefore, a criterion for not selecting a face area that does not move as an attention area may be provided. Accordingly, it is possible to prevent the face of a person on a wall poster or the face of a photograph in a photo stand on the desk from being erroneously selected as the attention area. In order to identify a face area that does not move, the tracking unit 42 examines the history of the face information held in the face area history storage unit 44, and whether the position of the face area has fluctuated compared to the past frame, It may be detected whether a part of the face such as eyes, nose, or mouth has changed on the image. The face verification unit 40 notifies the ROI determination processing unit 46 of the determination result of the movement of the face area by the tracking unit 42 so that the ROI determination processing unit 46 does not set the face area without movement as the attention area. Alternatively, the face verification unit 40 may discard the face area determined to have no movement by the tracking unit 42 from the beginning and may not supply the ROI determination processing unit 46.

  FIGS. 11A and 11B are diagrams illustrating a method of determining a region of interest when the user leaves the seat. As shown in FIG. 11A, when the user is sitting on the seat in front of the camera, the face area 420 including the user's face 410 is determined as the attention area. For example, when the ROI determination processing unit 46 manages the attention area in units of GOPs, the same attention area is used in other frames within the same GOP. As shown in FIG. 11B, in the frame immediately after the user leaves the seat, the same attention area 420 is continuously used. It will be displayed on the other party's display.

  Therefore, when the face area is no longer detected by the face area detection unit 12 and the face verification unit 40, the ROI determination processing unit 46 is interrupted so as to forcibly reset the attention area even during the GOP. Give a signal. Thereby, after the user disappears, it is possible to eliminate the inconvenience that the state of the room is projected in detail.

  As described later, for a region that the user does not want to show to the communication partner, the user can set a prohibited region, and when a face region is detected in the prohibited region by the face region detection unit 12 and the face verification unit 40, The ROI determination processing unit 46 sets the face area as the attention area even within the prohibited area, but immediately releases the attention area set as the prohibited area when the face area is no longer detected from within the prohibited area. You may control as follows.

  FIGS. 12A and 12B are diagrams illustrating a method of determining a region of interest when a user moves around a room. As shown in FIG. 12A, while the user is sitting on the seat, an area 420a including the detected face 410a is set as the attention area. As shown in FIG. 12B, when the user temporarily leaves the seat and moves around the room, if the attention area is set in accordance with the detected movement of the face 410c, the state of his / her room is moved by the user's movement. Will be projected clearly. Therefore, even if the face area 420c at a position away from the center of the screen is detected, a criterion for not selecting the face area 420 may be provided.

  In addition, if you set a standard that does not set the face area detected outside the center of the screen as the attention area, even if a family other than yourself enters the room, there is no worry that the family face will be set as the attention area. It also helps protect user privacy.

  FIGS. 13A and 13B are diagrams for explaining a method for designating a region for which setting of a region of interest is permitted and a region for which prohibition is to be set. As shown in FIG. 13A, a region 450 (in this case, the center region of the screen) in which the user permits setting of the attention region is set in advance, and the face region is detected within the permission region 450 set by the user. In this case, a reference area may be set, but if the face area is detected outside the permission area 450, a determination criterion that the area is not the attention area may be provided. Further, as shown in FIG. 13B, the user may be able to set an area 460 that prohibits the setting of the attention area. Here, the area on the desk is set as the prohibited area 460 so that documents on the desk are not projected with high image quality.

  By providing a criterion that the face area detected in the permitted area 450 is the attention area, or a criterion that the face area is not detected even if the face area is detected in the prohibited area 460, the privacy of the user can be protected, Can be secured.

  Next, a criterion for determining a region of interest when a plurality of face regions are detected will be described. For example, a face region that satisfies at least one of the following criteria is determined as a region of interest.

(1) The face area with the largest area,
(2) a face area existing near the center of the image,
(3) the face area where the score of facialness is the maximum, or
(4) A face area in which the area, position, and score of the face area are normalized, and the weighted sum of these values is maximized.

As another criterion, the attention area may be determined as follows.
(5) Adopt face areas as separate attention areas,
(6) Adopting an area including all the face areas as the attention area, or
(7) The face areas that are close to each other are grouped together and adopted as the attention area.

  First, with reference to FIGS. 14A and 14B, a method of determining a region of interest when only one face region is detected will be described. Assume that a face 410 is detected in an image 400 and a rectangular face area 420 including the face 410 is detected as shown in FIG. At this time, the rectangular face area 420 may be used as the attention area as it is, or an area 420 including the face 410 and the upper body 412 may be used as the attention area as shown in FIG.

  FIGS. 15A and 15B are diagrams illustrating a method of determining a region of interest when a plurality of face regions are detected. As shown in FIG. 15A, when the detected faces 410a to 410c are close to each other, that is, when the detected face areas are within a predetermined distance from each other, the area 420 including these face areas is set as the attention area. As shown in FIG. 15B, when the plurality of detected face areas are separated from each other as shown in FIG. 15B, the areas 420a to 420c including the detected faces 410a to 410c are set as separate attention areas. And Even when a plurality of face regions are detected, a region including the face and upper body may be used as a region of interest as shown in FIG.

  FIGS. 16A and 16B are diagrams illustrating a method of determining a region of interest when face regions having different sizes are detected. As shown in FIG. 16A, when a large-sized face 410c and small-sized faces 410a and 410b are detected, a region 420 including the larger-sized face 410c is set as a region of interest. As shown in FIG. 16B, when two large faces 410a and 410b and three small faces 410c, 410d, and 410e are detected, attention is paid to regions 420a and 420b that include the two large faces 410a and 410b, respectively. This is an area. As an example of the determination criterion, when the ratio between the maximum size and the minimum size is larger than a predetermined threshold for a plurality of detected face regions, the face region of the maximum size is set as the attention region. The second and subsequent large face areas may be set as the attention area.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

It is a lineblock diagram of a chat system concerning an embodiment. It is a block diagram of the information processing apparatus of FIG. FIG. 3 is a configuration diagram of an encoding processing block in FIG. 2. It is a function block diagram of the ROI specific | specification part of FIG. FIGS. 5A to 5C are diagrams illustrating a method for eliminating discontinuity at the boundary between the attention area and the non-attention area. FIG. 4 is a flowchart illustrating a processing procedure of ROI encoding by the encoding processing block of FIG. 3. FIG. It is a flowchart explaining the procedure of the face verification process of FIG. It is a flowchart explaining the procedure of the misdetection determination processing by tracking of FIG. It is a flowchart explaining the procedure of the face interpolation process by tracking of FIG. It is a figure explaining a mode that an attention area is set up in an image. FIGS. 11A and 11B are diagrams illustrating a method of determining a region of interest when the user leaves the seat. FIGS. 12A and 12B are diagrams illustrating a method of determining a region of interest when a user moves around a room. FIGS. 13A and 13B are diagrams for explaining a method for designating a region for which setting of a region of interest is permitted and a region for which prohibition is to be set. FIGS. 14A and 14B are diagrams illustrating a method for determining a region of interest when only one face region is detected. FIGS. 15A and 15B are diagrams illustrating a method of determining a region of interest when a plurality of face regions are detected. FIGS. 16A and 16B are diagrams illustrating a method of determining a region of interest when face regions having different sizes are detected.

Explanation of symbols

  10 frame memory, 12 face area detection unit, 14 display control unit, 18 ROI identification unit, 20 band information acquisition unit, 22 non-ROI filter, 24 video encoder, 26 video packetization unit, 28 audio encoder, 30 audio packetization unit , 32 multiplexing unit, 40 face verification unit, 42 tracking unit, 44 face area history storage unit, 46 ROI determination processing unit, 48 judgment criterion storage unit, 50 ROI parameter adjustment unit, 100 information processing device, 200 encoding processing block , 220 decoding processing block, 230 microphone, 240 camera, 250 speaker, 260 display, 270 communication unit, 300 network.

Claims (10)

A detection unit for detecting a face area in a frame of a movie;
A recording unit in which the history of the detected face area is recorded in units of frames;
The detection result of the face area by the detection unit by tracking the face area detected by the detection unit over a plurality of continuous frames with reference to the detection history of the face area in units of frames recorded in the recording unit A tracking unit for correcting
A region of interest determination unit that determines a region of interest based on a predetermined reference based on the face region corrected by the tracking unit;
The region of interest and the other region is coded with different quality, see contains an encoding unit which generates a video coded stream,
When the attention area determination unit sets an area including the face area detected while the user is sitting on the seat by the detection unit as the attention area, the tracking unit detects that the user has moved away from the seat. When this is the case, the attention area determination section does not set the detected face area as the attention area even if the detection section detects the face area .   Even when the tracking unit detects a face region in a certain frame, the tracking unit does not continuously detect the face region in a predetermined number of consecutive frames after the frame. The encoding processing apparatus according to claim 1, wherein the face area detection history detected in the frame is determined to be invalid and is deleted from the recording unit.   If the tracking unit continues to detect a face area in which the detection history is present in the recording unit and the detection unit does not detect the face area in a predetermined number of consecutive frames, the tracking unit The encoding processing apparatus according to claim 1, wherein the detection history is determined to be unnecessary and is deleted from the recording unit.   The tracking unit determines whether or not the detected face area in the frame is a false detection by collating the detection result of the face area in a frame with the detection history of the face area in the frames before and after the frame. The encoding processing apparatus according to claim 1, wherein when it is determined that the detection is erroneous, the history of the face area detected in the frame is deleted.   The tracking unit compares the face area detected in a certain frame with the face areas detected in frames before and after the frame with respect to at least one of the position and size of the face, thereby detecting the face detected in the frame. The encoding processing apparatus according to claim 4, wherein it is determined whether or not the area is a false detection.   Even if the detection unit does not detect a face region in the current frame, the tracking unit may be a detection failure of the face region in the current frame if the detection history of the face region exists in the past frame. By determining and reusing the information on the position of the face area of the past frame as the position information of the face area of the current frame, the information on the face area of the current frame determined to be undetected is interpolated. The encoding processing apparatus according to claim 1, wherein   The code according to any one of claims 1 to 6, further comprising an imaging control unit that controls a direction of a camera for photographing a face in accordance with a position of a face area tracked by the tracking unit. Processing equipment.   The imaging control unit according to any one of claims 1 to 6, further comprising an imaging control unit that controls zooming of a camera for capturing a face in accordance with a size of a face region tracked by the tracking unit. Encoding processing device. Detecting a face area in a frame of the video;
A step of recording the history of the detected face area in units of frames;
A tracking step for correcting a detection result of the face area by tracking the detected face area over a plurality of continuous frames with reference to the recorded face area detection history in a frame unit,
Determining a region of interest on a predetermined basis based on the corrected face region;
The region of interest and the other region is coded with different quality, seen including a step of generating a video coded stream,
When the step of determining the attention area is set as the attention area, the area including the face area detected while the user is sitting in the seat by the detecting step, the user has moved away from the seat by the tracking step. And the step of determining the attention area does not set the detected face area as the attention area even if the face area is detected by the detecting step. Method. A detection function for detecting a face area in a frame of a video;
A recording function that records the history of the detected face area in units of frames;
A tracking function that corrects the detection result of the face area by tracking the detected face area over a plurality of continuous frames with reference to the recorded face area detection history of the frame unit,
A region-of-interest determination function for determining a region of interest based on the corrected face region on a predetermined basis;
Encoding the region of interest with a different image quality from other regions, and causing the computer to realize an encoding function for generating a video encoded stream ,
When the attention area determination function sets an area including a face area detected while the user is sitting in the seat by the detection function as the attention area, the tracking function detects that the user has moved away from the seat. When this is the case, the attention area determination function does not set the detected face area as the attention area even if a face area is detected by the detection function .

Images