JP5894055B2 - VIDEO DATA CONTROL DEVICE, VIDEO DATA CONTROL METHOD, AND VIDEO DATA CONTROL PROGRAM - Google Patents

Description

  The present invention relates to a technique for controlling video data for a user who communicates using sign language or spoken language, such as a hearing impaired person or his family.

  The act of “conversing with family while watching TV” can be performed naturally by a listener who is not impaired in hearing. This is because, for the listeners, the communication method using only sound is a general communication method and can be performed without interrupting the viewing of the video content.

  On the other hand, it is almost impossible for the hearing impaired and their family members who communicate using sign language or speech to perform the above actions. This is because the sign language and the spoken language are communication methods established by looking at the other party, and it is necessary to interrupt the viewing and face the other party in order to have a conversation when viewing the video content.

  Here, as a method of solving such a problem, by synthesizing the camera video taken by a plurality of users with the video content being viewed, the users can see each other while viewing the video content, A method of communicating in sign language or spoken language over the video is conceivable (see Patent Document 1).

  As a simpler method, as shown in FIG. 11, a mirror is placed on the TV and the user's appearance is reflected by the mirror, so that the user can use sign language or spoken language while watching the video on the TV screen. It is also possible to have a method of communicating with each other.

JP 2008-217536 A

  However, the viewing style of video content at home is not limited to the style of sitting near a television screen as shown in FIG. In the floor plan room as shown in FIG. 11, for example, there are a style of lying on a sofa and watching, a style of watching about a dining table, a style of watching while cooking in the kitchen, a style of watching while moving around the whole room for cleaning, etc. Conceivable.

  Therefore, depending on how the user is reflected in the camera and mirror, the lower body of the user and extra background may be included in the camera image and mirror image. Therefore, it is not necessarily a video or mirror image suitable for sign language or verbal communication. However, there is a problem that the user who performs the communication must carefully read detailed expressions of the hand and fingertips, facial expressions, and delicate movements of the lips from the image.

  The present invention has been made in view of the above circumstances, and it is an object of the present invention to realize communication in sign language or spoken language while viewing video content even when viewing the video content in various viewing styles. .

The video data control device according to claim 1 receives a camera video of a user who is photographed from a screen side of a video display device capable of displaying video content and is viewing the video content in an arbitrary viewing style from a video camera. In the case of a mode for extracting image areas of the user's face and upper body from the camera image and communicating with sign language according to the mode selected and determined by the user , the camera image receiving means and the face and upper body In a mode in which a user video consisting only of the image area is generated and communication is carried out by verbal speech, user video generation means for generating a user video consisting only of the face image area and the user video are displayed. Video synthesizing means for superimposing the user video on the video content, and the superimposed user video And has a video output means for outputting the video content to the video display device, the user image generation unit, when the distance between the user photographed in the camera image is larger than the threshold value, divides the plurality of users A plurality of user videos are generated .

According to the present invention, it extracts the image area from the camera image the face and upper body of a user viewing the video content of the video display device, depending on the selected determined mode by the user, performs communication by sign language mode In this case, a user video consisting only of the image area of the face and the upper body is generated, and in a mode in which communication is carried out by spoken language, a user video consisting of only the image area of the face is generated and superimposed on the video content. Thus, since it is output to the video display device, it is possible to realize sign language or spoken language communication while viewing the video content without depending on the viewing style of the video content. In addition, according to the present invention, when the distance between users captured in the camera video is larger than the threshold value, a plurality of user videos are generated by dividing the plurality of users. It is possible to prevent the excessive background of the image from being included.

The video data control device according to claim 2 receives a camera video of a user who is photographed from a screen side of a video display device capable of displaying video content and is viewing the video content in an arbitrary viewing style from a video camera. In the case of a mode for extracting image areas of the user's face and upper body from the camera image and communicating with sign language according to the mode selected and determined by the user , the camera image receiving means and the face and upper body A user video generation means for generating a user video consisting only of the image area of the face and a user video consisting of only the image area of the face in a mode in which communication is carried out by spoken language, and positioned within the field of view of the user a video output means for outputting the other of the user image to the image display device for, the, the user image generation hands The distance between the user photographed in the camera image is larger than the threshold value, and generating a plurality of user video obtained by dividing the plurality of users.

According to the present invention, it extracts the image area from the camera image the face and upper body of a user viewing the video content of the video display device, depending on the selected determined mode by the user, performs communication by sign language mode in the case of, it generates a user image consisting of only the image area of the face and upper body, when the mode for communication with the mouth talk generates the user image consisting of only the image area of the face, the user's field of view Because it is output to another video display device located at, communication with sign language or spoken language can be realized while viewing the video content without depending on the viewing style of the video content. In addition, according to the present invention, when the distance between users captured in the camera video is larger than the threshold value, a plurality of user videos are generated by dividing the plurality of users. It is possible to prevent the excessive background of the image from being included.

The video data control method according to claim 3 , wherein a video of a camera of a user who is photographed from a screen side of a video display device capable of displaying video content by a computer and who views the video content in an arbitrary viewing style. In the case of a mode in which the camera image reception step received from the camera image and the image area of the user's face and upper body are extracted from the camera image and the communication is performed in sign language according to the mode selected and determined by the user , In a mode in which a user video consisting only of the face and upper body image area is generated and communication is made through oral speech, a user video generation step for generating a user video consisting only of the face image area; and A video composition process for superimposing the user video on the video content to be displayed. And-up, has a video output step of outputting the user image and video content the superposed on the image display device, in the user image generation step, the distance between the user photographed in the camera image threshold In the case of larger than this, a plurality of user videos obtained by dividing a plurality of users are generated .

5. The video data control method according to claim 4 , wherein a video of a camera of a user who is photographed from a screen side of a video display device capable of displaying video content by a computer and is viewing the video content in an arbitrary viewing style. In the case of a mode in which the camera image reception step received from the camera image and the image area of the user's face and upper body are extracted from the camera image and the communication is performed in sign language according to the mode selected and determined by the user , A user video generation step of generating a user video consisting only of the face and upper body image area and generating a user video consisting only of the image area of the face in a mode in which communication is made through oral speech; Video output step for outputting the user video to another video display device located within Has, in the user image generation step, the distance between the user photographed in the camera image is larger than the threshold value, and generating a plurality of user video obtained by dividing the plurality of users.

A video data control program according to claim 5 causes a computer to function as the video data control apparatus according to claim 1 or 2 .

  According to the present invention, even when video content is viewed in various viewing styles, it is possible to communicate in sign language or spoken language while viewing the video content.

It is a figure showing the whole viewer picture display system composition concerning a 1st embodiment. It is a sequence diagram which shows operation | movement of a viewer image | video display system. It is a figure which shows the functional block structure of a user image | video production | generation part. It is a figure which shows the example of production | generation (one person) of a user image | video. It is a figure which shows the example of a production | generation of a user image | video (2 persons). It is a figure which shows the example of production | generation (3 persons) of a user image | video. It is a figure which shows the example before the effect process of a user image | video. It is a figure which shows the example (mirroring) after the effect process of a user image | video. It is a figure which shows the example (translucent) after the effect process of a user image | video. It is a figure which shows the whole structure of the viewer video display system which concerns on 2nd Embodiment. It is a figure which shows the example of the viewing-and-listening style of video content.

  The present invention focuses on the fact that communication by sign language or spoken language is performed by paying attention to facial changes and finger movements, and the upper body that is the range of motion of the face and fingertips from the camera video of the user viewing the video content. The image area is extracted and superimposed on the video content or displayed alone in the user's field of view, so that the user can communicate in sign language or spoken language while viewing the video content without depending on the specific viewing style. Is realized.

  Also, paying attention to the above-mentioned communication style among users who are distant from each other among various viewing styles, depending on the distance between the photographed users, the user video may be displayed in a divided state so that the user can see the camera video. It prevents the extra background of the lower body and between users from being included.

  Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings. However, the present invention can be implemented in many different modes and should not be construed as being limited to the description of the present embodiment.

[First Embodiment]
First, the overall configuration of the viewer video display system according to the first embodiment will be described with reference to FIG.

  This viewer video display system views video content, a video display device 5 that displays video content on a screen, a video camera 3 that captures a viewer (hereinafter referred to as a user) 50 from the screen side of the video display device 5, and the video content. The video data control device 1 that performs image processing on the camera video of the user 50, the remote control terminal 7 that remotely operates the video data control device 1, and the video content output device 9 that outputs video content are mainly configured. .

  The video data control device 1 plays a central role in extracting an image area related to the user 50 from the camera video in the viewer video display system, and all other devices (video display device 5, video camera 3, remote controller). The terminal 7 and the video content output device 9) are connected to each other via a wired or wireless communication path so that data communication is possible.

  As the video content output device 9, for example, a terrestrial digital television broadcast receiver that outputs a terrestrial digital signal received from a broadcasting station as a video content, or a Blu-ray / DVD that outputs video content recorded on a hard disk, a DVD, or the like. An example is a recorder.

  Next, functions of the video data control apparatus 1 will be described. The video data control apparatus 1 includes a camera video reception unit 11, a camera video storage unit 12, a camera video analysis unit 13, a user video generation unit 14, a video synthesis unit 15, a video output unit 16, and adjustment data reception. Unit 17 and adjustment data storage unit 18.

  The camera video reception unit 11 has a function of receiving the camera video of the user 50 viewing the video content from the video camera 3 via the camera video input interface 31 and temporarily storing it in the camera video storage unit 12. Yes.

  The camera video storage unit 12 has a function of storing the camera video received from the video camera 3 in a readable manner. For example, when a camera video is received in units of frames, the frame images are stored in time series.

  The camera video analysis unit 13 receives the camera video from the camera video reception unit 11 (can be obtained from the camera video storage unit 12), and detects an image area including the face part and upper body part of the user 50 from the camera video. The detection result is output to the user video generation unit 14 at the subsequent stage. In addition, it has a function of correcting, for example, rotation of the camera image as necessary.

  The user video generation unit 14 extracts an image area detected based on the detection result from the camera video, generates a user video including only the face portion and upper body portion of the user 50, and performs adjustment in the adjustment data storage unit 18. After executing processing according to data (described later), it has a function of outputting to the video composition unit 15 in the subsequent stage.

  The video composition unit 15 superimposes the user video output from the user video generation unit 14 and the video content output from the video content output device 9 via the video content input interface 33 so that the user video is displayed. It has a function.

  The video output unit 16 has a function of outputting the user video and video content superimposed by the video synthesis unit 15 from the video output interface 32 and displaying them on the video display device 5.

  The adjustment data receiving unit 17 receives the adjustment data such as the display position of the user video selected and determined by the user operation of the remote control terminal 7 via the adjustment data input interface 34, and stores the adjustment data in the adjustment data storage unit 18. have.

  The adjustment data storage unit 18 has a function of storing the adjustment data received by the adjustment data receiving unit 17 in a readable manner. The adjustment data includes, for example, selection of a communication mode (sign language mode or spoken language mode), presence / absence of division of user video, selection of effect pattern, size of user video, display position of user video, and the like.

  The above is the function of the video data control apparatus 1 according to the present embodiment. Such a video data control device 1 can be realized by a computer having a memory and a CPU. For example, it can be realized using a personal computer connected to the video display device 5, a browser mounted on a television, or the like. Each process of each functional unit is executed by a program.

  Next, the operation of the viewer video display system will be described with reference to FIG. First, a video camera 3 installed near the screen of the video display device 5 captures a user 50 who is viewing video content in an arbitrary viewing style, and the camera video is input to the video data control device 1. (Step S101).

  Next, in the video data control apparatus 1, the camera video reception unit 11 receives the camera video output from the video camera 3 via the camera video input interface 31 (step S102).

  Next, the camera image analysis unit 13 detects the image area of the face portion and upper body portion of the user 50 from the received camera image (step S103). At this time, if the face inclination is detected, the camera image including the upper body is corrected so that the face inclination becomes vertical, and the corrected camera image is converted into the detection result (the coordinate position of the detected image area). Output to the user video generation unit 14.

  Next, the user video generation unit 14 extracts the image region of the face and upper body of the user 50 from the video image (or the corrected video image if corrected) based on the detection result, and only from the image region. A user video is generated and output to the video synthesis unit 15 (step S104). Details of this process will be described later.

  Next, the generated video image is superimposed on the video content from the video content output device 9 by the video composition unit 15 with the size and position specified by the user 50 (step S105).

  Next, the video output unit 16 outputs the superimposed user video and video content as new video data to the video display device 5 through the video output interface 32 (step S106).

  Finally, the new video data is received by the video display device 5, and the user video is displayed on the screen and the video content is displayed on the screen (step S107).

  Note that the technique for detecting the face part and upper body part of the user 50 described in step S103 can be realized using a known technique. For example, it is possible to use a method of extracting a feature amount according to the face direction and calculating a similarity using the feature amount, and recognizing the face based on the calculated similarity (Japanese Patent Laid-Open No. 2009-157766). Issue gazette).

  A method for detecting the tilt of the face and correcting the image can also be realized using a known technique. For example, it is possible to use a method of extracting the outline of the mouth, which is a facial component, and estimating the inclination of the mouth by performing Hough transformation on the outline, thereby estimating the inclination of the face (Li Shoubun, “Detection of face image using Hough transform of mouth edge image and length difference between eyes”, IEICE Technical Report, IEICE, March 2006, HIP2005-161, p.43- 48).

  Subsequently, functions and operations (details of step S104) of the user video generation unit 14 will be described in detail. As shown in FIG. 3, the user video generation unit 14 includes a user video extraction unit 14a, a user video division processing unit 14b, a user video effect processing unit 14c, a user video size determination unit 14d, and a user video position coordinate determination. And 14e.

  The user 50 uses the remote control terminal 7 to (1) a communication mode (sign language mode or spoken language mode), (2) a display method when a plurality of users are displayed on the user video (displayed or divided as it is) It is possible to select and determine each item such as (displayed one by one), (3) pattern of effect processing applied to user video, (4) size of user video, and (5) display position of user video.

  Here, when the adjustment data selected and determined by the user 50 as appropriate for each of these items and selected / determined by pressing the determination button or the like is transmitted, the adjustment data receiving unit 17 receives the adjustment data via the adjustment data input interface 34. And stored in the adjustment data storage unit 18. When the adjustment data is received a plurality of times, the previous adjustment data value is rewritten and updated each time the adjustment data is received.

  The adjustment data is acquired by the user video generation unit 14 every time it is updated, and the following processing is executed. A preset value stored in advance in the adjustment data storage unit 18 may be used.

  First, the user video extraction unit 14a extracts an image area corresponding to the selected / determined communication mode from the camera video and outputs it to the user video division processing unit 14b (step S104a). Specifically, in the sign language mode (when communicating in sign language), not only the facial expression but also the hand and fingertip expressions are watched, so the image area that combines both the face and upper body of the user 50 To extract. On the other hand, in the speech mode (when communicating by speech), the movement of the lips is watched, so that only the image area of the face of the user 50 is extracted.

  Here, extraction patterns of user videos are shown in FIGS. 4 to 6 illustrate the case of the sign language mode as an example. First, when there is one user shown in the camera image (the input result of the user's face part and upper body part is one area), as shown in FIG. 4, the detected image area is directly used as the user image. Extract as

  There are two users shown in the camera image (detection result is two regions), and a distance d between users (for example, a distance between the center points of each detected region) based on the detection result is a preset value. If it is smaller than (hereinafter referred to as a threshold), as shown in FIG. 5A, the minimum rectangle including the detection areas for the two persons is extracted as the user video.

  On the other hand, when there are two users reflected in the camera image (detection result is two regions) and the distance d between users is larger than the threshold value from the detection result, the detection is performed as shown in FIG. One user video is extracted from each image area.

  When there are three or more users appearing in the camera video (the detection result is three or more areas), as shown in FIG. 6, users whose user distance d is smaller than the threshold value are the detected image areas of those users. The smallest rectangle that is included is extracted as the user video, and the other users extract the detected image area relating to the user as the user video as it is.

  Next, when a user video including a plurality of users is extracted, whether the user video including the plurality of users is displayed as it is based on the display method selected and determined by the user 50 by the user video division processing unit 14b. It is determined whether to divide a plurality of users and display them as one user video, and when division is not necessary, the extracted user video is output to the user video effect processing unit 14c as it is, and division is necessary. In this case, a plurality of user videos subjected to the division process are output (step S104b).

  Next, the effect processing of the pattern selected and determined by the user 50 is executed on the user video input from the user video division processing unit 14b by the user video effect processing unit 14c, and output to the user video size determination unit 14d. (Step S104c).

  Note that a known video processing technique can be applied to the effect processing applied to the user video. FIG. 7 shows a user video that has not been subjected to the effect processing, and FIGS. 8 and 9 show examples of user images to which the effect has been applied. FIG. 8 shows an effect process for making a mirror image, and FIG. 9 shows an effect process for making it translucent.

  Next, the user video size determination unit 14d adjusts the size of the user video input from the user video effect processing unit 14c based on the video size selected and determined by the user 50, and the user video position coordinate determination unit 14e (step S104d).

  In addition, since the size of the user's face part and upper body part included in the detection result varies depending on the distance between the user and the camera, the size of the user video input to the user video size determination unit is also considered to vary. . The user video size determination unit 14d may adjust the size so that each user video has the same magnification.

  Finally, the user video position coordinate determination unit 14e rewrites the position coordinates of the user video input from the user video size determination unit 14d based on the position coordinates of the video selected and determined by the user 50, and the video composition unit 15 (step S104e).

  As described above, according to the present embodiment, when the face and upper body image areas are extracted from the camera video of the user 50 who is viewing the video content of the video display device 5 and communication is performed using sign language, When a user video consisting only of the upper body image area is generated and communication is carried out through verbal speech, a user video consisting only of the face image area is generated and superimposed on the video content and output to the video display device 5. Therefore, it is possible to realize sign language or verbal communication while viewing the video content without depending on the viewing style of the video content.

  In addition, according to the present embodiment, when the distance between users photographed in the camera video is larger than the threshold, a plurality of user videos are generated by dividing the plurality of users. It is possible to prevent an extra background or the like from being included.

[Second Embodiment]
Next, a viewer video display system according to the second embodiment will be described with reference to FIG.

  In the first embodiment, the mode in which the user video is displayed superimposed on the video content has been described. On the other hand, the above-mentioned effect of “enabling communication in sign language or spoken language while viewing video content without depending on a specific viewing style” allows the user video showing the user 50 to be viewed together with the video content. Therefore, it can be obtained by extracting and displaying the face and upper body area necessary for communication by sign language or spoken language.

  Therefore, in the second embodiment, instead of displaying the user video superimposed on the video content, a user video video display device 5b dedicated to the user video is provided in the vicinity of the video display device 5a that displays the video content. The user video is displayed on the screen of the user video video display device 5b.

  The video data control apparatus 1 according to the present embodiment has basically the same function as the function described in the first embodiment, but does not require a process of superimposing user video on video content. As shown in FIG. 10, it is not necessary to provide the video content input interface 33 or the video composition unit 15.

  On the other hand, a part of the function of the video output unit 16 is modified in accordance with the deletion of the video synthesis unit 15 from the components of the viewer video display system. In the first embodiment, the superimposed user video and video content are output and displayed on the video display device 5a. In the second embodiment, the user video generated by the user video generation unit 14 is displayed. Are displayed on the video display device 5b for user video different from the video display device 5a displaying the video content.

  The video content output device 9 directly outputs the video content to the video display device 5a for display. Such video content output device 9 and video display device 5a are, for example, home digital terrestrial television.

  It should be noted that the installation location of the user video image display device 5b may be installed so as to be located within the field of view of the user viewing the video content, such as on, next to, or in front of the video display device 5a.

  As described above, according to the present embodiment, when the face and upper body image areas are extracted from the camera video of the user 50 who is viewing the video content of the video display device 5a and communication is performed using sign language, When a user video consisting only of the upper body image area is generated and communication is made through oral speech, a user video consisting only of the face image area is generated, and a video display for user video located within the visual field of the user 50 is displayed. Since it is output to the apparatus 5b, it is possible to realize sign language or verbal communication while viewing the video content without depending on the viewing style of the video content.

DESCRIPTION OF SYMBOLS 1 ... Video data control apparatus 3 ... Video camera 5, 5a ... Video display apparatus 5b ... User video display apparatus 7 ... Remote control terminal 9 ... Video content output apparatus 11 ... Camera video reception part 12 ... Camera video storage part 13 ... Camera Video analysis unit 14 ... User video generation unit 15 ... Video synthesis unit 16 ... Video output unit 17 ... Adjustment data reception unit 18 ... Adjustment data storage unit 31 ... Camera video input interface 32 ... Video output interface 33 ... Video content input interface 34 ... Adjustment data input interface 50 ... User S101 to S107, S104a to S104e ... Step

Claims (5)

A camera video reception means for receiving from a video camera a camera video of a user who is shot from a screen side of a video display device capable of displaying video content and viewing the video content in an arbitrary viewing style;
Extract the face and upper body of the image area of the user from the camera image, depending on the selected determined mode by the user, if the mode for communication with sign language consists of only the image area of the face and upper body In the case of a mode in which user video is generated and spoken communication is performed, user video generation means for generating a user video consisting only of the face image area;
Video composition means for superimposing the user video on the video content so that the user video is displayed;
Video output means for outputting the superimposed user video and video content to the video display device ,
The user video generation means includes
A video data control device that generates a plurality of user videos obtained by dividing a plurality of users when a distance between users photographed in the camera video is larger than a threshold . A camera video reception means for receiving from a video camera a camera video of a user who is shot from a screen side of a video display device capable of displaying video content and viewing the video content in an arbitrary viewing style;
Extract the face and upper body of the image area of the user from the camera image, depending on the selected determined mode by the user, if the mode for communication with sign language consists of only the image area of the face and upper body In the case of a mode in which user video is generated and spoken communication is performed, user video generation means for generating a user video consisting only of the face image area;
Video output means for outputting the user video to another video display device located within the field of view of the user ,
The user video generation means includes
A video data control device that generates a plurality of user videos obtained by dividing a plurality of users when a distance between users photographed in the camera video is larger than a threshold . By computer
A camera video reception step for receiving from a video camera a camera video of a user who is photographed from a screen side of a video display device capable of displaying video content and viewing the video content in an arbitrary viewing style;
In the mode in which the user's face and upper body image areas are extracted from the camera image, and communication is performed in sign language according to the mode selected and determined by the user, the image area includes only the face and upper body image areas. In the case of a mode in which user video is generated and spoken communication is performed, a user video generation step of generating a user video consisting only of the face image area;
A video composition step of superimposing the user video on the video content so that the user video is displayed;
A video output step of outputting the superimposed user video and video content to the video display device,
In the user video generation step,
When the distance between users captured in the camera video is larger than a threshold, a plurality of user videos are generated by dividing the plurality of users.
And a video data control method. By computer
A camera video reception step for receiving from a video camera a camera video of a user who is photographed from a screen side of a video display device capable of displaying video content and viewing the video content in an arbitrary viewing style;
In the mode in which the user's face and upper body image areas are extracted from the camera image, and communication is performed in sign language according to the mode selected and determined by the user, the image area includes only the face and upper body image areas. In the case of a mode in which user video is generated and spoken communication is performed, a user video generation step of generating a user video consisting only of the face image area;
A video output step of outputting the user video to another video display device located within the field of view of the user,
In the user video generation step,
When the distance between users captured in the camera video is larger than a threshold, a plurality of user videos are generated by dividing the plurality of users.
And a video data control method. A video data control program for causing a computer to function as the video data control device according to claim 1.

Images