JP2007300452A - Image and television broadcast receiver with sound communication function - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow a display dimension by each screen on a synthetic screen to be switchable in cooperation with a program content and the importance degree of each attendant in the conversation of a video conference. <P>SOLUTION: A television broadcast receiver with a sound communication function includes: a video decoding part 22 for decoding a television broadcast; an image and sound adjusting part 13 for converting an input signal into a form suitable for communication processing; a communication decoding part 34 for decoding the input signal from a communication line; a communication control part 32 for controlling an input/output signal relative to the communication line; an image composing part 14 for outputting the composite image signal of image signals from the video decoding part 22 and the communication decoding part 34; a display part 15 for displaying the composite image signal; and a conversation state determining part 41 for determining a conversation state between a user and a communication opposite party, based on respective sound signals from the image and sound adjusting part 13 and the communication decoding part 34, and outputting a synthetic control signal. The image composing part 14 composes the composite image signal, based on the composite control signal to be inputted from the conversation state determining part 41, and displays it in the display part 15. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a television broadcast receiver having an image and audio communication function.

  A device having an image and voice communication function capable of performing a pseudo face-to-face conversation with a communication partner displayed on a screen via a communication line such as a telephone line is generally known as a videophone device. . Similarly, a device having an image and voice communication function capable of performing face-to-face listening to the speech of a conference conversation participant displayed on a screen via a communication line is generally known as a video conference system. ing. There is also known a television broadcast receiver that receives a television broadcast from a television station, reproduces a program, and displays the program on a screen.

  In addition, video and audio communications that enable a simulated face-to-face conversation with a communication partner displayed on the screen while listening to a television broadcast program displayed on the screen in combination with a television broadcast receiver and a videophone device A television broadcast receiver with an image and audio communication function having a function (video phone function) is also known. As such a conventional television broadcast receiver with an image and audio communication function, for example, it is added to an expansion slot of a home television broadcast receiver and is also connected to a telephone line, so that a general telephone function and an image are received. In addition, a videophone device that can be switched by operating a remote control device while a voice communication function is in use is known (see, for example, Patent Document 1). In such a videophone device, in order to reduce the installation space, the videophone can be used as a fixed sub-screen formed by cutting out the lower right part of the main screen of the television broadcast with the aspect ratio of the screen. When a communication partner is displayed and a videophone call is received, it is initially displayed on a large sub-screen. You can choose to reduce the display size of the small screen by operating the remote control device. Output from the speaker of the television broadcast receiver.

  In addition, as a conventional video conference system, each speaker's terminal device and its number of points are automatically detected from a voice signal input from each conference conversation participant's terminal device connected to a communication line, and the speaker In the case of one person, the image of the speaker is sent to each terminal device other than the speaker, and the screen is simply divided into two to four as the number of speakers in the conference conversation increases. ... There is known a video conference system including a multi-point control device that sends a composite image of each inventor with n divisions and the number of divisions to each terminal device (for example, see Patent Document 2). In a video conference system equipped with the multipoint control device, when there are a plurality of speakers, a plurality of images of each speaker are combined into one equally divided screen, and the combined images are displayed in each conference conversation participant. In this case, the image of the speaker who is the transmission destination is replaced with the image of another conference conversation participant from the composite image and transmitted to each terminal device.

JP-A-10-56675 (paragraphs 0014 to 0017, FIG. 1) Japanese Patent No. 3305037 (paragraphs 0026 to 0029, FIG. 1)

  Patent Document 1 describes that in a videophone device added to a conventional television broadcast receiver, the voice of the communication partner of the videophone is output from the speaker of the television broadcast receiver. It does not describe how to process the sound of a television broadcast program at that time. For this reason, in Patent Document 1, it is considered that it is not assumed that a user talks with a communication partner of a videophone at the same time while listening to a television broadcast. Assuming that the television broadcast audition and the conversation with the communication partner of the videophone are performed simultaneously instead of simultaneously, it is common sense that during a videophone call, the audio of the television broadcast is muted and not played back, or the conversation is disturbed. It is considered that the data is reproduced at a fixed small volume that does not become a signal and is mixed with the voice of the communication partner of the videophone and output. In addition, the child screen on which the other party of the videophone is displayed at that time can be adjusted slightly as described above, but it is fixedly arranged at the lower right of the main screen of the television broadcast. is there.

  For the volume of the videophone device added to the television broadcast receiver, for example, when talking while watching the contents of the television broadcast of the same station as the communication partner of the videophone, for the user of the videophone device Is desired to reproduce the sound of the television broadcast being viewed at a volume close to the normal volume, and it may be considered that the user's desire cannot be satisfied only by the conventional mute or volume reduction. In such a case, in such a case, no consideration is given to automatically changing the volume in accordance with the conversation situation or television viewing situation or manually selecting the volume.

  As for the screen display, if the image of the communication partner of the videophone is fixedly arranged in the lower right of the main screen as in Patent Document 1, for example, in the case of a soccer game relay, from the lower right of the screen It is inconvenient because it is impossible to view the goal scene, kick scene, and the like. On the other hand, for example, as shown in Patent Document 2, it is also possible to divide the screen of the television broadcast receiver into two equally and display the screen of the TV broadcast program and the screen of the communication partner of the videophone. However, the screen size of a TV broadcast program is half the size of the top, bottom, left and right, so the area is reduced to 1/4. It becomes difficult to see movement. In such a case, the conventional apparatus automatically changes the screen size (vertical and horizontal dimensions in the entire screen) and layout (arrangement) according to the conversation situation and the television viewing situation, or selects manually. This is not considered at all.

  In Patent Document 2, the number of speakers is simply increased from 2 to 4 as the number of speakers increases from the audio signals of the participants of the conference conversation. However, it is completely described that changing the screen size (vertical and horizontal display dimensions) according to the conversation and TV viewing status, and automatically changing the screen size. It has not been.

  Based on the above-described conventional technology, a method for displaying a TV broadcast viewing program and a plurality of TV conference speakers at a fixed display position and size by dividing the display screen evenly, There are several methods for displaying a communication partner of a viewing program and a videophone at a fixed display position and size. For example, a television broadcast viewing program is displayed in the center of the display screen and a conference speaker or communication partner is arranged around the screen, or a television broadcast viewing program is displayed at the top of the display screen. And a method of synthesizing the screen so that the speaker or the communication partner of the conference is arranged at the bottom. Since the videophone device is considered to be a one-to-one case in a videoconference system, the following description describes only the case where the device used by the user is a videoconference system, and is particularly necessary for the case of a videophone. The description is omitted except in cases.

  However, there are a wide variety of programs that are viewed on television broadcasts, such as sports programs, varieties, dramas, etc., and the degree of attention (importance) of television broadcasts to viewers (= users of videophones / conference devices) is It depends on the contents of the program, and also varies depending on the interests and hobbies of the individual at that time. For example, it varies depending on the timing and is very difficult to predict, define and quantify. The user's attention on the screen when watching a TV broadcast program is different from the case of a TV conference system where the user's line of sight is simple. The ratio of the display dimensions of the screen changes every moment and is not easy to detect. In addition, the importance for each user also changes from moment to moment depending on the status of the television broadcast program and the status of the conversation with the communication partner (or the content of other speakers in the conference). It is not something that can be simply judged.

  For example, when using a TV conference system, if there are sports program attack scenes and player substitution scenes, drama introduction parts and ending parts, main program and commercials, etc., depending on the contents Whether the importance for TV viewers (users of video conferencing systems) is centered on program viewing, or on the speech of other conversation participants (conversations with communication partners). change.

  In a conventional apparatus, for example, when a TV broadcast viewing program and a TV conference speaker are combined and displayed at a fixed ratio, the user is not very interested in the TV broadcast viewing program. A speaker who made an unimportant statement was displayed on the screen with a relatively large fixed size. That is, the size of each display screen is not linked to the importance felt by each conference conversation participant with respect to the contents of the program. In such a case, the user may not be able to view his / her own interests and wishes for the screen of a fixed-size viewing program that does not link with the interest of viewing or the screen of each communication partner with a fixed size that does not link with the content of speech or the presence of speech. It feels unnatural and unsatisfactory that the degree of importance that you feel is not consistent with what is displayed on the screen, and it is difficult to distinguish between the speaker who made the important speech and other speakers at the meeting. As a result, there is a problem that the apparatus is difficult to use.

  Similarly, for speech, for example, the content of a speaker in a video conference is diverse, and the level of importance of each content for a user of a video conference is various. For example, when the audio of a television broadcast viewing program and the audio of a conference call speaker are mixed at a fixed ratio as in the conventional apparatus, the television broadcast viewing program is not very interested. The sound is output at a high fixed volume, or the speaker who made an unimportant speech is output at a relatively high fixed volume. There is a problem that the importance of the conference conversation participant is not linked, so that the user may feel unnatural and difficult to use.

  The present invention has been made in order to solve the above-described problems, and is capable of both viewing a television broadcast program screen and sound, and listening to a person through a video conference (videophone) speaker screen. In a television broadcast receiver with an image and voice communication function that can be simultaneously displayed on the composite screen, at least the display dimensions of each screen on the composite screen, the content of the program, and the importance of each conversation participant in the video conference (videophone) It is an object of the present invention to provide a television broadcast receiver with an image and audio communication function that is switched in cooperation with the user and is natural and easy to use for the user.

A television broadcast receiver with an image and audio communication function according to the present invention,
A television decoder for decoding a received television broadcast digital signal to detect a television video signal and a television audio signal and a television control signal attached to them;
A video / audio adjustment unit that converts an input video signal from the imaging unit and an input audio signal from the audio input unit into a format suitable for image and audio communication processing;
A communication decoder that decodes an input signal from a communication line to detect a communication video signal and a communication audio signal;
A communication control unit for controlling input / output of video and audio input / output signals and communication control signals attached to the communication line;
A TV video signal from the TV decoder unit and a communication video signal from the communication decoder unit are input, and at the time of image and audio communication, the program image received by TV broadcasting and the image input from the communication line are combined. A video composition unit for outputting a video signal;
A television broadcast receiver with an image and audio communication function, comprising at least a display unit for displaying an image of a program received by television broadcasting and displaying a composite video signal during image and audio communication,
A conversation status determination unit that receives at least an input audio signal from the video / audio adjustment unit and a communication audio signal from the communication decoder unit, determines a conversation status between the user and the communication partner, and outputs a synthesis control signal;
The video synthesis unit further receives a synthesis control signal from the conversation state determination unit, synthesizes a synthesized video signal based on the synthesis control signal,
The display unit displays a composite video signal based on the composite control signal.

  The television broadcast receiver with an image and sound communication function according to the present invention associates at least the display size of each screen on the composite screen with the contents of the program and the importance of each conversation participant of the video conference (videophone). Therefore, it is possible to provide a television broadcast receiver with an image and sound communication function that is natural and easy to use for the user.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a schematic configuration of an example of a television broadcast receiver with an image and audio communication function according to Embodiment 1 of the present invention.
In FIG. 1, a television broadcast receiver 1 with an image and sound communication function receives a television broadcast from a television station, reproduces a program and displays it on a screen, and similarly, for example, a screen via a communication line. Communication that is displayed on the screen via a video conference system having an image and voice communication function, or a communication line such as a telephone line, etc., which enables face-to-face listening to the speech of the conference conversation participant displayed on the screen A videophone device having an image and voice communication function capable of a face-to-face conversation with a partner via a screen is provided. In this embodiment as well, the videophone device is considered to be one-to-one in the video conference system. Therefore, in the following description, only the case where the device used by the user is the video conference system will be described. In the case of, the description is omitted unless particularly necessary.

  The remote control device 2 is a button (or a button used for operating the television broadcast receiver and the video conference system) in order to remotely control the television broadcast receiver with an image and sound communication function by a radio signal using radio waves or infrared rays. Key). Furthermore, a button is provided that enables an input operation to turn on / off the output of the synthesis control signal from the conversation status determination unit 41, which will be described later, or to change the control content of the synthesis control signal. When the user operates the remote control device 2, the user's operation content is transmitted to the conversation state determination unit 41 via a radio signal using radio waves or infrared rays, and a composite control signal from the conversation state determination unit 41 is further received. The screen layout and the sound mixing method can be finally switched according to the user's request.

  Further, the remote control device 2 is provided with a button that can be automatically adjusted to a program that other conversation participants are viewing. When the user presses the button, the television broadcast receiver with audio communication function 1 displays the image of the program being watched by another conversation participant and the television broadcast image with audio communication function. The system automatically switches the program to be watched to the same program as other conversation participants from the inquiry and reply contents.

  The system control unit 11 is connected (not shown) to each unit in the television broadcast receiver 1 with an image and audio communication function, and controls the operation of the entire receiver.

  The video / audio input unit 12 is connected with, for example, an image input unit that captures a user such as a video camera and a voice input unit that inputs a user's speech such as a microphone, and inputs the video and audio of the user. And output to the audio / video adjustment unit 13.

  The video / audio adjustment unit 13 receives an input video signal from the imaging unit and an input audio signal from the audio input unit via the video / audio input unit 12 and converts them into a format suitable for subsequent image and audio communication processing. The data is sent to the communication encoder unit 33 and the conversation state determination unit 41.

  The video synthesis unit 14 receives a television video signal from the television decoder unit 22 and a communication video signal from the communication decoder unit 34, and further receives a synthesis control signal from the conversation status determination unit 41 during image and audio communication. Then, a synthesized video signal obtained by synthesizing an image of a program received by television broadcasting and an image inputted from a communication line based on the synthesis control signal is output. Specifically, the video synthesizing unit 14 outputs a synthesized video signal synthesized according to the screen layout of the synthesis control signal including the contents of a plurality of synthesized layout templates having different dimensional ratios of the synthesized images, and the display unit 15 Output to and display.

  The display unit 15 displays an image of a program received by television broadcasting, and displays a composite video signal based on the composite control signal during image and audio communication.

  The audio mixing unit 16 receives the TV audio signal from the TV decoder unit 22, the communication audio signal from the communication decoder unit 34, the input audio signal from the video / audio adjustment unit 13, and the synthesis control signal from the conversation state determination unit 41. Thus, a mixed audio signal is output by changing the volume level of each conversation participant based on the synthesis control signal. Specifically, the sound mixing unit 16 outputs a mixed sound signal mixed according to the ratio of the synthesis control signal including the contents of a plurality of sound mixing templates in which the sound levels corresponding to the respective images are mixed. It is output to the output unit 17 and output as sound.

  The remote operation receiving unit 18 receives an image from the remote operation device 2 and a radio signal for remotely operating the television broadcast receiver with a voice communication function, and outputs it to the system control unit 11. Thereby, the user's operation content is received by the television broadcast receiver with an image and audio communication function.

  The tuner unit 21 is connected to an antenna (not shown) and the television decoder unit 22 and receives broadcast waves received by the antenna. Thereafter, a reception signal of a channel (program) tuned (tuned) according to a user instruction or the like is output to the television decoder unit 22.

  The television decoder unit 22 decodes the digital signal of the tuned channel (program) of the received television broadcast, and includes a television video signal and a television audio signal, and various modes and program classifications attached thereto. The information and the television control signal are detected, the decoded video signal is input to the video synthesis unit 14, and the decoded audio signal is input to the audio mixing unit 16 and output.

  The communication interface unit 31 is connected to a communication line (not shown) and the communication control unit 32. A signal input from the communication control unit 32 is transmitted to a communication partner during a call via the communication interface unit 31 and the communication line, and a signal input from the communication line is transmitted to the communication control unit 32.

  The communication control unit 32 is connected to the communication interface unit 31, the communication encoder unit 33, the communication decoder unit 34, the video / audio adjustment unit 13, and the conversation state determination unit 41. Controls the input / output of the accompanying communication control signals.

  The communication control unit 32 outputs, for example, a user's speech volume information and voice volume level information, channel information and supplementary information of a TV broadcast program viewed by the user, user attention point information, , Including the contents instructed by the radio signal from the remote operation device 2. On the other hand, the communication control unit 32 receives, for example, the communication partner's utterance volume information and utterance volume level information, the channel information and supplementary information of the TV broadcast program viewed by the user, and the user's attention from the input communication control signal. Information and contents instructed by a radio signal from the remote control device of the communication partner are detected.

  The communication encoder 33 encodes the video signal (camera input) and the audio signal (microphone input) input from the video / audio adjustment unit 13 and sends them to the communication control unit 32.

The communication decoder 34 decodes an input signal from the communication line and detects a communication video signal and a communication audio signal.
The video decoded by the communication decoder unit 34 is input to the video synthesis unit 14, and the audio is input to the audio mixing unit 16.

  The conversation status determination unit 41 is connected to the video / audio adjustment unit 13, the television decoder unit 22, the communication control unit 32, and the communication decoder unit 34 (not shown), and determines video information / audio information and other information to determine the conversation status. Additional information is collected. For example, an input audio signal from the video / audio adjusting unit 13, channel information from the TV decoder unit 22, a TV audio signal and a TV control signal, a communication audio signal from the communication decoder unit 34, and a communication control signal from the communication control unit 32 are included. The input signal is used to determine the conversation status between the user and the communication partner using these signals, and a composite control signal is output. The TV control signal includes genre information of the TV broadcast program being viewed by the user and sound mode information of the TV broadcast program being viewed by the user. The communication control signal includes genre information and / or audio mode information in the incidental information of the TV broadcast program viewed by the communication partner.

  For example, in the conversation status determination unit 41, the user's speech volume and / or utterance volume level detected from the input audio signal from the video / audio adjustment unit 13 and the communication detected from the communication audio signal from the communication decoder unit 34. The conversation status is determined based on the other party's utterance volume and / or utterance volume level, and a synthesis control signal is generated so that an image of a person with a higher utterance volume and / or a person with a higher utterance volume level is displayed larger.

  The conversation state determination unit 41 receives a communication control signal from the communication control unit 32 and determines the number of conversation participants from the communication control signal. When the number of conversation participants is three or more, for example, the composite control signal is generated so that the images of the conversation participants increase in the order of the largest amount of speech, or the person who has the largest amount of speech The synthesis control signal is generated so that the image of the image becomes larger.

  The conversation state determination unit 41 determines the point of interest in which the user is paying attention on the display unit 15 based on the input video signal from the video / audio adjustment unit 13 and / or the remote control device instruction input information. A portion 54 is provided. The conversation status determination unit 41 can determine the conversation status and output a synthesis control signal by using the attention point information of all the conversation participants. For example, judgment of conversation status, synthesis control of the screen of the TV broadcast program to be watched, synthesis control of the screen of the most watched person in the entire conversation participant, each paying attention to each other in the entire conversation participant The information on the attention point can be used for the composition control of the conversation participant's screen.

  The synthesis control unit 42 receives the determination information from the conversation state determination unit 41 and controls the video synthesis unit 14 and the audio mixing unit 16. Specifically, the composition control unit 42 can control the layout and dimensions of each of the images to be synthesized, and can be synthesized in accordance with a synthesis layout template stored in advance, The volume level can be controlled individually for each voice to be mixed, and a synthesis control signal that can be mixed according to a voice mixing template held in advance is output.

  The synthesis control unit 42 outputs an instruction (synthesis control signal) for synthesizing the screen to the video synthesis unit 14 in correspondence with the content of the template as the synthesis control signal received from the conversation state determination unit 41, and the received template of the received template. In response to the content, an instruction (synthetic control signal) for mixing sounds is output to the sound mixing unit 16.

FIG. 2 is a block diagram illustrating an example of a schematic internal configuration of the conversation state determination unit 41 in FIG.
The net search unit 51 is connected to the communication control unit 32 and a program information determination unit 56 to be described later, and searches through the communication control unit 32 for a site that is open on the Internet or the like and is viewing or is scheduled to view the site. By collecting program information of a program, for example, genre information of an analog broadcast program being viewed can be searched. In many cases, the genre information of a program is added to a control signal or the like of a program that is broadcast in digital broadcasting, but a control signal is not added in analog broadcasting. Therefore, when searching for program information of such an analog broadcasting program In addition, when the program information in the control signal added to the digital broadcast is insufficient, for example, when investigating more detailed genre classification than the genre classification in the program additional information, The search used is effective.

  The person-of-interest determination unit 52 is connected to the communication control unit 32, a later-described attention point determination unit 54, and a later-described combination mode selection unit 73, and the attention point determination unit 54 receives the determination result of the user's attention point. The On the other hand, from the communication control unit 32, information on which communication partner each communication partner is paying attention to is input. For example, communication that is most noticed among all video conference conversation participants (conversation participants). Detect (estimate) the other party. Or you may detect so that it may rank in steps according to the attention degree of each conversation participant, for example. Regarding the detection result of the attention point of the user of the image and audio communication function-equipped television broadcast receiver 1, the communication partner 32 and the communication partner image and audio communication function-equipped television broadcast receiver 1 are connected via the communication control unit 32 and the communication line. To send. Further, for example, in order to improve the accuracy of detection and to reduce the calculation load in the television broadcast receiver 1 with the image and audio communication function of the communication partner, the communication result of the communication partner who has received the most attention You may send to the other party's image and audio | voice communication function-equipped television broadcast receiver 1.

  The line-of-sight detection unit 53 is also connected to the communication control unit 32, the point-of-interest detection unit 54, and the synthesis mode selection unit 73. The target point detection unit 54 receives the detection result of the user's point of interest, and the communication control unit 32. The information on which communication partner is paying attention to is obtained from each communication partner. In the line-of-sight coincidence detection unit 53, for example, the attention point of the user of the television broadcast receiver 1 with this image and voice communication function is obtained. And the image of each communication partner and the attention point of the user (communication partner) in the television broadcast receiver 1 with the voice communication function mutually pay attention to each other, that is, in a pseudo manner via the TV screen. It is detected (estimated) whether each communication partner and the line of sight match, thereby detecting the communication partner whose line of sight matches.

  The attention point detection unit 54 is connected to the video / audio adjustment unit 13, the video composition unit 14, the person-of-interest detection unit 52, the attention point detection unit 54, and the synthesis mode selection unit 73. This image input to the input unit 12 and the video of the user of the television broadcast receiver 1 with a voice communication function are input. Information about what image is displayed on the screen is input from the video composition unit 14. The attention point detection unit 54 detects, for example, the position of the eyes with respect to the contour of the face, the position of the black eyes with respect to the entire eye, the contour of the black eyes, and the like from the input user image. The point of interest on which the user is paying attention is detected by comparing with the information on what image is displayed where on the screen. As another method for detecting the attention point, for example, when a light source can be installed in the vicinity of the display unit 15, the attention point is detected by using retinal reflected light as disclosed in JP-A-6-319701. A detection method can be used.

  The audio mode detection unit 55 is connected to the television decoder unit 22 and the synthesis mode selection unit 73, and audio related information in the incidental information of the broadcast signal is input from the television decoder unit 22. From the audio-related information, for example, it is detected whether the audio mode of the program currently viewed by the user is the bilingual mode or the stereo mode, and the detection result is output to the synthesis mode selection unit 73.

  The program information detection unit 56 is connected to the television decoder unit 22, the net search unit 51, and the synthesis mode selection unit 73, and program related information in the incidental information of the broadcast signal is input from the television decoder unit 22. The genre of the program currently viewed by the user is detected from the program related information, and the detection result is output to the synthesis mode selection unit 73. When the program related information from the TV decoder unit 22 such as analog broadcasting is not input, or when it is desired to obtain program related information (genre division) more detailed than the program related information in the accompanying information, the program information detecting unit 56 The detailed search genre of the program currently viewed by the user can be checked by searching the program information published on the net by the net search unit 51.

  The climax detection unit 61 is connected to the television decoder unit 22, the program information detection unit 56, and the composition mode selection unit 73, and the genre information of the program currently viewed by the user is input from the program information detection unit 56. The program swell situation and highlight scene specific to the program genre are detected from the audio signal from the television decoder unit 22, and the detection result is output to the synthesis mode selection unit 73. For example, in a soccer broadcast as an example of a program of a specific genre, the announcer increases the volume (volume) and the tone in the goal scene. It is often high. Similarly, in the case of a baseball home run, the announcer's volume (volume) increases and the tone is often high. From this, it can be said that there is a correlation between the excitement of soccer and baseball games and the volume and tone of the announcer. Using the correlation between the announcer's voice volume and tone and the game's excitement, the program's excitement can be detected.

  The total call volume detecting unit 62 is connected to the video / audio adjusting unit 13, the communication decoder unit 34, and the synthesis mode selecting unit 73. The video / audio adjusting unit 13 inputs the image and audio communication function. The voice of the user of the attached television broadcast receiver 1 is input, and the voice of the communication partner is input from the communication decoder unit 34. The total volume of the voice of the user and the voice of the communication partner is combined. (Speech volume) and duration are detected, the total amount of calls in the communication function is detected from the overall volume and duration statistical processing results, and the detection results are output to the synthesis mode selection unit 73. For the total call volume, the total call volume detection unit 62 detects, for example, statistics of the volume of conversational voice for the past 1 minute as the duration. Further, for example, it may be possible to detect the excitement of the conversation from the entire audio signal using the voice volume and tone of the communication partner as in the excitement detection unit 61.

  The most-speaker detection unit 63 is also connected to the audio / video adjustment unit 13, the communication decoder unit 34, and the synthesis mode selection unit 73, and the image and audio communication input to the audio / video input unit 12 from the audio / video adjustment unit 13. While the voice of the user of the television broadcast receiver with function 1 is input and the voice of the communication partner is input from the communication decoder 34, the individual volumes of the user's voice and the voice of the communication partner are input. (Speech volume) and duration are detected, and from the statistical processing results of the individual volume and duration, the communication partner who speaks the most is detected or the communication partner is divided by ratio in order of the amount of speech. The detection result and the ratio are output to the synthesis mode selection unit 73.

  The detection content determination table 71 is connected to the synthesis mode selection unit 73, and in order to select the synthesis mode of the screen layout, the determination criteria for the detection content such as the total amount of calls determined in advance through experiments and examinations and the specific gravity of the detection content Coefficients are stored in table format.

  The layout / sound mixed template 72 is also connected to the synthesis mode selection unit 73, and a screen layout template in which each image of a conceivable situation is arranged in various sizes at various positions, and the voice of each conversation participant. Audio mixing templates to be mixed at various levels are stored.

  The synthesis mode selection unit 73 is connected to the detection content determination table 71 and the layout / sound mixing template 72, and further includes a subject detection unit 52, a line-of-sight detection unit 53, a point of interest detection unit 54, a voice mode detection unit 55, The program information detection unit 56, the climax detection unit 61, the total call amount detection unit 62, and the most frequent speaker detection unit 63 are connected. The synthesis mode selection unit 73 includes a person-of-interest detection unit 52, a line-of-sight detection unit 53, a point of interest detection unit 54, a voice mode detection unit 55, a program information detection unit 56, a climax detection unit 61, a total call amount detection unit 62, Each detection result is input from the most-speaker detection unit 63, corrected by multiplying each detection result by the detection criterion and specific gravity of the detection content determination table 71, and based on each corrected detection result. A synthesis mode is selected, a screen layout and a voice mixing template that match the selected synthesis mode are selected, and the selection result is output to the synthesis control unit 42 as an instruction (synthesis control signal).

  Thus, in addition to the signals input / output via the communication line, audio signals and control signals from the TV decoder unit 22 are also input to the synthesis mode selection unit 73. In the description of the block, the operation content of each block will be described by simplifying only the operation based on signals input / output through a basic communication line.

  The broadcast / call ratio calculation unit 81 inside the synthesis mode selection unit 73 uses the determination result and the coefficient of the detection content determination table 71 for the input detection result of the total call amount detection unit 62, and the ratio of the conversation priority and the broadcast priority. By calculating, it is determined which of the broadcast viewing and the video conference conversation is focused, and the determination result is output to the template selection unit 84. As the determination result, for example, a candidate template is selected from the layout / sound mixing template 72 and the selection result is output to the template selection unit 84.

  The specific speaker emphasis detection unit 82 calculates the communication partner to be displayed with emphasis by using the determination criteria and coefficient of the detection content determination table 71 based on the detection result of the input most frequent speaker detection unit 63, thereby specifying a specific communication. It is determined whether or not the partner's emphasis display is performed and which communication partner's emphasis display is performed, and the determination result is output to the template selection unit 84. As the determination result, for example, a candidate template is selected from the layout / sound mixing template 72 and the selection result is output to the template selection unit 84.

  The user selection mode storage unit 83 receives the video synthesis mode or the audio mixing mode selected by the remote scanning device 2 from the system control unit 11, and selects candidates from the layout / audio mixing template 72 according to the selected mode, for example. Select a template to be stored.

  The template selection unit 84 selects a final template from the layout / audio mixed template 72 based on the results of the broadcast / call ratio calculation unit 81, the specific speaker priority detection unit 82 and the stored contents of the user selection mode storage unit 83. The content of the template is selected and transmitted to the synthesis control unit 42 as a synthesis control signal.

  The general operation of the television broadcast receiver 1 with an image and audio communication function when a television broadcast program viewing and a conversation by the video conference system are simultaneously performed will be described.

  First, an operation related to television broadcast program viewing will be described. In the television broadcast receiver 1 with an image and audio communication function, a broadcast wave received by an antenna is tuned by a tuner unit 21 and decoded by a television decoder unit 22, where video signals and audio signals of a television broadcast program are obtained. And a control signal including incidental information including various modes and program classifications is detected and output. The detected video signal of the program is displayed as a video on the display unit 15 via the video synthesis unit 14. The detected audio signal of the program is output from the audio output unit 17 via the audio mixing unit 16. In this way, video and audio of a television broadcast program are provided by the television broadcast receiver 1 with an image and audio communication function.

  Next, the operation related to the conversation by the video conference system will be described. The conversation data of the video conference system transmitted via the communication line is received by the communication interface unit 31 and data necessary for control is detected by the communication control unit 32, and then decoded by the communication decoder unit 34. Are output as video and audio signals. When there are a plurality of communication partners, the output (video signal and audio signal) is separated for each communication partner based on the control signal in the communication control unit 32 or the decoding result in the communication decoder unit 34. The detected video signal is displayed on the display unit 15 via the video synthesis unit 14, and the detected audio signal is output from the audio output unit 17 via the audio mixing unit 16. In this way, the video and audio related to the communication partner in the video conference system are provided by the television broadcast receiver 1 with the image and audio communication function.

  In the conversation by the video conference system, the user's own video is also displayed for confirmation, and the operation will be described. For example, a video signal input from a video camera and an audio signal input from a microphone or the like are processed by the video / audio adjustment unit 13 into a format suitable for use in a video conference system via the video / audio input unit 12. Are input to the communication encoder 33 and encoded. The video signal and the audio signal encoded by the communication encoder unit 33 are sent to the communication partner via the communication control unit 32 and the communication interface unit 31. At this time, the video signal output from the video / audio adjustment unit 13 is displayed on the display unit 15 via the video synthesis unit 14, and the audio signal is output from the audio output unit 17 via the audio mixing unit 16. Is done. In this way, video and audio related to the user himself / herself in the video conference system is provided by the television broadcast receiver 1 with an image and audio communication function.

  In an apparatus that simultaneously performs both viewing of a program and conversation of a video conference system as in the present embodiment, a television broadcast program, one or a plurality of communication partners, and a plurality of videos of the user himself / herself are displayed simultaneously. It is necessary to output a plurality of sounds at the same time. Each video of these pieces of information is synthesized and displayed as one image by the video synthesizing unit 14, and each audio is output as audio mixed by the audio mixing unit 16.

  Here, when the video of the above-mentioned television broadcast program, the video of the communication partner in the video conference system, and the video of itself in the video conference system are combined and displayed on one screen, the layout is a combination of the videos. There are many possible patterns. For example, if the program viewing is important, the program screen is displayed large and the communication partner or yourself is displayed small, but if the conversation is important, the communication partner or self is displayed large and the program screen is small. Will be displayed. In the present embodiment, a template is created in accordance with each of the many patterns and stored in the conversation state determination unit 41.

FIG. 3 is a diagram showing a screen layout in a case where four people including the person have a conversation as an example of the case where the television viewing receiver with the image and voice communication function of FIG.
When considering the case of having a conversation with a video conference system while watching the same program, it is considered that the optimum screen layout varies depending on the program content. For example, when watching a movie or a drama as a program, it is generally considered that the emphasis is on viewing the program, and less emphasis is placed on the conversation in the video conference system. FIG. 3 shows an example of the screen layout in such a case. In the composite display screen 101, the program screen 601 to be emphasized is displayed with a relatively large display size, and the four conversation participants 201 of the video conference system are displayed. 301, 401, and 501 are displayed with relatively small display dimensions. That is, when the program to be viewed is a movie or a drama, the program screen 601 has a relatively large display size, and the display sizes of the four conversation participants 202, 302, 402, and 502 in the video conference system are relatively small. A composite screen such as the layout shown in FIG. 3 is considered suitable.

FIG. 4 is a diagram showing a screen layout in a case where conversation is made by four persons including the person himself as an example of the case where importance is placed on conversation in the television broadcasting receiver with the image and voice communication function of FIG.
For example, when viewing a variety program, it is generally considered that the program viewing is not so focused, and the ratio of the conversation emphasis is equal to the emphasis placed on the conversation in the video conference system. In an extreme case, the content of the television broadcast program is hardly focused on the topic-providing view, and it is also possible that most of the focus is placed mainly on the conversation in the video conference system. FIG. 4 shows an example of the screen layout in such a case. On the composite display screen 102, the program screen 602 that is not emphasized is displayed with a relatively small display size, and the four conversation participants 202 of the video conference system are displayed. , 302, 402, and 502 are displayed with relatively large display dimensions. That is, when the program to be viewed is a variety program, the display size of the program screen 601 is relatively small, and the display size of the four conversation participants 202, 302, 402, and 502 of the video conference system is relatively large. It is considered that a composite screen like the layout of FIG.

  The layout shown in FIGS. 3 and 4 is an example, and various screen layouts other than the layouts shown in FIGS. 3 and 4 may be used as the layout when the program viewing is important or the conversation is important. Can be considered. As an extreme layout, for example, a layout that does not display a communication partner or a layout that does not display a television broadcast program can be considered.

  In the present embodiment, as described above, it is desired that the layout of the composite screen is changed to an optimal layout in accordance with the contents of the program. Therefore, the television broadcast receiver with a voice communication function of the present embodiment is used. Then, according to the contents of the program, the layout when the program viewing is important or the layout when the conversation is important is automatically switched. For this purpose, the TV decoder unit 22 outputs a control signal including various accompanying information transmitted together with the program by broadcasting, and the conversation state detecting unit 41 detects the genre information of the program from the accompanying information. A screen layout corresponding to the genre of the program is output, and the synthesis control unit 42 automatically outputs a control signal (indicating a layout and audio mixing method that can output the screen layout corresponding to the genre of the program). Template).

  In the present embodiment, program information such as supplementary information of a program is not only obtained from a broadcast signal of a received radio wave, for example, but also a TV station connected to a communication line such as the Internet or a site for handling TV-related information Program information can be acquired via the communication control unit 32 and the communication interface unit 31. Then, the conversation status determination unit 41 switches the control signal (template) to be output after detecting the program genre from the program information, and the composition control unit 42 automatically selects the screen layout and sound mixing method according to the program genre. You can switch to

  In addition, the conversation state determination unit 41 according to the present embodiment identifies the state of the voice mode in which the sound mode is switched, such as stereo or bilingual broadcasting, according to the content or situation of the program, and the composition control unit 42 determines the state of the voice mode. It is possible to automatically switch to a sound mixing method according to the.

  Further, in the present embodiment, by utilizing the correlation between the excitement of a game such as a sports broadcast program and the volume / tone of the announcer's voice, the conversation status determination unit 41 can output the sound of the television broadcast program. By detecting and determining, the program excitement status can be determined, and the composition control unit 42 can automatically switch to a screen layout and an audio mixing method according to the program excitement status.

  Further, in the present embodiment, the voice of the video conference system can be checked by the conversation status determination unit 41, and when each utterance amount (voice information amount) through the video conference system is large, it is more than the case of watching TV. It is determined that the focus has been shifted to the conversation, the conversation state determination unit 41 checks the conversation excitement, and the composition control unit 42 automatically switches to a screen layout and a voice mixing method according to the conversation excitement. it can.

  In the present embodiment, the position on the screen that the user is currently gazing at in the conversation status determination unit 41 using the screen layout of the user image and the composite video output from the video / audio adjustment unit 13. By detecting (line of sight), the screen position that the user is looking at can be detected, and the composition control unit 42 can automatically switch to the screen layout and audio mixing method according to the gaze status of the screen. For example, when the line of sight is directed to the screen of a television broadcast program, the screen of the television broadcast program is switched to a larger screen layout. You can switch to a larger layout of the video conferencing system screen.

  In this embodiment, when detecting the user's line of sight, the information on the screen currently being watched is exchanged with the communication partner (video conference conversation participant) via the communication line. The determination unit 41 can detect the degree of attention of all the members participating in the conference. In this case, the composition control unit 42 can automatically switch to a screen layout and a sound mixing method according to the screen gaze situation of all the conference participants.

  In this embodiment, the screen layout and the sound mixing method can be automatically switched according to the sound and line of sight, but the screen layout and the sound mixing method are not necessarily optimal for the user. Therefore, in the present embodiment, when the user further operates the remote control device 2, the screen layout and the voice mixing method desired by the user are transmitted to the conversation state determination unit 41, and the screen layout is determined by the conversation state determination unit 41. In addition, the voice mixing method can be switched according to the user's wishes.

  Further, in the present embodiment, information regarding the screen layout desired by the user and the switching of the voice mixing method is transmitted to the communication partner via the communication line, and the combined screens of both the user and the communication partner are set to the same screen layout. Can be aligned.

Up to this point, the screens of each conversation participant in the video conference system have the same dimensions, and the screen layout switching of the composite screen with the screen of the television broadcast program has been described. However, in the case of a video conference system with multiple members In some cases, it is better to individually control the size relationship of the screen dimensions of each conversation participant in the video conference system.
For example, it is possible to control such that a conversation participant having a large amount of speech at the center of the conversation is displayed in a large size and a conversation participant having a small amount of speech is displayed in a small size.

  FIG. 5 is a diagram showing a screen layout of a composite screen in which the screens of the four conversation participants of the video conference system have the same dimensions and are combined with the screen of the television broadcast program, and FIG. 6 shows the screen layout of the video conference system. It is a figure which shows the screen layout of the synthetic | combination screen which displayed only the specific conversation participant among four conversation participants large, and was synthesize | combined with the screen of the program of a television broadcast.

  In the composite display screen 103 of FIG. 5, the program screen 603 is displayed with a relatively large display size in the center of the composite display screen 103, and the four conversation participants 203, 303, 403, and 503 of the video conference system are combined. Display dimensions are displayed relatively small in the four corners of the display screen 103. On the other hand, in the composite display screen 104 of FIG. 6, the program screen 604 is similarly displayed at a relatively large display size in the center of the composite display screen 104, but the four conversation participants 204, 304, Concerning 404 and 504, the conversation participants 204, 404, and 504 are similarly displayed with relatively small display dimensions at each corner of the composite display screen 103, but the conversation participant 304 is enlarged to the lower left corner of the composite display screen 103. Has been displayed larger.

  In the present embodiment, when the screen layout is switched from the composite screen of FIG. 5 to the composite screen of FIG. 6, for example, in the situation where the composite screen of FIG. The unit 63 detects the most-speaker from the speech of each conversation participant, determines that the most-speaker has a high level of attention, selects a template in which the image of the most-speaker is enlarged, and performs synthesis control as a synthesis control signal To the unit 42. The composition control unit 42 automatically switches the composition control signal so that the image synthesized by the image composition unit 14 has a screen layout in which the image of the most-speaker is displayed large.

  Further, in the present embodiment, when switching the screen layout from the composite screen of FIG. 5 to the composite screen of FIG. 6, instead of the method of detecting the most-speaker from the voice and controlling the screen layout, the conversation status determination The attention point detection unit 54 of the unit 41 detects the user's line of sight to detect the screen of the conversation participant being watched by the user, and determines that the conversation participant has a high degree of attention. It is possible to use a method of selecting a template in which an image of a conversation participant having a high level is enlarged and outputting it as a synthesis control signal to the synthesis control unit 42. In that case, the synthesis control unit 42 automatically switches the synthesis control signal so that the image synthesized by the video synthesis unit 14 has a screen layout in which a video of a conversation participant having a high degree of attention is displayed large.

  In this embodiment, when detecting the user's line of sight, the information on the screen currently being watched is exchanged with the communication partner (video conference conversation participant) via the communication line. The determination unit 41 can detect the degree of attention of all the members participating in the conference. In such a case, the composition control unit 42 generates a composition control signal so that a screen layout for displaying a video image of a communication partner that many conversation participants are paying attention to according to the screen gaze situation of all the conference participants. It can be switched automatically.

  In this embodiment, when detecting the user's line of sight, the information on the screen currently being watched is exchanged with the communication partner (video conference conversation participant) via the communication line. The determination unit 41 detects (estimates) whether the lines of sight coincide with each other in a pseudo manner via the TV screen, and only the screens of the conversation participants whose lines of sight match each other are arranged large. The screen layout can be automatically switched to.

For example, when the conversation participant 204, the conversation participant 304, the conversation participant 404, and the conversation participant 504 in FIG. 6 are having a conversation, the conversation participant 204 watches the screen of the conversation participant 304. It is assumed that the conversation participant 304 is also watching the screen of the conversation participant 204, the conversation participant 404 is watching the conversation participant 204, and the conversation participant 504 is watching the television screen. In this case, the screen in which the conversation participant 204 is gazing is displayed with a large image of the conversation participant 304, and the screen in which the conversation participant 304 is gazing is displayed in a large size. It becomes. In this case, FIG. 6 shows a screen at which 204 is gazing.
Further, in the screen in which the conversation participants 404 and 504 are watching, the images of all four conversation participants are displayed in an equal size.

  When there are four or more conversation participants, for example, the conversation participant 204 and the conversation participant 304 in FIG. 6 pay attention to each other and have a conversation, and the conversation participant 404 and the conversation participant 504 interact with each other. As in the case of having a conversation while paying attention to each other, there is a case where a process of specifying each communication partner by visual line detection and displaying the communication partner in large size is performed in a plurality of sets. In this case, the screen layout can be switched to a screen layout in which the other party who is gazing at each other is displayed greatly by detecting the line of sight, but since the volume of the two conversational voices is mixed and output for the voice, There are cases where it is difficult to hear the voices of each party in each group. In such a case, in the present embodiment, when adjusting the screen size of each pair of individual opponents, it is possible to make adjustments so that the mixing ratio of the other party's voice is increased. Therefore, in this embodiment, even when there are a plurality of conversations by a plurality of conversation participants in one screen of the video conference system, the mixing ratio of the other party's voice corresponding to each conversation participant is increased. Therefore, it is easy for individual conversation participants to easily identify the voice of their conversation partner (easy to hear).

  As described above, in the television broadcast receiver with the image and audio communication function according to the present embodiment, the display size of each screen on the composite screen is switched in association with the importance of each conference conversation participant with respect to the content of the program. In addition, the screen display size of each conference conversation participant on the composite screen can be switched in association with the individual importance for each conference conversation participant, and the voice of the corresponding conversation partner can also be switched. Therefore, it is possible to provide a television broadcast receiver with an image and audio communication function that is natural and easy to use for the user.

Embodiment 2. FIG.
In Embodiment 1 described above, the attention point and line-of-sight detection results of each conference conversation participant were used for switching the screen layout. A common topic can be provided to each conversation participant by marking each conversation participant's attention point on the screen of the same television program on the composite screen. In the second embodiment described below, a case will be described in which the attention point of each conversation participant is marked on the screen of the same television program in such a composite screen.

The configuration of the present embodiment is basically the same as that of the first embodiment shown in the block diagrams of FIGS. 1 and 2, but some of the configuration blocks have differences described below.
A specific symbol indicating the attention point of each conversation participant is stored in advance in the layout / audio mixed template 72 in the conversation state determination unit 41, and the attention point of each conversation participant detected by the attention point detection unit 54 is Synthesize video with a specific symbol. For example, a first specific symbol indicating each of the attention points of each conversation participant and a second specific symbol indicating the most noticed point most noticed from each attention point are stored in the layout / voice mixed template 72. In this case, the video of the first specific symbol is synthesized with the attention point of each conversation participant detected by the attention point detection unit 54, and the video of the second specific symbol is synthesized with the most noticed point.

FIG. 7 is a diagram in which a specific symbol indicating the attention point of the conversation participant is added to the screen layout when the conversation is performed with four persons including the person in FIG. 3.
FIG. 7 shows a case where emphasis is placed on program viewing as in FIG. 3 and less emphasis is placed on conversation in the video conference system. In the composite display screen 105, the program screen 605 has a relatively small display size. The four conversation participants 205, 305, 405, and 505 of the video conference system are displayed with a relatively small display size and the same size.

The result of detecting the line of sight (attention point) of each conversation participant 205, 305, 405, 505 in the present embodiment is exchanged via a communication line, and each conversation is displayed on the program screen 605 of the synthesized screen to be watched. A specific symbol such as a “◯” mark indicating the gazing point of the participants 205, 305, 405, and 505 is displayed.
As the specific symbol, a symbol having a shape and tone that is difficult to be blended with the surrounding image and tone such as a circle “◯” and a cross “+” is used so that the display position can be emphasized. Further, the specific symbol may be emphasized by a change in luminance, a change in color, or the like. This specific symbol display function allows each conversation participant 205, 305, 405, 505 to know the position on the screen where each participant is paying attention, so that a common topic is provided to each participant.

  The mark on the screen allows each conversation participant to be displayed with the same specific symbol so that an individual cannot be identified, but depending on the purpose, it is better to know who is viewing the line of sight among each conversation participant Since it may be good, in that case, a different specific symbol is displayed for each conversation participant. In this case, the specific symbol that is different for each conversation participant may be, for example, who has the same line-of-sight mark displayed on the screen by color coding or partial modification of the shape. Can be easily identified.

  When performing identification by color coding, for example, the border lines of the face display of the conversation participants 205, 305, 405, and 505 displayed on the screen of FIG. 7 are displayed in different colors. The frame of the conversation participant 205 is red, the frame of the conversation participant 305 is blue, the frame of the conversation participant 405 is green, the frame of the conversation participant 505 is yellow, and the like. Next, a specific symbol (circle) indicating the line of sight of each conversation participant 205, 305, 405, 505 is displayed in a circle with the same color as the color of the frame of each participant. By color-coding in this way, the line of sight of each conversation participant 205, 305, 405, 505 is easily compared with the color of the specific symbol and the frame of the face display, so that the specific symbol indicating the line of sight can be easily determined. Can be identified.

  Thus, in this Embodiment, a common topic can be provided to each conversation participant by marking the attention point of each conversation participant.

Embodiment 3 FIG.
In Embodiment 2 described above, the attention point of each conversation participant is marked on the screen of the same TV program on the composite screen, but the same TV program screen is displayed on the composite screen viewed by each conversation participant. The method was not described. Even in other cases, in a television broadcast receiver with an image and audio communication function, a conversation may be performed on the assumption that conversation participants are watching the same television broadcast program. In Embodiment 3 described below, a method of displaying the same TV program screen on the composite screen viewed by each conversation participant will be described.

The configuration of the present embodiment is basically the same as that of the first embodiment shown in the block diagrams of FIGS. 1 and 2, but some of the configuration blocks have differences described below.
The program information detection unit 56 in the conversation status determination unit 41 uses the information of the switched channel when the user switches the channel and the like, and the television broadcast receiver with an image and voice communication function of other conversation participants. Send to. In addition, the remote control device 2 inquires about the program being viewed from the television broadcast receiver with image and voice communication function of other conversation participants, receives the result, and automatically selects the channel of the program. And a button for matching the program that the other conversation participant is viewing with the program that the user is viewing.

  As a method of displaying the same TV program screen on the composite screen viewed by each conversation participant, for example, when the user switches the channel, the information of the switched channel is displayed as the image of the other conversation participant. It is also necessary to transmit to a television broadcast receiver with a voice communication function. Therefore, when a channel change scan is performed by the remote operation device 2, the program information detection unit 56 obtains channel information currently being viewed from the control information of the television decoder unit 22, for example, a conversation status determination unit The channel information is transmitted to the other conversation participants from the communication control unit 32 and the communication interface unit 31 via the communication line using the program information detection unit 56 and the net search unit 51 in 41. On the other hand, in a television broadcast receiver with an image and audio communication function that has received the channel information, the channel information is automatically switched to the channel by using the program information detection unit 56 and the net search unit 51 in the conversation state determination unit 41. Like that.

  In the television broadcast receiver with the image and voice communication function of each conversation participant, depending on the purpose, the channel of the television broadcast receiver with the own image and voice communication function without ignoring or transmitting the channel switching information. You can also switch only.

  In addition, when a new conversation participant is added to the current conversation participant, for example, the remote control device is provided with a button for adjusting to the program that the other conversation participant is viewing, and the button is pushed. Inquiries are made to the television broadcast receiver with image and audio communication functions of other communication partners about the program being viewed, and the channel is automatically switched to that channel.

  Thus, in the present embodiment, it is possible to display the same television program screen on the composite screen viewed by each conversation participant.

Embodiment 4 FIG.
In the first embodiment described above, the main communication partner is easily identified by mainly switching the size of each screen, the screen layout, and the mixing ratio of the audio. However, for example, the capability of the communication decoder unit 34 is poor. When it is difficult to change the image display size of each conversation participant, for example, the main communication partner can be easily identified by switching the frame rate of the image of each conversation participant. In the fourth embodiment described below, a case where the frame rate of each conversation participant's image is switched will be described.

FIG. 8 is a block diagram showing a schematic configuration of an example according to a television broadcast receiver with an image and audio communication function according to the fourth embodiment of the present invention.
The configuration of the present embodiment shown in FIG. 8 is basically the same as that of the first embodiment shown in FIG. 1, but has the differences described below. The same applies to FIG.
The layout / sound mixing template 72 stores a screen layout template in which each image of a conceivable situation is increased or decreased at various positions at various positions.
The synthesis control unit 42 includes a frame rate control unit 43 that uses the synthesis control signal to increase the frame rate for an image to be displayed in an enhanced manner.
The frame rate control unit 43 reduces the frame rate for an image that does not need to be displayed with emphasis.

In the present embodiment, for example, the image of the communication partner of interest is displayed at 30 frames per second, and the other communication partner is displayed at one frame per second. The communication partner can be easily identified.
This method of changing the frame rate can be used in combination with the above-described method of changing the image size.

  As described above, in this embodiment, it is possible to easily identify the main communication partner by switching the frame rate of the image of each conversation participant.

FIG. 1 is a block diagram showing a schematic configuration of an example of a television broadcast receiver with an image and audio communication function according to Embodiment 1 of the present invention. It is a block diagram which shows an example of the schematic internal structure of the conversation condition determination part 41 of FIG. It is a figure which shows the screen layout in the case of having a conversation with four persons including the person as an example in the case of placing importance on program viewing on the television broadcast receiver with the image and voice communication function of FIG. It is a figure which shows the screen layout in the case of having conversation with four persons including the principal as an example in the case where importance is placed on conversation in the television broadcast receiver with the image and voice communication function of FIG. It is a figure which shows the screen layout of the synthetic | combination screen which synthesize | combined with the screen of the program of a television broadcast by making the screen of four conversation participants of a video conference system into the same dimension, respectively. It is a figure which shows the screen layout of the synthetic | combination screen which displayed only the specific conversation participant among the four conversation participants of a video conference system, and was synthesize | combined with the screen of the program of a television broadcast. It is the figure which added and added the specific symbol which shows the attention point of a conversation participant to the screen layout in the case of having a conversation with four persons including the principal of FIG. It is a block diagram which shows schematic structure of an example which concerns on the television broadcast receiver with the image and audio | voice communication function of Embodiment 4 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Television broadcast receiver with an image and audio | voice communication function, 2 Remote operation apparatus, 11 System control part, 12 Image | video audio | voice input part, 13 Image | video audio | voice adjustment part, 14 Image | video synthetic | combination part, 15 Display part, 16 Audio | voice mixing part, 17 Audio output unit, 18 remote operation receiving unit, 21 tuner unit, 22 TV decoder unit, 31 communication interface unit, 32 communication control unit, 33 communication encoder unit, 34 communication decoder unit, 41 conversation state determination unit, 42 synthesis control unit, 43 frame rate control unit, 51 net search unit, 52 subject detection unit, 53 line-of-sight detection unit, 54 attention point detection unit, 55 voice mode detection unit, 56 program information detection unit, 61 climax detection unit, 62 call total amount Detection unit, 63 Most-speaker detection unit, 71 Detection content determination unit, 72 Layout / sound mixing template 73 Synthesis mode selection unit, 81 broadcast-call ratio calculator, 82 specific speaker focus detection unit, 83 user selection mode storage unit, 84 template selecting section.

Claims (27)

A television decoder for decoding a received television broadcast digital signal to detect a television video signal and a television audio signal and a television control signal attached to them;
A video / audio adjustment unit that converts an input video signal from the imaging unit and an input audio signal from the audio input unit into a format suitable for image and audio communication processing;
A communication decoder that decodes an input signal from a communication line to detect a communication video signal and a communication audio signal;
A communication control unit for controlling input / output of video and audio input / output signals and communication control signals attached to the communication line;
The TV video signal from the TV decoder unit and the communication video signal from the communication decoder unit are input, and at the time of image and audio communication, the image of the program received by the television broadcast and the image input from the communication line are combined. A video synthesis unit for outputting the synthesized video signal,
A television broadcast receiver with an image and audio communication function including at least a display unit for displaying an image of a program received by the television broadcast and displaying the composite video signal at the time of image and audio communication,
A conversation status determination unit that receives at least an input audio signal from the video / audio adjustment unit and a communication audio signal from the communication decoder unit, determines a conversation status between a user and a communication partner, and outputs a synthesis control signal; ,
The video synthesis unit further receives a synthesis control signal from the conversation situation determination unit, synthesizes the synthesized video signal based on the synthesis control signal,
The display unit displays a composite video signal based on the composite control signal. A television broadcast receiver with an image and audio communication function. The conversation status determination unit is configured to perform a conversation based on a user's utterance detected from an input audio signal from the video / audio adjustment unit and a communication partner's utterance detected from a communication audio signal from the communication decoder. The television broadcast receiver with an image and a voice communication function according to claim 1, wherein the composite control signal is generated so that a situation is determined and an image of a person with a large amount of speech is displayed in a large size. The said video composition part outputs the synthesized video signal synthesize | combined according to the said synthesis | combination control signal containing the content of the some synthetic | combination layout template from which the dimension ratio of each image synthesize | combined differs. A television broadcast receiver with the described image and audio communication function. The communication control unit outputs at least the communication control signal to be output including user's speech amount information, and detects the speech amount information of the communication partner from the input communication control signal. Item 5. A television broadcast receiver with an image and audio communication function according to any one of Items 1 to 3. The conversation status determination unit receives a communication control signal from the communication control unit, determines the number of conversation participants from the communication control signal, and when there are three or more, each conversation participation in the order of the amount of speech The television broadcast receiver with an image and audio communication function according to claim 4, wherein the synthesis control signal is generated so that a person's image becomes larger. The conversation status determination unit receives a communication control signal from the communication control unit, determines the number of conversation participants from the communication control signal, and when there are three or more, the image of the person who has the largest amount of speech is displayed. The composite control signal is generated so as to be large. The television broadcast receiver with an image and sound communication function according to claim 4. A television audio signal from the television decoder unit, a communication audio signal from the communication decoder unit, an input audio signal from the video / audio adjustment unit, and a synthesis control signal from the conversation state determination unit are input, and the synthesis control signal An audio mixing unit that outputs a mixed audio signal mixed based on
The television broadcast with an image and voice communication function according to any one of claims 4 to 6, wherein a volume level of each conversation participant is also changed by the voice mixing unit in accordance with the synthesis control signal. Receiver. The sound mixing unit outputs a mixed sound signal mixed in accordance with the synthesis control signal including the contents of a plurality of sound mixing templates in which each sound level corresponding to each image is mixed and each volume level is different. The television broadcast receiver with an image and audio communication function according to claim 7. In the communication control unit, the communication control signal to be output is output including the voice volume level information of the user, and the voice volume level information of the communication partner is detected from the input communication control signal.
In addition to the amount of speech, the conversation status determination unit sets the user's utterance volume level in the input audio signal from the video / audio adjustment unit and the utterance volume level of the communication partner in the communication audio signal from the communication decoder unit. The television broadcast receiver with an image and audio communication function according to any one of claims 2 to 8, wherein a conversation state is determined based on the image. The conversation state determination unit further receives a TV audio signal from the TV decoder unit together with channel information,
The television broadcast receiver with an image and audio communication function according to any one of claims 1 to 9, wherein the television audio signal is used to determine a conversation state and a synthesis control signal is output. The conversation status determination unit further receives a TV control signal from the TV decoder unit together with channel information,
11. The television broadcast receiver with an image and audio communication function according to claim 10, wherein at least genre information of a television broadcast program that is being viewed by a user is used as the television control signal. The conversation status determination unit further receives a TV control signal from the TV decoder unit together with channel information,
The television broadcast receiver with an image and audio communication function according to claim 10, wherein at least audio mode information of a television broadcast program being viewed by a user is used as the television control signal. In the communication control unit, the communication control signal that is output includes the channel information and supplementary information of the TV broadcast program that the user views, and the communication control signal that is input determines the TV broadcast program of the communication partner. Detect channel information and incidental information,
The communication status determination unit further receives a communication control signal from the communication control unit, determines the conversation status, and outputs a composite control signal. A television broadcast receiver with the image and audio communication function according to the description. The image and sound according to claim 13, wherein the conversation status determination unit uses at least genre information in the incidental information of a television broadcast program viewed by a communication partner as a communication control signal from the communication control unit. Television broadcast receiver with communication function. The image and audio communication function-equipped according to claim 13, wherein the conversation status determination unit uses at least audio mode information of a television broadcast program viewed by a communication partner as a communication control signal from the communication control unit. Television broadcast receiver. The conversation state determination unit includes an attention point determination unit that receives an input video signal from the video / audio adjustment unit and determines an attention point on which part of the display unit the user is paying attention to. The television broadcast receiver with an image and audio communication function according to any one of claims 1 to 15, wherein a conversation control state is determined using the information, and a synthesis control signal is output. In the communication control unit, the communication control signal to be output is output including the attention point information of the user, the attention point information of the communication partner is detected from the input communication control signal,
The television with an image and voice communication function according to claim 16, wherein the conversation state determination unit determines a conversation state using the attention point information of all conversation participants and outputs a composite control signal. John broadcast receiver. 18. The television with an image and audio communication function according to claim 16, wherein the conversation state determination unit uses the attention point information at least for composition control of a television broadcast program screen to be viewed. Broadcast receiver. The image according to claim 16 or 17, wherein the conversation status determination unit uses the attention point information for at least the synthesis control of the screen of a person to be watched most of all conversation participants. And a television broadcast receiver with a voice communication function. The said conversation status determination part uses the said attention point information with respect to the synthetic | combination control of the screen of each conversation participant mutually paying attention at least in the whole conversation participant. TV broadcast receiver with image and audio communication function. The conversation status determination unit stores in advance a specific symbol indicating the attention point of each conversation participant,
The television broadcast receiver with an image and audio communication function according to any one of claims 16 to 20, wherein an image of the specific symbol is synthesized with the attention point determined by the attention point determination unit. In the conversation status determination unit, a first specific symbol indicating each of the attention points of each conversation participant and a second specific symbol indicating the most attention point that is most noticed from each of the attention points are stored in advance. Remember,
The video of the first specific symbol is synthesized with the attention point of each conversation participant determined by the attention point determination unit, and the video of the second specific symbol is synthesized with the most noticeable point. The television broadcast receiver with an image and sound communication function according to claim 21. The image processing apparatus according to any one of claims 1 to 22, further comprising: a frame rate control unit that increases a frame rate for an image to be displayed with emphasis using the synthesis control signal. Television broadcast receiver with voice communication function. The television broadcast receiver with an image and audio communication function according to claim 21, wherein the frame rate control unit reduces the frame rate for an image that does not need to be displayed with emphasis. A remote operation device capable of remotely operating a television broadcast receiver with an image and audio communication function by a radio signal using radio waves or infrared rays, and a remote operation receiving unit 18 for receiving the radio signal;
25. The remote control device according to claim 16, wherein the remote control device is capable of turning on / off the output of the composite control signal from the conversation state determination unit or changing the control content of the composite control signal. The television broadcast receiver with an image and audio communication function according to any one of the preceding claims. In the communication control unit, the communication control signal to be output includes the contents instructed by the wireless signal from the remote operation device 2 and is output from the remote control device of the communication partner from the input communication control signal. The contents indicated by the signal
26. The television broadcast receiver with an image and voice communication function according to claim 25, wherein the conversation status determination unit uses remote control device instruction input information of all conversation participants to determine the attention point. . The remote control device inquires about the program being viewed from the television broadcast receiver with image and voice communication function of other conversation participants, receives the result, and automatically selects the channel of the program, 27. The television broadcast receiver with an image and audio communication function according to claim 25 or 26, further comprising a button for matching a program being watched by another conversation participant with a program being watched by a user.

Images