JP2012054693A - Video audio transmission/reception device, video audio transmission/reception system, computer program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize a smooth conversation between users.SOLUTION: A video audio transmission/reception device that is used by a user and accommodated in a network at each of plural bases, comprises: own base user video audio acquisition means for acquiring a video and audio of an own base user; transmission means for transmitting the acquired video and audio of the own base user to a video audio transmission/reception device at another base via the network; another-base user video audio acquisition means for acquiring a video and audio of an another-base user via the network; and control means for controlling predetermined video output means and audio output means so that the video and audio of the another-base user to be acquired are output with information synchronizing with the video and audio of the another-base user to be acquired on a time axis of the another base among own base user information to be generated based on at least one of the acquired video and audio of the own base user.

Description

  The present invention relates to, for example, a video / audio transmission / reception apparatus and a video / audio transmission / reception system capable of smoothly proceeding with a conversation with video between users at a remote location, and a computer program and a recording medium for realizing the apparatus. Related to the field.

  An example of this type of device is a videophone device disclosed in Patent Document 1. According to the videophone device of Patent Document 1, a self-image signal after encoding and decoding stored in an image memory having a motion compensation variable delay function in the self-image encoding circuit is used, and a received image is transmitted during communication. At the same time, by displaying the encoded and decoded own image on the monitor of the own terminal, the encoded and decoded own image can be displayed with a simple configuration.

JP-A-6-334997

  According to the videophone device disclosed in Patent Document 1, since the video in the encoder is PinP (Picture in Picture) into the received video, a plurality of decoders are not required, but the self-image displayed on the monitor of the own terminal It is not synchronized in time with the image of the communication partner.

  By the way, in data transmission via a network, there are often some transmission delays in the network. For this reason, video and audio at one site are transmitted to the other site after a fixed or indefinite time delay. In addition, when the transmitted video or audio of one site is received and an utterance is made at the other site, the video or audio corresponding to this utterance is also transmitted to the one site after an appropriate time delay. .

  However, a user at one base tends to think that the content related to the utterance has been transmitted to the other party when the utterance is completed. In addition, since it is understood that there is this kind of transmission delay, it is not practically easy to know when the content of the utterance is transmitted to the other party. That is, the conventional conversation system performed via the network has a technical problem that it is difficult for the users to grasp the consistency between the utterance content of the user and the utterance content of the other party.

  Due to such technical problems, conversations between users must continue with a time axis shift between them. For example, conversations do not mesh at all or conversations are not established. Problems that are difficult to overlook can arise.

  The present invention has been made in view of such technical problems, and a video / audio transmission / reception apparatus and a video / audio transmission / reception system capable of smoothly advancing conversation between users via a network, and a computer program capable of realizing the apparatus. It is another object of the present invention to provide a recording medium.

  In order to solve the above-described problem, a video / audio transmission / reception device according to claim 1 is a video / audio transmission / reception system including a plurality of video / audio transmission / reception devices accommodated in a network and used by a user at each of a plurality of sites. The local audio / video transmission / reception device, the local user video / audio acquisition means for acquiring the video and audio of the local user, and the local site acquired via the network with respect to the video / audio transmission / reception device at another site The transmission means for transmitting the video and audio of the user, the other base user video / audio acquisition means for acquiring the video and audio of the other base user via the network, and the acquired video and voice of the other base user Of the acquired local user information generated based on at least one of the acquired video and audio of the local user, the acquired And a control means for controlling the predetermined video output means and the audio output means so as to be output together with the information synchronized with the video and audio of the other base user on the time base of the other base. .

  In order to solve the above-described problem, the video / audio transmission / reception system according to claim 11 includes a plurality of video / audio transmission / reception apparatuses accommodated in a network and used by a user at each of a plurality of bases. Each of the local site user video and audio acquisition means for acquiring the video and audio of the local site user, and the video and audio of the local site user acquired via the network to the video and audio transmission / reception device of another site Transmission means for transmitting, other-site user video / audio acquisition means for acquiring the video and audio of the other-site user via the network, and the acquired local-site user's video and audio of the acquired own-site user Among the local site user information generated based on at least one of the video and audio of As it will be output along with the information to be synchronized on the time axis of the different hub and voice, characterized by a control means for controlling a predetermined video output means and audio output means.

  In order to solve the above-described problem, a computer program according to a twelfth aspect causes a computer system to function as the video / audio transmission / reception apparatus according to any one of the first to tenth aspects.

  In order to solve the above-described problem, a recording medium according to a thirteenth aspect records the computer program according to the twelfth aspect.

1 is a schematic configuration diagram conceptually showing a configuration of a conversation system according to a first example of the present invention. It is a timing chart which illustrates the progress state of conversation between users in face-to-face conversation without going through a network. It is a timing chart which illustrates the progress state of conversation between users in conversation via a network. It is a timing chart which illustrates the progress state of conversation between users in the 1st and 2nd examples. It is a block diagram showing the structure of the audio-video transmission / reception apparatus in the conversation system of FIG. It is a block diagram showing the structure of the transmission / reception apparatus in the video / audio transmission / reception apparatus of FIG. It is a flowchart of the transmission control performed in the transmission / reception apparatus of FIG. It is a flowchart of the reception control performed in the transmission / reception apparatus of FIG. FIG. 9 is a diagram illustrating a display image of the base A in the process of executing the reception control of FIG. 8 according to the effect of the first embodiment. It is a block diagram showing the structure of the transmission / reception apparatus which concerns on 2nd Example. It is a conceptual diagram of the speech data hold | maintained in the transmission / reception apparatus of FIG. It is a flowchart of the speech data generation process performed in the transmission / reception apparatus of FIG. 11 is a flowchart of video / audio output processing executed in the transmission / reception apparatus of FIG. 10. It is a figure which concerns on the effect of 2nd Example and illustrates the various aspects of an utterance state video frame. It is a block diagram showing the structure of the transmission / reception apparatus which concerns on 3rd Example. It is a flowchart of the transmission control performed in the transmission / reception apparatus of FIG.

  <Embodiment of Video / Audio Transmission / Reception Device>

  An embodiment of the video / audio transmission / reception apparatus according to the present invention is the video / audio transmission / reception system in the video / audio transmission / reception system including a plurality of video / audio transmission / reception apparatuses accommodated in a network and used by a user at each of a plurality of bases. A local user video / audio acquisition unit that acquires video and audio of the local user, and the acquired video and audio of the local user via the network with respect to the video / audio transmission / reception device at another site. Transmitting means for transmitting, video data and audio acquisition means for other base users that acquire video and audio of other base users via the network, and the acquired local base video and audio of the other base users The other site user acquired from the local site user information generated based on at least one of the user's video and audio As is output together with information to be synchronized on the time axis of the different hub with video and audio, and control means for controlling a predetermined video output means and audio output means.

  The embodiment of the video / audio transmission / reception apparatus of the present invention is one video / audio transmission / reception apparatus that constructs a video / audio transmission / reception system as a system capable of establishing a conversation between a plurality of users. The video / audio transmission / reception apparatus according to the embodiment is limited to a case where the network is always installed or some condition (this kind of condition is not limited in any way) is satisfied at the installation site. It is configured to be accommodated.

  The “network” according to the embodiment is, for example, a WAN (Wide Area Network) network, a LAN (Local Area Network) network, or a WAN, an ADSL (Asymmetric Digital Subscriber Line), or an optical fiber via the WAN network or the LAN network. It is a concept encompassing various data communication networks such as the Internet network appropriately connected via a cable or the like.

  Note that “accommodated” is practically, for example, provided with various communication I / Fs according to various communication standards corresponding to the network, and always or limited via the communication I / F. In addition, it means that various types of information, data, data files, etc. can be transmitted / received among a plurality of video / audio transmitting / receiving apparatuses. Note that the data transmission / reception mode via the network may be, for example, via an appropriate server device, or may not be via a server device such as P2P (Peer To Peer). Inevitably, the components of the video / audio transmitting / receiving apparatus may also be ambiguous.

  In the embodiment according to the video / audio transmission / reception apparatus of the present invention, the local user video / audio acquisition means means the local site (that is, one of the plurality of bases where the video / audio transmission / reception apparatus according to the embodiment is installed). The video and audio of the user at the local site, which means the user who participates in the conversation, is acquired. Note that the acquired video and audio of the local user are synchronized with each other on the time axis of the local site.

  The acquired video and audio of the local user are transmitted via the network to the video / audio transmitting / receiving apparatus at another base (that means a base different from the local base among a plurality of bases) by the transmission means. The In this case, these video and audio may be transmitted after being subjected to compression coding based on a predetermined standard, for example, as a suitable form.

  On the other hand, in the embodiment according to the video / audio transmission / reception apparatus, the other site user means a user who participates in the conversation at the other site (that is, a user corresponding to the conversation partner of the own site user) by the other site user video / audio acquisition unit. Video and audio are acquired via the network.

  As a preferred embodiment, the individual video / audio transmission / reception devices constituting the video / audio transmission / reception system are equivalent in the part relating to the invention-specific matters, and the video and audio of other base users acquired via the network are preferable. As one form, it is acquired in the video / audio transmission / reception apparatus of another base.

  Here, in the embodiment of the video / audio transmission / reception apparatus, basically, the video output means such as various display devices and the audio output means such as various speakers (theses) based on the acquired video and audio of the user at the other site. May be completely independent hardware, or at least part of the hardware may be integrated. As a result, at the own site, the video of the user at the other site is output together with the audio synchronized with the video on the time axis and perceived by the user at the local site. That is, if the user at the other site is performing an utterance operation, the content of the utterance is transmitted to the user at the local site with a video.

  By the way, in data transmission via a network, it is known that a transmission delay of the network occurs in addition to the processing delay of the video / audio transmitting / receiving apparatus itself. For example, the utterance operation of the local user at the local site is, for example, the transmission delay time of one way between the bases (from the local site to another site) or the video / audio transmission / reception at the other site from the time when the speech operation occurs at the local site. It is recognized by the user at another site with a delay corresponding to the processing delay time of the apparatus. In addition, the utterance operation of the user at the other site is also delayed by a time corresponding to the transmission delay time for one way between the sites (from the other site to the own site) or the processing delay time of the video / audio transmitting / receiving device at the own site. Recognized by the base user. That is, a time lag occurs when a conversation is actually progressed between users.

  This time lag is a factor that causes the conversation to pass, regardless of whether or not the user knows in advance that this kind of delay will occur, and reduces the conversation quality in terms of reducing the comfort of the conversation. It becomes a factor to make. This is because it is almost impossible for the user to constantly keep track of when and when the other party recognized his / her speech movement. On the other hand, in order to eliminate this time lag itself, this time lag is an unavoidable phenomenon as long as data transmission through a network is performed.

  In view of these points, it is necessary to take measures to prevent each user from perceiving this time lag in order to smoothly advance conversations between users accompanied by data transmission via a network. The embodiment according to the video / audio transmission / reception apparatus of the present invention has a configuration for realizing such measures.

  That is, the embodiment according to the video / audio transmission / reception apparatus of the present invention includes a control unit. The control means includes the local site user information generated based on at least one of the acquired local site user's video and audio by the other site user's video and audio acquired by the other site user video and audio acquisition unit. The predetermined video output means and audio output means are controlled so as to be output together with information acquired on the time axis of the other base on the acquired video and audio of the other base user.

  The local site user information refers to the status, operation, state, or action of the local site user that is generated based on at least one of the acquired video and audio of the local site user (that is, video and / or audio). Associated information.

  For example, the local site user information may be a video of the local site user itself, or may be various derived videos or similar videos obtained by appropriately performing image processing or the like on the video of the local site user. Further, the derived video or similar video may be, for example, a reduced version of the original video, or an indicator or icon according to some criteria for the user's local user's utterance operation (or just the state without utterance). Alternatively, it may be represented by a mark or the like. Further, when the video is reduced, the resolution of the video may be reduced. Alternatively, the local user information may be, for example, the voice of the local user itself, or is output according to the user's voice level (which simply means the sound pressure level) or according to the utterance timing. Various signal sounds and alarms may be used.

  The local site user information may be generated on the local site side or on the other site side, or may be generated by an appropriate server device or the like interposed between them in the network. Further, the time synchronization between the generated local user information and the video and audio of the other site user acquired by the other site user video and audio acquisition means is before the video and audio of the other site user is transmitted at the other site. It may be made after the video and audio of the user at another site are acquired at the site. That is, the essential technical idea according to the embodiment is that the video and audio of the user at another site are provided for output at the site together with the user information of the user at the time base synchronized with these.

  According to the video / audio transmission / reception apparatus according to the embodiment, when the local user recognizes the video and audio of the other base user, various videos and sounds related to the local user synchronized with the time base of the other base (easily understood) (Speaking, it is a video and sound about yourself in the past). Therefore, the user at the local site can advance the conversation with the user at the other site while recognizing in real time what kind of utterance operation the user at the other site has brought out the utterance operation. . That is, the conversation can be smoothly advanced.

  Note that each means provided in the video / audio transmission / reception apparatus according to the first embodiment is at least partially, for example, various arithmetic processing apparatuses such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), for example, It may be configured as various processors, controllers or various functional modules.

  In one aspect of the embodiment of the video / audio transmission / reception device of the present invention, the local site user information is based on at least one of the transmitted video and audio of the local site user and the video / audio transmission / reception device of the other site. The local site user information generated by the video / audio transmitting / receiving apparatus at the other site is transmitted via the network together with the video and audio of the other site user acquired at the other site, and the other site user The video / audio acquisition unit acquires the transmitted video and audio of the other site user and the own site user information, and the control unit outputs the acquired video and audio of the other site user and the own site user information. In this manner, the video output unit and the audio output unit are controlled.

  According to this aspect, the local site user information is generated on the other site side based on the video and audio of the local site user transmitted from the local site. The local site user information generated on the other site side is transmitted to the video / audio transmitting / receiving device according to the embodiment together with the video and audio of the other site user acquired by the video / audio transmitting / receiving device of the other site, This is used for controlling the output means.

  That is, according to this aspect, the control means only needs to output the video and audio of the other site user and the local site user information transmitted from the video and audio transmission / reception device at the other site, and the video and audio transmission and reception according to the embodiment. The load on the control of the device is reduced.

  Here, according to this aspect, the local site user information is generated by causing the video / audio transmitting / receiving apparatus at the other site to generate the local site user information based on the video and / or audio of the local site user. The synchronization processing for synchronizing the video and audio of the other site user acquired at the own site on the time axis of the other site is substantially unnecessary. In other words, if the local site user information includes the base data, it will follow the path from the local site to another site to the local site, and the video and audio of the other site user acquired at the other site will be semi-automatically. Will synchronize with each other on the time base of other bases.

  Supplementally, the above-described time lag is caused by a network transmission delay, a processing delay of each video / audio transmission / reception device, or the like. Therefore, by generating the local user information based on at least one of the video and audio of the local user transmitted through the regular data transmission path including the network and the video / audio transmission / reception device, the video / audio of the other site is generated. This time lag can be offset without specially performing time synchronization processing in the transmission / reception device. Therefore, it is possible to reduce the processing load of the entire video / audio transmission / reception system, which is preferable.

  In particular, according to this aspect, the local user information is generated based on at least one of the video and audio of the local user actually transmitted via the network each time, and For example, even if the network transmission state or the communication bandwidth changes for some reason, the acquired local user information and local site user information are also returned to the local site. The temporal synchronization state with the video and audio of the other base users is always maintained. That is, a conversation quality that is robust to the state of the network is provided.

  Furthermore, in view of the fact that the time synchronization between the local site user information and the video and audio of the other site user is always maintained by the network itself and the video / audio transmission / reception device itself of the other site, which is the cause of the transmission delay. In the video / audio transmitting / receiving apparatus according to the embodiment, the acquired video and audio of the local user or the generated local user information or the like need not be held for a certain or indefinite period, and should be provided for this purpose. Does not require a buffer. That is, it is possible to prevent the device configuration from becoming enlarged.

  In this aspect, the information processing apparatus further includes another site user information generating unit that generates other site user information based on at least one of the acquired video and audio of the other site user, and the transmission unit is configured to acquire the acquired site information. The generated other site user information may be transmitted together with the video and audio of the own site user.

  As described above, when the local site device and the local site device generate the local site user information and local site user information to be used for the output from the local site device and the local site device, respectively, Smooth conversations are realized for both the base user and other base users, and the practical benefits provided by the video / audio transmission / reception system are greater.

  In other words, for example, in the case where the own site user is clearly dominant or superior to other site users, the benefit of generating user information in the other device is, It may be provided only for the user at the local site. In other words, when the video / audio transmission / reception apparatus according to the embodiment adopts a configuration that does not include the other-site user information generation unit, the quality of the conversation between users is ensured depending on the use environment of the video / audio transmission / reception system. Can exist.

  In another aspect of the embodiment of the video / audio transmission / reception apparatus of the present invention, the specifying means for specifying a delay time required for data transmission via the network with the video / audio transmission / reception apparatus at the other site, and the acquisition The own site user information generating means for generating the own site user information based on at least one of the video and audio of the own site user, and the own site user information generating means based on the specified delay time Selection means for selecting information acquired on the time axis of the other site and the video and audio of the acquired other site user among the generated own site user information, and the control unit is the acquired The video output unit and the audio output unit are controlled so that the video and audio of the user at the other site and the selected information are output.

  In the one aspect described above, time synchronization on the time base of the other base between the video and voice of the other base user acquired at the other base and the own base user information based on the video and / or voice of the own base user. However, it was realized in a simple manner in the process of executing normal data transmission processing in the video / audio transmission / reception apparatus at another site. On the other hand, in this aspect, instead of or in addition to such a configuration, such time synchronization is realized by the action of the own base side (that is, the action of the video / audio transmission / reception apparatus according to the embodiment).

  According to this aspect, the specifying unit specifies the delay time required for data transmission between a plurality of bases via the network. This delay time is information necessary for defining the time axis on the other site side. As a preferred form, the transmission delay time on the network such as RTT (Round Trip Time) and each video / audio transmission / reception apparatus The processing delay time is defined in a dominant manner.

  Note that “specific” is a concept encompassing the final determination of a specific target as a reference value, and its practical aspect is not limited in any way. For example, the specifying unit may measure the RTT using, for example, an RTCP (Real-time Transport Control Protocol) SR (Sender Report) or RR (Receiver Report). Further, the processing delay time may be estimated based on device information (processing speed or busy state) held by the video / audio transmitting / receiving apparatus as its performance value.

  According to this aspect, the local site user information that can have the various aspects described above is generated on the local site side by the local site user information generation means.

  Here, when the local site user information is generated on the local site side, unlike the case where the local site user information is generated on the local site side, time synchronization processing (that is, acquired in the near future) for the video and audio of the acquired other site user. Or processing for appropriately selecting the local site user information corresponding to the utterance operation of the local site user, which causes the generation of the video and audio of the other local site user already acquired. According to this aspect, the selection process is realized by the selection unit.

  The selecting means realizes time synchronization between them based on the delay time specified by the specifying means. As described above, since this delay time is information that defines the time axis for other bases, the selection means can always select appropriate local user information. Note that the selection operation related to the selection means is a case where the local user information is generated in an appropriate amount before it is necessary and stored in the appropriate storage means using a time delay corresponding to the delay time. However, the present invention is not necessarily limited thereto. That is, the local site user information generation means based on the specified delay time so that the generated local site user information is constantly time synchronized with the video and audio of the acquired other site users. You may include the effect | action of controlling the production | generation timing of user information.

  As a result, according to this aspect, when the user at the local site recognizes the video and audio of the user at the other site, various videos and sounds related to the user at the local site synchronized with the time axis of the other site (which can be said in an easy-to-understand manner). For example, it is possible to simultaneously recognize video and sound related to the past. Therefore, the user at the other site (other site) recognizes in real time what kind of utterance operation the user at the other site (own site) has come to exhibit in real time. You can make conversations with other users. That is, the conversation can be smoothly advanced.

  In another aspect of the embodiment of the video / audio transmission / reception apparatus of the present invention, the local user information is a video corresponding to the acquired video of the local user.

  According to this aspect, since the local site user information is generated as a video corresponding to the video of the local site user, in the user of the local site (preferably, another site), the other site (preferably, further It is possible to easily understand visually at what timing the user at his / her own site has started to speak for what kind of speech.

  In this aspect, the video corresponding to the acquired video of the local user may be the acquired video of the local user with a reduced display area.

  If the display area is reduced in this way, the local site user information does not hinder the display of the video regarding the other site users, and the quality of the conversation is suitably maintained.

  In another aspect of the embodiment of the video / audio transmitting / receiving apparatus of the present invention, the local user information is a video corresponding to the acquired voice of the local user.

  Here, the video corresponding to the voice is an audio level, an icon or mark indicating whether or not the utterance is being performed, an icon or mark indicating whether or not the utterance has ended, numerical information or an indicator indicating the remaining time of the utterance, etc. In any case, the amount of information may be significantly smaller than the case where the local user information is generated based on the user's video. Therefore, the load required for the output control related to the control means can be relatively light.

  Also, according to this aspect, compared to the case where the user's video is subjected to some processing and the local site user information is generated, the practical output mode related to the actually displayed video is free, and the screen display Can be given flexibility. Also, customization by the user is relatively free.

  It should be noted that, in a mode in which the local site user information is generated at the local site, the acquired video, audio or self is at least the time corresponding to the delay time specified by the specifying means, unlike the case where the local site user information is generated at the other site. It may be necessary to appropriately store the base user information and the like (in a mode in which the local base user information is generated at another base, the network assumes this role). In this case, as this kind of buffer, for example, various kinds of known storage means such as various high-speed memories having a volatile storage area and HDD having a nonvolatile storage area can be applied. The smaller the better. In view of this point, the present aspect in which the information amount of the local site user information is small is applied to a configuration in which the local site user information is generated at the local site and is remarkably effective.

  In this aspect, the video corresponding to the acquired voice of the local user may be a video representing the speech state of the local user.

  When the video corresponding to the voice of the user at the local site is a video representing the utterance state of the user at the local site, the determination as to what utterance operation the video and audio of the user at the other site is made Is easier and effective in further improving the conversation quality. In this case, depending on the case, special processing is not required for a period in which the user at the local site is not in an utterance state, and the processing load on the entire apparatus can be reduced.

  Note that the utterance state includes a qualitative state such as whether or not the utterance is being made and a quantitative state such as the volume of the voice and the gesture size at the time of the utterance. Meaning an icon, an indicator or a mark indicating whether or not, an icon indicating a remaining utterance time, an indicator or a mark, an icon indicating an audio level, an indicator or a mark, and the like.

  In this aspect, the video corresponding to the acquired voice of the local user may be a video representing the remaining time until the utterance of the local user is completed.

  The utterance end timing of the local user may have a close relationship with the utterance start timing of the other local user. Therefore, the remaining time until the completion of the utterance that allows the user at the local site to effectively perceive the utterance end timing is suitable as an image corresponding to the voice of the user at the local site.

  In another aspect of the embodiment of the video / audio transmitting / receiving apparatus of the present invention, the local user information is a notification sound for notifying that the utterance of the local user is completed.

  The utterance end timing of the local user may have a close relationship with the utterance start timing of the other local user. Therefore, the notification sound for notifying the utterance end timing is suitable as the local user information. In particular, this kind of notification sound is effective in that the amount of information can be reduced compared to the effect thereof, and the conversation quality can be maintained while preventing the apparatus configuration from becoming enlarged.

  <Embodiment of Video / Audio Transmission / Reception System>

  An embodiment of the video / audio transmission / reception system of the present invention includes a plurality of video / audio transmission / reception devices accommodated in a network and used by a user at each of a plurality of bases. A local user video / audio acquisition unit that acquires video and audio of the local user, and a transmission unit that transmits the acquired video and audio of the local user via the network to the video / audio transmission / reception apparatus at another site; , Another site user video / audio acquisition means for acquiring the video and audio of the other site user via the network, and the acquired video and audio of the other site user Among the local site user information generated based on at least one of them, the acquired video and audio of the other site user and the other site user information As is output together with information to be synchronized on during shaft, and a control means for controlling a predetermined video output means and audio output means.

  According to the embodiment related to the voice transmission / reception system, since there are a plurality of embodiments related to the voice transmission / reception apparatus, practical benefits brought about by the embodiment related to the voice transmission / reception apparatus can be similarly obtained, and between users via the network Smooth conversations are realized.

  <Embodiment of Computer Program>

  The embodiment according to the computer program of the present invention causes a computer system to function as the video / audio transmission / reception apparatus according to any one of the above.

  The computer program is loaded into the computer system from a solid-state storage device such as a ROM, CD-ROM, DVD-ROM, hard disk or other storage medium storing the computer program or a USB (Universal Serial Bus) memory. Or the computer program is downloaded to a computer system via a communication means or the like and executed, for example, the above-described embodiment of the video / audio transmitting / receiving apparatus of the present invention is relatively simple. Can be realized.

  Incidentally, in response to the various aspects of the embodiment of the video / audio transmitting / receiving apparatus of the present invention described above, the embodiment of the computer program of the present invention can also adopt various aspects.

  <Embodiment of Recording Medium>

  The embodiment according to the computer program of the present invention records the embodiment according to the computer program.

  According to the embodiment of the recording medium of the present invention, the computer program of the present invention recorded by being attached to or connected to a computer system, or by being inserted into an appropriate reader provided in or connected to the computer system. The embodiments according to the present invention can be read and executed by a computer system, and the embodiments according to the video / audio transmission / reception apparatus of the present invention described above can be realized relatively easily.

  As described above, according to the first embodiment of the video / audio transmission / reception apparatus of the present invention, the local-site user video / audio acquisition means, the transmission means, the other-site user video / audio acquisition means, and the control means are provided. A smooth conversation between users via the network is realized.

  As described above, according to the embodiment of the video / audio transmission / reception system of the present invention, since there are a plurality of embodiments of the video / audio transmission / reception apparatus of the present invention, smooth conversations between users via a network are possible. Realized.

  As described above, according to the embodiment of the computer program of the present invention, the computer system can function as the embodiment of the video / audio transmission / reception apparatus of the present invention.

  As described above, according to the embodiment of the recording medium of the present invention, the computer system functions as the embodiment of the video / audio transmitting / receiving apparatus of the present invention by recording the embodiment of the computer program of the present invention. Can be made.

  These effects and other advantages of the present invention will become apparent from the embodiments described below.

Hereinafter, various preferred embodiments of the present invention will be described with reference to the drawings as appropriate.
<First embodiment>
<Configuration of Example>
First, the configuration of the conversation system 1 according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a schematic configuration diagram conceptually showing the configuration of the conversation system 1.

  In FIG. 1, a conversation system 1 is a base A (that is, an example of the “own base” according to the present invention) and a base B (that is, an example of the “other base” according to the present invention) that are separated from each other. 1 is a system that is an example of the “video / audio transmission / reception system” according to the present invention, which is accommodated in a wide area network 10 (IP (Internet Protocol) network).

  The conversation system 1 is installed at the site A, and the video / audio transmission / reception device 30A (that is, according to the present invention) used for the user 20A existing at the site A (that is, an example of the “own site user” according to the present invention). An example of such “video / audio transmission / reception device” and an audio transmission / reception device 30B used for the user 20B installed at the base B and existing at the base B (that is, an example of the “other base user” according to the present invention). (That is, another example of the “video / audio transmitting / receiving apparatus at another site” according to the present invention). The user 20 </ b> A and the user 20 </ b> B can make a conversation while confirming mutual video and audio by the conversation system 1.

  Here, in order to make the detailed configuration of the conversation system 1 easier to understand, an outline of the operation of the conversation system 1 will be described with reference to FIGS. 2 and 3. FIG. 2 is a timing chart illustrating the progress of conversation between users in face-to-face conversation not via a network. FIG. 3 is a timing chart illustrating the progress of conversation between users in a conversation via a network. In addition, in each of these drawings, portions that overlap with those in FIG. 1 and portions that overlap each other are denoted by the same reference numerals, and description thereof is omitted as appropriate.

  In FIG. 2, the user 20A (base A) is shown on the left side, and the user 20B (base B) is shown on the right side. The vertical axis is a time axis, and a time axis is set for each of the base A and the base B.

  Here, it is assumed that the base A and the base B are so close that the users can talk to each other without going through the network 10 (for example, in a conference room). In this case, the user 20 </ b> A and the user 20 </ b> B can face-to-face conversation, i.e., face each other and exchange conversations while confirming facial expressions and gestures.

  That is, when the utterances A1, A2, and A3 are sequentially performed along the time axis at the base A, the user 20B can instantly recognize the utterance contents of the user 20A at the base B and respond to these. The utterances B1, B2 and B3 can be performed sequentially. That is, both of these utterances are synchronized on the time axis, and the user can have a smooth conversation with each other.

  On the other hand, FIG. 3 shows a configuration in which the base A and the base B are connected via the network 10 (that is, the configuration according to the present embodiment).

  In the conversation environment shown in FIG. 3, the utterance A1 of the user 20A at the site A is transmitted from the occurrence time of the utterance A1 at the site A to the transmission delay time of the network 10 and the processing delay time of the video / audio transmitting / receiving device 30B. Is output as an utterance A1B delayed by a delay time DLY_AB including (meaning that the utterance is A1 at the base B).

  On the other hand, it is assumed that the user 20B responds to the utterance A1B with the utterance B1. In this case, the utterance B1 of the user 20B at the base B is delayed at the base A by a delay time DLY_BA including the transmission delay time of the network 10 and the processing delay time of the video / audio transmitting / receiving apparatus 30A from the time when the utterance B1 at the base B occurs. Utterance B1A (meaning utterance B1 at base A). That is, when compared with the face-to-face conversation illustrated in FIG. 2, the delay time corresponding to “DLY_AB + DLY_BA” (that is, the “delay time” according to the present invention) from when the user 20A performs the utterance A1 until the user confirms the utterance B1. Only an extra time will elapse.

  On the other hand, on the time axis of the base A, the utterances A2 and A3 are sequentially performed following the utterance A1, and the utterance B1A is output at a timing overlapping with the utterance A3. However, since the utterance B1 that is the basis of the utterance B1A is originally made in correspondence with the utterance A1 of the user 20A, from the user 20A, the utterance is not synchronized on the time axis. Such a phenomenon can occur in the base B as well. That is, in the conversation via the network 10, unless any countermeasure is taken, the conversation between the user 20A and the user 20B has a time lag, and smooth conversation becomes difficult.

  The conversation system according to the first embodiment and the second embodiment, which will be described later, is for solving such problems. Here, prior to detailed description of each embodiment, the principle of solving the problems in these embodiments will be described with reference to FIG. FIG. 4 is a timing chart illustrating the progress of conversation between users in the first and second embodiments. In the figure, the same reference numerals are given to the same portions as those in FIG. 3, and the description thereof is omitted as appropriate.

  4, according to the conversation system according to the first and second embodiments, information corresponding to the utterance (for example, A1B) of the user 20A at the base A output at the base B (in FIG. 4, the utterance A1B itself) ) Is output to the base A while maintaining the time relationship with the utterance (for example, B1) of the user 20B at the base B (that is, synchronized with the time axis of the base B) (see the utterance A1BA shown in the figure). ). That is, in the conversation between the user 20A and the user 20B, although the output time of the other party's utterance is delayed at both bases, the mutual relation between the utterances in each is not broken at all. For this reason, regardless of the delay time, users at each site can always check at what timing the other party's utterance was made, and smooth conversation is possible. It becomes.

  In FIG. 4, the time relationship between the utterances B1A, B2A and B3A of the user 20B made at the site B and the utterances A1BA, A2BA and A3BA of the user 20A at the site A outputted at the site B in the site A A state in which the time relationship between the utterances B1, B2, and B3 of the user 20B made at the base B and the utterances A1B, A2B, and A3B of the user 20A made at the base A at the base B is shown.

  Based on these points, the detailed configuration and operation will be described for each embodiment.

  First, the configuration of the video / audio transmission / reception device 30A (the hardware configuration is the same as that of the video / audio transmission / reception device 30B) will be described with reference to FIG. FIG. 5 is a block diagram of the video / audio transmitting / receiving apparatus 30A. In the figure, the same reference numerals are given to the same portions as those in FIG. 1, and the description thereof will be omitted as appropriate.

  In FIG. 5, the video / audio transmission / reception device 30 </ b> A (or 30 </ b> B) includes a camera 31, a microphone 32, a monitor 33, a speaker 34, and the transmission / reception device 100.

  The camera 31 is an imaging device configured to be able to image the user 20A. The camera 31 has an imaging lens (not shown) disposed opposite to the upper body including the face of the user 20A, and can image the upper body of the user 20A. The video of the user 20A obtained as a result of this imaging is configured to be sent to the transmission / reception device 100 as a video signal for each frame.

  The microphone 32 is a sound collecting device configured to collect the voice of the user 20A. The microphone 32 has a sound collection unit arranged toward the face of the user 20A, and can collect the sound of the user 20A. The voice of the user 20A obtained as a result of the sound collection is configured to be sent to the transmission / reception device 100 as a voice signal.

  The monitor 33 is a video display device such as a liquid crystal display type display device, for example, and is configured to display a video in a predetermined display area in accordance with a predetermined video signal. The monitor 33 is electrically connected to the transmission / reception device 100, and the video output is controlled by the transmission / reception device 100.

  The speaker 34 is an audio output device and is configured to be able to output audio in accordance with a predetermined audio signal. The speaker 34 is electrically connected to the transmission / reception device 100, and the sound output is controlled by the transmission / reception device 100.

  The transmission / reception device 100 is a computer system configured to be able to control the entire operation of the video / audio transmission / reception device 30A, and functions in the narrow sense as an example of the “video / audio transmission / reception device” according to the present invention.

  The transmission / reception device 100 includes an electronic control unit including a CPU, a ROM, a RAM, and the like (not shown). The ROM stores various control programs configured to be executable by the CPU. When the CPU executes the control program, the transmission / reception device is an example of the “video / audio transmission / reception device” according to the present invention. It has a functioning configuration. The control program is an example of the “computer program” according to the present invention.

  This control program is recorded on various recording media (an example of the “recording medium” according to the present invention) such as HD (Hard Disk), CD-ROM, DVD-ROM, or BD (Blu-ray Disk). This kind of recording medium can be appropriately read by the computer system in a state where the recording medium is installed, stored or inserted in an appropriate recording / reproducing apparatus incorporated in or attached to various computer systems, for example. It is also possible for a computer system to function as an example of the “video / audio transmitting / receiving apparatus” according to the present invention.

  Next, a detailed configuration of the transmission / reception device 100 will be described with reference to FIG. FIG. 6 is a block diagram of the transmission / reception device 100. In the figure, the same reference numerals are assigned to the same parts as those in FIG. 5, and the description thereof is omitted as appropriate.

  6, the transmission / reception apparatus 100 includes an input unit 110, a transmission unit 120, a reception unit 130, a reception buffer 140, an output unit 150, a video processing unit 160, and a video synthesis unit 170.

  The input unit 110 is a video and audio input interface unit, and includes a video input unit 111, an audio input unit 112, and a video / audio encoding unit 113.

  The video input unit 111 is a video input interface in which an input terminal (reference numeral omitted) is connected to the camera 31 described above, and a video signal in frame units of the user 20A sequentially transmitted from the camera 31 (that is, “ It is an example of “own site user video / audio acquisition means” according to the present invention configured to be able to acquire “example of own site user's video”. The video input unit 111 is electrically connected to a video synthesis unit 170 described later, and the acquired video signal is transmitted to the video synthesis unit 170.

  The audio input unit 112 is an audio input interface whose input terminal (not shown) is connected to the microphone 32 described above, and an audio signal that defines the audio of the user 20A via the microphone 32 (that is, “ It is another example of the “own site user video and audio acquisition means” according to the present invention configured to be able to acquire “example of own site user's voice”.

  The video / audio encoding unit 113 includes an encoder (not shown), and receives a video signal and an audio signal obtained via the video input unit 111 and the audio input unit 112 or a video signal and an audio signal corresponding to the video signal and the audio signal respectively. This is a data generation unit that generates video and audio data that complies with a predetermined standard (for example, MPEG2TS (Transport Stream), etc.) by compression encoding and multiplexing at a bit rate. Further, the video / audio encoding unit 113 is configured to convert the generated video / audio data into a data packet conforming to a predetermined transmission protocol (RTP (Real-time Transfer Protocol or the like in this embodiment)). Yes.

  The transmission unit 120 is a transmission interface configured to be able to transmit the data packet regarding the video / audio data generated by the video / audio encoding unit 113 to the transmission / reception device 100 at the base B via the network 10, and according to the present invention. It is an example of “transmission means”.

  The reception unit 130 is a reception interface that acquires a data packet related to the video / audio data generated by the video / audio encoding unit 113 in the transmission / reception apparatus 100 at the site B via the network 10. It is an example of “voice acquisition means”.

  The reception buffer 140 suppresses variations in the transmission delay time of each data packet, changes the order of each data packet (that is, the arrival time of each data packet varies), and absorbs defects caused by the loss of the data packet. This is a storage means having a temporary storage area provided for this purpose. The reception buffer 140 is configured to sequentially transmit acquired data packets using the RTP sequence numbers and time stamps while maintaining the transmission order and time interval in the transmitting / receiving apparatus 100 at the site B.

  The output unit 150 is an output interface unit for video and audio, and is an example of a “control unit” according to the present invention including a video / audio decoding unit 151, a video output unit 152, and an audio output unit 153.

  The video / audio decoding unit 151 converts the data packet in the RTP format into video / audio data that is compression-encoded data in the MPEG2TS format, and further decodes and separates the video / audio data into a video signal (into one video frame). Corresponding video signals) and audio signals can be extracted. The extracted video signal is sent to the video output unit 152 and the video processing unit 160 described later, and the audio signal is sent to the audio output unit 153.

  The video output unit 152 is a video output interface whose output terminal (reference numeral omitted) is connected to the monitor 33 described above, and is configured to be able to supply a video signal sent from the video / audio encoding unit 151 to the monitor 33. ing.

  The audio output unit 153 is an audio output interface whose output terminal (reference numeral omitted) is connected to the speaker 34 described above, and is configured to be able to supply the audio signal sent from the video / audio encoding unit 151 to the speaker 34. ing.

  The video processing unit 160 should be used for synthesis with one video frame of the user 20A based on the video signal supplied from the video / audio decoding unit 151 (video signal corresponding to one video frame of the user 20B at the base B). It is an image processing apparatus that generates a video frame for composition (that is, an example of “other site user information” according to the present invention).

  The composition video frame according to the present embodiment is a video in which the vertical and horizontal display sizes are reduced by a predetermined ratio with respect to the original video frame, and the display position is defined so as to be displayed in the lower left corner of the original video frame. It is assumed that the video frame has been processed. The remaining area of the frame is subjected to processing such as making the background color transparent. The generated composite video frame is sent to the video composition buffer of the video composition unit 170.

  The video synthesizing unit 170 is an image processing apparatus that includes a video synthesizing buffer 171 and is capable of synthesizing one video frame of the user 20A at the site A and a synthesizing video frame related to the user B.

  The video composition buffer 171 is a storage device that temporarily stores the composition video frame transmitted from the video processing unit 160.

  The video synthesis unit 170 synthesizes the video frame for synthesis temporarily stored in the video synthesis buffer 171 and one video frame of the user 20A generated based on the video signal sent from the video input unit 111. It is configured to be able to generate a composite video frame by executing processing. The video signal related to the generated composite video frame is sequentially sent to the video / audio encoding unit 113.

<Operation of Example>
Hereinafter, as operations according to the first embodiment of the present invention, details of transmission control and reception control realized by the transmission / reception apparatus 100 executing the various control programs described above will be described.

  First, details of transmission control will be described with reference to FIG. FIG. 7 is a flowchart of transmission control.

  In FIG. 7, imaging and sound collection processing are executed (step S101). In the imaging and sound collection processing, the imaging processing of the user 20A by the camera 31 and the sound collection processing of the user 20A by the microphone 32 are executed in parallel while maintaining time synchronization. The video signal obtained as a result of these processes is sent to the video synthesizing unit 170 via the video input unit 111, and the audio signal is sent to the video / audio encoding unit 113 via the audio input unit 112.

  Next, video composition processing is executed by the video composition unit 170 (step S102). In the video synthesis process, as described above, one video frame of the user 20A and the video frame for synthesis related to the user 20B are synthesized to generate a synthesized video frame. The video signal related to the composite video frame is sent to the video / audio encoding unit 113 while maintaining time synchronization between the original video signal and the audio signal.

  Next, the video / audio encoding unit 113 executes encoding and multiplexing processing of the video signal and the audio signal related to the synthesized video frame, and generates the above-described video / audio data (step S103). The generated video / audio data is sent to the transmission unit 120 as the data packet described above.

  The transmitting unit 120 transmits the transmitted data packet to the transmitting / receiving apparatus 100 at the site B via the network 10 (step S104). When the transmission process ends, the process returns to step S101, and a series of processes is repeated. Transmission control is executed as described above.

  Next, details of reception control will be described with reference to FIG. FIG. 8 is a flowchart of reception control.

  In FIG. 8, the receiving unit 130 receives data packets transmitted from the transmitting / receiving device 100 at the base B, and sequentially stores the received data packets in the reception buffer 140 (step S201).

  The data packet conforming to RTP includes a time stamp that defines the time relationship of the original data. In the reception buffer 40, the data packet randomly transmitted via the network 10 is converted into the original data packet. Can be rearranged in order. The reception buffer 140 manages the data packets to be held with time stamps, and sequentially sends them to the video / audio decoding unit 151 according to the time stamps (step S202).

  The video / audio decoding unit 151 decodes and separates the data packet as described above, and sends the video signal to the video output unit 152 and the video processing unit 160, and the audio signal to the audio output unit 153 (step S203). ). After the processing by the video / audio decoding unit 151, the processing is divided into two systems.

  That is, on the other hand, a video frame for synthesis is generated by the video processing unit 160 (step S204) and stored in the video synthesis buffer 171 (step S205). The stored video frame for synthesis is used for the synthesis process in step S102 in the transmission control described above.

  On the other hand, the video output unit 152 and the audio output unit 153 send the video signal and the audio signal obtained through the video / audio decoding unit 151 to the monitor 33 and the speaker 34, respectively. As a result, the synthesized video frame synthesized at the site B is displayed on the monitor 33, and the sound of the user B corresponding to the synthesized video frame is output from the speaker 34.

  When these processes are completed, the process returns to step S201. Reception control is executed as described above.

<Effect of Example>
Next, effects of the first embodiment will be described with reference to FIG. FIG. 9 is a diagram exemplifying a display image of the base A in the execution process of the reception control.

  In FIG. 9, let us say that one video frame of the user 20 </ b> A captured by the camera 31 at the site A is PICT_A. In addition, at the site B, one video frame of the user 20B captured by the camera 31 is defined as PICT_B.

  In the process in which the reception control is executed at the site B, a composite video frame PICT_A ′ (here, a reduced video frame of PICT_A) is generated based on this PICT_A. The composition video frame PICT_A ′ is an example of “own site user information” according to the present invention. This composite video frame PICT_A ′ is combined with PICT_B in the process of performing the previous transmission control at the base B, and a composite video frame PICT_BA to be displayed at the base A is generated.

  This composite video frame PICT_BA is displayed on the monitor 33 as a video of the site B in the reception control at the site A.

  Here, the composite video frame PICT_BA is different from a simple composite video of the user A video and the user B video in that the composite video PICT_A ′ (the same applies to the composite video PICT_B ′ generated at the site A). It has a time element.

  That is, the composition video frame PICT_A ′ is based on the video frame PICT_A originally obtained at the site A. However, if the transmitting / receiving device 100 at the site A generates the compositing video frame PICT_A ′ and combines it with the video frame PICT_B obtained from the site B, as described above with reference to FIG. Therefore, the time relationship between these frames is shifted. In this case, the technical significance of displaying the composite video frame PICT_A ′ relating to the user A at the site A becomes extremely small.

  On the other hand, in this embodiment, a composition video frame PICT_A ′ is generated at the site B. That is, in the process of transmitting the data packet from the base A to the base B, the delay time for one way from the base A to the base B is cancelled. That is, the composition video frame PICT_A is a video frame synchronized with the time axis at the base B.

  On the other hand, the composite video frame PICT_A ′ is combined with the video frame PICT_B of the user B whose time axis is synchronized at the base B and returned to the base A as a data packet. At this time, the composite processing is performed at the base B. Therefore, in the process of transmitting the network 10 from the base B to the base A, the delay time for one way from the base B to the base A never breaks this temporal synchronization relationship.

  As a result, at the base A, the state of the user A and the user B synchronized with the time axis at the base B (in other words, the state of the conversation at the base B at a certain point in the past) is the composite video frame PICT_BA. Is displayed. That is, the user A makes the utterance of the user B defined by the voice of the user B (synchronized with the video frame PICT_B) output in synchronization with the synthesized video frame PICT_BA for any utterance of the user A. It is possible to clearly recognize whether or not the timing has been made by the composite video frame PICT_A ′ synchronized with the time axis of the base B. Of course, such a profit is also enjoyed in the base B as well.

Therefore, according to the conversation system 1 according to the present embodiment, a plurality of users can smoothly advance the conversation by the action of the transmitting / receiving device 100 installed at the base A and the base B.
<Second embodiment>
Next, a conversation system according to the second embodiment of the present invention is described. The conversation system according to the second embodiment is basically the same as the conversation system according to the first embodiment, but the configuration of the video / audio transmission / reception apparatus installed at each site, particularly the transmission / reception apparatus, is related to the first embodiment. Different from the transmitting / receiving device 100. More specifically, it is different from the first embodiment in that a transmission / reception device 200 is provided instead of the transmission / reception device 100.

  First, the configuration of the transmitting / receiving apparatus according to the second embodiment will be described with reference to FIG. FIG. 10 is a block diagram showing the configuration of the transmitting / receiving apparatus 200. In the figure, the same reference numerals are given to the same portions as those in FIG. 6, and the description thereof is omitted as appropriate. As in the first embodiment, the transmission / reception device 200 is assumed to be a transmission / reception device at the site A.

  10, the transmission / reception device 200 includes an input unit 110, a transmission unit 120, a reception unit 130, a reception buffer 140, an output unit 210, a delay measurement unit 220, and a synthesis processing unit 230.

  The output unit 210 includes a video processing unit 211 and an audio processing unit 212 in addition to the video / audio encoding unit 151, the video output unit 152, and the audio output unit 153.

  Under the control of the utterance data processing unit 233 described later, the video processing unit 211 performs utterance described later at a predetermined timing for one video frame of the user B based on the video signal transmitted from the video / audio encoding unit 151. It is an image processing apparatus that synthesizes a state video frame (that is, an example of “own site user information” according to the present invention). The video signal related to the video frame in which the speech state video frame is combined is sent to the video output unit 152.

  Under the control of the utterance data processing unit 233, which will be described later, the audio processing unit 212, for the voice of the user B based on the audio signal transmitted from the video / audio encoding unit 151, the utterance end sound described later at a predetermined timing. That is, it is a voice processing device that synthesizes (another example of “own site user information” according to the present invention). A voice signal related to the voice synthesized with the utterance operation sound is transmitted to the voice output unit 153.

  The delay measurement unit 220 includes a device delay information communication unit 221, a network delay measurement unit 222, and a total delay amount calculation unit 223.

  The device delay information communication unit 221 measures the processing delay time of the transmission / reception device 200 (mainly the sum of the delay time required for the input processing related to the input unit 110 and the delay time required for the output processing related to the output unit 210). The processing delay time information is transmitted to the transmission / reception device 200 at the base B, and similar processing delay time information can be acquired from the transmission / reception device 200 at the base B. The transmission / reception of the processing delay time information may be frequently executed at a constant cycle, or may be executed at a certain specific timing (for example, at the initial stage of starting the conversation system).

  The network delay measuring unit 222 is a unit that measures the transmission delay time (here, RTT) of the network 10. The network delay measuring unit 222 is configured to measure RTT using SR (Sender Report) and RR (Receiver Report) of RTCP (Real-time Transport Control Protocol).

  The total delay amount calculation unit 223 adds the processing delay time measured by the device delay information communication unit 221 and the transmission delay time measured by the network delay measurement unit 222 to obtain a total delay required for data transmission between the bases. It is an arithmetic processing unit that calculates time (an example of “delay time” according to the present invention).

  The synthesis processing unit 230 includes an utterance determination unit 231, an utterance data holding unit 232, and an utterance data processing unit 233.

  The utterance determination unit 231 is configured to acquire an audio signal from the audio input unit 112, detect the user A's audio level (sound pressure level), and determine whether the user A is in an utterance state. .

  The utterance data holding unit 232 is a unit that generates and temporarily stores utterance data obtained by converting the utterance state of the user A into data based on the determination result of the utterance determination unit 231. Here, speech data will be described with reference to FIG. FIG. 11 is a conceptual diagram of the utterance data held by the utterance data holding unit 232.

  In FIG. 11, the utterance data is data including information on the utterance start time of the user A and information on the utterance end time. The utterance data holding unit 232 employs a configuration in which the utterance data is stored by being temporarily stored. Note that utterance data corresponding to the past than the total delay time calculated by the total delay amount calculation unit 223 is sequentially discarded from the utterance data holding unit 223.

  Returning to FIG. 10, the utterance data processing unit 233 includes a predetermined utterance state video frame including various utterance marks mk associated with the utterance state of the user A based on the utterance data held in the utterance data holding unit. Is a control unit having an image processing function for generating an image, a function for transmitting the generated speech state video frame to the video processing unit 211 at an appropriate timing, and a function for controlling the operation of the video processing unit 211. The utterance data processing unit 233 is an example of each of the “own site user information generating unit” and the “selecting unit” according to the present invention.

  Here, in the transmission / reception apparatus 200 according to the second embodiment, the CPU executes a speech data generation process and a video / audio output process in accordance with a control program stored in the ROM.

  First, the speech data generation process will be described with reference to FIG. FIG. 12 is a flowchart of the utterance data generation process.

  In FIG. 12, the utterance determination unit 231 determines whether or not the detected voice level is equal to or higher than a threshold set in advance so as to determine whether or not the utterance state is present (step S301). When the sound level is less than the threshold (step S301: NO), step S301 is repeatedly executed, and the process is substantially in a standby state.

  When the voice level is equal to or higher than the threshold (step S301: YES), the utterance data holding unit 232 records the utterance start time (that is, the time when the voice level exceeds the threshold) (step S302), and the voice level is lower than the threshold. It is determined whether or not (step S303). During a period in which the voice level is equal to or higher than the threshold (step S303: NO), the process is in a standby state, and when the voice level falls below the threshold (step S303: YES), the utterance data holding unit 232 records the utterance end time. Thus, one utterance data is completed (step S304). When one utterance data is completed, the process returns to step S301. The speech data generation process is executed as described above.

  Next, the video / audio output processing will be described with reference to FIG. FIG. 13 is a flowchart of the video / audio output process.

  First, the utterance data processing unit 233 synchronizes on the time axis of the site B with the time when one video / audio data is decoded by the video / audio decoding unit 151 (hereinafter referred to as “decoding time” as appropriate). The voice state of the user A at the time (hereinafter referred to as “synchronization time” as appropriate) is confirmed via the utterance data holding unit 232 (step S401). This synchronization time is a time past the total delay time calculated by the total delay amount calculation unit 223 from the decoding time.

  The utterance data processing unit 233 determines whether the user A is in the utterance state based on the confirmation result (step S402). When the user A is in the utterance state (step S402: YES), the utterance data processing unit 233 further determines whether or not the utterance has been completed (step S403). Note that “whether or not the utterance has been completed” means that the utterance is not continuing at present, and specifically, the utterance data of the utterance data holding unit 232 has an end time. Means that it is described.

  When the utterance is still continuing (step S403: NO), the utterance data processing unit 233 generates an utterance state video frame and corresponds to the video signal that has passed through the video / audio decoding unit 151 via the video processing unit 211. The generated speech state video frame is synthesized with the video frame to be played (step S409). At this time, since the utterance is still continuing, the remaining time of the utterance described later is not displayed.

  On the other hand, when the utterance has already been completed (step S403: YES), the utterance data processing unit 233 calculates the remaining time of the utterance (step S404). The remaining time of the utterance is a difference between the above-described synchronization time and the utterance end time (specified by the utterance data).

  The utterance data processing unit 233 determines whether or not the calculated remaining time is less than a preset threshold value (for example, 5 seconds) (step S405). If the remaining time is equal to or greater than the threshold (step S405: NO), that is, if there is sufficient time until the end of the utterance, the process proceeds to step S409 described above.

  On the other hand, when the remaining time is less than the threshold (step S405: YES), that is, when the utterance is finished soon, the video data processing unit 233 generates an utterance state video frame including the remaining time display mark, and the video It is synthesized with a video frame transmitted from the video / audio decoding unit 151 via the processing unit 211 (step S406). The remaining time display mark is a mark (including an icon, an indicator, etc.) that visually represents the remaining time of the utterance.

  In step S402, when the user A is not in the utterance state at the synchronization time (step S402: NO), the utterance data processing unit 233 matches the synchronization time with the utterance end time, that is, the utterance is just finished. It is determined whether it is time (step S407). When the synchronization time and the utterance end time do not match (step S407: NO), the utterance data processing unit 233 outputs the video so that the video frame sent from the video / audio decoding unit 151 is sent to the video output unit 152 as it is. The processing unit 211 is controlled.

  When the synchronization time and the utterance end time match, the utterance data processing unit 233 generates an audio signal related to the utterance end sound, and adds the audio signal transmitted from the video / audio decoding unit 151 via the video processing unit 161 to the audio signal. Combining (step S408). The utterance end sound is a notification sound for notifying the user A that his / her past utterance synchronized with the time axis of the base B has ended, and its mode is not limited at all.

  If step S408 is executed, step S407 branches to the “NO” side, or if a video frame in which the speech state video frame is synthesized is displayed, the process returns to step S401. The video / audio output processing is executed as described above.

  Next, an utterance state video frame that is generated by the utterance data control unit 233 when appropriate conditions are satisfied and appropriately combined with the video frame at that time (synchronization time) will be described. FIG. 14 is a diagram illustrating various modes of the speech state video frame. In the figure, the same parts as those in FIG. 9 are denoted by the same reference numerals, and the description thereof is omitted as appropriate.

  In FIG. 14, the utterance state video frame is a composite display frame synthesized with the video frame PICT_BA of the user B displayed at the site A, and the video frame in which the utterance mark mk with various aspects is arranged at the left end portion of the frame. It is.

  FIG. 14A illustrates a state in which an utterance state video frame including the remaining time display mark mk1 as the utterance mark mk is synthesized.

  The remaining time display mark mk1 illustrated in FIG. 14A is a kind of indicator in which one display unit mark is assigned to a unit time (here, assumed to be 1 second as an example). In FIG. 14A, the remaining time of user A's utterance corresponds to the above-described threshold value (here, 5 seconds as an example), that is, five display unit marks, that is, five unit times (that is, five unit times). Second). That is, thereafter, every time the unit time elapses, the display unit mark gradually decreases and is not displayed when the user A's utterance ends (that is, a state corresponding to step S409 in FIG. 13).

  FIG. 14B illustrates a state where an utterance state video frame including the utterance display mark mk2 as the utterance mark mk is synthesized. The utterance display mark mk2 illustrated in FIG. 14B is an icon indicating whether or not the user A is in an utterance state, and is displayed when the user A is speaking, and is not displayed unless the user A is speaking. It has a configuration. Alternatively, the utterance display mark mk2 may be displayed in a highly appealing blinking manner so that the user A's attention can be drawn more.

  FIG. 14C illustrates a state in which an utterance state video frame including the voice level mark mk3 as the utterance mark mk is synthesized. The voice level mark mk3 illustrated in FIG. 14C is an indicator that represents the voice level (sound pressure level) of the user A in a plurality of stages, and in the figure, the voice level of the user A is level 3. (See hatching display). The voice level mark mk3 is an utterance mark that changes in real time with the voice level of the user A.

  In order to visually display the sound level (sound pressure level), naturally, it is necessary to store the sound level (either the sound level itself or the sound level after being divided into a plurality of stages).

  Therefore, when realizing the display of the audio level mark mk3 illustrated in FIG. 14C, it is preferable to provide a storage device that enables this type of storage in addition to the device configuration illustrated in FIG. It is.

  Alternatively, in the apparatus configuration illustrated in FIG. 10, the utterance data holding unit 232 holds voice level data corresponding to the voice level in addition to the utterance data illustrated in FIG. 11 or as part of the utterance data. May be.

  In this case, for example, when the utterance determination unit 231 determines that the speech signal corresponds to the utterance when performing the utterance determination, the utterance period (one utterance period between the utterance start time and the utterance end time (one In the period corresponding to the utterance data), the voice level value may be stored in the utterance data holding unit 232 in a predetermined cycle. At this time, for example, the speech data processing unit 233 may perform the work of dividing the stored voice level value into a plurality of stages corresponding to the voice level mark mk3.

  Note that one utterance state video frame may include a plurality of utterance marks including these utterance marks. For example, when the utterance marks mk1 and mk2 are included, if there is a relatively large space between the remaining teams until the end of the utterance, both the utterance marks mk1 and mk2 are displayed, and only the utterance mark mk2 is just before the end of the utterance. May be displayed.

  As described above, according to the conversation system according to the second embodiment, the utterance state of the user A synchronized with the video frame of the user 20B of the site B displayed at the site A on the time axis of the site B. An utterance state video frame that includes an utterance mark mk that prescribes the For this reason, the user 20A at the site A can visually recognize at what timing the user 20B at the site B uttered what kind of speech he / she himself / herself. In addition, such an effect is the same in the base B. Therefore, it is possible to make the conversation between both parties proceed smoothly.

  In the conversation system according to the second embodiment, unlike the first embodiment, the utterance data processing unit 233 uses the total delay time measured by the delay measurement unit 220 to convert the utterance data of its own base. By managing, it is possible to perform all measures within the own site to display the video related to the user 20A at the timing synchronized with the time axis of the site B. Therefore, for example, even if only one transmission / reception device installed in one base among the transmission / reception devices in the conversation system has the configuration according to the present embodiment, at least the user in the one base Can proceed comfortably. Such an effect is remarkably exerted, for example, when the right to speak and the amount of speech are uneven among users.

  Also, according to the conversation system according to the second embodiment, attention is paid to the voice that has actually established the conversation, and the utterance determination based on the voice and the video (utterance mark mk) display based on the voice are performed. . For this reason, the user of each base can reliably compare his / her utterance with the utterance of the other party, and can further improve the smoothness of the conversation. In other words, by simply synthesizing a video based on one's own video (for example, the video frame for synthesis according to the first embodiment) with the other's video, even if the time axis synchronization is ensured, what kind of utterance is performed by itself? It may be difficult to confirm whether or not

  In the transmission / reception apparatus 200 according to the second embodiment, it is necessary to manage speech data by the synthesis processing unit 230, and it is necessary to measure the delay time required for data transmission by the delay measurement unit 220. Compared with the transmission / reception apparatus 100 according to the example, the processing load of the apparatus tends to increase.

For this reason, as in the first embodiment, when a video frame for synthesis based on the video of the user at the local site is used, the video signal obtained by the video input unit 111 for at least the time corresponding to the delay time, A storage area for holding a relatively large amount of data such as a video generated based on the video signal or a composite video frame generated based on the video is required. In view of this point, as in the second embodiment, a video frame including a video related to sound (here, the utterance mark mk) that is relatively light and clear even if it is a video is used as a video frame for synthesis. The effect of using it is great.
<Third embodiment>
The technical features according to the first embodiment and the second embodiment are not mutually exclusive, and can also be coordinated with each other. The third embodiment of the present invention will be described below.

  The conversation system according to the third embodiment is basically the same as the conversation system according to the second embodiment, but the configuration of the video / audio transmission / reception apparatus installed at each base, particularly the transmission / reception apparatus, is related to the second embodiment. It is different from the transmission / reception device 200. More specifically, the second embodiment differs from the second embodiment in that a transmission / reception device 300 is provided instead of the transmission / reception device 200.

  First, the configuration of the transmitting / receiving apparatus according to the third embodiment will be described with reference to FIG. FIG. 15 is a block diagram showing the configuration of the transmission / reception device 300. In the figure, the same reference numerals are given to the same parts as those in FIG. 10, and the description thereof will be omitted as appropriate. As in the second embodiment, the transmission / reception device 300 is assumed to be a transmission / reception device at the site A.

  In FIG. 15, the transmission / reception device 300 according to the third embodiment is different from the transmission / reception device 200 according to the second embodiment in that an input unit 310 is provided instead of the input unit 110. The input unit 310 is different from the input unit 110 in that an audio level display synthesis unit 311 is provided in a signal path between the video input unit 111 and the video / audio encoding unit 113.

  The audio level display synthesis unit 311 takes in the audio signal of the user B sent from the video / audio decoding unit 151 and detects the audio level (sound pressure level) (that is, with the same function as the speech determination unit 231). A video frame for synthesis (for example, a video frame corresponding to FIG. 14C) representing the audio level is generated. Further, the generated composite video frame is combined with the video frame of the user 20A (that is, functions as another example of the composite video generation unit and the composite video generation unit). Note that the video signal relating to the synthesized video frame is converted into video / audio data via the video / audio encoding unit 113 and transmitted to the site B via the transmission unit 120.

  In addition, the video frame synthesis processing related to the audio level is executed in the same way at the site B as well. That is, the audio of the user 20A is extracted from the downstream of the video / audio decoding unit 151 at the base B, and the video frame relating to the audio level is synthesized with the video frame of the user B to generate video / audio data. The data packet related to the generated video / audio data is transmitted to the site A.

  That is, according to the third embodiment, for the video related to the audio level, time synchronization with the time base on the other base side is performed on the other base side, as in the first embodiment. Therefore, the video frame corresponding to the video signal transmitted from the video / audio decoding unit 151 at the site A has already been synthesized with the video frame related to the audio level of the user 20A synchronized with the time axis of the site B.

  On the other hand, the utterance data processing unit 233 synthesizes a video with a relatively light processing load such as the remaining time display mark mk1 or the utterance display mark mk2 on the own site side. That is, an utterance state video frame further including a remaining time display mark mk1 or an utterance display mark mk2 is synthesized in synchronism with the time axis of the base B on the synthesized video frame obtained by synthesizing the synthesis video frame related to the audio level. Finally, it is output from the monitor 33 as one video frame. Supplementally, the sound level represented in the one video frame, the remaining time, and the presence or absence of utterance naturally define the utterance operation of the user A at the same time in the past at the base A.

  According to the third embodiment, for the video that can change in real time, such as the audio level mark mk3, from the viewpoint of the processing load and the storage capacity described above, as in the first embodiment, it is intentionally transmitted on the network 10. The processing load required for the time management can be reduced, and the data capacity of the composition processing unit 230 can be reduced. Therefore, higher conversation quality can be provided by taking advantage of both. In this example, the video related to the audio level is used. However, as in the first embodiment, a composite video frame derived from the user's video may be used.

  The transmission / reception device 300 can execute transmission control in the same manner as in the first embodiment, according to the control program stored in the ROM. Here, the transmission control according to the third embodiment will be described with reference to FIG. FIG. 16 is a flowchart of transmission control according to the third embodiment. In the figure, the same reference numerals are given to the same portions as those in FIG. 7, and the description thereof will be omitted as appropriate.

  In FIG. 16, when step S101 is executed, a video composition process is executed (step S501). In the video synthesis process according to step S501, as described above, the synthesis video frame synthesis process related to the audio level is executed in the input unit 310. When a video frame for synthesis based on the user's video is generated as in the first embodiment, a video frame for synthesis derived from the user's video is further generated by, for example, the video synthesizing unit 170 or the like. May be executed.

  After execution of the video composition processing, the video / audio encoding unit 113 generates video / audio data and data packets as in the first embodiment (step S103), and transmission via the transmission unit 120 is executed (step S103). Step S104). The transmission control according to the third embodiment is executed as described above.

  The present invention is not limited to the above-described embodiments, and can be appropriately changed without departing from the spirit or concept of the invention that can be read from the claims and the entire specification. The audio / video transmission / reception system, the computer program, and the recording medium are also included in the technical scope of the present invention.

  The present invention is applicable to an apparatus or system that establishes a conversation between users at remote locations.

  DESCRIPTION OF SYMBOLS 1 ... Conversation system, 10 ... Network, 20A, 20B ... User, 30A, 30B ... Video / audio transmission / reception apparatus, 100 ... Transmission / reception apparatus, 110 ... Input part, 120 ... Transmission part, 130 ... Reception part, 140 ... Reception buffer, 150 ... Output unit, 160 ... Video processing unit, 170 ... Video synthesis unit, 171 ... Buffer for video synthesis, 200 ... Transmission / reception device, 210 ... Output unit, 220 ... Delay measurement unit, 230 ... Speech data processing unit, 300 ... Input unit 310. Audio level display synthesis unit.

Claims (13)

In the video / audio transmission / reception system including a plurality of video / audio transmission / reception devices accommodated in a network and used by a user at each of a plurality of bases,
Own site user video and audio acquisition means for acquiring video and audio of the own site user;
Transmitting means for transmitting the acquired video and audio of the local user via the network to the video / audio transmitting / receiving apparatus at another site;
Other site user video and audio acquisition means for acquiring video and audio of other site users via the network;
The acquired video and audio of the other site user is the acquired image of the other site user in the own site user information generated based on at least one of the acquired video and audio of the own site user and A video / audio transmitting / receiving apparatus comprising: a predetermined video output means and a control means for controlling the audio output means so that the audio and the information synchronized on the time axis of the other base are output. The local site user information is generated by the video / audio transmitting / receiving device at the other site based on at least one of the transmitted video and audio of the local site user,
The local site user information generated by the video / audio transmitting / receiving device at the other site is transmitted via the network together with the video and audio of the other site user acquired at the other site,
The other site user video and audio acquisition means acquires the transmitted video and audio of the other site user and the own site user information,
The said control means controls the said image | video output means and the said audio | voice output means so that the acquired image | video and audio | voice of the other site user, and own site user information may be output. Video audio transmission and reception device. Further comprising other site user information generating means for generating other site user information based on at least one of the video and audio of the acquired other site user,
The video / audio transmission / reception apparatus according to claim 2, wherein the transmission unit transmits the generated other-site user information together with the acquired video and audio of the local user. A specifying means for specifying a delay time required for data transmission via the network with the audio / video transmission / reception device at the other site;
Own site user information generating means for generating the own site user information based on at least one of the acquired video and audio of the own site user;
Information synchronized on the time axis of the other site with the acquired video and audio of the other site user among the own site user information generated by the own site user information generating means based on the specified delay time And a selection means for selecting
The said control means controls the said video output means and an audio | voice output means so that the acquired image | video and audio | voice of the other site user, and the said selected information are output. The video / audio transmission / reception device according to any one of the above. 5. The video / audio transmission / reception apparatus according to claim 1, wherein the local site user information is a video corresponding to the acquired video of the local site user. 6. The video / audio transmitting / receiving apparatus according to claim 5, wherein the video corresponding to the acquired video of the local user is the acquired video of the local user whose display area is reduced. The video / audio transmission / reception apparatus according to claim 1, wherein the local site user information is a video corresponding to the acquired voice of the local site user. The video / audio transmission / reception apparatus according to claim 7, wherein the acquired video corresponding to the voice of the local user is a video representing the utterance state of the local user. The video / audio transmitting / receiving apparatus according to claim 7 or 8, wherein the acquired video corresponding to the voice of the local user is a video representing a remaining time until the utterance of the local user is completed. The video / audio transmitting / receiving apparatus according to any one of claims 1 to 10, wherein the local site user information is a notification sound for notifying that the utterance of the local site user has been completed. A plurality of video / audio transmission / reception devices accommodated in a network and used by a user at each of a plurality of bases,
Each of the plurality of video / audio transmission / reception devices includes:
Own site user video and audio acquisition means for acquiring video and audio of the own site user;
Transmitting means for transmitting the acquired video and audio of the local user via the network to the video / audio transmitting / receiving apparatus at another site;
Other site user video and audio acquisition means for acquiring video and audio of other site users via the network;
The acquired video and audio of the other site user is the acquired image of the other site user in the own site user information generated based on at least one of the acquired video and audio of the own site user and A video / audio transmission / reception system comprising: a predetermined video output unit and a control unit that controls the audio output unit so that the audio and the information synchronized on the time axis of the other site are output.   A computer program for causing a computer system to function as the video / audio transmitting / receiving apparatus according to any one of claims 1 to 10.   A recording medium for recording the computer program according to claim 12.

Images