JP2013213931A - Communication system and terminal device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To understand a state of a wide area and a state of a specific area within a space, from another space.SOLUTION: A terminal device 10A installed in a space RA and a terminal device 10B installed in a space RB communicate with each other, and then a sound or an image recorded in one of the space RA and the space RB is reproduced in the other of the spaces R. A first operation is performed in which the sound or the image recorded for a first recording area of the space RA is preferentially reproduced in the space RB. A second operation is performed in which the sound or the image recorded for a second recording area of the space RA that is larger than the first recording area is preferentially reproduced in the space RB.

Description

  The present invention relates to a technology for mutually transmitting and receiving audio and images between a plurality of terminal devices.

  For example, Patent Document 1 discloses a communication system in which a plurality of terminal devices installed in geographically separated spaces mutually transmit and receive sound and images to realize a duet between users at a remote location and user audition. It is disclosed.

JP 2002-366169 A

  Just by transmitting the voice and image of one user singing in the space to another terminal device as in the technique of Patent Document 1, the other users in the space (listening listeners) Since voice and images are not transmitted to users in other spaces, there is a problem that it is difficult for users in each space to grasp the atmosphere in other spaces. By using a sound collection device that picks up sound over a wide area in the space and an image pickup device that picks up a wide area, it is possible to transmit a wide atmosphere in the space to users in other spaces. It is difficult to clearly grasp the voice and image of a specific user who is singing in each space. In view of the above circumstances, an object of the present invention is to make it possible to grasp a state in a wide range in a space and a state in a specific range in another space.

  Means employed by the present invention to solve the above problems will be described. In order to facilitate understanding of the present invention, in the following description, the correspondence between each element of the present invention and the element of each of the embodiments described later is indicated in parentheses, but the scope of the present invention is not limited to the embodiment. It is not intended to limit the example.

  The communication system of the present invention includes a first terminal device (eg, terminal device 10A) installed in a first space (eg, space RA) and a second terminal device (eg, terminal device) installed in a second space (eg, space RB). 10B) communicates with each other to reproduce sound or images recorded in each of the first space and the second space in the other space, and the first recording range in the first space. A first operation for generating a reproduction signal (for example, reproduction signal Y [A], reproduction signal Z [A]) for preferentially reproducing the recorded sound or image in the second space, and the first of the first spaces. A reproduction control unit (for example, an acoustic processing unit 62) that executes a second operation for generating a reproduction signal for preferentially reproducing the sound or image recorded in the second recording range in the second space compared to the recording range. And control processing unit 46, image processing unit 82 and a control processing unit 46). In the above configuration, the first operation for preferentially reproducing the sound or image recorded in the first recording range in the first space in the second space, and the second recording range that is wider than the first recording range. And a second operation for preferentially reproducing the sound or image recorded in the first space in the second space, so that the sound or image in the first recording range in the first space (for example, specific Singer's sound and image) and sound or image within the second recording range (for example, sound and image of a plurality of listeners) can be grasped in the second space.

  In a preferred aspect of the present invention, the first terminal device includes a first recording signal (for example, an acoustic signal S1 and an image signal V1) indicating sound or an image within the first recording range, and an acoustic or image within the second recording range. 2 includes a signal acquisition unit (for example, an acoustic acquisition unit 52, an image acquisition unit 54) that acquires a second recording signal (for example, an acoustic signal S2 and an image signal V2) from the recording device in the first space, and performs reproduction control. In the first operation, the unit selects the first recording signal and generates a reproduction signal. In the second operation, the unit selects the second recording signal and generates a reproduction signal. In the above aspect, since the first recording signal is selected in the first operation and the second recording signal is selected in the second operation, the operation of the control processing unit is simplified and within the first recording range. There is an advantage that the sound and the image and the sound and the image within the second recording range can be clearly grasped. In addition, the specific example of the above aspect is later mentioned, for example as 1st Embodiment or 4th Embodiment.

  In a preferred aspect of the present invention, the first terminal device includes a first acoustic signal (for example, an acoustic signal S1) indicating sound within the first recording range and a second acoustic signal (for example, acoustic signal S1) indicating the sound within the second recording range. The sound acquisition unit (for example, the sound acquisition unit 52) that acquires the sound signal S2) from the sound collecting device in the first space, and the reproduction control unit in the first operation has a gain (for example, the first sound signal) The reproduction signal is generated by mixing the first acoustic signal and the second acoustic signal so that the adjustment value GA1) exceeds the gain of the second acoustic signal (for example, the adjustment value GA2). In the second operation, the second acoustic signal is generated. The first acoustic signal and the second acoustic signal are mixed so that the gain of the first acoustic signal exceeds the gain of the first acoustic signal to generate a reproduction signal. In the above aspect, since the reproduction signal is generated by mixing the first acoustic signal and the second acoustic signal, it is compared with the configuration in which the reproduction signal is generated by selecting either the first acoustic signal or the second acoustic signal. Thus, it is possible to reproduce acoustically natural sound in the second space reflecting the overall sound of the first space. In addition, the specific example of the above aspect is later mentioned, for example as 2nd Embodiment.

  In a preferred aspect of the present invention, the first terminal device includes a first acoustic signal (for example, an acoustic signal S1) indicating sound within the first recording range and a second acoustic signal (for example, acoustic signal S1) indicating the sound within the second recording range. The sound acquisition unit (for example, the sound acquisition unit 52) that acquires the sound signal S2) from the sound collection device in the first space, and the reproduction control unit in the first operation has an intensity of the second sound signal (for example, When the intensity P) exceeds the threshold, the first acoustic signal and the second acoustic signal are mixed so that the gain of the first acoustic signal exceeds the gain of the second acoustic signal to generate a reproduction signal, and the second acoustic signal is generated. When the signal strength is below the threshold, the first acoustic signal is selected to generate a reproduction signal. In the above aspect, when the intensity of the second acoustic signal is lower than the threshold, the reproduction signal is generated by selecting the first acoustic signal, and when the intensity of the second acoustic signal exceeds the threshold, the first acoustic signal and the first acoustic signal are generated. A reproduction signal is generated by mixing with two acoustic signals. That is, when the intensity of the second sound signal increases, the sound within the second recording range is reflected in the reproduced sound in the second space. Therefore, for example, when the user's sound within the second recording range increases, the atmosphere can be appropriately transmitted to the second space. In addition, the specific example of the above aspect is later mentioned, for example as 3rd Embodiment.

  In a preferred aspect of the present invention, the first terminal device includes a first image signal (for example, an image signal V1) indicating an image in the first recording range and a second image signal (for example, an image in the second recording range). And an image acquisition unit (for example, the image acquisition unit 54) that acquires the image signal V2) from the imaging device in the first space, and the reproduction control unit performs an image (for example, an image of the second image signal) in the first operation. FU2) generates a reproduction signal by synthesizing the first image signal and the second image signal so that the display size of the image of the first image signal (for example, image FU1) is smaller than the display size of the first image signal. The reproduction signal is generated by combining the first image signal and the second image signal so that the display size of the image of the one image signal is smaller than the display size of the image of the second image signal. In the above aspect, since the reproduction signal is generated by combining the first image signal and the second image signal, the reproduction signal is generated by selecting either the first image signal or the second image signal. And there exists an advantage that the user in 2nd space can visually recognize the image of both the 1st recording range and the 2nd recording range. In addition, the specific example of the above aspect is later mentioned, for example as 5th Embodiment.

  In each of the above aspects, the trigger for switching between the first operation and the second operation is arbitrary. For example, in a configuration in which the first terminal device and the second terminal device play the same music, the first operation is started at the time corresponding to the start of the music performance, and the second operation is performed at the time corresponding to the end of the music. The structure which starts and the 1st operation | movement is performed within a song section among music, and the structure which performs a 2nd operation outside a song section among music is suitable.

  The present invention is also specified as a terminal device used in the communication system according to each of the above-described embodiments. The terminal device of the present invention is a terminal device installed in the first space and capable of communicating with other terminals installed in the second space, and the sound or image recorded for the first recording range in the first space. A first operation for preferentially generating a reproduction signal to be reproduced in the second space, and a sound or image recorded for a second recording range that is wider than the first recording range in the first space is preferentially used. A reproduction control unit for executing a second operation for generating a reproduction signal to be reproduced in the two spaces; According to the above terminal device, the same operation and effect as the communication system according to the present invention are realized.

  The configuration according to each of the aspects described above is realized by hardware (electronic circuit) such as a dedicated DSP (Digital Signal Processor), and cooperation between a general-purpose arithmetic processing device such as a CPU (Central Processing Unit) and a program. It is also realized by. The program of the present invention is a computer installed in the first space and capable of communicating with the second terminal device installed in the second space, and the sound or image recorded in each of the first space and the second space is recorded on the other side. A first operation for generating a reproduction signal for preferentially reproducing the sound or image recorded in the first recording area in the first space in the second space for reproduction in the space; It functions as a reproduction control unit that executes a second operation for generating a reproduction signal for preferentially reproducing the sound or image recorded in the second recording range, which is wider than the first recording range, in the second space. According to the above program, the same operation and effect as the communication system according to the present invention are realized. Note that the program of the present invention is provided in a form stored in a computer-readable recording medium and installed in the computer, or is provided in a form distributed via a communication network and installed in the computer.

1 is a block diagram of a communication system according to a first embodiment of the present invention. It is a block diagram of the terminal device in a 1st embodiment. It is a block diagram which shows the functional structure of a terminal device. It is a flowchart of operation | movement of a control processing part. It is a flowchart of operation | movement of the control process part in 3rd Embodiment. It is a block diagram of the terminal device in a 4th embodiment. It is a block diagram which shows the functional structure of the terminal device in 4th Embodiment. It is explanatory drawing of operation | movement of 5th Embodiment.

<First Embodiment>
FIG. 1 is a block diagram of a communication system 100 according to the first embodiment of the present invention. As shown in FIG. 1, a communication system 100 according to the first embodiment includes a plurality of terminal devices 10 (10A, 10B) installed in spaces R (RA, RB) that are geographically separated from each other. Composed. Each terminal device 10 communicates with each other via the communication network 200. The communication network 200 is a broadband communication network with a sufficiently small transmission delay, for example. Although the terminal device 10A in the space RA and the terminal device 10B in the space RB are shown for convenience in FIG. 1, in practice, a large number of terminal devices 10 are connected to the communication network 200 and communicate with each other.

  The communication system 100 according to the first embodiment is a karaoke system in which each terminal device 10 functions as a karaoke device. In each space R (RA, RB) where the terminal device 10 is installed, there are a singer U1 who sings music and a plurality of listeners (audiences) U2 who listen to the singing of the singer U1. The terminal device 10A and the terminal device 10B play the same music, and the terminal device 10A and the terminal device 10B communicate with each other, so that the sound and images recorded in each of the space RA and the space RB are the other space. Played in R. Therefore, the singer U1 in the space RA and the singer U1 in the space RB can perform a duet, for example, while being located in a remote place.

  FIG. 2 is a block diagram of each terminal device 10. As shown in FIG. 2, each terminal device 10 is realized by a computer system including an arithmetic processing device 20, a storage device 22, an input device 24, a signal processing circuit 26, and a communication device 28. It is also possible to realize the terminal device 10 as a plurality of separate devices. For example, the communication device 28 can be configured as a separate device from the other elements of the terminal device 10.

  The arithmetic processing unit 20 executes various control processes and arithmetic processes by executing the programs stored in the storage device 22. The storage device 22 stores a program executed by the arithmetic processing device 20 and various data used by the arithmetic processing device 20. For example, the storage device 22 stores music data for specifying the accompaniment sound and lyrics of the music for each music. A known recording medium such as a semiconductor recording medium or a magnetic recording medium or a combination of a plurality of types of recording media is arbitrarily adopted as the storage device 22.

  The input device 24 is an operation device that receives instructions from the user (singer U1 or listener U2) for the terminal device 10, and includes, for example, a plurality of operators that can be operated by the user. For example, an operation panel installed in the casing of the terminal device 10 or a remote control device separate from the terminal device 10 is employed as the input device 24.

  The signal processing circuit 26 generates an acoustic signal SOUT and a reproduction signal Y indicating an acoustic waveform, and an image signal VOUT and a reproduction signal Z indicating an image (moving image). A specific configuration of the signal processing circuit 26 will be described later. The communication device 28 communicates with other terminal devices 10 via the communication network 200. Specifically, the communication device 28 exchanges the reproduction signal Y and the reproduction signal Z generated by the signal processing circuit 26 with other terminal devices 10. For example, assuming communication between the terminal device 10A in the space RA and the terminal device 10B in the space RB, as shown in FIG. 2, the reproduction signal Y [A] generated by the signal processing circuit 26 of the terminal device 10A and the reproduction are generated. The communication device 28 of the terminal device 10A transmits the signal Z [A] to the terminal device 10B, and the reproduction signal Y [B] and the reproduction signal Z [B] generated by the signal processing circuit 26 of the terminal device 10B are transmitted to the terminal device 10A. The communication device 28 receives from the terminal device 10B.

  As shown in FIGS. 1 and 2, a recording device 12 and a playback device 14 are connected to each terminal device 10. The recording device 12 is a device that records sound and images in the space R, and includes a sound collecting device 32-1, a sound collecting device 32-2, and an imaging device 34. The imaging device 34 supplies an image signal V of an image (moving image) captured in the space R to the terminal device 10. The imaging device 34 of the first embodiment captures an image of the singer U1 in the space R.

  The sound collecting device 32-1 supplies the terminal device 10 with an acoustic signal S1 that picks up the surrounding sound, and the sound collecting device 32-2 supplies the sound signal S2 that picks up the surrounding sound to the terminal device 10. To do. The sound collection device 32-1 and the sound collection device 32-2 have different sound collection ranges (recording ranges) in the space R. Specifically, the sound collection device 32-2 collects sound over a wide range compared to the sound collection device 32-1. For example, a directional (typically unidirectional) microphone is employed as the sound collection device 32-1, and a non-directional microphone (for example, a boundary microphone) is employed as the sound collection device 32-2. The sound collecting device 32-1 is used for collecting the singing sound of the singer U1 in the space R, and the sound collecting device 32-2 is expressed as overall sound in the space R (hereinafter referred to as “background sound”). Used). The background sound in the space R is, for example, sound such as the utterance sound (conversation sound or shout) or applause sound of each listener U2. As described above, the acoustic signal S1 generated by the sound collecting device 32-1 corresponds to the singing sound of the singer U1, and the acoustic signal S2 generated by the sound collecting device 32-2 is the background sound in the space R (that is, Corresponding to the overall atmosphere in the space R).

  The playback device 14 is an element that plays back sound and images in the space R, and includes a sound emitting device 142 and a display device 144. The sound emitting device 142 (speaker) reproduces sound corresponding to the sound signal SOUT generated by the signal processing circuit 26. The D / A converter that converts the acoustic signal SOUT from digital to analog is not shown for convenience. The display device 144 (for example, a liquid crystal display panel) displays an image corresponding to the image signal VOUT generated by the signal processing circuit 26.

  FIG. 3 is a block diagram illustrating a functional configuration of the terminal device 10 according to the first embodiment. In the following description, the configuration of the terminal device 10A in the space RA will be described, but the configuration of each terminal device 10 (terminal device 10B) in the communication system 100 is the same. When the terminal device 10A and the terminal device 10B communicate with each other, one of the terminal device 10A and the terminal device 10B is designated as a main terminal (master) and the other is designated as a sub terminal (slave).

  As illustrated in FIG. 3, the arithmetic processing device 20 of the terminal device 10 </ b> A executes a program stored in the storage device 22, thereby obtaining a plurality of elements (accompaniment playback unit 42, lyrics playback unit 44, control processing unit 46). Function. Further, the signal processing circuit 26 of the terminal device 10 </ b> A includes an acoustic acquisition unit 52, an image acquisition unit 54, an acoustic processing unit 62, an acoustic processing unit 64, and an image processing unit 84. A configuration in which part or all of the functions of the arithmetic processing device 20 are realized by the signal processing circuit 26 and a configuration in which part or all of the functions of the signal processing circuit 26 are realized by the arithmetic processing device 20 may be employed. It is also possible to distribute each function of the signal processing circuit 26 to a plurality of integrated circuits.

  The sound acquisition unit 52 acquires the sound signal S1 generated by the sound collection device 32-1 and the sound signal S2 generated by the sound collection device 32-2. The acoustic acquisition unit 52 can include an A / D converter that converts the acoustic signal S1 and the acoustic signal S2 from analog to digital, and a limiter that limits the intensity of the acoustic signal S1 and the acoustic signal S2 to a predetermined range. . An echo canceller that removes echoes generated in the acoustic signal S1 and the acoustic signal S2 can be mounted in the acoustic acquisition unit 52. The image acquisition unit 54 acquires the image signal V generated by the imaging device 34. The image signal V acquired by the image acquisition unit 54 is transmitted from the communication device 28 to the terminal device 10B as a reproduction signal Z [A]. Further, the communication device 28 of the terminal device 10A receives the reproduction signal Z [B] transmitted from the terminal device 10B.

  The music data stored in the storage device 22 includes accompaniment data DM1 and lyrics data DM2. The accompaniment data DM1 designates accompaniment sounds of music in time series. The lyrics data DM2 designates the lyrics of the music in time series. The accompaniment data DM1 and the lyrics data DM2 are time-series data described in a format compliant with, for example, the MIDI (Musical Instrument Digital Interface) standard. The accompaniment reproducing unit 42 in FIG. 3 generates an accompaniment sound of a song selected by the user (hereinafter referred to as “selected song”). Specifically, the accompaniment reproducing unit 42 generates the acoustic signal SX of the accompaniment sound by sequentially processing the accompaniment data DM1 of the selected song. The generation of the accompaniment sound by the accompaniment playback unit 42 is executed when the terminal device 10A is designated as the main terminal. When the terminal device 10A is designated as the sub-terminal, the accompaniment sound of the selected tune is reproduced in the space RA according to the acoustic signal SX generated by the accompaniment reproduction unit 42 of the terminal device 10B. The lyrics reproduction unit 44 generates the image signal VX by sequentially processing the lyrics data DM2 of the selected song. The image signal VX indicates an image of the lyrics of the selected song (an image in which a character string of the lyrics is wiped over time). A known technique is arbitrarily employed for generation of the acoustic signal SX (accompaniment sound reproduction) by the accompaniment reproduction unit 42 and generation of the image signal VX (reproduction of lyrics) by the lyrics reproduction unit 44.

  The image processing unit 84 generates the image signal VOUT by combining the reproduction signal Z [B] received by the communication device 28 from the terminal device 10B and the image signal VX generated by the lyrics reproduction unit 44. By supplying the image signal VOUT generated by the image processing unit 84 to the display device 144, an image obtained by synthesizing the lyrics of the selected song with the image in the space RB is displayed on the display device 144 in the space RA. Therefore, the singer U1 and the listener U2 in the space RA can visually confirm the state in the space RB.

  The sound processing unit 62 generates a reproduction signal Y [A] for controlling sound reproduction within the space RB. Specifically, the acoustic processing unit 62 generates a reproduction signal Y [A] corresponding to the acoustic signal S1 and the acoustic signal S2 acquired by the acoustic acquisition unit 52. The acoustic processing unit 62 of the first embodiment mixes the acoustic signal SX with the reproduced signal Y [A] when the terminal device 10A is designated as the main terminal (when the accompaniment reproducing unit 42 generates the acoustic signal SX). To do.

  The acoustic processing unit 62 of the first embodiment is a mixer that mixes the acoustic signal S1, the acoustic signal S2, and the acoustic signal SX with a variable mixing ratio, and includes three adjustment units 72 (72-1 to 72-3). ) And an adder 74. The adjustment unit 72-1 adjusts the intensity of the acoustic signal S1 according to the adjustment value (gain) GA1. Similarly, the adjustment unit 72-2 adjusts the intensity of the acoustic signal S2 according to the adjustment value GA2, and the adjustment unit 72-3 adjusts the intensity of the acoustic signal SX according to the adjustment value GA3. The adder 74 generates the reproduction signal Y [A] by adding the signals after adjustment by the adjustment units 72. The reproduction signal Y [A] generated by the acoustic processing unit 62 is transmitted from the communication device 28 to the terminal device 10B.

  The acoustic processing unit 64 generates an acoustic signal SOUT that is reproduced in the space RA. The acoustic processing unit 64 of the first embodiment includes an acoustic signal S1 and an acoustic signal S2 acquired by the acoustic acquisition unit 52, a reproduction signal Y [B] received by the communication device 28 from the terminal device 10B, and an accompaniment reproduction unit 42. A mixer that mixes the generated acoustic signal SX with a variable mixing ratio, and includes four adjustment units 76 (76-1 to 76-4) and an addition unit 78. The adjusting unit 76-1 adjusts the intensity of the acoustic signal S1 according to the adjustment value GB1. Similarly, the adjustment unit 76-2 adjusts the intensity of the acoustic signal S2 according to the adjustment value GB2, and the adjustment unit 76-3 adjusts the intensity of the acoustic signal SX according to the adjustment value GB3, and the adjustment unit 76-4. Adjusts the intensity of the reproduction signal Y [B] according to the adjustment value GB4. The adder 78 generates the acoustic signal SOUT by adding the signals adjusted by the respective adjusters 76. Sound corresponding to the sound signal SOUT generated by the sound processing unit 64 is reproduced from the sound emitting device 142 into the space RA. As understood from the above description, in each of the space RA and the space RB, a mixed sound of the sound in the space RA, the sound in the space RB, and the accompaniment sound is reproduced. Therefore, each singer U1 in the space RA and the space RB can sing (ie, duet) the selected tune while listening to the singing sound of the singer U1 in the other space R.

  The control processing unit 46 in FIG. 3 variably controls the adjustment values G (GA1 to GA3, GB1 to GB4) applied to the acoustic processing unit 62 and the acoustic processing unit 64. Specifically, the control processing unit 46 determines each adjustment value GB (GB1 to GB4) applied to generation of the acoustic signal SOUT (acoustic processing unit 64) according to, for example, an instruction from the user to the input device 24. Control. Further, the control processing unit 46 sings the singing sound (acoustic signal) of the singer U1 that the sound collecting device 32-1 has collected in the space RA during the period in which the selected music is played (hereinafter referred to as “music playing period”). S1) is preferentially reproduced in the space RB as compared to the sound of the listener U2 in the space RA, and the background sound (sound) collected by the sound pickup device 32-2 in the space RA outside the music performance period. Each applied to the generation of the reproduction signal Y [A] (acoustic processing unit 62) so that the signal S2) is preferentially reproduced in the space RB in comparison with the singing sound of the singer U1 in the space RA. Adjust the adjustment value GA (GA1, GA2). The adjustment value GA3 is set according to an instruction from the user, for example.

  FIG. 4 is a flowchart of an operation executed by the control processing unit 46 in relation to the generation of the reproduction signal Y [A]. The process shown in FIG. 4 is executed every predetermined time during the operation of the terminal device 10A. When the processing of FIG. 4 is started, the control processing unit 46 determines whether or not the current time is within the music performance period (QA1). Specifically, the control processing unit 46 starts from the time point corresponding to the start of the performance of the selected song (for example, the start point of the performance or the time point before or after the start point of the performance) (eg, the time point corresponding to the end of the performance of the selected song) The determination in step QA1 is executed with the period from the end point of the performance to the time point before and after the end point of the performance as the music performance period.

  When the current time is within the music performance period (QA1: YES), the control processing unit 46 sets the adjustment value GA1 to 1 (that is, the numerical value that allows the adjustment unit 72-1 to pass the acoustic signal S1) and the adjustment value GA2. Is set to 0 (that is, a numerical value that the adjusting unit 72-2 blocks the acoustic signal S2) (QA2). That is, the acoustic signal S1 obtained by collecting the singing sound of the singer U1 in the space RA is selectively applied to the generation of the reproduction signal Y [A], and the acoustic signal S2 obtained by collecting the background sound in the space RA is reproduced. It does not apply to the generation of signal Y [A]. Therefore, the sound reproduced in the space RB in accordance with the acoustic signal SOUT generated from the reproduced signal Y [A] by the acoustic processing unit 64 of the terminal device 10B does not include the background sound in the space RA, but in the space RA. The sound of the singer U1 can be heard clearly.

  On the other hand, when the current time is outside the musical performance period (QA1: NO), the control processing unit 46 sets the adjustment value GA1 to 0 and sets the adjustment value GA2 to 1 (QA3). That is, the acoustic signal S2 obtained by collecting the background sound in the space RA is selectively applied to the generation of the reproduction signal Y [A], and the acoustic signal S1 obtained by collecting the singing sound of the singer U1 in the space RA is reproduced. It does not apply to the generation of signal Y [A]. Therefore, the sound reproduced in the space RB according to the acoustic signal SOUT generated from the reproduced signal Y [A] by the acoustic processing unit 64 of the terminal device 10B does not include the singing sound of the singer U1 in the space RA. The background sound in the space RA can be sufficiently heard. As understood from the above description, the sound processing unit 62 and the control processing unit 46 of the first embodiment preferentially use the sound (singing sound) in the space RA collected by the sound collecting device 32-1 as a space. The first operation (QA2) for generating a reproduction signal Y [A] to be reproduced in RB and the sound (background sound) in the space RA picked up by the sound collecting device 32-2 are preferentially reproduced in the space RB. It functions as an element (reproduction control unit) that executes the second operation (QA3) for generating the reproduction signal Y [A] to be performed.

  As described above, in the first embodiment, the first operation (QA2) for reproducing the singing sound in the space RA with priority over the background sound in the space RB, and the singing sound for the background sound in the space RA. The second operation (QA3) for preferentially reproducing in the space RB is selectively executed. Therefore, there is an advantage that the singer U1 and the listener U2 in the space RB can grasp both the singing sound of the singer U1 in the space RA and the wide range atmosphere (background sound) in the space RA. Also, since the first operation is performed within the music performance period and the second operation is performed outside the music performance period, the music performance is performed while reliably listening to the singing sound of the selected music during the music performance period. Outside the period, it is possible to listen to the utterance sound (for example, shout or conversation sound) and applause sound of the listener U2.

Second Embodiment
A second embodiment of the present invention will be described below. In the first embodiment, one of the acoustic signal S1 and the acoustic signal S2 is selected to generate the reproduction signal Y [A]. In the second embodiment, the reproduction signal Y [A] is generated by mixing the acoustic signal S1 and the acoustic signal S2. In addition, about the element which an effect | action and a function are equivalent to 1st Embodiment in each form illustrated below, the reference | standard referred by description of 1st Embodiment is diverted, and each detailed description is abbreviate | omitted suitably.

  The control processing unit 46 of the second embodiment sets the adjustment value GA1 to 1 as in the first embodiment and sets the adjustment value GA2 to a predetermined value C in step QA2 (first operation) of FIG. . The predetermined value C is set to a positive number (a positive number less than 1) lower than the adjustment value GA1. For example, the predetermined value C is set to 0.25. As a result of the above control, the sound processing unit 62 mixes the sound signal S1 and the sound signal S2 so that the gain (GA1) of the sound signal S1 exceeds the gain (GA2) of the sound signal S2 (the sound signal S1 and the sound). The reproduced signal Y [A] is generated by mixing the signal S2 with a mixing ratio of 4: 1. Therefore, in the space RB, the background sound in the space RA is reproduced with a small volume together with the singing sound of the singer U1 in the space RA. As understood from the above description, in the second embodiment as well, as in the first embodiment, the first operation (in which the singing sound in the space RA is preferentially reproduced in the space RB with respect to the background sound ( QA2) is executed.

  Further, the control processing unit 46 of the second embodiment sets the adjustment value GA2 to 1 as in the first embodiment, while setting the adjustment value GA1 to the predetermined value C in step QA3 (second operation) of FIG. Set. The predetermined value C is set to a positive number (for example, 0.25) lower than the adjustment value GA2. As a result of the above control, the sound processing unit 62 mixes the sound signal S1 and the sound signal S2 so that the gain (GA2) of the sound signal S2 exceeds the gain (GA1) of the sound signal S1 (the sound signal S2 and the sound). The reproduction signal Y [A] is generated by mixing the signal S1 with a mixing ratio of 4: 1. Therefore, in the space RB, the singing sound of the singer U1 in the space RA is reproduced together with the background sound in the space RA at a low volume. As understood from the above description, in the second embodiment as well, in the same manner as in the first embodiment, the second operation for preferentially reproducing the background sound in the space RA with respect to the singing sound in the space RB ( QA3) is executed.

  Also in the second embodiment described above, the same effect as in the first embodiment is realized. In the second embodiment, the background sound is mixed with the singing sound at a low volume when the first operation is executed, and the singing sound is mixed with the background sound at the low volume when the second operation is executed. Therefore, there is an advantage that sound of an acoustically natural impression reflecting the overall sound in the space RA can be reproduced in the space RB.

<Third Embodiment>
A third embodiment of the present invention will be described. The control processing unit 46 of the third embodiment executes the process of FIG. 5 at predetermined intervals instead of the process of FIG. When the processing of FIG. 5 is started, the control processing unit 46 determines whether or not the current time is within the music performance period as in the first embodiment (QA1). When the current time is within the music performance period (QA1: YES), the control processing unit 46 calculates the intensity (for example, average value of power) P of the acoustic signal S2 acquired by the sound acquisition unit 52 from the sound pickup device 32-2. (QA4). Then, the control processing unit 46 determines whether or not the intensity P calculated in step QA4 exceeds a predetermined threshold value PTH (QA5).

  When the intensity P of the acoustic signal S2 exceeds the threshold value PTH (QA5: YES), the control processing unit 46 sets the adjustment value GA1 of the acoustic processing unit 62 to 1 and sets the adjustment value GA2 to the predetermined value C (QA6). ). The predetermined value C is set to a positive number less than 1 (for example, 0.25) as in the second embodiment. That is, the acoustic processing unit 62 generates the reproduction signal Y [A] by mixing the acoustic signal S1 and the acoustic signal S2 so that the gain (GA1) of the acoustic signal S1 exceeds the gain (GA2) of the acoustic signal S2. To do. Since the acoustic signal SOUT corresponding to the reproduction signal Y [A] generated under the above conditions is reproduced in the space RB, the background sound in the space RA is small and the singing sound of the singer U1 in the space RA. It is reproduced in the space RB.

  On the other hand, when the intensity P of the acoustic signal S2 is lower than the threshold value PTH (QA5: NO), the control processing unit 46 executes the same step QA2 as in the first embodiment. As described above, in the third embodiment, when the intensity P of the acoustic signal S2 exceeds the threshold value PTH, the reproduction signal Y [A] is generated by mixing the acoustic signal S1 and the acoustic signal S2 (QA6), When the intensity P is lower than the threshold value PTH, the reproduction signal Y [A] is generated by selecting the acoustic signal S1 (QA2). If the current time is outside the musical performance period (QA1: NO), the reproduction signal Y [A] is generated by selecting the acoustic signal S2 (QA3), as in the first embodiment.

  In the third embodiment, the same effect as in the first embodiment is realized. In the third embodiment, when the intensity P of the acoustic signal S2 containing the background sound in the space RA exceeds the threshold value PTH (for example, when the atmosphere of each listener U2 rises), it is within the music performance period. However, since the background sound is reproduced in the space RB, for example, the singer U1 and the listener U2 in the space RB can listen to the sound of the listener U2 generated in the space RA during the musical performance period (for example, shout and applause). it can. Therefore, the effect that the singer U1 and the listener U2 in the space RB can grasp the atmosphere in the space RA is particularly remarkable. On the other hand, since the background sound is reproduced at a low volume, listening to the singing sound of the singer U1 in the space RA in the space RB is not hindered.

<Fourth embodiment>
FIG. 6 is a block diagram of each terminal device 10 (terminal device 10A) in the fourth embodiment. As shown in FIG. 6, the recording device 12 connected to the terminal device 10 of the fourth embodiment includes a sound collection device 32, an imaging device 34-1 and an imaging device 34-2. The imaging device 34-1 generates an image signal V1 captured in the space RA, and the imaging device 34-2 generates an image signal V2 captured in the space RA. The imaging device 34-1 and the imaging device 34-2 differ in the imaging range (recording range) in the space R. Specifically, the imaging device 34-2 images a wider range than the imaging device 34-1. In the following description, it is assumed that the imaging device 34-1 images the singer U1 in the space R, and the imaging device 34-2 images a plurality of listeners U2 in the space R. The sound collection device 32 collects the singing sound of the singer U1 in the space RA and generates the acoustic signal S, similarly to the sound collection device 32-1 of the first embodiment.

  FIG. 7 is a block diagram of a functional configuration of the terminal device 10 (terminal device 10A) according to the fourth embodiment. As shown in FIG. 7, the signal processing circuit 26 of the fourth embodiment includes an acoustic acquisition unit 52, an image acquisition unit 54, an acoustic processing unit 62, an acoustic processing unit 64, an image processing unit 82, and an image processing unit 84. Consists of. Note that part or all of the functions of the signal processing circuit 26 can be realized by the arithmetic processing unit 20. The sound acquisition unit 52 acquires the sound signal S generated by the sound collection device 32 as in the first embodiment. The image acquisition unit 54 acquires the image signal V1 generated by the imaging device 34-1 and the image signal V2 generated by the imaging device 34-2.

  The sound processing unit 62 generates a reproduction signal Y [A] for controlling sound reproduction within the space RB. Specifically, the acoustic processing unit 62 mixes the acoustic signal S acquired by the acoustic acquisition unit 52 and the acoustic signal SX generated by the accompaniment reproduction unit 42 when the terminal device 10A is designated as the main terminal. Thus, when the reproduction signal Y [A] is generated and the terminal device 10A is designated as the sub-terminal, the acoustic signal S acquired by the acoustic acquisition unit 52 is output as the reproduction signal Y [A]. The reproduction signal Y [A] generated by the acoustic processing unit 62 is transmitted from the communication device 28 to the terminal device 10B.

  The acoustic processing unit 64 generates an acoustic signal SOUT that is reproduced in the space RA. Specifically, when the terminal device 10A is designated as the main terminal, the acoustic processing unit 64 uses the acoustic signal S acquired by the acoustic acquisition unit 52 and the reproduction signal Y [B] received by the communication device 28 from the terminal device 10B. And the acoustic signal SX generated by the accompaniment reproducing unit 42 are mixed to generate the acoustic signal SOUT. When the terminal device 10A is designated as the sub-terminal, the acoustic processing unit 64 mixes the acoustic signal S acquired by the acoustic acquisition unit 52 and the reproduction signal Y [B] received by the communication device 28 from the terminal device 10B. As a result, the acoustic signal SOUT is generated. Sound corresponding to the sound signal SOUT generated by the sound processing unit 64 is reproduced from the sound emitting device 142 into the space RA. Therefore, in both the space RA and the space RB, a mixed sound of the singing sound of the singer U1 in the space RA, the singing sound of the singer U1 in the space RB, and the accompaniment sound of the selected tune is reproduced.

  The image processing unit 82 in FIG. 7 generates a reproduction signal Z [A] that controls reproduction of an image in the space RB. The image processing unit 82 of the first embodiment selects one of the image signal V1 and the image signal V2 acquired by the image acquisition unit 54 as the reproduction signal Z [A]. Further, the image processing unit 84 generates an image signal VOUT that controls reproduction of an image in the space RA. Specifically, the image processing unit 84 generates the image signal VOUT by combining the reproduction signal Z [B] received by the communication device 28 from the terminal device 10B and the image signal VX generated by the lyrics reproduction unit 44. By supplying the image signal VOUT generated by the image processing unit 84 to the display device 144, an image obtained by synthesizing the lyrics of the selected song with the image in the space RB is displayed on the display device 144 in the space RA.

  The control processing unit 46 of FIG. 7 controls the operation of the image processing unit 82 (selection of the image signal V1 or the image signal V2). Specifically, the control processing unit 46 controls the image processing unit 82 to select the image signal V1 within the music performance period, and controls the image processing unit 82 to select the image signal V2 outside the music performance period. Control. As understood from the above description, the image processing unit 82 and the control processing unit 46 of the fourth embodiment preferentially use the image of the singer U1 in the space RA imaged by the imaging device 34-1 in the space RB. And a second operation (image signal) that preferentially reproduces images of a plurality of listeners U2 in the space RA imaged by the imaging device 34-2 in the space RB. V2 selection) functions as an element (reproduction control unit).

  As described above, in the fourth embodiment, the first operation for preferentially reproducing the image of the singer U1 in the space RA in the space RB and the image of each listener U2 in the space RA are prioritized. A second operation for reproducing in the space RB is executed. Therefore, there is an advantage that the singer U1 and the listener U2 in the space RB can grasp both the image of the singer U1 in the space RA and the listener U2 (the overall atmosphere in the space RA) in the space RA. is there. In addition, since the first operation is performed within the music performance period and the second operation is performed outside the music performance period, it is outside the music performance period while visually confirming that the singer U1 sings during the music performance period. Then, for example, it is possible to visually recognize how the listener U2 is excited.

<Fifth Embodiment>
The image processing unit 82 of the fourth embodiment selects one of the image signal V1 and the image signal V2 as the reproduction signal Z [A]. The image processing unit 82 of the fifth embodiment generates a reproduction signal Z [A] by combining the image signal V1 and the image signal V2. Similar to the fourth embodiment, an image corresponding to the reproduction signal Z [A] generated by the terminal device 10A is displayed in the space RB, and according to the reproduction signal Z [B] generated by the terminal device 10B. The displayed image is displayed in the space RA.

  FIG. 8 is an explanatory diagram of operations of the image processing unit 82 and the control processing unit 46 in the fifth embodiment. As shown in FIG. 8, the image processing unit 82 generates a reproduction signal Z [A] indicating one of the image FA and the image FB by combining the image signal V1 and the image signal V2. Each of the images FA and FB is an image (picture-in-picture) obtained by combining the image FU1 of the singer U1 indicated by the image signal V1 and the image FU2 of the listener U2 indicated by the image signal V2. In the image FA, the display size of the image FU2 is smaller than the display size of the image FU1. That is, the image FU1 of the singer U1 is preferentially displayed as compared with the image FU2 of the listener U2. On the other hand, in the image FB, the display size of the image FU1 is smaller than the display size of the image FU2. That is, the image FU2 of the listener U2 is preferentially displayed as compared with the image FU1 of the singer U1. The control processing unit 46 generates the reproduction signal Z [A] of the image FA when the first operation is performed (for example, within the music performance period), and the image FB when the second operation is performed (for example, outside the music performance period). The image processing unit 82 is controlled so that the reproduction signal Z [A] is generated.

  In the fifth embodiment, the same effect as in the fourth embodiment is realized. In the fourth embodiment, images (FA, FB) obtained by combining the image FU1 of the singer U1 in the space RA and the image FU2 of the listener U2 in the space RA are displayed in the space RB. There is an advantage that the overall state of the singer U1 and the listener U2 in RA can be confirmed in the space RB.

<Modification>
Each of the above-described embodiments can be variously modified. Specific modifications are exemplified below. Two or more aspects arbitrarily selected from the following examples can be appropriately combined.

(1) In the first embodiment to the third embodiment, the recording device 12 has exemplified the configuration including the sound collection device 32-1 and the sound collection device 32-2, but the sound collection range of the single sound collection device 32. By variably controlling (directivity), it is possible to generate the acoustic signal S1 and the acoustic signal S2 corresponding to different sound collection ranges. Further, in the fourth embodiment and the fifth embodiment, the configuration in which the recording device 12 includes the imaging device 34-1 and the imaging device 34-2 is illustrated, but the imaging range of the single imaging device 34 is variably controlled. By controlling the focal length of the imaging lens (for example, it is possible to generate the image signal V1 and the image signal V2 corresponding to different imaging ranges).

(2) In each of the above-described embodiments, the reproduction control unit (the acoustic processing unit 62 and the control processing unit 46) that generates the reproduction signal Y [A] and the reproduction control unit (the image processing unit 82) that generates the reproduction signal Z [A]. Although the control processing unit 46) is installed in the terminal device 10A, the position of the reproduction control unit is changed as appropriate. For example, the playback control unit can be installed in a management device (for example, a server device) connected to the communication network 200 separately from the terminal device 10A and the terminal device 10B. For example, the acoustic signal S1 and the acoustic signal S2 are transmitted from the terminal apparatus 10A to the management apparatus, and the reproduction control unit in the management apparatus generates the reproduction signal Y [A] and transmits it to the terminal apparatus 10B. A configuration is adopted in which the image signal V2 is transmitted from the terminal apparatus 10A to the management apparatus, and the reproduction control unit in the management apparatus generates the reproduction signal Z [A] and transmits it to the terminal apparatus 10B. As can be understood from the above description, the playback control unit is included as an element installed in the communication system 100, and a specific position in the communication system 100 is arbitrary.

(3) In each of the above-described embodiments, the first operation is performed during the music performance period and the second operation is performed outside the music performance period. The conditions for switching between the first operation and the second operation are as exemplified above. It is not limited. For example, the structure which performs a 1st operation | movement within the song area in which a song part exists among selection music, and performs a 2nd operation | movement outside a song area (for example, in the area of a prelude, an interlude, or a postlude) may be employ | adopted. According to the above configuration, for example, it is possible to listen to the sound such as the applause sound of the listener U2 generated in the space RA in the section of the prelude, the interlude, and the postlude in the selected tune, even in the space RB. In addition, the operation specification data (for example, MIDI data in which event data for instructing operation switching are arranged in time series) specifying the time point of switching between the first operation and the second operation in time series is included in the music data of each song. It is also possible to switch from one of the first operation and the second operation to the other at the time specified by the operation specifying data. According to the above structure, there exists an advantage that the period which should pay attention to the sound and image of singer U1, and the period which should pay attention to the sound and image of each listener U2 can be set separately for every music. In addition, a configuration in which the first operation and the second operation are switched in accordance with an instruction from the user to the input device 24 is also employed.

(4) A configuration in which which of the first operation and the second operation is being performed is notified to the singer U1 and the listener U2 in each space R is also preferable. For example, during execution of the first action, the fact that the voice or image of the singer U1 in the space RA is preferentially reproduced in the space RB is displayed on the display device 144 in the space RA, and the second action During execution of the above, it is displayed on the display device 144 in the space RA that the background sound in the space RA or the image of each listener U2 is preferentially reproduced in the space RB.

(5) In each of the above-described embodiments, the configuration in which the echo canceller for removing the echoes of the acoustic signal S1 and the acoustic signal S2 is mounted in the acoustic acquisition unit 52 is exemplified. However, the transmission delay between the terminal device 10A and the terminal device 10B Is sufficiently small, it is difficult for echoes to be generated in the acoustic signal S1 and the acoustic signal S2, so that the echo canceller can be omitted.

(6) In the first to third embodiments, if the adjustment value GA1 or the adjustment value GA2 is changed discontinuously, noise may be generated in the reproduced sound and the sound quality may be degraded. Therefore, a configuration in which the adjustment value GA1 and the adjustment value GA2 are crossfade is suitable. For example, when changing from the first operation to the second operation, the adjustment value GA1 is changed from 1 to 0 over time and the adjustment value GA2 is changed from 0 to 1 over time within a predetermined length of time. When the second operation is changed to the first operation, the adjustment value GA2 is changed from 1 to 0 with time and the adjustment value GA1 is changed from 0 to 1 with time within a predetermined length of time. According to the above configuration, there is an advantage that deterioration in sound quality due to discontinuous changes in the adjustment value GA1 and the adjustment value GA2 is suppressed.

(7) In each form mentioned above, although the karaoke system which utilized the terminal device 10 as a karaoke apparatus was illustrated, the use of the communication system 100 is not limited to karaoke. For example, even when a user in the space RA and a user in the space RB perform an ensemble (session) of musical instruments, the communication system 100 similar to the above embodiments can be used. That is, the accompaniment reproducing part 42 and the lyrics reproducing part 44 peculiar to karaoke can be abbreviate | omitted among each above-mentioned form. In the case where a user in the space RA and a user in the space RB perform together, for example, a configuration in which the user in each space R switches between the first operation and the second operation by operating the input device 24. Adopted. It is also possible to switch the second operation to the first operation when the start of the ensemble by each user is detected. For example, when the volume of a specific musical instrument (for example, a percussion instrument sound that tends to appear at the beginning of a musical piece) exceeds a threshold, a configuration in which the start of an ensemble is determined, or when a musical sound with a specific rhythm or melody occurs (for example, a musical piece) A configuration is adopted in which the start of the ensemble is determined when a percussion instrument count occurs at the beginning of.

(8) The acoustic control exemplified in the first to third embodiments and the image control exemplified in the fourth and fifth embodiments may be combined. In each of the above-described embodiments, attention is paid to communication between the terminal device 10A and the terminal device 10B. However, each of the above-described embodiments is also applicable when three or more terminal devices 10 installed in different spaces R communicate with each other. The same configuration is adopted. For example, the reproduction signal Y [A] and the reproduction signal Z [A] generated by the terminal device 10A are transmitted to two or more terminal devices 10 that are communication partners of the terminal device 10A.

DESCRIPTION OF SYMBOLS 100 ... Communication system 200 ... Communication network, 10 (10A, 10B) ... Terminal device, 12 ... Recording device, 14 ... Reproduction device, 20 ... Arithmetic processing device, 22 ... Storage device, 24 ... ... Input device, 26 ... Signal processing circuit, 28 ... Communication device, 32 (32-1, 32-2) ... Sound pickup device, 34 (34-1, 34-2) ... Imaging device, 42 ... ... Accompaniment playback unit, 44 ... Lyrics playback unit, 46 ... Control processing unit, 52 ... Sound acquisition unit, 54 ... Image acquisition unit, 62, 64 ... Sound processing unit, 82, 84 ... Image processing unit 142 ... Sound emitting device, 144 ... Display device.

Claims (6)

The first terminal device installed in the first space and the second terminal device installed in the second space communicate with each other, so that sounds or images recorded in each of the first space and the second space are recorded. Is a communication system for reproducing in the other space,
A first operation for generating a reproduction signal for preferentially reproducing the sound or image recorded in the first recording area in the first space in the second space; and the first recording area in the first space. And a reproduction control unit for executing a second operation for generating a reproduction signal for preferentially reproducing the sound or the image recorded in the second recording range in the second space. The first terminal device includes a first recording signal indicating sound or image within the first recording range, and a second recording signal indicating sound or image within the second recording range, within the first space. It has a signal acquisition unit that acquires from the recording device,
The reproduction control unit selects the first recording signal to generate the reproduction signal in the first operation, and generates the reproduction signal by selecting the second recording signal in the second operation. Item 4. The communication system according to Item 1. The first terminal device transmits a first acoustic signal indicating sound within the first recording range and a second acoustic signal indicating sound within the second recording range from a sound collecting device within the first space. It has a sound acquisition unit to acquire,
In the first operation, the reproduction control unit mixes the first acoustic signal and the second acoustic signal so that the gain of the first acoustic signal exceeds the gain of the second acoustic signal. In the second operation, the reproduction signal is generated by mixing the first acoustic signal and the second acoustic signal so that the gain of the second acoustic signal exceeds the gain of the first acoustic signal. The communication system according to claim 1. The first terminal device transmits a first acoustic signal indicating sound within the first recording range and a second acoustic signal indicating sound within the second recording range from a sound collecting device within the first space. It has a sound acquisition unit to acquire,
In the first operation, the reproduction control unit is configured so that, when the intensity of the second acoustic signal exceeds a threshold value, the gain of the first acoustic signal exceeds the gain of the second acoustic signal. And the second acoustic signal is mixed to generate the reproduction signal, and when the intensity of the second acoustic signal is lower than the threshold, the first acoustic signal is selected to generate the reproduction signal. 1 communication system. The first terminal device acquires a first image signal indicating an image in the first recording range and a second image signal indicating an image in the second recording range from the imaging device in the first space. An image acquisition unit
In the first operation, the reproduction control unit sets the first image signal and the second image signal so that an image display size of the second image signal is smaller than an image display size of the first image signal. The reproduction signal is generated by combining the first image signal and the second image signal so that the display size of the image of the first image signal is smaller than the display size of the image of the second image signal in the second operation. The communication system according to claim 1, wherein the reproduction signal is generated by combining an image signal. A terminal device installed in the first space and capable of communicating with other terminals installed in the second space,
A first operation for generating a reproduction signal for preferentially reproducing the sound or image recorded in the first recording area in the first space in the second space; and the first recording area in the first space. A playback control unit that executes a second operation for generating a playback signal for preferentially playing back sound or images recorded in the second recording range in the second recording range.

Images