JP5338350B2 - Information processing apparatus and voice correction program - Google Patents

Description

  The present invention relates to an information processing apparatus and a sound correction program.

  2. Description of the Related Art An information processing apparatus is known in which a space of the same quality is provided for both communication parties and a shadow of a communication partner is generated for each space.

  For example, the information processing apparatus disclosed in Patent Document 1 is an apparatus that assists communication between a user and a communication partner by transmitting and receiving voice and video of the user to and from a communication partner via the Internet and the like. Each is provided with a homogeneous space at two points, and is connected to a camera for photographing a user and a projection device for projecting a shadow of a communication partner to the space. Thus, the user is different from the explicit information such as the voice of the communication partner or the video of the communication partner shot by the movement of the shadow of the communication partner projected from the projection device in addition to the explicit information by the voice or video. Implicit information can be obtained.

JP 2006-157959 A

  An object of the present invention is to provide an information processing apparatus and a sound correction program capable of transmitting a user's emotion to a communication partner and suppressing inappropriate sound transmission.

[1] and the emotion extraction unit configured to extract each emotion of the plurality of users based on the plurality of video signals obtained by photographing the user's facial expressions and / or audio signal obtained from the voice of the plurality of users ,
Storage means for storing a plurality of emotional expression images expressing different emotions,
And emotional expression image selecting means for respectively selecting the emotion image from the plurality of emotional expressions image based on feelings of the emotion extraction unit extracts the plurality of users,
Voice correction means for correcting the voice signal so as to suppress transmission of the emotion based on the emotion extracted by the emotion extraction means;
Information processing comprising: video processing means for generating and processing video information displaying the selected emotion expression image on a display image of the video signal and displaying a code indicating each user corresponding to the emotion expression image apparatus.

[2] The above [1], further comprising: an audio processing unit that generates audio information from the audio signal corrected by the audio correcting unit; and a communication unit that transmits the video information and the audio information to an external information processing apparatus. ]. The information processing apparatus according to claim.

[3] The information processing apparatus according to [1], further including an audio processing unit that reproduces audio from the audio signal corrected by the audio correcting unit, and the video processing unit reproduces the video information.

[4] The information processing apparatus according to [1], wherein the emotion extraction unit further acquires the user's biometric information and extracts the user's emotion based on the biometric information.

[5] A computer capable of referring to a plurality of emotion expression images expressing different emotions,
An emotion extraction function for respectively extracting emotions of the plurality of users based on video signals obtained by photographing the facial expressions of a plurality of users and / or audio signals obtained from the voices of the users;
An emotion expression image selection function for selecting each of the emotion expression images based on the extracted emotions of the plurality of users ;
A voice correction function for correcting the voice signal based on the extracted emotion to suppress transmission of the emotion;
In order to display the selected emotion expression image on the display image of the video signal , and to execute a video processing function for generating and processing video information displaying a code indicating each user corresponding to the emotion expression image Voice correction program.

  According to the invention which concerns on Claim 1 or 5, while showing a user's emotion with an emotion expression image, an inappropriate audio | voice can be correct | amended.

  According to the invention which concerns on Claim 2, while transmitting a user's emotion to a communication other party, transmission of an improper voice can be suppressed.

  According to the invention which concerns on Claim 3, a user's emotion can be transmitted to a user.

  According to the invention which concerns on Claim 4, a user's emotion can be extracted more correctly.

It is the schematic which shows the structural example of the electronic conference system which concerns on the 1st Embodiment of this invention. It is the schematic which shows the structural example of the information processing apparatus which concerns on the 1st Embodiment of this invention. (A)-(f) is the schematic which shows the operation example of the audio | voice correction program which concerns on the 1st Embodiment of this invention. (A)-(d) is the schematic which shows the operation example of the audio | voice correction program which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the operation example of the audio | voice correction program which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the operation example of the audio | voice correction program which concerns on the 2nd Embodiment of this invention.

[First Embodiment]
(Configuration of electronic conference system)
FIG. 1 is a schematic diagram showing a configuration example of an electronic conference system according to the first embodiment of the present invention.

  The electronic conference system 1 is configured by connecting information processing apparatuses 2 </ b> A and 2 </ b> B installed at a place A and a place B, respectively, via the Internet 10 so that they can communicate with each other. Since the configurations of the devices installed at the points A and B are the same, only the configuration of the location A will be described below for the sake of omission.

  The information processing device 2A is a server device that communicates with the information processing device 2B via the Internet 10 and processes information transmitted and received by communication. Display unit 3A for displaying video, camera 4A for photographing the situation at place A, speaker 5A for outputting sound collected by microphone 6B and information processing apparatus 2B, microphone 6A for collecting sound at place A Connected to.

  FIG. 2 is a schematic diagram illustrating a configuration example of the information processing apparatus according to the first embodiment of the present invention.

  The information processing apparatus 2A is configured by a CPU (Central Processing Unit) or the like, and controls each unit and executes various programs, and a storage device such as an HDD (Hard Disc Drive) and the like, and stores information. 21A and a communication unit 22A that communicates with the outside. The information processing apparatus 2A includes a display unit 3A including a display device such as an LCD (Liquid Crystal Display), an optical system such as a sensor such as a CMOS (Complementary Metal Oxide Semiconductor), a CCD (Charge Coupled Device), and a lens. 4A, a speaker 5A that outputs sound, a microphone 6A that collects sound, an infrared camera that measures the body temperature of users A1 and A2, and a band type that acquires biological information such as blood pressure and body temperature The sensor is connected to a sensor 7A which is a vibration sensor, a pressure sensor or the like installed on a chair, a desk, a floor or the like on which the users A1 and A2 sit.

  The control unit 20A operates an emotion extraction unit 200, an emotion expression image selection unit 201, an audio correction unit 202, a video processing unit 203, and an audio processing unit 204 by executing a later-described audio correction program 210. Let

  The emotion extraction unit 200 is a unit that extracts the emotions of the users A1 and A2 based on information obtained from the camera 4A, the microphone 6A, and the sensor 7A. The emotion expression image selection unit 201 is a unit that selects an emotion expression image used for expressing an emotion in accordance with the emotion extracted by the emotion extraction unit 200. The voice correction unit 202 is a unit that corrects the voice signal according to the emotion extracted by the emotion extraction unit 200 or the characteristics of the voice collected by the microphone 6A. The video processing unit 203 processes the signal captured by the camera 4A, displays the image selected by the emotion expression image selection unit 201, and displays the video information received via the communication unit 22A as a video. It is a means to display. The audio processing unit 204 processes the signal collected by the microphone 6A or the signal corrected by the audio correction unit 202, converts the audio information received via the communication unit 22A into an audio signal, and outputs it to the speaker 5A. It is a means for displaying.

  The storage unit 21A stores a sound correction program 210 that causes the control unit 20A to operate as the above-described units, and an emotion expression image 211 that stores a plurality of images that express emotions extracted by the emotion extraction unit 200. The emotion expression image 211 uses an icon indicating an expression, an avatar indicating an expression or a whole body, a photograph, a character, or the like.

(Operation)
Hereinafter, the operation of the electronic conference system according to an embodiment of the present invention will be described with reference to the drawings.

  The users A1 and A2 advance the conference while viewing the videos of the users B1 to B3 displayed on the display unit 3A and the voices of the users B1 to B3 output from the speaker 5A. The video displayed on the display unit 3A and the sound output from the speaker 5A are reproduced based on information received by the information processing apparatus 2A from the information processing apparatus 2B via the Internet 10. At the same time, the information processing apparatus 2A processes the video signals of the users A1 and A2 photographed by the camera 4A and the voice signals of the voices of the users A2 and A2 collected by the microphone 6A to obtain video information and audio. Information is transmitted to the information processing apparatus 2B via the Internet 10.

  The emotion extraction unit 200 of the information processing apparatus 2A performs the above operation, extracts emotions of the users A1 and A2 based on information obtained from the camera 4A, the microphone 6A, and the sensor 7A, and an emotion expression image selection unit 201. The voice correction unit 202 executes an operation described below according to the extracted emotion.

  The emotion extraction means 200 analyzes the video information obtained from the camera 4A and extracts, for example, emotions such as “anger”, “fun”, and “sad” from the facial expression. Also, emotions such as “indifference” when there is no change in facial expression, “boring” when yawning, and “sleepy” when depressed for a long time are extracted.

  The emotion extraction means 200 analyzes the voice information obtained from the microphone 6A. For example, the emotion extraction means 200 is “excited” when the inflection of the voice waveform is large, “tense” when the key is high, or the voice pitch is fast. Extract emotions such as

  The emotion extraction means 200 analyzes the information obtained from the sensor 7A and extracts emotions such as “anger”, “excited”, and “tensed” when the blood pressure and body temperature deviate from normal values. To do.

  FIGS. 3A to 3F are schematic diagrams illustrating an operation example of the audio correction program according to the first embodiment of the present invention.

  For example, as shown in FIG. 3A, the emotion extraction unit 200 determines that the voice is rough due to “anger” when a voice having a large waveform amplitude is input, and the voice is shown in FIG. 3B. As described above, the sound correction unit 202 is made to smooth the sound having a volume greater than or equal to the predetermined amplitude A1, so that an unpleasant volume is not given to the other party.

  Further, for example, as shown in FIG. 3 (c), the emotion extraction means 200 determines that the voice is “tense” and becomes a quick mouth when a voice having a fast waveform pitch is input, and FIG. 3 (d) shows. As shown, the voice correction means 202 is changed to a voice interval T2 having a pitch equal to or smaller than a predetermined interval T1, so that tension is not transmitted to the other party.

  Further, for example, as shown in FIG. 3 (e), the emotion extraction means 200 determines that the voice is low because the voice is small because the waveform has a small amplitude. As shown in (f), a voice having a volume equal to or lower than a predetermined amplitude A2 is expanded to the amplitude A2 to the voice correction unit 202 so as not to convey a lack of energy.

  4A to 4D are schematic diagrams illustrating an operation example of the audio correction program according to the first embodiment of the present invention.

  When the emotion extraction means 200 of the information processing device 2A does not extract emotions, the video information photographed by the camera 4A and processed by the video processing means 203 is transmitted to the information processing device 2B, as shown in FIG. As shown, the display video 30B is displayed on the display unit 3B.

  When the emotion extraction means 200 of the information processing apparatus 2A extracts the emotion, for example, as shown in FIG. 4B, the emotion expression image 300a that is an icon having a facial expression based on the emotion extracted on the display video 30B, for example. And 300b, and reference numerals 301a and 301b are displayed so as to indicate the users A1 and A2 corresponding to the emotion expression images 300a and 300b.

  Further, when the emotion extraction means 200 of the information processing apparatus 2A extracts the emotion, for example, as shown in FIG. 4C, the emotion extraction means 200 displays a real video 31B and, based on the extracted emotion, an emotion that is a whole body avatar. The emotion expression video 32B may be displayed using the expression images 320a and 320b.

  Further, when the emotion extraction unit 200 of the information processing apparatus 2A extracts the emotion, for example, as shown in FIG. 4D, the emotion extraction unit 200 may superimpose the emotion on the face of the user A2 based on the emotion extracted on the display video 30B. An emotion expression image 330a which is an icon having a facial expression may be displayed.

  FIG. 5 is a flowchart showing an operation example of the sound correction program according to the first embodiment of the present invention.

  First, the emotion extraction means 200 extracts the emotions of the users A1 and A2 using the camera 4A, the microphone 6A, and the sensor 7A (S10). Next, the emotion expression image selection means 201 selects an emotion expression image as shown in FIGS. 4B to 4D based on the extracted emotion (S11), and superimposes it on the display video (S12). .

  Next, the sound correction unit 202 corresponds to, for example, the examples shown in FIGS. 3A, 3C, and 3E, and when it is determined that correction is necessary (S13; Yes), FIG. The voice is corrected as shown in (d) and (f) (S14). When correction is not necessary (S13; No), the process proceeds to the next step S15.

  Next, the video processing unit 203 and the audio processing unit 204 send the video information superimposed in step S12 and the audio information corrected or not corrected in step S14 to the information processing apparatus 2B via the communication unit 22A and the Internet 10. Transmit (S15).

[Second Embodiment]
In the second embodiment, the emotion of the user is manipulated by feeding back the emotion expression image and the corrected audio information transmitted to the communication partner in the first embodiment to the user.

  FIG. 6 is a flowchart showing an operation example of the sound correction program according to the second embodiment of the present invention.

  First, the emotion extraction means 200 extracts the emotions of the users A1 and A2 using the camera 4A, the microphone 6A, and the sensor 7A (S20). Next, the emotion expression image selection means 201 selects an emotion expression image as shown in FIGS. 4B to 4D based on the extracted emotion (S21), and superimposes it on the display video (S22). .

  Next, the sound correction unit 202 corresponds to, for example, the example shown in FIGS. 3A, 3C, and 3E, and when it is determined that correction is necessary (S23; Yes), FIG. The voice is corrected as shown in (d) and (f) (S24). When correction is not necessary (S23; No), the process proceeds to the next step S25.

  Next, the video processing unit 203 and the audio processing unit 204 output the video information superimposed in step S22 and the audio information corrected or not corrected in step S24 from the display unit 3A and the speaker 5A to output the user A1. And the feeling of confidence is fed back to A2 (S25).

  The feedback is not limited to the correction of the emotional expression image and sound, but the illuminance may be adjusted by connecting the information processing device 2A and the illumination installed at the place A, or the air temperature is connected to adjust the temperature. Also good. Moreover, you may be connected to the apparatus which ejects the fragrance | flavor which suppresses a user's emotion.

[Other embodiments]
The present invention is not limited to the above embodiment, and various modifications can be made without departing from the spirit of the present invention. For example, the emotion expression image is not limited to an image but may be a moving image.

  Moreover, the emotion extraction means 200, the emotion expression image selection means 201, the sound correction means 202, the video processing means 203, and the sound processing means 204 used in the above embodiment are a storage medium such as a CD-ROM. Or may be downloaded to a storage unit in the apparatus from a server apparatus connected to a network such as the Internet. Moreover, you may implement | achieve part or all of the means used by the said embodiment by hardware, such as ASIC.

DESCRIPTION OF SYMBOLS 1 ... Electronic conference system, 2A ... Information processing apparatus, 2B ... Information processing apparatus, 3A ... Display part, 3B ... Display part, 4A ... Camera, 4B ... Camera, 5A ... Speaker, 5B ... Speaker, 6A ... Microphone, 6B ... Microphone, 7A ... sensor, 7B ... sensor, 10 ... Internet, 20A ... control unit, 21A ... storage unit, 22A ... communication unit, 30B ... display video, 31B ... real video, 32B ... emotion expression video, 200 ... emotion extraction means 201 ... emotion expression image selection means, 202 ... sound correction means, 203 ... video processing means, 204 ... sound processing means, 210 ... sound correction program, 211 ... emotion expression image, 300a ... emotion expression image, 301a ... code, 320a ... emotion expression image, 330a ... emotion expression image

Claims (5)

Emotion extraction unit configured to extract a plurality of video signals obtained by photographing the user's facial expressions and / or the plurality of the plurality based on the audio signal obtained from the user voice user emotions, respectively,
Storage means for storing a plurality of emotional expression images expressing different emotions,
And emotional expression image selecting means for respectively selecting the emotion image from the plurality of emotional expressions image based on feelings of the emotion extraction unit extracts the plurality of users,
Voice correction means for correcting the voice signal so as to suppress transmission of the emotion based on the emotion extracted by the emotion extraction means;
Information processing comprising: video processing means for generating and processing video information displaying the selected emotion expression image on a display image of the video signal and displaying a code indicating each user corresponding to the emotion expression image apparatus. Voice processing means for generating voice information from the voice signal corrected by the voice correction means;
The information processing apparatus according to claim 1, further comprising a communication unit that transmits the video information and the audio information to an external information processing apparatus. Voice processing means for reproducing the voice from the voice signal corrected by the voice correction means;
The information processing apparatus according to claim 1, wherein the video processing unit reproduces the video information.   The information processing apparatus according to claim 1, wherein the emotion extraction unit further acquires the user's biological information and extracts the user's emotion based on the biological information. A computer that can refer to multiple emotional expressions that express different emotions,
An emotion extraction function for respectively extracting emotions of the plurality of users based on video signals obtained by photographing the facial expressions of a plurality of users and / or audio signals obtained from the voices of the users;
An emotion expression image selection function for selecting each of the emotion expression images based on the extracted emotions of the plurality of users ;
A voice correction function for correcting the voice signal based on the extracted emotion to suppress transmission of the emotion;
In order to display the selected emotion expression image on the display image of the video signal , and to execute a video processing function for generating and processing video information displaying a code indicating each user corresponding to the emotion expression image Voice correction program.

Images