JP2010041485A - Video/voice output device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a voice localization technology without causing an uncomfortable feeling even when a scene change occurs while the same person is speaking. <P>SOLUTION: This video/voice output device 1 includes: a video analysis part 11 analyzing video to identify the position of a speaker, and detecting presence of a scene change to determine whether the identified speaker is the same person before and after the scene change; a speaker voice localization parameter setting part 12 setting the value of a speaker voice localization parameter to localize the voice at the position of the identified speaker; a speaker voice localization parameter adjustment part 14 adjusting the value of the speaker voice localization parameter to reduce the change of the localized position with respect to the value of the speaker voice localization parameter set by the speaker voice localization parameter setting part 12 when the identified speaker is determined to be the same person before and after the scene change; a localization process part 15 executing a localization change process of the voice in accordance with the adjusted value of the speaker voice localization parameter; and a voice output part 17 outputting the voice changed in localization. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a video / audio output apparatus that outputs content data including video and audio, and more particularly to a video / audio output apparatus that determines sound localization according to a speaker position of video and performs audio output control.

  When a program content such as a television broadcast is received, an image is displayed on a display and a sound is output from a speaker, in monaural sound, a human voice can be heard from the position of the speaker. In stereo / surround sound, in many cases, a human voice is localized at the center of the screen so that the human voice can be heard from the center of the screen.

  However, since it is generally known that the presence of a person's voice is localized at the speaker position on the display, it is known that the sense of presence increases. An audio localization technique for localizing audio is disclosed.

  For example, in Patent Document 1, the position of a speaker is detected, and the volume of sound output from a plurality of speakers is controlled according to the detected position. Moreover, in patent document 2, the position of a speaker is specified, an effect and volume adjustment are performed according to the specified position, and audio | speech data is output from the optimal speaker.

JP-A-11-313272 JP 2007-110582 A

  However, in the above-described prior art, since the sound is localized at the speaker position without considering the contents of the scene, depending on the scene, there is a case where stress is felt instead of enhancing the sense of reality. For example, in a scene where the camera angle changes during the dialogue and the speaker position of the same speaker changes suddenly, the localization position of the voice is changed during the dialogue of the same person. The viewer who is doing this has a problem of feeling stress on the contrary.

  In this way, in the prior art, since the voice is uniformly localized at the speaker position without considering the contents of the scene, a scene change occurs during the speech of the same person, and the speaker position suddenly changes. In a scene that has changed to, there is a problem that rather than enhancing the sense of presence, a sense of incongruity occurs.

  The present invention has been made in view of the above circumstances, and as an example of the problem, in the speech localization technology that specifies the speaker position and localizes the voice to the specified speaker position, the same person is speaking It is an object of the present invention to provide a video / audio output device that does not give a sense of incongruity even when a scene change occurs and the speaker position changes suddenly.

  In order to achieve the above object, a video / audio output device according to claim 1 is a video / audio output device that controls audio localization based on an audio localization parameter, and identifies a speaker position by analyzing video. Speaker localization specifying means, voice localization parameter setting means for setting the value of the voice localization parameter so as to localize the voice to the position of the speaker specified by the speaker position specifying means, and analyzing the video, Same as scene change detection means for detecting the presence or absence of a scene change, and analyzing video or audio to determine whether or not the speaker specified by the speaker position specifying means is the same person before and after the scene change The speaker determination means and the scene change detection means detect that there is a scene change, and the speaker specified by the speaker position specification means by the same speaker determination means If it is determined that the same person is present before and after the change, the value of the voice localization parameter is set so that the change of the localization position is smaller than the value of the voice localization parameter set by the voice localization parameter setting means. A sound localization parameter adjusting unit for adjusting the sound localization parameter, and a sound localization parameter adjusting unit for performing sound localization change processing according to the value of the sound localization parameter adjusted, and for outputting a video and a sound. It is characterized by.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a schematic configuration diagram of a video / audio output apparatus 1 according to an embodiment of the present invention. The video / audio output device 1 is a device that outputs a voice with a sound localization in accordance with a speaker position while taking into account a scene change during the utterance of the same speaker. A parameter setting unit 12, a speaker voice localization parameter adjustment unit 14, a localization processing unit 15, a video display unit 16, and a voice output unit 17 are provided.

  Here, the video / audio output device 1 may be any device as long as it has a function of reproducing content data including video and audio input from the outside and outputting the content data to the outside. A television (TV), a DVD player and recorder, a BD player and recorder, a personal computer (PC), and the like are assumed. “Speaker” refers to the person speaking in the video data (on the screen), and “speaker position” refers to the position of the speaker on the screen. A position near the face (especially the mouth). In addition, “output the voice with the sound localization according to the speaker position” means that, for example, when the speaker exists on the left side of the screen, the volume of the sound output from the speaker provided on the left side of the screen is increased. This means that the sound is output so that the sound can be heard from the position of the speaker.

  The video analysis unit 11 outputs the input video data to the video display unit 16 (to synchronize with the audio data, the video data is delayed and output to the video display unit 16 as necessary) and the input video data The speaker position is specified from the above. The method for specifying the speaker position is performed using a known technique. For example, a speaker may be specified by detecting a region of a human face from video data and detecting a mouth movement in the face. At this time, in detecting the movement of the mouth, using the video data of several frames before and after, the difference such as the brightness of the mouth area is calculated as a feature amount, and the person with the mouth region having the largest feature value is calculated. Is determined as a speaker, the speaker can be specified even when a plurality of faces are detected.

  The video analysis unit 11 detects the presence / absence of a scene change from the input video data. About the detection method of the presence or absence of a scene change, it is performed using a well-known technique. For example, the luminance difference Vd between the corresponding frame and the previous frame is calculated for each pixel, the number of pixels Vdcnt where the difference Vd is equal to or greater than a threshold is counted, and the number of pixels Vdcnt is a predetermined ratio with respect to the total number of pixels If the value exceeds the threshold, a method of determining that there is a scene change may be used.

  Further, when it is determined that there is a scene change, the video analysis unit 11 determines whether or not the specified speakers are the same person. Specifically, when the facial feature amount of the speaker specified in the current scene is calculated by the video analysis unit 11 and the facial feature amount is equal to the speaker facial feature amount specified in the previous scene (before the scene change). The determined speaker is determined to be the same person. Note that the speaker's facial feature amount in the previous scene is stored in a temporary storage area as will be described later. Here, the face feature amount is a value calculated based on, for example, the shape and positional relationship of the facial organ, and a known technique is used as a method for calculating the face feature amount.

  In addition, the video analysis unit 11 outputs the specified speaker position to the speaker voice localization parameter setting unit 12, and the determination information regarding the presence / absence of a scene change and the determination information regarding whether or not the speaker is the same speaker are output. The sound is output to the sound localization parameter adjustment unit 14.

  The speaker voice localization parameter setting unit 12 sets a parameter value (hereinafter referred to as a speaker voice localization parameter) for localizing the voice data at the speaker position input from the video analysis unit 11. . Here, the “parameter value for localizing the voice data at the speaker position” means a parameter value for outputting a sound so that sound can be heard from the speaker position. For example, a plurality of speakers are provided. Among them, it means a parameter value related to volume adjustment (volume setting value for each of a plurality of speakers) that increases the volume of a speaker installed near the speaker position and decreases the volume of other speakers.

  The speaker voice localization parameter setting unit 12 outputs the set speaker voice localization parameter value to the speaker voice localization parameter adjustment unit 14.

  The speaker voice localization parameter adjustment unit 14 inputs the value of the speaker voice localization parameter set by the speaker voice localization parameter setting unit 12, and also includes determination information regarding the presence / absence of a scene change output from the video analysis unit 11, and The determination information as to whether or not they are the same speaker is input, and the value of the set speaker voice localization parameter is adjusted. Specifically, if there is a scene change and the specified speaker is the same person before and after the scene change, the speaker is set to reduce the amount of sound localization change that causes the sound to be localized at the current speaker position. The value of the sound localization parameter is adjusted (corrected).

  Here, “adjusting the value of the speaker's voice localization parameter so as to reduce the voice localization change amount that causes the voice to be localized at the current speaker position” means, for example, that there is a scene change and the same speaker is on the left side of the screen For example, when a scene change is not considered at all, a speaker sound localization parameter value P1 set so that the volume of the right speaker is output at the size of A1 is displayed on the right side. Is adjusted to the value P2 of the speaker voice localization parameter set so as to be output at the magnitude of A2 (<A1). That is, if there is a scene change and the same speaker moves from the left to the right, and the same speaker is speaking continuously before and after the scene, the value of the speaker voice localization parameter is set to the speaker position. The value of the speaker voice localization parameter is adjusted so as to change gently, for example, by moving the voice to the center position of the screen or the like without changing it extremely. As a result, even if a scene change occurs while the same speaker is speaking and the speaker position is changed, the viewer does not feel uncomfortable. Note that “decreasing the audio localization change amount” may be configured such that the audio localization change amount is reduced with respect to input audio data (usually audio data that is often localized at the center position of the screen). Alternatively, the voice localization change amount may be reduced with respect to the value of the speaker voice localization parameter set immediately before.

  In the above description of the speaker audio localization parameter adjustment, the case where the position of the same speaker is changed before and after the scene change has been described as an example. However, the same speaker before and after the scene change is described. The position may not be changed. In this case, since the value of the speaker voice localization parameter does not change before and after the scene change, the voice localization change amount does not occur. Therefore, the adjustment of the speaker voice localization parameter described above may include a case where the speaker position of the same speaker is not changed. In this case, since the voice localization change amount is 0, substantially The speaker voice localization parameters are not adjusted.

  The speaker voice localization parameter adjusting unit 14 outputs the adjusted speaker voice localization parameter value to the localization processing unit 15.

  The localization processing unit 15 inputs voice data, inputs the value of the speaker voice localization parameter output from the speaker voice localization parameter adjustment unit 14, and based on the adjusted value of the speaker voice localization parameter, Audio data localization change processing is performed. In addition, the localization processing unit 15 outputs the audio data subjected to the localization change process to the audio output unit 17.

  The video display unit 16 outputs the video data output from the video analysis unit 11 to be displayed on a display or the like.

  The audio output unit 17 outputs audio data that has been subjected to the localization change process to a speaker.

Yes.

  Next, referring to FIG. 2, the function of the speaker voice localization parameter adjusting unit 14, that is, the speaker voice localization parameter when the same speaker moves when a scene change occurs while the same speaker is speaking. The adjustment will be specifically described.

  Note that the specific example shown in FIG. 2 will be described using a coordinate system as shown in FIG. That is, in an image size of 1440 × 1080, the coordinate system is configured in pixel units with the upper left corner of the screen as the origin, the horizontal direction as the X axis, and the vertical direction as the Y axis. Here, the position of the speaker SP specified on the screen is the position of the face, and in this embodiment, the coordinates of the four corners of the rectangular face area F are the positions of the speaker SP. Specifically, the upper left vertex S0 (X0, Y0), the upper right vertex S1 (X1, Y1), the lower left vertex S2 (X2, Y2), and the lower right vertex S3 (X3, Y3) of the face area F Thus, the position of the speaker SP is specified.

  In the specific example shown in FIG. 2, the speaker voice localization parameter described above is described as the speaker voice localization position P (Px, Py), and the voice is adjusted so that the voice can be heard from the speaker voice localization position P. Output. In the specific example shown in FIG. 2, in the normal state, the speaker voice localization position P is set to the center position of the face area F of the specified speaker, and a scene change occurs during the same speaker's utterance. When the speaker position is moved, the speaker voice localization position P is set to the center position of the screen.

  FIG. 2A shows the speaker position of the scene 1 before the scene change, that is, the speaker position when the speaker A is present at the left position on the screen. Specifically, as shown in FIG. 2A, the face area F of the speaker A is S0 (200, 220), S1 (580, 220), S2 (200, 600), S3 (580, 600). Therefore, the speaker voice localization position P is P1 (390, 410) which is the center of the face area F.

  On the other hand, FIG. 2B shows the speaker position of the scene B after the scene change, that is, the speaker position when the speaker A moves from the left side to the right side on the screen and exists on the right side. Specifically, as shown in FIG. 2 (b), the face area F of the speaker A is S0 (860, 220), S1 (1240, 220), S2 (860, 600), S3 (1240, 600). Therefore, the center of the face area F is P2 (1050, 410), but the speaker voice localization position P is P3 (720, 540) which is the center position of the screen.

  As described above, when a scene change occurs while the speaker A is speaking and the speaker A moves before and after the scene change, the speaker voice is localized at the center position of the screen, and the viewer feels uncomfortable. Is not caused. If the scene change is not taken into consideration, the speaker voice follows the speaker position and is localized at the speaker position, so the speaker voice localization position P is P2 (1050, 410).

  That is, when the speaker voice localization position P is determined in consideration of the scene change, the speaker voice localization position P is changed from P1 (390, 410) to P3 (720, 540), but the scene change is considered. Without determining the speaker voice localization position P, the speaker voice localization position P is changed from P1 (390, 410) to P2 (1050, 410). Here, the position change of P1 (390,410) → P3 (720,540) is smaller than the position change of P1 (390,410) → P2 (1050,410). Specifically shows the above-mentioned "Adjust the value of the speaker voice localization parameter so as to reduce the voice localization change amount".

  Next, with reference to FIG. 4, the video / audio output processing of the video / audio output device 1 of the present embodiment will be described. FIG. 4 is a flowchart showing the flow of the video / audio output processing for performing the audio localization control in consideration of the scene change during the utterance of the same speaker of the video / audio output device 1.

  First, the video analysis unit 11 of the video / audio output device 1 analyzes the input video data, and specifies the speaker position of the video data (step S10).

  Next, the speaker voice localization parameter setting unit 12 of the video / audio output device 1 sets the value of the speaker voice localization parameter based on the specified speaker position (step S20).

  Next, the video analysis unit 11 of the video / audio output device 1 performs a scene change detection process (step S30). In the scene change detection process, the input video data is analyzed, a scene change is detected, and the presence or absence of a scene change is determined.

  Next, when the video analysis unit 11 of the video / audio output device 1 determines that there is a scene change, the same story that determines whether or not the specified speakers are the same person before and after the scene change. A person determination process is performed (step S40).

  Here, the same speaker determination process will be described with reference to FIG. FIG. 5 is a flowchart showing in detail the flow of the same speaker determination process in step S40 of FIG.

  The video analysis unit 11 of the video / audio output device 1 extracts the facial feature amount of the speaker specified in the current scene (the scene after the scene change) (step S41), and specifies the previous scene (the scene before the scene change). It is compared with the face feature amount of the speaker who has been made (step S42).

  Next, the video analysis unit 11 of the video / audio output device 1 determines whether or not the facial feature amount of the speaker specified in the current scene is equal to the facial feature amount of the speaker specified in the previous scene (step S1). S43) If equal (step S43: YES), it is determined that there is no change of the speaker, that is, the same speaker (step S44), and if not equal (step S43: NO), It is determined that there is a change, that is, they are not the same speaker (step S45).

  Finally, the video analysis unit 11 of the video / audio output device 1 stores the facial feature amount of the speaker of the current scene in the temporary storage area (step S46).

  Returning to FIG. 4, the speaker audio localization parameter adjustment unit 14 of the video / audio output device 1 receives the determination information of the presence / absence of the scene change and the same speaker from the video analysis unit 11. It is then determined whether or not the same speaker is present before and after the scene change (step S60).

  When there is a scene change and the same speaker is present before and after the scene change (step S60: YES), the speaker audio localization parameter adjustment unit 14 of the audio / video output apparatus 1 performs audio to the speaker position. The value of the speaker sound localization parameter is adjusted so that the localization change amount becomes small (step S70).

  Next, the localization processing unit 15 of the video / audio output device 1 changes the audio localization of the audio data in accordance with the value of the set speaker audio localization parameter (step S80). That is, when there is a scene change and the same speaker is present before and after the scene change (step S60: YES), the speaker voice adjusted so that the voice localization change amount to the speaker position is reduced. The voice localization of the voice data is changed with the value of the localization parameter. If not (step S60: NO), the voice localization of the voice data is changed with the value of the speaker voice localization parameter set in step S20.

  Next, the video display unit 16 of the video / audio output apparatus 1 outputs video data, and the audio output unit 17 outputs audio data that has undergone audio localization change.

  In the present embodiment, the video analysis unit 11 analyzes the video data and performs the same speaker determination process, but the method of the same speaker determination process is not limited to this. For example, when the video analysis unit 11 determines that there is a scene change, the audio data may be analyzed to determine whether or not the specified speaker is the same person before and after the scene change.

  FIG. 6 is a schematic configuration diagram of the video / audio output device 2 that performs the same speaker determination processing based on the audio data. The video / audio output device 2 is a device that outputs a voice with a sound localization adapted to the speaker position while taking into account a scene change during the utterance of the same speaker, and more specifically, the video analysis unit 11 and the speaker sound localization. A parameter setting unit 12, a voice analysis unit 13, a speaker voice localization parameter adjustment unit 14, a localization processing unit 15, a video display unit 16, and a voice output unit 17 are provided. In other words, the video / audio output apparatus 2 is different from the video / audio output apparatus 1 in that the audio analysis unit 13 is provided, and the other points are substantially the same as the video / audio output apparatus 1. In the following, only configurations and functions different from those of the above-described embodiment will be described, and with regard to other configurations and functions, the same portions are denoted by the same reference numerals, and description thereof will be omitted.

  The voice analysis unit 13 outputs the input voice data to the localization processing unit 15, and if the video analysis unit 11 determines that there is a scene change, the input voice data is analyzed and the identified speaker is the same It is determined whether or not it is a person. Specifically, the voice analysis unit 13 calculates the voice feature amount of the speaker specified in the current scene (after the scene change), and compares it with the voice feature amount of the speaker specified in the previous scene (before the scene change). When the voice feature amounts are equal, it is determined that the specified speakers are the same person. Note that the voice feature amount of the speaker in the previous scene is stored in a temporary storage area as will be described later. Here, the speech feature amount is, for example, a frequency intensity in a spectrogram analysis of speech, and a known technique is used for a speech feature amount calculation method.

  Further, the voice analysis unit 13 outputs determination information as to whether or not they are the same speaker to the speaker voice localization parameter adjustment unit 14.

  FIG. 7 is a flowchart showing the flow of the same speaker position determination process of the video / audio output device 2. FIG. 7 is a process corresponding to step S40 of FIG.

  The video analysis unit 11 of the video / audio output device 1 extracts the voice feature amount of the speaker specified in the current scene (the scene after the scene change) (step S51), and specifies the previous scene (the scene before the scene change). It is compared with the voice feature amount of the speaker who has been made (step S52).

  Next, the video analysis unit 11 of the video / audio output device 1 determines whether or not the voice feature amount of the speaker specified in the current scene is equal to the voice feature amount of the speaker specified in the previous scene (Step S1). S53) If they are equal (step S53: YES), it is determined that there is no alternation of speakers, that is, they are the same speaker (step S54). If they are not equal (step S53: NO), the voice analysis unit 11 determines that there is a change of speakers, that is, the speakers are not the same (step S55).

  Finally, the voice analysis unit 13 stores the voice feature amount of the speaker of the current scene in the temporary storage area (step 56).

  As described above, according to the video and audio output apparatuses 1 and 2 according to the above-described embodiments, the video analysis unit 11 that analyzes the video and identifies the position of the speaker, and the story that is identified by the video analysis unit 11 A speaker voice localization parameter setting unit 12 for setting the value of the speaker voice localization parameter so as to localize the voice at the position of the person, a video analysis unit 11 for analyzing the video and detecting the presence or absence of a scene change, and a video Alternatively, the video analysis unit 11 or the voice analysis unit 13 that analyzes the voice and determines whether or not the speaker specified by the speaker position specifying unit is the same person before and after the scene change, and the video analysis unit 11 When a scene change is detected by the video analysis unit 11 and the voice analysis unit 13 determines that the speaker specified by the video analysis unit 11 is the same person before and after the scene change. The speaker voice localization parameter for adjusting the value of the speaker voice localization parameter so as to reduce the change of the localization position with respect to the value of the speaker voice localization parameter set by the speaker voice localization parameter setting unit 12. The adjustment unit 14, the localization processing unit 15 that performs the localization process of the speech according to the adjusted speaker voice localization parameter value by the speaker voice localization parameter adjustment unit 14, and the localization that has been localized by the localization processing unit 15 And an audio output unit 17 for outputting, so that even if a scene change occurs during the same speaker's utterance and the speaker position changes suddenly, the viewer does not feel uncomfortable.

  In addition, the video analysis unit 11 calculates the facial feature amount of the speaker specified by the video analysis unit 11 from the video data, and determines whether the calculated facial feature amount is the same before and after the scene change. It may be. In this case, it is possible to easily determine whether or not the specified speakers before and after the scene change are the same person by extracting the facial feature amount from the video data.

  Further, the video analysis unit 11 calculates the voice feature amount of the speaker specified by the video analysis unit 11 from the voice data, and determines whether or not the calculated voice feature amount is the same before and after the scene change. It may be. In this case, it is possible to easily determine whether or not the specified speakers before and after the scene change are the same person by extracting the audio feature amount from the audio data.

  The speaker voice localization parameter adjustment unit 14 may adjust the value of the speaker voice localization parameter so that the voice is localized at a position in the center direction of the display screen. Even if a scene change occurs during the same speaker's utterance and the speaker position suddenly changes, the sound is localized around the screen, so the viewer can comfortably enjoy the content without feeling uncomfortable. Can be watched.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made to the embodiments of the present invention without departing from the gist of the present invention. Such modifications and changes can be made, and those accompanying such modifications and changes are also included in the technical scope of the present invention.

1 is a schematic configuration diagram of a video / audio output device according to an embodiment of the present invention. It is a figure which shows a mode that a speaker position changes in the video data input into the video / audio output device video / audio output device which concerns on embodiment of this invention. It is an example of the video data input into the video / audio output device which concerns on embodiment of this invention. It is a flowchart which shows the flow of the video / audio output process which performs audio localization control in consideration of the scene change in the same speaker utterance of the video / audio output device which concerns on embodiment of this invention. It is a flowchart which shows in detail the flow of the same speaker determination process of FIG.4 S40. It is a schematic block diagram of the video / audio output device which concerns on other embodiment of this invention. It is a flowchart which shows in detail the flow of the same speaker determination process of the video / audio output device which concerns on other embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 2 Video audio output apparatus 11 Video analysis part 12 Speaker audio localization parameter setting part 13 Voice analysis part 14 Speaker audio localization parameter adjustment part 15 Localization processing part 16 Video display part 17 Audio | voice output part

Claims (5)

An audio / video output device that controls audio localization based on audio localization parameters,
A speaker position specifying means for analyzing a video and specifying a speaker position;
Voice localization parameter setting means for setting the value of the voice localization parameter so that the voice is localized at the position of the speaker specified by the speaker position specifying means;
Scene change detection means for analyzing the video and detecting the presence or absence of a scene change;
Analyzing video or audio, the same speaker determining means for determining whether the speaker specified by the speaker position specifying means is the same person before and after the scene change,
The scene change detecting means detects that there is a scene change, and the same speaker determining means determines that the speaker specified by the speaker position specifying means is the same person before and after the scene change. A voice localization parameter adjusting unit that adjusts the value of the voice localization parameter so as to reduce a change in the localization position with respect to the value of the voice localization parameter set by the voice localization parameter setting unit;
Localization change output means for performing audio localization change processing according to the value of the adjusted audio localization parameter by the audio localization parameter adjusting means, and outputting video and audio;
A video / audio output device comprising:   The same speaker determination unit calculates the facial feature amount of the speaker specified by the speaker position specifying unit from the video, and determines whether the calculated facial feature amount is the same before and after the scene change. The video / audio output device according to claim 1.   The same speaker determination unit calculates the voice feature amount of the speaker specified by the speaker position specifying unit from the voice, and determines whether the calculated voice feature amount is the same before and after the scene change. The video / audio output device according to claim 1.   4. The video according to claim 1, wherein the audio localization parameter adjusting unit adjusts the value of the audio localization parameter so that the audio is localized at a position in a center direction of the display screen. 5. Audio output device. The speaker position specifying means detects the position of a human face in the video, specifies the speaker from the detected movement of the mouth of the face, and sets the vicinity of the specified speaker's mouth as the speaker position. The video / audio output device according to claim 1, wherein:

Images