JP2011250100A - Image processing system and method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To visually present environmental sound to a user more effectively.SOLUTION: An analysis part 24 subjects sound data of sound constituting a content to a blind sound source separation processing to extract the sound data of sound of each sound source and generates direction data indicating a direction of the sound source based on the sound data of sound of each sound source. Also, the analysis part 24 determines whether or not the sound of the sound source is an environmental sound which is not a human voice, and converts the environmental sound into a text. A visual information generation part 25 generates an effect image for visually presenting the environmental sound based on the text-converted environmental sound. An image composition part 26 overlays the effect image on a position defined by the direction data on a content image constituting the content. The present invention can be adopted to a video reproduction device.

Description

  The present invention relates to an image processing apparatus, method, and program, and more particularly, to an image processing apparatus, method, and program capable of visually presenting environmental sounds to a user more effectively.

  In commercial package media such as DVD (Digital Versatile Disc) and BD (Blu-ray (registered trademark) Disc), text information and image information can be displayed at a desired position on the content image. There is no particular consideration for ambient environmental sounds that are different from utterances.

  For example, when viewing content, the effect of producing environmental sounds such as door opening / closing sound, car approaching sound, telephone ringing sound, etc. has no meaning for hearing impaired people. In addition, it is difficult to add such an environmental sound effect for content that has been personally shot without the authoring procedure or a program broadcast in a live broadcast format.

  In the US, where the addition of closed captions is obligatory, closed captions may be created by a person listening to and typing content audio. In such a case, the user can read the converted character as visual information, but with closed captioning, it expresses where the environmental sound or sound is coming from or on the screen. I can't.

  As a technique using voice recognition, an apparatus that converts voice-recognized voice into text and displays it on a head-mounted display for persons with disabilities has also been proposed (see, for example, Patent Document 1). This head-mounted display also displays the approximate sound source position of the voiced text, so that the user can know from which direction the sound is coming from.

JP 2007-334149 A

  However, with the above-described technology, it has been impossible to effectively visually present the environmental sound included in the content to the user for any content.

  For example, even when a user wears a head-mounted display that displays surrounding voice as text and views the content, the recognized voice text is simply displayed at the center of the display. Therefore, there is no sense of unity between the content and the displayed text, and it cannot be said that the voice is effectively presented.

  This invention is made | formed in view of such a condition, and enables it to present an environmental sound visually with respect to a user more effectively.

  An image processing apparatus according to an aspect of the present invention includes a sound source direction estimating unit that estimates a direction of a sound source of a sound with respect to a predetermined reference position based on sound data of sound forming content, and the sound data includes the sound data Separating the sound data of the environmental sound excluding the speech sound generated by the person's speech and the sound data of the speech sound, and performing speech recognition processing on the sound data of the environmental sound, The environmental sound identification means for converting the text into the text and the effect image for visually presenting the environmental sound are displayed in the text so as to be displayed at a position determined by the estimation result of the direction of the sound source on the image constituting the content. Image synthesizing means for synthesizing the effect data of the effect image generated based on the environmental sound and the image data of the image

  The image processing apparatus further includes sound source distance estimating means for estimating a distance from the reference position to the sound source based on the sound data of the sound, and the effect image is stored in the image synthesizing means. The effect data and the image data can be combined such that the effect data and the image data are displayed at the position determined by the estimation result of the direction of the sound source in the size determined by the estimation result of the distance of the sound source.

  The image processing apparatus further includes a sound source separation unit that performs a blind sound source separation process on the sound data of the sound and separates the sound data into sound data of the sound of each sound source, and the separation unit includes: For each of the sound data separated by the sound source separation means, by determining whether the sound data is the sound data of the environmental sound, the sound data of the environmental sound and the speech sound The voice data can be separated.

  The image composition means has the size, color, or brightness of the effect image displayed on the image according to at least one of the direction of the sound source, the distance of the sound source, and the volume of the environmental sound. The effect data and the image data can be combined so as to change.

  In the image processing apparatus, text is converted so that supplementary information, which is predetermined for the environmental sound that has been converted into text, and includes a character string that supplements the content of the environmental sound is displayed on the effect image. Visual information generating means for generating the effect data based on the environmental sound can be further provided.

  The image processing apparatus further includes speech recognition means for performing speech recognition processing on the speech data of the utterance sound separated by the separation means to convert the speech sound into text, and the image synthesis means includes The effect data and the image data can be combined so that the effect image of the environmental sound and the effect image that visually presents the speech sound are displayed on the image.

  An image processing method or program according to one aspect of the present invention estimates a direction of a sound source of a sound with respect to a predetermined reference position based on sound data of sound that constitutes content, and uses the sound data of a person of the sound. Separating the voice data of the environmental sound excluding the voice generated by the utterance and the voice data of the voice, performing voice recognition processing on the voice data of the environmental sound, converting the environmental sound into text, The effect image that visually presents the sound is generated based on the environmental sound generated as text so that the effect image is displayed at a position determined by the estimation result of the direction of the sound source on the image constituting the content. A step of synthesizing the effect data of the effect image and the image data of the image;

  In one aspect of the present invention, the direction of the sound source of the sound with respect to a predetermined reference position is estimated based on the sound data of the sound that constitutes the content, and the sound data includes the utterance sound generated by the person's utterance in the sound. The voice data of the environmental sound is separated from the voice data of the utterance sound, and voice recognition processing is performed on the voice data of the environmental sound, the environmental sound is converted into text, and the environmental sound is visually recognized. Of the effect image generated based on the environmental sound that has been converted into text so that the effect image to be presented is displayed at a position determined by the estimation result of the direction of the sound source on the image constituting the content. The effect data and the image data of the image are combined.

  According to one aspect of the present invention, it is possible to visually present environmental sounds to a user more effectively.

It is a figure which shows the structural example of one Embodiment of the image processing apparatus to which this invention is applied. It is a figure which shows the structural example of an analysis part. It is a flowchart explaining a content reproduction process. It is a figure which shows the example of a display of a composite content image. It is a figure which shows the example of a display of a composite content image. It is a flowchart explaining an analysis process. It is a figure which shows the other structural example of an analysis part. It is a flowchart explaining an analysis process. It is a block diagram which shows the structural example of a computer.

  Embodiments to which the present invention is applied will be described below with reference to the drawings.

<First Embodiment>
[Configuration of image processing apparatus]
FIG. 1 is a diagram showing a configuration example of an embodiment of an image processing apparatus to which the present invention is applied.

  The image processing device 11 reads content data of content from a recording medium 12 such as an optical disk mounted on the image processing device 11 and reproduces it. For example, the image processing apparatus 11 includes a television receiver, a video playback device, a personal computer, and the like, and realizes functions to be implemented on a video playback application program.

  In particular, the image processing device 11 converts sound effects used for production into image information without taking time when authoring the content, and overlays the image information on an appropriate position, so that the content of the hearing impaired person's content can be displayed. The purpose is to help viewing.

  The content data is composed of moving image data for displaying a moving image as content and audio data of audio accompanying the moving image (hereinafter also referred to as a content image). These moving image data and audio data Are encoded by a predetermined method.

  The image processing apparatus 11 includes a reading unit 21, a moving image decoder 22, an audio decoder 23, an analysis unit 24, a visual information generation unit 25, an image synthesis unit 26, and a display unit 27.

  The reading unit 21 reads content data from the recording medium 12, supplies moving image data constituting the content data to the moving picture decoder 22, and supplies audio data constituting the content data to the audio decoder 23.

  The moving picture decoder 22 decodes the moving image data supplied from the reading unit 21 and supplies it to the image composition unit 26. The audio decoder 23 decodes the audio data supplied from the reading unit 21 and supplies the decoded audio data to the analysis unit 24 and the display unit 27.

  The analysis unit 24 performs analysis processing on the audio data supplied from the audio decoder 23, and direction data indicating the direction of the sound source of the sound reproduced by the sound data, and distance data indicating the distance to the sound source of the sound Are generated and supplied to the image composition unit 26. Here, the direction and distance of the sound source are the direction and distance with reference to a sound collection unit such as a microphone that picks up the sound.

  In addition to the sound directly picked up by the sound pickup unit, the sound constituting the content may include sound such as sound effects added (synthesized) to the sound later. Sounds such as sound effects are considered to have been collected by the sound collection unit. That is, the direction and distance of the sound source of sound such as sound effects are estimated using the sound collection unit as a reference.

  The analysis unit 24 performs voice recognition processing on the voice data supplied from the audio decoder 23, and supplies a word string indicating the result of the voice recognition processing to the visual information generation unit 25. In other words, the analysis unit 24 converts the voice reproduced by the voice data into text. For example, the word sequence that shows the results of the voice recognition process, "Hello", such as speech contents of the people, not only the word string representing the recognition result of the voice of the people, "folk folk", such as siren sound of an ambulance, A word string indicating the recognition result of the environmental sound emitted around the sound collection unit is also included.

  In the following, in particular, the voice uttered by a person such as a person's utterance or onomatopoeia is also referred to as a utterance sound, and all other surroundings except the utterance sound collected when the sound constituting the content is collected. Is also referred to as environmental sound. Also, hereinafter, text data indicating a word string obtained as a result of speech recognition processing for an uttered sound is also referred to as utterance sound data, and text data indicating a word string obtained as a result of speech recognition processing for environmental sound is referred to as environmental sound data. I will call it. Accordingly, the utterance sound data and the environmental sound data are supplied to the visual information generation unit 25.

  The visual information generation unit 25 uses the utterance sound data and the environmental sound data supplied from the analysis unit 24 to generate effect data for visually presenting a word string indicated by the data, and sends it to the image composition unit 26. Supply.

  For example, the effect data is image data such as text images obtained by converting speech sounds and environmental sounds into text, images such as illustrations and textures predetermined for speech recognition results of speech sounds and environmental sounds. The Hereinafter, an image displayed by effect data is particularly referred to as an effect image.

  It should be noted that an image such as an illustration as an effect image may be a moving image or a still image as long as it recalls the utterance content and the environmental sound content. In addition, an image such as a text image or an illustration as an effect image may have a display effect such that the display format changes with time. In this case, for example, the color and position of characters displayed on the effect image change with time.

  The image synthesis unit 26 uses the direction data and distance data supplied from the analysis unit 24 to synthesize the moving image data supplied from the video decoder 22 and the effect data supplied from the visual information generation unit 25. The obtained moving image data is supplied to the display unit 27. For example, the synthesis of the moving image data and the effect data is performed such that the effect image is overlaid at a position determined by the direction data on the content image with a size determined by the distance data. Hereinafter, in particular, a content image on which an effect image is overlaid is also referred to as a synthesized content image, and moving image data of the synthesized content image is also referred to as synthesized moving image data.

  The display unit 27 includes, for example, a liquid crystal display and a speaker. The display unit 27 displays a synthesized content image based on the synthesized moving image data supplied from the image synthesizing unit 26, and based on the audio data supplied from the audio decoder 23. , Output audio.

[Configuration of analysis unit]
The analysis unit 24 in FIG. 1 is configured as shown in FIG. 2 in more detail.

  That is, the analysis unit 24 includes a sound source separation unit 51, a sound source direction estimation unit 52, a sound source distance estimation unit 53, an environmental sound / utterance sound identification unit 54, an environmental sound identification unit 55, and an utterance content identification unit 56. The sound source separation unit 51 is supplied with audio data from the audio decoder 23.

  The sound source separation unit 51 performs blind sound source separation processing based on independent component analysis on the sound data supplied from the audio decoder 23, extracts sound sound data of one or more sound sources from the sound data, It supplies to the direction estimation part 52 thru | or environmental sound / speech sound discrimination part 54.

  For example, the sound of the sound data constituting the content data is mixed with sound emitted from one or a plurality of sound sources such as a person who speaks or an ambulance that sounds a siren. When the blind sound source separation process is performed on the audio data of the content, the audio data of the audio generated from the sound sources is obtained for each audio source included in the audio of the content. Hereinafter, the sound from the sound source is also referred to as individual sound, and the sound data of the individual sound is also referred to as individual sound data.

  The sound source direction estimation unit 52 performs sound source direction estimation processing on each individual sound data supplied from the sound source separation unit 51 to generate direction data indicating the direction of the sound source that emits the individual sound, and the image composition unit 26. To supply. The sound source distance estimation unit 53 performs sound source distance estimation processing on each individual audio data supplied from the sound source separation unit 51 to generate distance data indicating the distance to the sound source that emits the individual audio, and an image synthesis unit 26.

  The environmental sound / speech sound discriminating unit 54 determines, for each individual voice data supplied from the sound source separation unit 51, whether the individual voice based on the individual voice data is a speech sound or an environmental sound. The output destination of the individual audio data is switched according to the result. That is, the environmental sound / speech sound identification unit 54 supplies the individual sound data of the environmental sound to the environmental sound identification unit 55 and supplies the individual sound data of the utterance sound to the utterance content identification unit 56.

  The environmental sound identification unit 55 performs voice recognition processing on the individual audio data supplied from the environmental sound / speech sound identification unit 54 and supplies environmental sound data indicating the result to the visual information generation unit 25. The utterance content identification unit 56 performs voice recognition processing on the individual voice data supplied from the environmental sound / speech sound identification unit 54, and supplies the utterance sound data indicating the result to the visual information generation unit 25.

[Description of content playback processing]
By the way, when the user attaches the recording medium 12 on which the content is recorded to the image processing apparatus 11 and operates the image processing apparatus 11 to instruct the reproduction of the content, the image processing apparatus 11 receives the content from the recording medium 12. The content reproduction process of reading and reproducing is started.

  Hereinafter, content reproduction processing by the image processing apparatus 11 will be described with reference to the flowchart of FIG.

  In step S <b> 11, the reading unit 21 reads the content data of the content instructed to be played by the user from the recording medium 12. Then, the reading unit 21 supplies the moving image data and audio data of the read content data to the moving picture decoder 22 and the audio decoder 23.

  In step S <b> 12, the moving picture decoder 22 decodes the moving image data supplied from the reading unit 21 and supplies it to the image composition unit 26. In step S 13, the audio decoder 23 decodes the audio data supplied from the reading unit 21 and supplies the decoded audio data to the analysis unit 24 and the display unit 27.

  In step S <b> 14, the analysis unit 24 performs an analysis process, and generates direction data, distance data, environmental sound data, and speech sound data from the sound data supplied from the audio decoder 23. The generated direction data and distance data are supplied to the image composition unit 26, and the environmental sound data and speech sound data are supplied to the visual information generation unit 25. Details of the analysis process will be described later.

  In step S <b> 15, the visual information generation unit 25 generates effect data using the environmental sound data and the speech sound data supplied from the analysis unit 24, and supplies the effect data to the image composition unit 26. This effect data is generated for each individual sound such as a speech sound and an environmental sound.

  For example, the environmental sound data includes a word string obtained as a result of the voice recognition process and supplementary information related to the word string. The supplementary information is information that supplements the contents of the individualized audio that has been converted into text so that the user who views the content can understand in detail (accurately) the phenomenon occurring in the content with respect to the individual audio. is there.

  Specifically, it is assumed that the word string “Peepy Peep” representing the siren sound of an ambulance is obtained as a result of the voice recognition process of the individual voice data of the environmental sound. A predetermined character string “(ambulance siren)” is associated with the word string “Peepy Peep” as supplementary information. From the environmental sound identification unit 55, the word string “Peepy Peep” and the supplementary information “ Environmental sound data consisting of “Ambulance siren” is output.

  When such environmental sound data is supplied, the visual information generation unit 25 displays an image of an effect image that displays, for example, characters of the word string “Peepy Peep” and characters of supplementary information “(ambulance siren)”. Data is generated as effect data. As described above, by generating image data of an effect image that includes a word string obtained by converting individual speech into text and supplementary information that supplements the word string as effect data, a user who views the content is more accurate. It is possible to grasp the contents of the contents.

  In step S <b> 16, the image composition unit 26 synthesizes the moving image data from the moving picture decoder 22 and the effect data from the visual information generation unit 25 using the direction data and distance data from the analysis unit 24. Then, the image composition unit 26 supplies the composite moving image data of the composite content image obtained by the composition to the display unit 27.

  For example, it is assumed that the horn of a passenger car is picked up by the sound pickup unit as individual sound, and the position of the sound source indicated by the direction data of the individual sound is left front as viewed from the sound pickup unit. In this case, the image synthesizing unit 26 displays the effect image of the individual sound (horn) in the upper left part of the content image, that is, in the upper left part as viewed from the user viewing the content image from the front. Combine the image with the content image.

  At this time, the image synthesizing unit 26 adjusts the size of the effect image synthesized with the content image according to the distance indicated by the distance data of the individual sound. Specifically, the synthesis is performed so that the effect image is displayed smaller as the distance from the sound collection unit to the sound source is longer.

  When synthesizing the content image and the effect image, it is assumed that the image capturing unit that captures the content image and the sound collecting unit that collects the sound of the content are located at substantially the same position, and the content image determined by the direction data of the individual sound The effect image of the individual sound is overlaid at the upper position in a size determined by the distance data. That is, the effect image of the individual sound from each sound source is displayed at a position near the sound source on the content image.

  Further, the size of the effect image displayed on the content image may be changed according to the size of the individual sound of the effect image. In such a case, for example, the sound source separation unit 51 supplies information indicating the volume of each individual sound to the image synthesis unit 26, and the image synthesis unit 26 has a high volume based on the information indicating the supplied volume. As the effect image becomes larger, the effect image is synthesized.

  Further, the image synthesis unit 26 may change the color or brightness of the effect image of the individual sound in accordance with the direction and distance of the sound source of the individual sound based on the direction data and distance data of the individual sound. Good.

  Further, for example, when the audio data constituting the content data is multi-channel stereo such as 5.1 channel, the individual audio can be heard from the rear of the user, that is, the sound collecting unit is behind the sound collecting unit. Individual sound sources may be located at the same time. In such a case, the effect image of the individual sound cannot be displayed on the content image.

  Therefore, in such a case, the image composition unit 26 displays the effect image near the end of the content image. Further, in this case, the image composition unit 26 supplements that the user viewing the content is an arrow symbol or a sound from the back so that the user can grasp that the effect image is an individual sound from the back. Display information with effect images. Thereby, the user can know the sound source position of the individual sound indicated by the effect image more reliably and accurately.

  When the image composition unit 26 generates the combined moving image data of the combined content image, the image combining unit 26 supplies the combined moving image data to the display unit 27, and the process proceeds from step S16 to step S17.

  In step S <b> 17, the display unit 27 displays the synthesized content image based on the synthesized moving image data from the image synthesizer 26, and outputs the audio based on the audio data from the audio decoder 23, thereby reproducing the content. To do.

  As a result, the composite content image shown in FIGS. 4 and 5 is displayed on the display unit 27.

  For example, in the example of FIG. 4, an ambulance is displayed on the left side of the composite content image C <b> 11, and the lower side of the ambulance corresponds to the siren sound (environmental sound) emitted from the ambulance as a sound source. The effect image EF11 is displayed. In this effect image EF11, the text “Peepy Peep” representing the ambulance siren in text and the character “(ambulance siren)” as supplementary information of the siren are displayed.

  The displayed effect image EF11 moves together with the ambulance as the sound source. For example, when the ambulance moves to the back side of the screen and is displayed small, and the siren sound is reduced, the effect image EF11 is also displayed on the ambulance position and siren volume. It is displayed small in accordance with the change of.

  Further, in the example of FIG. 5, an explosion has occurred in the approximate center of the composite content image C12, and an effect image EF12 for the explosion sound is displayed on the lower side in the drawing.

  In the effect image EF12, the character “Dokaa Aan” obtained by converting the explosion sound into text is displayed as a texture of a decorative character. For example, the effect image EF12 is displayed so as to decrease with time according to a change in the volume when the volume of the explosion sound as the environmental sound gradually decreases.

  In the image processing apparatus 11, the processing in steps S <b> 11 to S <b> 17 described above is repeatedly performed for each predetermined unit such as a frame of a content image. And the size also changes.

  Returning to the description of the flowchart of FIG. 3, in step S <b> 18, the image processing apparatus 11 determines whether or not to end the reproduction of the content. For example, when the user operates the image processing apparatus 11 and gives an instruction to end content reproduction, it is determined to end.

  If it is determined in step S18 that the reproduction is not terminated, the process returns to step S11, and the above-described process is repeated. That is, the next frame of the content is read and reproduced.

  On the other hand, if it is determined in step S18 that the content reproduction is to be terminated, the image processing apparatus 11 terminates the content reproduction, and the content reproduction process is terminated.

  In this way, the image processing apparatus 11 separates the sound of the content for each sound source, converts each individual sound into text by speech recognition, and includes characters (word strings) and supplementary information obtained by text conversion. Generate an effect image. The image processing apparatus 11 determines the display position, size, color, and the like of the effect image for each individual sound according to the direction and distance of the sound source of the individual sound, and overlays the effect image on the content image.

  Therefore, according to the image processing apparatus 11, the effect image can be displayed in the vicinity of the sound source of each individual sound on the content image. As a result, it is possible to give the effect image a sense of unity with the content image compared to simply displaying closed captions or textual sound and the sound source position, and more effectively providing environmental sounds to the user. Can be presented visually.

  In particular, since the image processing apparatus 11 displays the effect image not only for the utterance sound but also for the environmental sound, the auditory system that can be perceived by a hearing impaired person, which is conventionally limited to only the caption of the utterance sound. Information can be extended to ambient sounds. As a result, the user can read even the intention of the content producer, and can fully enjoy viewing the content.

  In addition, supplementary information is displayed along with textual environmental sounds as necessary, so that the user can more accurately grasp the contents and enjoy viewing the contents. It becomes like this.

  Furthermore, since the image processing apparatus 11 analyzes the audio data of the content and generates an effect image, it can also be applied to a personal video or a live broadcast program that has been originally recorded with a camcorder and has no captions added. The effect image can be displayed during playback.

[Description of analysis processing]
Next, analysis processing corresponding to the processing in step S14 in FIG. 3 will be described with reference to the flowchart in FIG.

  In step S41, the sound source separation unit 51 performs blind sound source separation processing based on independent component analysis on the sound data supplied from the audio decoder 23, and extracts sound data of each individual sound from the sound data.

  For example, it is assumed that audio data constituting the content data is composed of audio data of the R channel and the L channel, that is, the left and right channels. In this case, the sound source separation unit 51 performs Fourier transform on the audio data of the R and L channels, and converts the audio data into frequency information composed of frequency components. This frequency information is information indicating the power of each frequency component of the sound.

  Then, the sound source separation unit 51 divides the entire frequency band indicated by the frequency information into a plurality of frequency bands based on the frequency information, and shows a frequency division spectrum indicating the power of the sound of each frequency in each frequency band after the division Generate ingredients. The frequency division spectrum component is generated for each frequency band after division for each of the R and L channels.

  Furthermore, the sound source separation unit 51 calculates the ratio of the power of each frequency for the frequency division spectrum components in the same frequency band of the R and L channels, and the obtained ratio is determined in advance among the frequency division spectrum components. Select the frequency division spectral component that is the value. The sound composed of the frequency division spectrum components selected in this way is assumed to be the individual sound to be extracted.

  The sound source separation unit 51 performs inverse Fourier transform on the selected frequency division spectrum components of the R and L channels, and the audio data of each of the R and L channels obtained as a result is converted to the audio of the R and L channels of the individual audio. Data.

  The value of the ratio of the frequency powers used for the extraction of the individual audio is determined in advance by the distribution rate of the individual audio levels allocated to the audio data of the R and L channels. The blind sound source separation processing is described in detail in, for example, Japanese Patent Application Laid-Open No. 2008-104240.

  When the sound source separation unit 51 separates the sound data of the content into the sound data of each individual sound, the sound data of the individual sound is separated into the sound source direction estimation unit 52, the sound source distance estimation unit 53, and the environmental sound / utterance sound identification. Supplied to the unit 54.

  Thus, by performing the blind sound source separation processing and extracting the sound data of each individual sound from the sound data, the direction and distance of the sound source of each individual sound can be obtained more accurately, and the sound source The effect image can be displayed at a position close to.

  In step S42, the sound source direction estimation unit 52 performs sound source direction estimation processing on the sound data of each individual sound supplied from the sound source separation unit 51, and estimates the direction of the sound source of the individual sound.

  For example, when the audio data constituting the content data is composed of audio data of R and L channels, the sound source direction estimating unit 52 performs Fourier transform on the audio data of the individual audio (individual audio data). The sound source direction estimation unit 52 compares the obtained frequency information of the R and L channels and detects the phase shift of the audio data of the R and L channels, thereby determining the direction of the sound source of the individual sound. presume.

  The sound source direction estimation unit 52 generates direction data indicating the direction of the sound source of each individual sound obtained and supplies the direction data to the image composition unit 26. Note that the estimation of the direction of the sound source is described in detail in, for example, Japanese Patent Application Laid-Open No. 2010-20294.

  In step S43, the sound source distance estimation unit 53 performs a sound source distance estimation process on each individual audio data supplied from the sound source separation unit 51, and estimates a distance to the sound source of each individual sound.

  For example, when the audio data constituting the content data is composed of audio data of the R and L channels, the sound source distance estimation unit 53 performs discrete Fourier transform on the individual audio data of the R and L channels, and Obtain the phase difference spectrum.

  Furthermore, the sound source distance estimation unit 53 obtains the standard deviation of the phase difference at each frequency from this phase difference spectrum, and calculates the average value of the standard deviation of the frequency in a predetermined frequency band as the feature amount. The sound source distance estimation unit 53 estimates the distance from the sound source to the sound collection unit by substituting the feature quantity obtained in this way into a function obtained in advance.

  The sound source distance estimation unit 53 generates distance data indicating the distance to the sound source of each obtained individual sound and supplies the distance data to the image composition unit 26.

  In step S <b> 44, the environmental sound / speech sound identification unit 54 determines, for each individual audio data supplied from the sound source separation unit 51, whether the individual audio based on the individual audio data is an utterance sound or an environmental sound. Do.

  For example, when the audio data constituting the content data is composed of audio data of the R and L channels, the environmental sound / speech sound identifying unit 54 calculates the sum of the R channel individual audio data and the L channel individual audio data. By calculating, the sum signal of the individual audio data is determined. In addition, the environmental sound / speech sound discriminating unit 54 performs a filtering process on the obtained sum signal using a filter that removes a frequency band component of a general human voice.

  Further, the environmental sound / speech sound discriminating unit 54 obtains a difference signal between the individual audio data by obtaining a difference between the individual audio data of the R channel and the individual audio data of the L channel, and is subjected to the difference signal and the filter processing. Find the difference from the sum signal.

  When the difference between the obtained difference signal and the sum signal is equal to or greater than a predetermined threshold, the environmental sound / speech sound identification unit 54 determines that the individual sound to be processed is an environmental sound.

  When a person's voice is picked up by the two sound pickup parts for the R and L channels, the person who becomes the sound source is often located approximately in the middle of the two sound pickup parts. Therefore, human voices included in the individual voices of the R and L channels should have substantially the same level (volume). Therefore, when the difference between the individual voice data is obtained, the obtained difference signal includes: There should be little human voice.

  Therefore, when the human voice component is removed from the sum signal of the individual audio data of the R and L channels by filtering and the difference between the filtered sum signal and the difference signal is obtained, the difference is R or L The sound data should include only the environmental sound of the channel. Therefore, the environmental sound / speech sound identification unit 54 determines that the individual sound to be processed is an environmental sound when the obtained difference is equal to or greater than the threshold value, and conversely, if the difference is less than the threshold value, the individual sound is a speech sound. Suppose there is.

  The environmental sound / speech sound identification unit 54 supplies, to each utterance content identification unit 56, the audio data of the individual sound determined to be the utterance sound among the individual sounds, and the individual sound determined to be the environmental sound. The sound data is supplied to the environmental sound identification unit 55.

  In step S45, the utterance content identification unit 56 performs voice recognition processing on the voice data of each individual voice supplied from the environmental sound / speech sound identification unit 54, and converts the utterance content of the individual voice into text.

  For example, the utterance content identification unit 56 performs acoustic analysis processing on the audio data for each predetermined frame, and extracts a feature amount of a predetermined feature from the audio data. For example, as an acoustic analysis process, discrete Fourier transform is performed, and a power spectrum is extracted as a feature amount.

  Next, the utterance content identification unit 56 performs a matching process using the obtained feature amount and a previously recorded acoustic model database, dictionary database, and grammar database to recognize individual speech.

  Here, the acoustic model database includes an acoustic model such as an HMM (Hidden Markov Model) representing acoustic features for each unit (PLU (Phoneme Like Units)) such as individual phonemes and syllables in a speech language.

  The dictionary database includes a word dictionary in which phoneme information related to pronunciation for each word is described for each word to be recognized, and information indicating a probability that a feature amount is observed from each acoustic model. The grammar database is composed of grammar rules (language model) that describe how words registered in the word dictionary of the dictionary database are chained together.

  The utterance content identification unit 56 refers to the word dictionary in the dictionary database, connects the acoustic model in the acoustic model database, and constructs an acoustic model (word model) of the word. Then, the utterance content identification unit 56 connects several word models by referring to the grammar rules of the grammar database, and the likelihood obtained from the feature amount is the highest from the series of word models thus connected. A word string corresponding to a series of high word models is output as an individual speech recognition result. That is, utterance sound data indicating a word string obtained as a result of speech recognition is supplied to the visual information generation unit 25.

  In step S46, the environmental sound identification unit 55 performs voice recognition processing on the voice data of each individual voice supplied from the environmental sound / speech sound identification unit 54, and converts the individual voice, that is, the environmental sound into text. .

  In the voice recognition process by the environmental sound identification unit 55, the same process as the voice recognition process in the utterance content identification unit 56 is performed. That is, feature amounts are extracted from the audio data, and matching processing between the extracted feature amounts and each database is performed.

  However, in the dictionary database recorded in the environmental sound identification unit 55, a word different from the word registered in the dictionary database of the utterance content identification unit 56, for example, an ambulance siren “Peepy Peep” is registered. The environmental sound identification unit 55 does not have to be provided with a grammar database.

  When the environmental sound is converted into text and the environmental sound data obtained as a result is supplied from the environmental sound identification unit 55 to the visual information generation unit 25, the analysis process ends, and then the process proceeds to step S15 in FIG. move on.

  In this way, the analysis unit 24 separates the audio data of the content into audio data of speech sounds and environmental sounds, and performs estimation of the sound source direction, estimation of the distance of the sound source, audio recognition processing, etc. for each audio data. Do. Therefore, according to the analysis unit 24, information for each individual voice such as the utterance content and the direction of the sound source can be obtained more reliably. In addition, the analysis unit 24 determines whether the sound is an utterance sound or an environmental sound for each individual voice, and performs voice recognition processing using a different dictionary according to the determination result. Voice can be converted into text.

<Second Embodiment>
[Configuration of analysis unit]
In the above description, it has been described that the effect images of both the utterance sound and the environmental sound constituting the content are displayed. However, since the utterance sound may have a closed caption or the like, the effect image of only the environmental sound is included. May be displayed.

  In such a case, the analysis unit 24 is configured as shown in FIG. 7, for example.

  That is, the analysis unit 24 shown in FIG. 7 includes a sound source direction estimation unit 52, an environmental sound / speech separation unit 81, and an environmental sound identification unit 55. The audio data from the audio decoder 23 is converted into the sound source direction estimation unit 52 and It is supplied to the environmental sound / sound separation unit 81.

  In FIG. 7, parts corresponding to those in FIG. 2 are denoted by the same reference numerals, and description thereof is omitted as appropriate.

  The environmental sound / audio separation unit 81 extracts environmental sound audio data from the audio data supplied from the audio decoder 23, and supplies it to the environmental sound identification unit 55.

[Description of analysis processing]
Next, analysis processing when the analysis unit 24 has the configuration shown in FIG. 7 will be described with reference to the flowchart shown in FIG.

  In step S <b> 71, the sound source direction estimation unit 52 generates direction data indicating the direction of the sound source of the environmental sound from the audio data supplied from the audio decoder 23, and supplies the direction data to the image synthesis unit 26.

  For example, the sound source direction estimation unit 52 performs a Fourier transform on the supplied audio data, and is generated from each sound source from the frequency information obtained thereby and a separation matrix obtained by a learning process to which independent component analysis is applied. Generate separated environmental sound signals. The sound source direction estimating unit 52 calculates a mutual covariance matrix between the frequency information of the corresponding time interval and the separated signal of the environmental sound among the separated signals, and determines the positions of the elements of the mutual covariance matrix. By calculating the phase difference, the sound source direction of the environmental sound is obtained and direction data is generated.

  In this case, since the environmental sound distance data is not generated, the image synthesis unit 26 overlays the environmental sound effect image at a position determined by the direction data in the content image. Of course, a sound source distance estimation unit 53 may be provided to generate environmental sound distance data.

  In step S <b> 72, the environmental sound / audio separation unit 81 extracts environmental sound audio data from the audio data supplied from the audio decoder 23, and supplies the environmental sound identification data to the environmental sound identification unit 55.

  For example, when the audio data constituting the content data is composed of R and L channel audio data, the environmental sound / audio separation unit 81 obtains the sum of the R channel audio data and the L channel audio data. The sum signal of the audio data is obtained. In addition, the environmental sound / speech separation unit 81 performs a filtering process using a filter that removes a component of a general human voice frequency band on the obtained sum signal.

  Further, the environmental sound / audio separation unit 81 obtains a difference signal of the audio data by subtracting the L channel audio data from the R channel audio data, and calculates the sum of the difference signal and the filtered sum signal. To obtain the sound data of the R channel of the environmental sound. The environmental sound / sound separation unit 81 generates L channel sound data of the environmental sound by subtracting the difference signal from the filtered sum signal.

  Similar to the processing in the environmental sound / speech sound discriminating unit 54 described above, the sum signal and the difference signal do not include the speech sound. Therefore, by obtaining the difference or sum of the signals, The audio data of the L channels can be extracted. That is, the audio data of the content audio can be separated into audio data of the utterance sound and audio data of the environmental sound. The environmental sound / speech sound identification unit 54 supplies the obtained environmental sound data to the environmental sound identification unit 55.

  When the sound data of the environmental sound is obtained in this way, the process of step S73 is performed thereafter, and the analysis process ends. However, the process of step S73 is the same as the process of step S46 of FIG. Description is omitted. When the analysis process ends, the process thereafter proceeds to step S15 in FIG.

  In this manner, the analysis unit 24 extracts only the sound data of the environmental sound from the sound data of the content, and converts the environmental sound into text. Thereby, the effect image of the environmental sound can be displayed on the content image, and the user can grasp the content of the environmental sound and the sound source position more accurately.

  In FIG. 7, the utterance content identification unit 56 is not provided in the analysis unit 24, but the utterance content identification unit 56 may also be provided in the analysis unit 24 in FIG. 7.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 9 is a block diagram illustrating a hardware configuration example of a computer that executes the above-described series of processing by a program.

  In a computer, a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, and a RAM (Random Access Memory) 303 are connected to each other by a bus 304.

  An input / output interface 305 is further connected to the bus 304. The input / output interface 305 includes an input unit 306 including a keyboard, a mouse, and a microphone, an output unit 307 including a display and a speaker, a recording unit 308 including a hard disk and a nonvolatile memory, and a communication unit 309 including a network interface. A drive 310 that drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

  In the computer configured as described above, the CPU 301 loads, for example, the program recorded in the recording unit 308 to the RAM 303 via the input / output interface 305 and the bus 304, and executes the above-described series. Is performed.

  The program executed by the computer (CPU 301) is, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor. It is recorded on a removable medium 311 which is a package medium composed of a memory or the like, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  The program can be installed in the recording unit 308 via the input / output interface 305 by attaching the removable medium 311 to the drive 310. Further, the program can be received by the communication unit 309 via a wired or wireless transmission medium and installed in the recording unit 308. In addition, the program can be installed in advance in the ROM 302 or the recording unit 308.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 11 Image processing apparatus, 12 Recording medium, 24 Analysis part, 25 Visual information generation part, 26 Image composition part, 27 Display part, 51 Sound source separation part, 52 Sound source direction estimation part, 53 Sound source distance estimation part, 54 Environment sound / utterance Sound identification unit, 55 Environmental sound identification unit, 56 Utterance content identification unit, 81 Environmental sound / speech separation unit

Claims (8)

Sound source direction estimating means for estimating the direction of the sound source of the sound relative to a predetermined reference position based on the sound data of the sound constituting the content;
Separating means for separating the voice data into the voice data of the environmental sound excluding the utterance sound caused by the utterance of the person in the voice and the voice data of the utterance sound;
An environmental sound identifying means for performing speech recognition processing on the audio data of the environmental sound and converting the environmental sound into text;
An effect image that visually presents the environmental sound is generated based on the textual environmental sound so that the effect image is displayed at a position determined by an estimation result of the direction of the sound source on the image constituting the content. An image processing apparatus comprising: effect data of the effect image; and image combining means for combining the image data of the image. Further comprising sound source distance estimating means for estimating a distance from the reference position to the sound source based on the sound data of the sound;
The image synthesis means is configured to display the effect data and the effect data so that the effect image is displayed at a position determined by the estimation result of the direction of the sound source on the image and in a size determined by the estimation result of the distance of the sound source. The image processing apparatus according to claim 1, wherein the image processing apparatus combines the image data. Further comprising sound source separation means for performing a blind sound source separation process on the sound data of the sound and separating the sound data into sound data of the sound of each sound source;
The separation means determines, for each of the sound data separated by the sound source separation means, whether or not the sound data is the sound data of the environmental sound, The image processing apparatus according to claim 1, wherein the voice data of the uttered sound is separated. The image composition means changes the size, color, or brightness of the effect image displayed on the image according to at least one of the direction of the sound source, the distance of the sound source, or the volume of the environmental sound. The image processing apparatus according to claim 2, wherein the effect data and the image data are combined. Based on the environmental sound converted to text so that supplementary information consisting of a character string supplementing the content of the environmental sound, which is predetermined for the environmental sound converted to text, is displayed on the effect image. The image processing apparatus according to claim 2, further comprising visual information generation means for generating the effect data. Further comprising speech recognition means for performing speech recognition processing on the speech data of the utterance sound separated by the separation means, and converting the utterance sound into text,
The image synthesis means synthesizes the effect data and the image data so that the effect image of the environmental sound and the effect image that visually presents the speech sound are displayed on the image. Item 3. The image processing apparatus according to Item 2. Sound source direction estimating means for estimating the direction of the sound source of the sound relative to a predetermined reference position based on the sound data of the sound constituting the content;
Separating means for separating the voice data into the voice data of the environmental sound excluding the utterance sound caused by the utterance of the person in the voice and the voice data of the utterance sound;
An environmental sound identifying means for performing speech recognition processing on the audio data of the environmental sound and converting the environmental sound into text;
An effect image that visually presents the environmental sound is generated based on the textual environmental sound so that the effect image is displayed at a position determined by an estimation result of the direction of the sound source on the image constituting the content. An image processing method for an image processing apparatus comprising: image data for combining the effect data of the effect image and the image data of the image,
The sound source direction estimating means estimates a direction of a sound source of the sound;
The separation means separates the sound data into the sound data of the environmental sound and the sound data of the utterance sound;
The environmental sound identification means converts the environmental sound into text,
An image processing method including a step in which the image synthesis means synthesizes the effect data and the image data. Based on the sound data of the sound constituting the content, the direction of the sound source of the sound with respect to a predetermined reference position is estimated,
Separating the voice data into the voice data of the environmental sound excluding the utterance sound caused by the utterance of the person in the voice and the voice data of the utterance sound;
Performing a speech recognition process on the sound data of the environmental sound, converting the environmental sound into text,
An effect image that visually presents the environmental sound is generated based on the textual environmental sound so that the effect image is displayed at a position determined by an estimation result of the direction of the sound source on the image constituting the content. A program for causing a computer to execute processing including a step of combining effect data of the effect image and image data of the image.

Images