JP5383056B2 - Sound data recording / reproducing apparatus and sound data recording / reproducing method - Google Patents

Description

  The present invention relates to a sound data recording / reproducing apparatus and a sound data recording / reproducing method capable of easily reproducing desired sound data from recorded sound data.

  It is more difficult to reproduce desired sound data from recorded sound data than to reproduce desired image data from recorded image data.

  The first reason is that it is difficult to overview sound data over time. In the case of image data, desired image data can be identified and reproduced relatively easily by fast-forwarding the data or by picking up and displaying the data at specific time intervals. Even if the image is fast-forwarded, the color tone of the image does not change, and a large recognition error does not occur even if the number of frames of the image is reduced. On the other hand, in the case of sound data, it becomes difficult to recognize if the data is fast-forwarded. Also, the picked-up short-time sound data cannot be recognized even if it is played back.

  The second reason is that it is difficult to distinguish sound sources. In the case of image data, even if two objects overlap on the screen, the front object can be recognized correctly, and the two objects can be discriminated. On the other hand, in the case of sound data, it is difficult to understand the content of the speech when the speech of a plurality of speakers overlaps.

  So far, for example, a system for estimating the position of a sound source and displaying the position of the estimated sound source on an image has been developed (for example, Patent Document 1).

However, a sound data recording / reproducing apparatus and a sound data recording / reproducing method configured to discriminate sound sources and to be able to overview sound data for each sound source over time have not been developed.
Japanese Patent Laid-Open No. 2003-111183

  Therefore, a sound data recording / reproducing apparatus configured so that sound data for each sound source can be overviewed over time so that desired sound data can be easily reproduced from the recorded sound data. There is also a need for a method for recording and reproducing sound data.

  The sound data recording / reproducing apparatus according to the present invention includes a sound data acquisition unit that acquires sound data, a sound source localization unit that specifies the direction in which the sound source exists, and a sound source separation unit that separates sound data for each sound source. The apparatus includes a time-series data storage unit that stores time-series sound data for each sound source, a stream data storage unit that stores stream data related to sound that indicates a direction of a predetermined sound source at a predetermined time, and the time And a data processing unit connected to the stream data storage unit and the stream data storage unit for processing data. The apparatus further includes a sound data reproduction unit that reproduces sound data, and a display unit that displays stream data.When the stream data displayed by the display unit is selected, the sound data reproduction unit It is configured to play sound data related to the selected stream data.

  The method of reproducing sound data by the sound data recording / reproducing apparatus according to the present invention acquires sound data, specifies the direction in which the sound source exists, and separates sound data for each sound source. Further, time-series sound data for each sound source is stored in the time-series data storage unit, and stream data related to sound indicating the direction of a predetermined sound source is created at a predetermined time and stored in the stream data storage unit. Furthermore, stream data is displayed, and when the displayed stream data is selected, sound data relating to the selected stream data is reproduced.

  According to the present invention, the stream data relating to the sound indicating the direction of the predetermined sound source is displayed at the predetermined time, so that the sound data for each sound source can be overviewed along the time, and the recorded sound data The desired sound data can be easily reproduced from the list.

  According to the embodiment of the present invention, voice is recognized from sound data, voice text information is generated, and the voice text information is displayed.

  According to the present embodiment, for example, a hearing-impaired person can easily reproduce and use desired sound data (sound) from the recorded sound data.

  According to an embodiment of the present invention, image data is acquired, time-series image data is stored in the time-series data storage unit, and data stored in the stream data storage unit is stored at a predetermined time at a predetermined time. It further includes stream data relating to an image indicating the direction of the object.

  According to the present embodiment, sound data (speech contents) of a sound source (speaker) and image data (speaker's facial expression) can be associated and reproduced.

  FIG. 1 is a diagram showing a configuration of a sound data recording / reproducing apparatus according to an embodiment of the present invention.

  In the sound data recording / reproducing apparatus, the sound data acquisition unit 101 acquires sound data. The acquired sound data is sent to the sound source localization unit 103, and the sound source localization unit 103 identifies the direction in which the sound source exists. The acquired sound data is sent to the sound source separation unit 105, and the sound source separation unit 105 separates the sound for each sound source. Further, the image data acquisition unit 107 acquires image data.

  The sound data recording / playback apparatus includes three types of memories: a time-series data storage unit 109, a stream data storage unit 111, and a stream list storage unit 113. The data structure of these memories will be described later.

  The sound data recording / reproducing apparatus includes a time series data storage unit 109, a stream data storage unit 111, a data processing unit 115 connected to the stream list storage unit 113, and a time series data storage unit 109 and an audio connected to the data processing unit 115. A recognition unit 121 is provided. The data processing unit 115 is further connected to a display / input unit 117 and a sound data reproduction unit 119. The data processing unit 115 performs processing using the data stored in the above three types of memory, the display / input unit 117 displays the sound data in an overview, and according to the operator input from the display / input unit 117. The sound data reproducing unit 119 is configured to reproduce desired sound data. The speech recognition unit 121 recognizes speech from the sound data stored in the time-series data storage unit 109 according to an instruction from the data processing unit 115, generates speech text information, and stores the speech text information in the time-series data storage unit 109. The display / input unit 117 may be a separate housing for the display unit and the input unit.

  As an example, the sound data acquisition unit 101 includes eight microphones and an acoustic signal processing processor that processes sound data collected by the microphones. The acoustic signal processing processor can collect, for example, 16-channel sound data at a predetermined frequency. The eight microphones are composed of seven normal microphones and one surround microphone. Seven normal microphones are attached to a base having a spherical shape, and the base is designed to easily collect sound from the direction in which each microphone is installed.

  The sound data recording / reproducing apparatus according to the present invention may be used, for example, when sound data is acquired and reproduced at a predetermined position such as a conference room. In that case, the sound data acquisition unit 101 is arranged at the predetermined position. In another embodiment, the sound data recording / reproducing apparatus according to the present invention may be used when sound data in a moving body is acquired and reproduced. In that case, the sound data acquisition unit 101 is attached to the moving body. Examples of the moving body include a vehicle, a monitoring robot, and a human. For example, in combination with GPS (Global Positioning System), the direction of the sound source can be recognized at the position of the moving body.

The sound source localization unit 103 performs localization by, for example, a steered beam former in order to specify the direction in which the sound source exists, and improves accuracy by a Kalman filter (Masamitsu Murase, Shun'ichi Yamamoto, Jean-Marc Valin, Kazuhiro Nakadai, Kentaro Yamada, Kazunori Komatani, Tetsuya Ogata, Hiroshi G. Okuno: Multiple Moving Speaker Tracking by
Microphone Array on Mobile Robot, Proceedings of the Ninth European Conference
on Speech Communication and Technology (Interspeech-2005), 249-252, Lisboa, Sep. 2005 p.10). The localization by the steered beam former is performed based on the cross-correlation between the microphone pairs by the following steps.

1) Consider a sphere centered at the origin of the coordinate system that defines the coordinates of the microphone, and place 2562 points on the surface at equal intervals.
2) For each point, find the sum of cross correlations for all microphone pairs. It is estimated that the direction of the point where this sum is the maximum is the direction of the sound source.
3) All estimated cross-correlation values are set to 0.
4) Repeat 2) and 3) to estimate all sound source directions. As a result, the estimated sound source direction is scanned on the time axis, and those having close directions are labeled as one sound source.

  The Kalman filter is for predicting current and future internal states from past observation values when a dynamically changing system such as a moving sound source is observed. For example, the current speaker state is estimated from the past speaker state. Thereby, when a single speaker moves, the same label can be reliably given to the single speaker.

The sound source separation unit 105 is, for example, Geometric to separate sound data for each sound source.
Source separation by Source Separation (GSS) and noise suppression processing by Post-Filter (J.-M. Valin, J. Rouat, F. Michaud: Enhanced Robot Audition Based on Microphone Array Source Separation with Post-Filter, Proc IEEE / RSJ International
Conference on Intelligent Robots and Systems (IROS), pp.2123-2128, 2004.).

ここでA(k)は変換を表す行列であり，n(k)はノイズである。これより推定した音源信号 y(k)は式（２）で表される。

x(k)は観測信号であるため、音源信号を求めることは変換行列W(k)を求める問題に帰着する。異なる音源間信号の独立性を仮定することにより式（３）が、音源とマイクの幾何学的制限により式（４）が得られる。

これらの式（３）および（４）は制限が強いため、Wの近似解を確率的勾配法により求める。 Assuming that the conversion from the acoustic signal (k) to the observation signal x (k) is linear at the frequency k, the transfer function from the sound source signal to the observation signal is expressed by Equation (1).

Here, A (k) is a matrix representing the transformation, and n (k) is noise. The sound source signal y (k) estimated from this is expressed by equation (2).

Since x (k) is an observation signal, obtaining a sound source signal results in a problem of obtaining a transformation matrix W (k). Equation (3) is obtained by assuming the independence of signals between different sound sources, and Equation (4) is obtained due to geometric limitations of the sound source and the microphone.

Since these equations (3) and (4) are severely limited, an approximate solution of W is obtained by the stochastic gradient method.

  Noise suppression by Post-Filter removes noise from separated sound by GSS. In this method, noise is estimated and removed by dividing it into stationary noise and non-stationary noise. Stationary noise is calculated by MCRA (Minima Controlled Recursive Average). Non-stationary noise is assumed to have leaked from another channel during the GSS process, and interference components are estimated adaptively.

  The data structures of the time series data storage unit 109, the stream data storage unit 111, and the stream list storage unit 113 will be described below.

  FIG. 2 is a diagram illustrating a data structure regarding sound data in the time-series data storage unit 109, the stream data storage unit 111, and the stream list storage unit 113.

  The time-series data storage unit 109 stores sound waveform data at each time. The sound waveform data includes waveform data that is not separated and waveform data that is separated for each sound source. Data of the direction of the sound source is given to the data separated for each sound source. Data at each time is linked in the time direction. For example, data at time t = 1 or time t = 3 can be referred to from data at time t = 2.

  The stream data storage unit 111 stores stream data related to sound. The stream related to sound is the sound of a predetermined sound source that continues for a certain period of time. The stream data relating to sound is data indicating the direction (position) of the sound source for each time with respect to the sound of a predetermined sound source that continues for a certain period of time. Specifically, the stream data includes data regarding a start time, an end time, identification of a sound source (for example, a person name), and a direction of the sound source at each time within the predetermined time. The direction of the sound source is expressed by, for example, an angle in a horizontal plane and a vertical plane. The stream data further includes a link to sound waveform data. For example, sound waveform data corresponding to the time can be referenced from stream data at a predetermined time.

  FIG. 3 is a diagram illustrating a data structure related to image data in the time-series data storage unit 109, the stream data storage unit 111, and the stream list storage unit 113.

  The time series data storage unit 109 stores image data (moving image information) at each time. Data of the direction of the target (for example, a person) is given to the image data. A person is identified using a general face recognition technique (for example, Japanese Patent Laid-Open No. 2002-216129). Data at each time is linked in the time direction. For example, data at time t = 1 or time t = 3 can be referred to from data at time t = 2.

  The stream data storage unit 111 stores stream data related to images. A stream relating to an image is an image of a predetermined target that continues for a certain period of time. The stream data relating to the image is data indicating the direction (position) of the target for each time with respect to the predetermined target image that is continuous for a predetermined time. Specifically, the stream data includes start time, end time, target identification (for example, person name), and data regarding the target direction at each time. The direction of the object is expressed by, for example, an angle in a horizontal plane and a vertical plane. The stream data further includes a link to image data (moving image information). For example, the image data corresponding to the time can be referred to from the stream data at a predetermined time.

  In the stream data storage unit 111, there are stream data related to sound and stream data related to images. When the direction of the sound source in the stream data related to sound and the target direction of the stream data related to the image coincide with each other for a predetermined time or more, the two stream data may be integrated to form integrated stream data. If the integrated stream data is used, for example, sound data (speech content) of a sound source (speaker) and image data (speaker's facial expression) can be associated and reproduced.

  The data in the stream list storage unit 113 is data related to a list of stream data. The list of stream data is for overview display. The total number of streams at the thinned time, the direction of the sound source of each stream, the type of each stream (stream related to sound, stream related to image, integrated stream), to each stream data Includes links. For example, stream data corresponding to the time can be referenced from a list of stream data at a predetermined time. 2 and 3, based on the data in the stream list storage unit 113, a function of sound source or target identification (person name) and time, or a function of sound source or target direction (angle) and time Each stream can be displayed on the display / input unit 117. 2 and 3, three stream data are indicated by a solid line, a dotted line, and an alternate long and short dash line.

  FIG. 4 is a diagram conceptually showing the interface function of the sound data recording / reproducing apparatus according to the present embodiment. The interface of the sound data recording / playback apparatus according to the present embodiment provides an overview display (Over first), a zoom (Zoom), a filter (Filter), and a detailed display upon request (Details on) so that the sound data can be reproduced in an easily understandable manner function for demand).

  FIG. 4A shows the contents of the overview display. The vertical axis represents sound source identification (person name), and the horizontal axis represents time. This display can be performed based on data in the stream list storage unit 113. With the overview display shown in FIG. 4A, the user can overview the sound data over time.

  FIG. 4B shows the zoomed content. FIG. 4B specifically shows the sound source and its direction at the zoomed time. These data are included in the stream data in the stream data storage unit 111. By specifying a specific time on the horizontal axis of the appearance display shown in FIG. 4A, the user can easily extract (zoom) data at the time. Since the list of stream data includes a link to each stream data, the above data can be easily extracted.

  FIG. 4C shows the contents of the filter process. FIG. 4C specifically shows that the sound source of BGM (background music) is removed by designating the range of the sound source.

  FIG. 4D is a diagram showing the contents of the detailed display. For example, the user can designate a predetermined sound source (speaker) from a sound source (speaker) at a predetermined time shown in FIG. 4B and reproduce only the sound data (speech content). .

  FIG. 5 is a flowchart showing a sound data recording / reproducing method according to an embodiment of the present invention.

  In step S010 in FIG. 5, the sound data acquisition unit 101 acquires sound data, and the image data acquisition unit 107 acquires image data.

  In step S020 in FIG. 5, the sound source localization unit 103 identifies the direction in which the sound source exists.

  In step S030 in FIG. 5, the sound source separation unit 105 separates sounds for each sound source.

  In step S040 in FIG. 5, the sound data and the image data are stored in the time-series data storage unit 109. The time-series data storage unit may further store voice text information generated by the voice recognition unit 121.

  In step S 050 in FIG. 5, the data processing unit 115 creates stream data based on the sound data and image data stored in the time-series data storage unit 109 and stores the stream data in the stream data storage unit 111.

  In step S 060 in FIG. 5, the data processing unit 115 creates a list of stream data based on the stream data stored in the stream data storage unit 111 and stores the list in the stream list storage unit 113.

  In step S070 in FIG. 5, the display / input unit 117 displays the stream data together with the time axis (FIG. 4A).

  In step S080 in FIG. 5, the display / input unit 117 displays the sound source and its direction at the time selected by the user on the displayed time axis (FIG. 4B).

  In step S090 in FIG. 5, the sound data reproducing unit 119 reproduces the sound data of the sound source selected by the user (FIG. 4 (d)).

  Instead of steps S070, S080, and step S090, the display / input unit 117 displays, in real time, a sound source that changes over time and its direction according to the stream data, for example, on a screen as shown in FIG. Alternatively, it may be configured to display at the same speed as real time, or at an increased or decreased speed.

  FIG. 6 is a diagram illustrating an example of a display screen of the display / input unit 117. The display screen includes, for example, an operation panel unit, a sound source direction display unit, and a stream display unit. An instruction to the sound data recording / playback apparatus by the user is input from the operation panel unit. The display unit and the input unit may be configured by one screen of one housing as shown in FIG. Or you may comprise by 2 housing | casing which uses the 2 screen of 1 housing | casing or a screen, or does not use it.

  The display screen may further include a text information display unit (not shown) that displays voice text information generated by the voice recognition unit 121. The display / input unit 117 displays voice text information on the text information display unit when the sound data playback unit 119 plays back the sound data (sound) of the sound source selected by the user (FIG. 4D). To do. The text information display unit may emphasize the text information of speech with a high recognition likelihood and display it in a dark color, and display the text information of speech with a low recognition likelihood in a light color. By displaying the audio text information, for example, a person with hearing impairment can easily reproduce and use desired sound data (sound) from the recorded sound data.

  FIG. 7 is a diagram illustrating details of the stream display unit of the display screen. The stream display unit displays the stream data based on the data in the stream list storage unit 113 or the stream data storage unit 111. The horizontal axis represents time, and the vertical axis represents the angle in the horizontal plane. The display content of the stream display unit corresponds to the display of FIG. In FIG. 7, the stream data of the stream A indicated by the solid line and the stream B indicated by the dotted line are displayed. The user can specify the playback time by specifying the time on the time axis (horizontal axis). Based on the data in the time-series data storage unit 109, the waveform data of the recorded sound may be displayed together on the stream display unit.

  FIG. 8 is a diagram showing details of the sound source direction display section of the display screen. The sound source direction display unit displays the sound source and its direction at the designated playback time based on the data in the stream data storage unit 111. The display screen shows, for example, a horizontal plane and the direction of the sound source with respect to the position of the microphone arranged at the center. In FIG. 8, the line attached with A indicates the direction of the sound source of stream A, and the line attached with B indicates the direction of the sound source of stream B. This display corresponds to the display of FIG. On the screen of the sound source display unit, for example, the angle range may be limited by an operation such as clicking, and sound data of the sound source in that range may be reproduced. This operation corresponds to the operation of FIG. Further, in FIG. 8, by clicking a line attached with A or a line attached with B, a sound source may be designated and only sound data of the sound source may be reproduced. This operation corresponds to the operation of FIG.

  Features of the embodiment of the present invention will be described below.

  According to the embodiment of the present invention, the stream data includes data related to a link to time-series sound data.

  According to the present embodiment, the sound data at the selected predetermined time can be easily reproduced by the link to the time-series sound data.

  According to the embodiment of the present invention, a list of stream data is stored in the stream list storage unit.

  According to this embodiment, it becomes easy to display the sound data for each sound source so that it can be viewed over time.

  According to the embodiment of the present invention, the time axis is displayed together with the stream data, and when a point on the displayed time axis is selected, the corresponding stream data is extracted from the stream data storage unit, and the corresponding stream data The sound source and its direction at the selected predetermined time are displayed.

  According to the present embodiment, the user can overview the recorded sound data along the time by the stream data, and the user can select the point by selecting the point on the time axis displayed together with the stream data. A sound source and its direction data at a predetermined time can be easily extracted.

  According to the embodiment of the present invention, when a predetermined sound source is selected from sound sources at a predetermined time displayed on the display unit, sound data of the predetermined sound source is extracted from the time-series data storage unit, and the corresponding Play sound data.

  According to the present embodiment, the user can easily extract the sound data of the predetermined sound source by selecting the predetermined sound source from the sound sources displayed at the predetermined time.

It is a figure which shows the structure of the sound data recording / reproducing apparatus by one Embodiment of this invention. It is a figure which shows the data structure regarding the sound data of a time series data storage part, a stream data storage part, and a stream list storage part. It is a figure which shows the data structure regarding the image data of a time series data storage part, a stream data storage part, and a stream list storage part. It is the figure which showed notionally the interface function of the sound data recording / reproducing apparatus by this embodiment. 4 is a flowchart illustrating a sound data recording / reproducing method according to an embodiment of the present invention. It is a figure which shows an example of the display screen of a display and an input part. It is a figure which shows the detail of the stream display part of a display screen. It is a figure which shows the detail of the sound source direction display part of a display screen.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Sound data acquisition part, 103 ... Sound source localization part, 105 ... Sound source separation part, 109 ... Time series data storage part, 111 ... Stream data storage part, 113 ... Stream list storage part, 115 ... Data processing part

Claims (8)

A sound data acquisition unit for acquiring sound data;
A sound source localization unit that identifies the direction in which the sound source exists,
A sound source separation unit for separating sound data for each sound source;
A time series data storage unit for storing time series sound data for each sound source;
A stream data storage unit for storing stream data relating to sound indicating the direction of a predetermined sound source at a predetermined time;
A data processing unit connected to the time-series data storage unit and the stream data storage unit for processing data;
A stream list storage unit for storing a list associating a time with a link to the stream data regarding the sound source at the time;
A sound data playback unit for playing back sound data;
A display for displaying stream data;
With
The display unit displays on the time axis a first display indicating a direction of a predetermined sound source at a predetermined time included in the stream data;
In response to selection of one of the first displays displayed on the time axis by the display unit, the sound data reproduction unit performs predetermined processing at a predetermined time corresponding to the selected first display. Play sound data of
A sound data recording / reproducing device ,
In response to the selection of one time on the time axis,
The data processing unit retrieves the stream data at the selected time from the stream data storage unit,
The display unit displays a second display indicating a sound source and its direction that existed at the selected time based on the extracted corresponding stream data,
A microphone is displayed at the center of the second display, and the direction of the sound source that existed at the selected time is shown as the direction relative to the position of the displayed microphone.
By selecting one of the sound sources displayed by the second display, one of the first displays is selected.
Sound data recording / reproducing device. Stream data, when the data about the links to the sound data of the series only including,
The data processing unit refers to the stream data of the sound source selected by the link in response to selection of one of the sound sources displayed in the second display, and the direction of the selected sound source Is within a predetermined angle range, the sound data of the selected sound source is extracted from the time-series data storage unit,
The sound data reproduction unit reproduces the sound data of the predetermined sound source at the extracted predetermined time;
The sound data recording / reproducing apparatus according to claim 1.   The sound data recording / reproducing apparatus according to claim 1, further comprising a speech recognition unit that recognizes speech from the sound data and generates speech text information, and wherein the display unit further displays the speech text information. An image data acquisition unit for acquiring image data is further provided, wherein the time-series data storage unit further stores time-series image data, and the data stored in the stream data storage unit has a predetermined time at a predetermined time. The sound data recording / reproducing apparatus according to any one of claims 1 to 3 , further comprising stream data relating to an image indicating a target direction. A method of reproducing sound data by a sound data recording / reproducing apparatus,
Get sound data,
Identify the direction in which the sound source exists,
Separate sound data for each sound source,
Store time-series sound data for each sound source in the time-series data storage,
At a predetermined time, create stream data relating to sound indicating the direction of a predetermined sound source, store it in a stream data storage unit,
A list that associates the time and a link to the stream data about the sound source at the time is stored in the stream list storage unit,
Displaying on the time axis a first display indicating a direction of a predetermined sound source at a predetermined time included in the stream data;
In response to selection of one of the first displays displayed on the time axis, sound data of a predetermined sound source at a predetermined time corresponding to the selected first display is reproduced.
A method for recording and reproducing sound data ,
In response to the selection of one time on the time axis,
The stream data at the selected time is extracted from the stream data storage unit,
Based on the retrieved corresponding stream data, a second display showing the sound source and its direction that existed at the selected time is displayed,
A microphone is displayed at the center of the second display, and the direction of the sound source that existed at the selected time is shown as the direction relative to the position of the displayed microphone.
By selecting one of the sound sources displayed by the second display, one of the first displays is selected.
Sound data recording and playback method. Stream data, when the data about the links to sound data of the series only including,
In response to selection of one of the sound sources displayed in the second display, the stream data of the selected sound source is referred to by the link, and the direction of the selected sound source falls within a predetermined angular range. When there is, take out the sound data of the selected sound source from the time-series data storage unit,
Reproducing the sound data of the predetermined sound source at the extracted predetermined time;
The sound data recording / reproducing method according to claim 5 . The sound data recording / reproducing method according to claim 5 or 6 , wherein a sound is recognized from sound data, sound text information is generated, and the sound text information is displayed. Stream data related to an image that acquires image data, stores time-series image data in the time-series data storage unit, and the data stored in the stream data storage unit indicates a predetermined target direction at a predetermined time The sound data recording / reproducing method according to any one of claims 5 to 7 , further comprising:

Images