JP5366043B2 - Audio recording / playback device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a voice reproduction device by which a content of voice information is easily stored in human memory, and easily adopted to human consideration, while linking feeling of reproduction operation when the voice information is reproduced, to the content of the voice information. <P>SOLUTION: The voice reproduction device for reproducing voice information recorded in a recording means (103) while associated with direction information indicating a direction, includes: a first direction specifying means (14, 100, S5) for specifying a direction which is indicated by using user's hands and body, when the voice information is reproduced; and a reproduction means (13, 100, S15) for reproducing the voice information recorded in the recording means while associated with the direction information indicating the direction corresponding to the direction specified by the first direction specifying means. The content of the voice information is easily stored in human memory, and easily adopted to human consideration, while linking feeling of reproduction operation when voice information is reproduced, to the content of the voice information. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

The present invention relates to an audio recording / reproducing apparatus, and more particularly to an audio recording / reproducing apparatus that records audio information in association with direction information and reproduces the recorded audio information.

An apparatus called an IC recorder (voice recorder) is known as an audio reproducing apparatus for reproducing audio information recorded in a recording means. As technologies relating to such an IC recorder, there are those in which various devices such as Patent Document 1, Patent Document 2, and Patent Document 3 have been devised.
JP 2007-325392 A [H02J 7/00, G10L 19/00, H01M 10/48] JP-A-2005-292893 [G10L 15/00, G10L 15/22, G10L 15/24, G10L 15/28] JP 2005-26133 A [H01H 15/16, G10L 19/00, H01H 3/52]

  However, conventionally, there has been no audio playback device such as an IC recorder that has been devised in terms of playback mode. In other words, in the conventional IC recorder, the serial number of the audio information and the recording date / time information are displayed on the small liquid crystal screen provided in the apparatus, and the desired audio information is played back. In this case, the sensation (experience) of the reproduction operation felt by the body was the same. For this reason, it has been impossible to make it easy to keep the contents of the sound information in memory or to incorporate the contents of the sound information into thoughts by linking the contents of the sound information with the experience of the reproduction operation.

Therefore, a main object of the present invention is to provide a novel audio recording / reproducing apparatus.

Another object of the present invention is to combine the experience of the reproduction operation when reproducing audio information with the content of the audio information, making it easy to keep the content of the audio information in memory, or incorporating the content of the audio information into thinking. Ru can or easy, is to provide a voice recording and reproducing apparatus.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like indicate correspondence relationships with embodiments described later to help understanding of the present invention, and do not limit the present invention in any way.

A first invention is an audio recording / reproducing device for recording audio information in a recording device and reproducing the audio information from the recording device, wherein the audio information generating means generates the audio information based on the sound emitted by the user, Recording direction specifying means for specifying the recording direction indicated by the user using the body at the time of recording, recording in which the audio information generated by the audio information generating means is recorded in the recording device in association with the direction information indicating the recording time direction means, recorded on the user recording apparatus in association with the direction information indicating the recording time direction corresponding to the playback direction specifying means, and playback direction identifying the reproduction direction shown with the body at the time of playback The audio recording / reproducing apparatus includes first audio reproducing means for reproducing the existing audio information from the recording apparatus .

In the first invention, the voice information generating means (12, S3, S35, S83, S133) generates voice information based on the voice uttered by the user, and the recording direction specifying means (14, 100, S5, S37, S85, S135) that identifies the direction indicated with the user body during recording of the audio information, recording means, the recording apparatus (103), the recording time of direction specifying means audio information is audio information generating means to generate Recording is performed in association with the direction information indicating the specified recording direction. Then, the reproduction direction specifying means (14, 100, S11, S63) specifies the direction indicated by the user using the body when reproducing the audio information, and the first audio reproduction means (100, S15, S67) is reproduced. The audio information recorded in the recording apparatus is reproduced from the recording apparatus in association with the direction information indicating the direction corresponding to the reproduction direction specified by the time direction specifying means.

According to the first invention, since the voice information is recorded in association with the direction indicated by the user using the body, when reproducing the voice information, the user records the voice information using the body. The user will be able to re-experience the physical feeling during recording during playback, make it easier to awaken thoughts during recording of audio information, and keep the contents of audio information in memory. It can be made easier, and the contents of audio information can be easily incorporated into thoughts. Furthermore, since the audio information associated with the direction information indicating the direction corresponding to the direction indicated by the user is reproduced, the experience of the reproduction operation when reproducing the audio information is combined with the content of the audio information. It is possible to make it easy to keep the contents of audio information in memory, and to make it easy to incorporate the contents of audio information into thoughts.

The second invention is an invention subordinate to the first invention, wherein the first sound reproduction means generates sound information based on direction information and distance information indicating a distance from the user to the sound information. This is an audio recording / playback apparatus that plays back in a localizable manner.

  In the second invention, the first reproduction means reproduces the audio information so that the sound source can be localized based on the direction information and the distance information indicating the distance from the user to the audio information.

According to the second aspect, since the audio information is reproduced so that the sound source can be localized, the user can specify the direction in which the audio information is located.
A third invention is an invention subordinate to the first or second invention, wherein a voice icon symbolized to indicate the presence of voice information is recorded in a recording device in association with the voice information and direction information. Audio icon reproduction for reproducing an audio icon recorded in the recording device in association with the audio information recorded in the recording device in association with the direction information indicating the recording direction not corresponding to the direction of reproduction in the icon recording means An audio recording / reproducing apparatus further comprising means.
According to the third aspect, since the sound icon is reproduced, it can be easily understood that the sound information is recorded in the recording direction that does not correspond to the reproduction direction of the user at that time.

A fourth invention is an invention subordinate to the third invention, wherein the sound icon reproducing means is configured to determine the sound icon based on direction information and distance information indicating a distance from the user to the sound information. It is an audio recording / reproducing device that can be generated .

In the fourth aspect of the invention, the audio icon reproducing means relates to the audio information recorded in the recording device in association with the direction information indicating the recording time direction not corresponding to the reproduction time direction specified by the reproduction time direction specifying means. A sound icon indicating the presence of the sound source is generated based on the direction information and the distance information so that sound source localization is possible.

According to the fourth invention, the audio information located in the recording time of a direction other than the playback direction indicated by the user, since reproduces audio icon that indicates the presence of the voice information sound source localization, the user recording device It is possible to grasp the existence of all audio information recorded in

A fifth invention is an invention dependent on any one of the first to fourth inventions, wherein the recording means is configured to record audio information into two or more layers having different depths, layer identification information indicating the layers, and Further, recording is performed in association with each other, and further includes a layer designation receiving unit that receives the designation of the layer, and the first sound reproduction unit is recorded in the recording apparatus in association with the direction information indicating the recording direction corresponding to the reproduction direction. It is an audio recording / reproducing apparatus that reproduces audio information associated with layer identification information indicating a layer received by a layer designation receiving unit of audio information.

In the fifth invention, the recording means records the audio information in association with the layer identification information for further identifying the layer. The layer designation accepting means (100, S23, S55) accepts the designation of the layer, and the first sound reproducing means records in association with the direction information indicating the recording direction corresponding to the reproduction direction specified by the reproduction direction specifying means. Of the audio information recorded in the apparatus, the audio information associated with the layer identification information indicating the layer received by the layer designation receiving means is reproduced. That is, the first sound reproducing means reproduces sound information from the designated layer.

  According to the fifth aspect, audio information can belong to a layer and can be distinguished for each layer.

A sixth invention is an audio recording / reproducing apparatus according to the fifth invention, wherein the distance from the user to the audio information is determined based on a layer depth indicated by the layer identification information. .

In the sixth invention, the distance to the audio information is determined based on the depth of the layer indicated by the layer identification information.

According to the sixth aspect , the distance to the audio information can be determined by the depth of the layer.

A seventh aspect of the invention, to no first invention an invention subordinate to any one of aspects 6, playback direction specifying means, the direction indicated by the direction of the face of the user, the orientation of the user's line of sight identifying one of the direction indicated, and the user's Pointing orientation shown, an audio recording and reproducing apparatus.

In the seventh invention, the reproduction direction specifying means specifies any one of a direction indicated by the face direction of the user, a direction indicated by the line-of-sight direction, and a direction indicated by the pointing direction as the reproduction direction .

According to the seventh aspect , the user can instruct the audio information to be reproduced by any one of the face direction, the line-of-sight direction, and the pointing direction.

Eighth aspect of the present invention is to not first invention an invention subordinate to any of the seventh invention, during recording direction specifying means, the direction indicated by the direction of the face of the user, the orientation of the user's line of sight identifying one of the direction indicated, and the user's Pointing orientation shown, an audio recording and reproducing apparatus.

In the seventh invention, the recording direction specifying means specifies any one of the direction indicated by the face direction of the user, the direction indicated by the direction of the line of sight, and the direction indicated by the pointing direction as the recording time direction .

According to the seventh aspect , the user can instruct the direction in the virtual space to which the audio information to be recorded is added by any one of the face direction, the line-of-sight direction, and the pointing direction.
According to a ninth aspect of the invention, there is provided a program executed by a voice recording / reproducing computer for recording voice information in a recording device and reproducing the voice information from the recording device, wherein the program executes the computer on the basis of voice produced by a user. Audio information generating means for generating audio information, recording direction specifying means for specifying the recording direction indicated by the user using the body during recording, direction information indicating the recording time direction for the audio information generated by the audio information generating means Corresponding to the recording means for recording in the recording device in association with the reproduction direction specifying means for specifying the reproduction direction indicated by the user using the body during reproduction, and the direction information indicating the recording direction corresponding to the reproduction direction The audio recording / reproducing program causes the audio information recorded in the recording apparatus to function as audio reproducing means for reproducing from the recording apparatus.
A tenth aspect of the invention is a method executed by a voice recording / reproducing computer for recording voice information on a recording device and reproducing the voice information from the recording device, and generating the voice information based on a voice uttered by a user. Audio information generating step, recording direction specifying step for specifying the recording direction indicated by the user using the body during recording, and audio information generated in the audio information generating step are recorded in association with the direction information indicating the recording direction A recording step for recording on the device, a direction specification step for reproduction for specifying a reproduction direction indicated by the user using the body at the time of reproduction, and direction information indicating a recording direction corresponding to the direction for reproduction are associated with the recording device. An audio recording / reproducing method including an audio reproducing step of reproducing recorded audio information from a recording apparatus.

The eleventh invention is an invention dependent on any one of the first to eighth inventions, the photographing means for photographing a peripheral image of the user at the time of recording, and the photographing means for photographing the direction at the time of recording. The sound recording / reproducing apparatus further includes conversion means for converting to a position on the peripheral image, and the recording means records the sound information in association with the peripheral image and the position information.

In the eleventh aspect of the invention, the photographing means (17) takes a peripheral image of the user at the time of recording audio information, and the photographing means determines the direction specified by the recording direction specifying means by the converting means (100, S99, S155). Convert to a position on the captured peripheral image. Then, the recording unit records the audio information generated by the audio information generation unit in association with the peripheral image captured by the imaging unit and the position information indicating the position converted by the conversion unit.

  According to the eleventh aspect, audio information can be recorded in association with a position on a peripheral image.

A twelfth invention is an invention subordinate to the eleventh invention, audio information photographing means is recorded in the recording apparatus on the peripheral image and the last shot is compared with the peripheral image that is to correspond, use the first, further comprising an environmental change judgment means for judging whether the user environment has changed, the recording means, the audio information when the first environmental change determination unit determines that the environment has changed, imaging means photographed the peripheral image and recorded in association, the audio information when the first environmental change unit determines that the environment is not changed, and the surrounding image audio information recorded in the last recording apparatus is attached corresponding Audio recording / playback device that records in correspondence.
It is.

In the twelfth invention, the first environment change determining means (100, S91) compares the peripheral image captured by the image capturing means with the peripheral image associated with the sound information recorded last in the recording device , and to determine whether the surrounding environment is changed, the recording means, the audio information is audio information generating means to generate when the first environmental change determination unit determines that the environment has changed, the photographing means is taken recorded in association with the peripheral image, the audio information audio information generating means is generated when the first environmental change unit determines that the environment is not changed, the speech information that was last recorded in the recording means corresponding Record in association with the attached peripheral image.

  According to the twelfth aspect, when the environment around the user changes, audio information can be added to the converted surrounding image of the environment.

A thirteenth invention is the invention that according to the eleventh invention, the time lapse determining means for determining whether or not the last predetermined time to record the audio information in the recording apparatus has passed, imaging means A second environment change judging means for judging whether or not the environment of the user has changed by comparing the photographed peripheral image with the peripheral image associated with the sound information recorded last in the recording device ; recording means, based on at least one of the determination result of the time lapse determining means and the second environmental change determination means, audio information, or recorded in association with the peripheral image photographing means is taken, or the audio information, the last audio information recorded in the recording apparatus is that records in association with the peripheral image that is to correspond, audio recording and reproducing apparatus.
It is.

In the thirteenth invention, the time lapse determining means (100, S14, S145) determines whether or not a predetermined time has elapsed since the last recording of the audio information on the recording device , and the second environment change determining means ( 100, S119) compares the peripheral image captured by the image capturing means with the peripheral image associated with the sound information recorded last in the recording device to determine whether the environment around the user has changed. . The recording unit records the audio information generated by the audio information generation unit in association with the peripheral image captured by the imaging unit based on the determination result of at least one of the time passage determination unit and the second environment change determination unit. or, recorded in association with the peripheral image audio information is audio information generating means stored the generated voice information to the end on the recording apparatus is to correspond.

  According to the thirteenth aspect, whether or not to update the peripheral image to which the audio information is added based on at least one of the time elapsed since the last recording of the audio information and the environment surrounding the user. Can be determined.

A fourteenth invention is an invention dependent on any one of the tenth invention to thirteenth invention of, and displays the peripheral image recorded in the recording apparatus, the peripheral image recorded in the recording device Display means for displaying audio information recorded in association with each other at a position on the peripheral image based on the position information of the audio information, icon designation means for accepting designation of the icon displayed on the display means, icon designation The audio recording / reproducing apparatus further includes second audio reproducing means for reproducing the audio information recorded in the recording apparatus indicated by the icon whose designation is accepted by the means.

In the fourteenth invention, the display means (106, S101, S103, S107 , S109) along with displaying a peripheral image recorded in a recording apparatus, is recorded in association on the peripheral image recorded in the recording device The icon indicating the voice information is displayed at a position on the peripheral image based on the position information of the voice information, and the icon designation means (105, S111) accepts designation of the icon displayed on the display means, and the second reproduction means (100, S113) reproduces the audio information recorded in the recording device indicated by the icon that has been designated by the icon designation means.

  According to the fourteenth invention, audio information corresponding to an icon can be reproduced by designating the icon displayed on the peripheral image.

The fifteenth invention is an invention dependent on the fourteenth invention, wherein the display means displays a plurality of peripheral images recorded in the recording device and accepts designation of the peripheral images displayed on the display means. The audio designating device further includes a designating unit, and the icon designating unit accepts designation of an icon displayed on a peripheral image whose designation is accepted by the image designating unit.

In the fifteenth aspect, the display means displays a plurality of peripheral images recorded in the recording device , the image designation means (105, S105) accepts designation of the peripheral images displayed on the display means, and the icon designation means The designation of the icon displayed on the peripheral image whose designation is accepted by the image designation means is accepted.

  According to the fifteenth aspect, it is possible to designate a peripheral image, designate an icon displayed on the peripheral image, and reproduce audio information corresponding to the icon.

A sixteenth aspect of the invention is an invention dependent on the fourteenth aspect of the invention or the fifteenth aspect of the invention, wherein the position designation means for accepting designation of the icon change position on the peripheral image displayed on the display means, the position designation means accepts it. The audio recording / reproducing apparatus further includes a changing unit that changes the position information recorded in the recording apparatus in association with the audio information of the icon based on the changed position.

In the sixteenth invention, the position designation means (105, S115) accepts designation of the icon change position on the peripheral image displayed on the display means, and the change means (100, S117) accepts the change position accepted by the position designation means. The position information recorded in the recording device is changed in association with the audio information of the icon based on the above.

  According to the sixteenth aspect, the position where the audio information is added to the peripheral image, that is, the position where the icon is displayed on the peripheral image can be changed.

According to the present invention, the voice information is recorded in association with the direction indicated by the user using the body. Therefore, when reproducing the voice information, the user uses the body to indicate the voice information when recording the voice information. The user will be able to re-experience the physical experience during recording, make it easier to awaken thoughts when recording audio information, and keep the contents of audio information in memory. Or make it easier to incorporate the contents of audio information into thoughts.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

<First Embodiment>
Referring to FIG. 1, an audio recording / reproducing apparatus 10 according to the present invention includes a personal information terminal (PDA: Personal Digital Assistants) 11, a bone conduction microphone 12, an ear-mounted stereo headphone 13, and a stereo headphone. 13 and a triaxial geomagnetic sensor 14 built in. The bone conduction microphone 12, the stereo headphones 13, and the triaxial geomagnetic sensor 14 are connected to a portable information terminal (hereinafter referred to as “PDA11”).

  Since the bone conduction microphone 12 directly detects the voice of the user wearing the microphone as a signal transmitted to the inner ear canal wall, the voice of the user can be picked up even in a noise. If the surrounding noise is not taken into account, a normal microphone may be used instead of the bone conduction microphone 12.

  As the triaxial geomagnetic sensor 14, a 3D sensor module TDS01V manufactured by Vitec Corporation can be used. The TDS01V includes a triaxial acceleration sensor in addition to the triaxial geomagnetic sensor 14.

  The PDA 11 is accommodated, for example, in a pocket 15 of a user's clothing. As shown in FIG. 2, the PDA 11 includes a CPU 100, ROM 101, RAM 102, flash memory 103, clock circuit 104, touch panel 105, display 106, audio input I / F 107, audio output I / F 108, geomagnetic sensor signal input I / F 109. It is composed of

  The PDA 11 can be operated by performing an input operation on the touch panel 105 disposed on the upper surface of the display 106 based on the display on the display 106. The audio input I / F 107 is an interface for inputting an audio signal from the bone conduction microphone 12 to the PDA 11. The audio output I / F 108 is an interface for outputting the audio signal reproduced by the PDA 11 to the stereo headphones 13. The geomagnetic sensor signal input I / F 109 is an interface for inputting a signal from the triaxial geomagnetic sensor 14 to the PDA 11.

According to such an audio recording / reproducing apparatus 10, audio information (audio information) uttered by a user picked up by the bone conduction microphone 12 can be recorded in the flash memory 103 of the PDA 11, and recorded in the flash memory 103. Audio information can be reproduced and output to the stereo headphones 13.

According to the audio recording / reproducing apparatus 10, it is possible to add and record audio information generated by the user at an arbitrary position in the virtual space, and to reproduce arbitrary audio information added to the virtual space. can do.

  More specifically, when recording the user's voice, the three-axis geomagnetic sensor 14 detects the face direction in the virtual space of the user at that time, and the direction information indicating the face direction is used as the voice information. And stored in the flash memory 103. When the audio information stored in the flash memory 103 is reproduced, the orientation of the face in the virtual space of the user at that time is detected by the three-axis geomagnetic sensor 14, and a predetermined range is determined from the detected face orientation. Audio information recorded in the flash memory 103 is generated in association with the direction information indicating the internal direction. At this time, the audio information is reproduced so that the user can perform sound source localization in the virtual space.

  Also, when reproducing audio information, the audio information recorded in the flash memory 103 in association with the direction information indicating the direction outside the predetermined range from the detected face orientation is the audio information instead of the audio information. A symbolized sound (hereinafter referred to as “icon sound”) indicating the presence of the icon is reproduced. This icon sound is also reproduced so that sound source localization is possible.

As described above, according to the audio recording / reproducing apparatus 10 of the present invention, the audio information is added to the position (direction) designated by the user in the virtual space by the face direction at the time of recording, and also designated by the face direction at the time of reproduction. Audio information added to the position (direction) can be reproduced. Therefore, the sense of face orientation when recording and playing back audio information is linked to the content of the audio information, making it easier to keep the content of the audio information in memory, and the content of the audio information into thoughts. Easy to incorporate.

  Also, since the audio information and the icon audio are reproduced so that the user can localize the sound source in the virtual space, the presence of all the audio information stored in the flash memory 103 of the PDA 11 is once detected. Can grasp.

The processing executed by the CPU 100 of the PDA 11 when recording audio and reproducing audio with the audio recording / reproducing apparatus 10 will be described below with reference to the flowcharts shown in FIGS. 3 and 4. Note that the processing for recording audio and the processing for reproducing audio are shown separately in the flowchart of FIG. 3 and the flowchart of FIG. 4, but the processing of the flowchart of FIG. 3 and the flowchart of FIG. The processing shown in the figure is executed in parallel by the CPU 100. That is, audio recording and audio reproduction can be performed simultaneously. However, although not shown in the flow diagram, when recording audio, the volume of the reproduced audio is kept low so as not to disturb the user's thought. Note that the processing shown in the flowcharts of FIGS. 3 and 4 is an example, and the order of the processing steps may be changed, or the order of those that can implement the invention may be changed.

  First, the CPU 100 of the PDA 11 determines whether or not there is a voice input in step S1 of FIG. The determination as to whether or not there is an audio input is made based on an audio signal input from the bone conduction microphone 12 via the audio input I / F 107. In this determination, an attempt is made to cut out a section (voice section) that is a voice signal (sound) from the voice signal input to the audio input I / F 107, and when the voice section is cut out, it is determined that there is a voice input. .

  In audio segmentation, the position where the time when the level of the audio signal exceeds the threshold exceeds 0.3 seconds is regarded as the start of speech (start of the audio interval), and the time when the level of the audio signal falls below the threshold after that is 1.0 second. The speech section is determined by regarding the position that starts to fall below the threshold when exceeding the threshold as speech stop (speech section end). However, after it is determined that the utterance has started, the level of the voice signal falls below the threshold within 0.5 seconds, and the duration of the state after the voice signal level falls below the threshold is 2.0 seconds or longer. In such a case, the determination that the utterance has been started is reversed, and the speech section is not considered.

  In this way, when the voice section is cut out from the voice signal and it is determined that there is voice input (step S1: YES), the CPU 100 then acquires the voice signal of the voice section from the voice signal in step S3. To generate voice information. The generated audio information is stored (recorded) in the flash memory 103. This audio information is, for example, an MP3 (MPEG Audio Layer-3) file.

In step S5, information (direction information) indicating the orientation of the face in the user's virtual space when the user's voice is input to the bone oscillating microphone 12 (when voice information is recorded) is obtained from the triaxial geomagnetic sensor 14. The direction information is generated based on the output, and the direction information is stored in the flash memory 103 in association with the voice information previously stored. Here, as shown in FIG. 5, the virtual space is a horizontal arc C centered on the user's head, and the arc C has a 90 ° spread from the front, which is the reference of the virtual space, to the left and right. Have. In addition, the front which becomes the reference of the virtual space is the use detected by the three-axis geomagnetic sensor 14 when the audio recording / reproducing apparatus 10 is turned on and the user performs a predetermined operation on the touch panel 105 of the PDA 11. This is the direction the person's face is facing.

The information (direction information) indicating the orientation of the face in the user's virtual space is an angle α formed by the reference (front) line of the virtual space and the straight line indicated by the orientation of the user's face. The angle α indicating the orientation of the user's face is represented by [I _x , I _y , I _z ] ′ as the output of the triaxial geomagnetic sensor 14 when the user's face is facing the reference direction (front direction). Assuming that the output of the triaxial geomagnetic sensor 14 at the time of measuring the orientation of the face is [M _x , M _y , M _z ] ′, _Equation 1 and Equation 2 can be obtained.

  According to the experiment, even if the user intends to input the voice with the face facing in one direction, the angle α is blurred by 5 ° to 7 ° between the start of the input and the end of the input. It was confirmed that there is. Therefore, the direction information indicating the direction of the user's face at the time of voice input is the average value of the face direction (angle α) from the start of voice input to the end of voice input (voice section). . The triaxial geomagnetic sensor 14 outputs a measurement value every 100 milliseconds.

  In step S7, icon sound information is generated and stored in the flash memory 103 in association with the sound information and direction information stored earlier. The icon audio information is, for example, an MP3 (MPEG Audio Layer-3) file. This icon sound information is a copy of the icon sound information recorded in the flash memory 103 in advance. The icon sound is a symbolized sound, for example, a simple bell sound. However, it is preferable that the musical sound of the bell sound includes 1000 Hz or more so that sound source localization is facilitated by the difference in volume between the left and right ears of the user.

  Next, in step S9, the modulation parameter of the icon sound to be reproduced is determined based on the icon sound information, and is stored in the flash memory 103 in association with the sound information previously stored. Here, the modulation parameter is for making the frequency of the icon sound different for each icon sound in accordance with the position of the icon sound in the virtual space so that the user can easily identify the icon sound to be reproduced. The modulation parameter is calculated based on the absolute value of the angle α (average value) of the user's face at the time of recording the sound, and the frequency of the icon sound is between 1.0 and 2.0 times according to Equation 3. Determined by conversion. By this frequency conversion, icon voices having a difference of 1 octave at maximum are generated.

  In this way, by recording the voice uttered by the user on the PDA 11, the flash memory 103 stores one piece of voice information, voice information, direction information, and icon voice information as shown in FIG. The modulation parameters are recorded.

  On the other hand, when reproducing audio information, the CPU 100 of the PDA 11 specifies the direction in which the user's face is facing in the virtual space based on the output of the triaxial geomagnetic sensor 14 in step S11 of FIG. That is, it is specified how much angle (α) the user's face is facing from the virtual space reference.

  Next, in step S13, for all audio information stored in the flash memory 103, how much the audio information is located in the virtual space with respect to the direction of the user's face specified in step S11. The relative direction (angle δ) (see FIG. 7) is calculated. This relative direction (angle δ) is calculated based on the direction information recorded in the flash memory 103 in association with the audio information and the information on the angle of the user's face orientation specified in step S11.

  In step S15, of the audio information recorded in the flash memory 103, the audio information whose relative direction calculated in step S13 is within a range of ± 10 °, for example, is reproduced and the audio output I / O is reproduced. The data is output to the stereo headphones 13 via F108. At this time, the sound information is reproduced so that the user can specify the position of the sound information in the virtual space, that is, the sound source can be localized.

  Specifically, the sound information is reproduced so that sound source localization can be performed based on the time difference between the arrival of sound to the left and right ears of the user and the sound volume difference.

  As shown in FIG. 7, when the distance between the left and right ears of the user is h [cm] and the distance from the left and right ears to the arc C, which is a virtual space, is c [cm], the sound source (voice information) The distance dL [cm] to the left ear and the distance dR [cm] to the right ear are determined by Equations 4 and 5, respectively. Then, the time required for the voice to reach the left and right ears is derived at dL / 34 and dR / 34, respectively, using the sound velocity 34 [milliseconds / cm], and the time difference is calculated based on these.

  FIG. 7A shows the case where the user's face is facing the front of the virtual space, and FIG. 7B shows the direction of the angle α from the front of the virtual space. The case where it faces is shown. In these figures, the angle γ indicates the angle from the front direction of the virtual space to the direction in which the audio information is located, and the angle δ indicates the direction in which the audio information is located from the direction in which the user's face is facing. The angle up to is shown.

  The distance h [cm] between the left and right ears of the user and the distance c [cm] from the user to the arc C are recorded in the ROM 101 of the PDA 11. This is the same in other embodiments.

On the other hand, if the volume difference is a general sound source, the total volume [db] reaching the left and right ears is not always the same when the localization changes even if the volume is always the same. However, the calculation of the wraparound voice or the like is not performed, and the calculation is simply based on the attenuation rate “atn” due to the distance. When the initial value (ATN _init ) of the attenuation rate is “0.8c ² ”, the attenuation rate “atn” based on the distance “d” between the sound source and the ear is calculated by Equation 6. However, d _min = c. For example, if c = 15 and h = 15, then ATN _init = 180.

  Next, in step S17, icon audio information is modulated for audio information whose relative direction calculated in step S13 is outside the range of ± 10 °, for example, of the audio information recorded in the flash memory 103. Modulated and reproduced based on the parameters and output to the stereo headphones 13 via the audio output I / F 108. As described above, the icon sound information is also reproduced so that sound source localization can be performed based on the time difference of sound arrival and the sound volume difference between the left and right ears of the user. Note that the reproduction of the audio information in step S15 and the reproduction of the icon audio information in step S17 are performed simultaneously in parallel. Also, the audio information and the icon audio information are repeatedly reproduced with a silent section of 500 milliseconds in between.

In this way, in the audio recording / reproducing apparatus 10 of the first embodiment, as shown in FIG. 8, the arc C in the virtual space is located within a range of ± 10 ° from the direction in which the user's face is facing. The audio information (audio information indicated by the black audio icon 201 in the figure) is reproduced as it is so that the sound source can be localized, and is located outside the range of ± 10 ° from the direction of the user's face. Audio information (audio information indicated by a white audio icon 202 in the drawing) is reproduced in place of the audio information so that the icon audio information can be localized.
<Second Embodiment>
In the audio recording / reproducing apparatus 10 of the second embodiment, a virtual space to which audio information is added is a three-dimensional space. That is, as the information (direction information) of the user's face orientation in the virtual space, the angle in the vertical direction is incorporated in addition to the same angle in the horizontal direction as in the first embodiment. In addition, the concept of layer is adopted, and the distance from the user (the center of the head) of the voice information in the virtual space is considered. In this embodiment, the layer is a hemispherical layer positioned at a certain distance with the center of the user's head concentric in the virtual space.

  Specifically, when recording new audio information (when adding it to the virtual space), when recording the new audio information, the layer to which the audio information belongs is the shallowest (closest distance) layer. For audio information that has already been recorded, if a predetermined time has elapsed since the time when the last audio information was recorded when recording new audio information, the layer to which the audio information belongs is deeper (the distance is Update to the far) layer. In other words, the already recorded audio information layer is a layer farther from the user.

Therefore, in the audio recording / reproducing apparatus 10 of the second embodiment, as shown in FIG. 9, it belongs to the layer designated by the user in the virtual space (layer 2 is designated in the example of FIG. 9). Audio information located within a range of ± 10 ° in the horizontal direction and within a range of ± 10 ° in the vertical direction from the direction in which the user's face is facing (the audio indicated by the black audio icon 201 in the figure) Information) is reproduced as it is so that sound source localization can be performed, and audio information belonging to a layer other than the layer specified by the user, and horizontal direction from the direction specified by the user but facing the user's face Voice information (voice information indicated by white voice icon 202 in the figure) that is outside the range of ± 10 ° and outside the range of ± 10 ° in the vertical direction is replaced with voice information. Plays sound source so that it can be localized.

  If the distance between the user's left and right ears is h [cm], the distance between adjacent layers is h / 2 [cm]. The distance from the user's ear to the latest layer is c [cm].

  In FIG. 9, the audio icon 202 is shown to be smaller for audio information positioned below the user's face direction and larger for audio information positioned above the face direction.

The configurations of the PDA 11, the bone conduction microphone 12, the stereo headphone 13, and the triaxial geomagnetic sensor 14 that constitute the audio recording / reproducing apparatus 10 are the same as those in the first embodiment.

Hereinafter, processing executed by the CPU 100 of the PDA 11 when recording audio and reproducing audio with the audio recording / reproducing apparatus 10 will be described with reference to the flowcharts shown in FIGS. 10 and 11. The processing for recording audio and the processing for reproducing audio are separately shown in the flowchart of FIG. 10 and the flowchart of FIG. 11, but the processing of the flowchart of FIG. 10 and the flowchart of FIG. The processing shown in the figure is executed in parallel by the CPU 100. That is, audio recording and audio reproduction can be performed simultaneously. However, although not shown in the flow diagram, when recording audio, the volume of the reproduced audio is kept low so as not to disturb the user's thought. Note that the processing illustrated in the flowcharts of FIGS. 10 and 11 is an example, and the order of the processing steps may be changed or the order of those that can implement the invention may be changed.

  When the process shown in the flowchart of FIG. 10 is executed and the voice uttered by the user is recorded in the PDA 11, the layer identification of one piece of voice information is stored in the flash memory 103 as shown in FIG. Information, audio information, direction information, icon audio information, modulation parameters, and time information are recorded.

  First, the CPU 100 of the PDA 11 determines whether or not there is an audio input in step S21 of FIG. The determination as to whether or not there is an audio input is made based on an audio signal input from the bone conduction microphone 12 via the audio input I / F 107. In this determination, an attempt is made to cut out a section (voice section) that is a voice signal (sound) from the voice signal input to the audio input I / F 107, and when the voice section is cut out, it is determined that there is a voice input. . Note that the method of segmenting the speech section is the same as that in the first embodiment.

  If it is determined that a voice section is cut out from the voice signal and there is a voice input (step S21: YES), then the CPU 100 performs voice recognition based on the voice signal in the voice section in step S23, and the user. It is determined whether or not the voice input by is a voice designating a layer. When the user designates a layer, the user is required to speak a predetermined phrase, for example, “designate layer 2”, and the CPU 100 determines that the voice of the voice signal in the voice section is the phrase. Is determined by voice recognition, and it is determined whether or not the input voice specifies a layer. If it is determined that the sound is for designating a layer (step S23: YES), the processing is taken over for sound reproduction in step S25. Note that the sound reproduction processing in step S25 is processing in step S59 and subsequent steps in FIG.

  On the other hand, if it is determined that the voice is not a layer designating voice (step S23: NO), then in step S27, the current time from the clock circuit 104 (at the time of recording voice information (more accurately, at the start of the voice section)). Is recorded in the flash memory 103.

  In step S29, the time information recorded lastly (previously) in association with the audio information recorded in the flash memory 103 is referred to, and the current time acquired from the clock circuit 104 in step S27 is For example, it is determined whether or not one hour has passed since the time when the voice information was last recorded. If there is no time information recorded in the flash memory 103 in association with the voice information, that is, if no voice information has been recorded yet, it is determined that one hour has not passed.

  If it is determined that one hour has passed (step S29: YES), the layer identification information indicating the layer of the audio information already recorded in the flash memory 103 is updated in step S31. That is, the number of layers (layer 1, layer 2, layer 3, etc.) indicated by the layer identification information is incremented by one. Note that the greater the number of layers, the farther the layer is from the user. On the other hand, if it is determined that one hour has not elapsed (step S29: NO), step S31 is skipped.

  In step S33, layer identification information of the audio information to be recorded at present, that is, information indicating the initial value of layer 1 is stored in the flash memory 103 in association with the time information stored in step S27.

  Next, CPU100 acquires the audio | voice signal of an audio | voice area from an audio | voice signal by step S35, and produces | generates audio | voice information. The generated audio information is stored (recorded) in the flash memory 103 in association with the time information and the layer identification information stored earlier. This audio information is, for example, an MP3 (MPEG Audio Layer-3) file.

In step S <b> 37, information (direction information) indicating the orientation of the face in the virtual space of the user when the user's voice is input to the bone oscillating microphone 12 (when recording voice information) is stored in the triaxial geomagnetic sensor 14. It is generated based on the output and stored in correspondence with the voice information previously stored in the flash memory 103. Here, the virtual space is a series of hemispherical layers centered on the user's head, as shown in FIG. In addition, the front which becomes the reference of the virtual space is the use detected by the three-axis geomagnetic sensor 14 when the audio recording / reproducing apparatus 10 is turned on and the user performs a predetermined operation on the touch panel 105 of the PDA 11. This is the direction the person's face is facing.
<Third Embodiment>
As shown in FIG. 13, the audio recording / reproducing apparatus 10a of the third embodiment includes a camera 16 having a fisheye lens in addition to the configurations of the audio recording / reproducing apparatus 10 of the first and second embodiments. Yes. The camera 16 shoots a peripheral image in the front direction of the user at the time of recording sound, and is lowered from the neck by a string (not shown) so as to be positioned in front of the user's torso. On the other hand, as shown in FIG. 14, the PDA 11 a includes an image signal input I / F 110 for inputting an image signal from the camera 16 in addition to the configuration of the PDA 11 of the first embodiment and the second embodiment. Yes. In addition, about the audio recording / reproducing apparatus 10a and PDA11a, the same code | symbol is used about the same structure as the audio recording / reproducing apparatus 10 and PDA11.

  The information (direction information) indicating the orientation of the face in the user's virtual space is the horizontal angle α (FIG. 5) formed by the reference (front) line of the virtual space and the straight line indicating the orientation of the user's face. Reference) and the vertical angle β (see FIG. 12). As described above, since the direction of the user's face is blurred between the start of the voice input and the end of the voice input, the direction indicating the direction of the user's face when the voice is input. The information includes the average value of the horizontal face direction (angle α) and the average value of the vertical face direction (angle β) from the start of voice input to the end of voice input (voice section). To do.

  In step S39, icon audio information is generated and stored in the flash memory 103 in association with the previously stored audio information, direction information, and the like. The icon sound information is a copy of the sound information of the bell sound previously recorded in the flash memory 103, as described above.

  Next, in step S41, a modulation parameter of the icon sound to be reproduced is determined based on the icon sound information, and is stored in the flash memory 103 in association with the sound information previously stored. Here, as described above, the modulation parameter is for changing the frequency of the icon sound in accordance with the position of the icon sound in the virtual space so that the user can easily identify the icon sound to be reproduced. .

  On the other hand, when reproducing audio information, the CPU 100 of the PDA 11 determines whether or not there is an audio input in step 51 of FIG. This determination method is the same as the determination in step 21 of FIG. If it is determined that no sound is input (step S51: NO), the number of layers to which the sound information to be reproduced belongs is set to an initial value in step S53. That is, it is considered that layer 1 is designated.

  On the other hand, if it is determined that there is a voice input (step S51: YES), then in step S55, it is determined by voice recognition whether or not the input voice is a voice designating a layer. This determination method is the same as the determination in step S23 of FIG. If it is determined that the sound is not a sound for designating a layer, that is, a sound to be recorded (step S55: NO), the processing is taken over for the sound recording in step S57. The audio recording process is a process after step S27 in the flowchart of FIG.

  On the other hand, if it is determined that the voice specifies a layer (step S55: YES), the number of layers is specified based on the result of speech recognition of the input voice in step S59, and the number of layers specified in step S61 is determined. This is set as the number of layers to which audio information to be reproduced belongs. That is, it is considered that the number of layers that have been voice-recognized is designated.

  Next, in step S63, based on the output of the triaxial geomagnetic sensor 14, the direction in which the user's face is facing in the virtual space is specified. That is, it is specified how much angle (α) the user's face faces in the horizontal direction from the reference of the virtual space and how much angle (β) faces in the vertical direction.

  In step S65, for all audio information stored in the flash memory 103, the relative direction of the direction in which the audio information is located in the virtual space with respect to the direction of the user's face specified in step S63 is calculated. This relative direction is based on the direction information (horizontal angle and vertical angle) recorded in the flash memory 103 in association with the audio information and the information on the angle of the face of the user specified in step S63. Is calculated.

  In step S67, the audio information recorded in the flash memory 103 belongs to the layer set (designated) in step S53 or step S61, and the relative directions calculated in step S65 are the horizontal direction and the vertical direction. Both audio information within ± 10 ° is reproduced and output to the stereo headphones 13 via the audio output I / F 108. At this time, the sound information is reproduced so that the user can specify the position of the sound information in the virtual space, that is, the sound source can be localized. Specifically, the sound information is reproduced so that sound source localization can be performed based on the time difference between the arrival of sound to the left and right ears of the user and the sound volume difference.

  The calculation method of the time difference of voice arrival and the volume difference of the voice can be realized by the same method as described in the first embodiment, but the process of reproducing the voice information so that sound source localization can be performed is Makoto Ouchi. , "Effects of spatial recognition ability training for visually handicapped by 3D acoustic VR edutainment system" by Yukio Iwatani, Yoichi Suzuki and Tetsuya Munakata, 3rd Science and Technology Forum FIT 2004 Information Technology Letters, pp. 283-284, 2004, Nobuyuki Iwanaga, Kennari Sakamoto, Jun Kobayashi, Takao Onoe, Isao Shirakawa "Headphone Stereo Out-of-head Sound Field Expansion Method for Embedded Systems and its Implementation", 17th Digital Signal Processing Symposium, B2- This is realized by a difference or change in sound volume, transmission time, and frequency characteristics based on the method proposed in US Pat.

Next, in step S69, of the audio information recorded in the flash memory 103, the audio information belonging to other than the designated layer and the relative direction belonging to the designated layer but calculated in step S65 is the horizontal direction. For the audio information that is out of the range of ± 10 ° in the vertical direction, the icon audio information is modulated and reproduced based on the modulation parameter, and is output to the stereo headphones 13 via the audio output I / F 108. As described above, this icon sound information is also reproduced so that sound source localization can be performed based on the above-described method. Note that the reproduction of the audio information in step S67 and the reproduction of the icon audio information in step S69 are simultaneously performed in parallel. Also, the audio information and the icon audio information are repeatedly reproduced with a silent section of 500 milliseconds in between.
<Third Embodiment>
As shown in FIG. 13, the audio playback device 10 a of the third embodiment includes a camera 16 having a fisheye lens in addition to the configurations of the audio playback devices 10 of the first embodiment and the second embodiment. The camera 16 shoots a peripheral image in the front direction of the user at the time of recording sound, and is lowered from the neck by a string (not shown) so as to be positioned in front of the user's torso. On the other hand, as shown in FIG. 14, the PDA 11 a includes an image signal input I / F 110 for inputting an image signal from the camera 16 in addition to the configuration of the PDA 11 of the first embodiment and the second embodiment. Yes. In addition, about the audio reproduction apparatus 10a and PDA11a, the same code | symbol is used about the same structure as the audio reproduction apparatus 10 and PDA11.

  In the audio recording / reproducing apparatus 10a of the third embodiment, audio information is added to and recorded on an image (peripheral image) around the user taken by the camera 16. More specifically, the orientation of the user's face when recording audio information is detected, the orientation of the face is converted into a position in the surrounding image, and the audio information is information on the position in the surrounding image (position information). And peripheral image information (peripheral image information). When newly recording audio information, the peripheral image added with the last (previous) recorded audio information is compared with the peripheral image captured by the camera 16 when recording the new audio information. If it is determined that the environment of the person has changed, new audio information is added to the surrounding image newly taken by the camera 16, and if not determined that the environment has changed, the last (previous) recorded audio information is stored. New audio information is added to the added peripheral image. That is, in the third embodiment, the peripheral image captured by the camera 16 serves as a layer.

  The peripheral image information (peripheral image information) is stored in the flash memory 103 in association with image identification information for identifying the peripheral image information, as shown in FIG. Are stored in association with each other. Thereby, the peripheral image to which the audio information is added is specified.

Hereinafter, processing executed by the CPU 100 of the PDA 11a when audio is recorded by the audio recording / reproducing apparatus 10a will be described with reference to a flowchart shown in FIG. Note that when the processing shown in FIG. 15 is executed and the voice uttered by the user is recorded in the PDA 11a, the flash memory 103 stores one piece of voice information, image identification information, voice, as shown in FIG. Information, position information, and time information are recorded. Separately from these, image identification information and peripheral image information are recorded in the flash memory 103. Furthermore, the processing shown in the flowchart of FIG. 15 is an example, and the order of the processing steps may be changed for those that can implement the invention even if they are changed.

  First, the CPU 100 of the PDA 11a determines whether or not there is a voice input in step S81 of FIG. The determination as to whether or not there is an audio input is made based on an audio signal input from the bone conduction microphone 12 via the audio input I / F 107. In this determination, an attempt is made to cut out a section (voice section) that is a voice signal (sound) from the voice signal input to the audio input I / F 107, and when the voice section is cut out, it is determined that there is a voice input. . Note that the method of segmenting the voice section is the same as in the first embodiment.

  If it is determined that the voice section is cut out from the voice signal and there is voice input (step S81: YES), the CPU 100 then acquires the voice signal of the voice section from the voice signal and generates voice information in step S83. To do. The generated audio information is stored (recorded) in the flash memory 103. This audio information is, for example, an MP3 (MPEG Audio Layer-3) file.

  Next, in step S85, the direction in which the user's face is facing in the virtual space is specified based on the output of the triaxial geomagnetic sensor 14, and direction information is generated. That is, it is specified how much angle (α) the user's face faces in the horizontal direction from the reference of the virtual space and how much angle (β) faces in the vertical direction. Here, the virtual space is a space indicated by a peripheral image having an angle of view of 180 ° that is captured by the camera 16 having a fisheye lens. The direction information (horizontal angle α and vertical angle β) is recorded in the RAM 102.

  In step S87, an image signal is acquired from the camera 16 via the image signal input I / F 110, peripheral image information in a direction in which the user's body is facing is generated, and this is temporarily recorded in the RAM 102. This peripheral image information is, for example, a JPEG file.

  Next, in step S89, information on the current time (at the time of recording the voice information (more accurately, at the start of the voice section)) is acquired from the clock circuit 104, and this corresponds to the voice information stored in step S83. And record it in the flash memory 103.

In step S91, the peripheral image information generated in step S87 and temporarily recorded in the RAM 102 is compared with the peripheral image information recorded last (previous) in the flash memory 103, and the environment surrounding the user is changed. Judge whether there is. The last (previous) peripheral image information is associated with the sound information if the sound information with the latest time indicated by the associated time information is identified from the sound information associated with the image identification information. It can be determined based on the image identification information. (The audio information stored in step S83 is excluded because the image identification information has not yet been associated.)
The current time information acquired from the clock circuit 104 when the peripheral image information is recorded in the flash memory 103 is recorded together with the peripheral image information, and the last (previous) peripheral image information is specified based on this time information. You may make it do.

  In addition, the determination of the presence or absence of the environmental change based on the peripheral image information is, for example, when the changes per unit time of the luminance and color information in the two peripheral image information are totaled, It can be realized by judging that the environment is changing. In addition, a scene change detection method in a moving image disclosed in JP-A-6-251147 (video feature processing method) can also be used.

  If it is determined that there is a change in the environment around the user (step S91: YES), the (new) peripheral image information once recorded in the RAM 102 is recorded in the flash memory 103 in step S93. At this time, image identification information for identifying the peripheral image information is assigned to the peripheral image information, and the peripheral image information is recorded in the flash memory 103 in association with the image identification information.

  Next, in step S95, the image identification information of the peripheral image information recorded in the flash memory 103 in step S93 is recorded in the flash memory 103 in association with the audio information stored in step S83. As a result, the newly recorded peripheral image information is associated with the audio information.

  On the other hand, if it is determined that there is no change in the surrounding environment of the user (step S91: NO), the (existing) peripheral image information recorded in the flash memory 103 at the end (used in the determination of step S91). The image identification information (one peripheral image information) is recorded in the flash memory 103 in association with the audio information stored in step S83, so that the last recorded (above) peripheral image information and audio information are associated with each other.

  In step S99, the horizontal direction angle α and the vertical direction angle β, which are the direction information generated in step S85 and recorded in the RAM 102, are set as the peripheral image (step S95 or step S97). Information) is converted into coordinates (x, y) indicating the position, and the position information (coordinates (x, y)) is recorded in the flash memory 103 in association with the previously stored audio information. The transformation from angle α and angle β indicating the orientation of the face to the coordinates (x, y) on the image is as follows: Hirotake Yamazoe, Akira Utsumi, Kenichi Hosaka, Masahiko Yachida, “A body-mounted camera system for head-pose estimation and user-view image synthesis, "Available online at www.sciencedirect.com ScienceDirect Image and Vision Computing 25 (2007) 1848-1855.

Next, processing executed by the CPU 100 of the PDA 11a when the audio recording / reproducing apparatus 10a visually browses audio information and reproduces audio will be described with reference to a flowchart shown in FIG. Note that the processing shown in the flowchart of FIG. 16 is an example, and the order of each processing step may be changed if the order of processing steps can be changed and the invention can be implemented.

  When the user performs an operation for browsing audio information on the touch panel 105 of the PDA 11a, the CPU 100 displays all peripheral images (information) (layers) recorded in the flash memory 103 in step S101. It is displayed in three dimensions on 106. At this time, as shown in FIG. 17, the oldest time recorded in the flash memory 103 is displayed below (back or far), and the newest time is displayed on top of this (close). In addition, a sticky note for selecting the peripheral image is displayed on the upper right of each peripheral image.

  The time when the peripheral image information is recorded in the flash memory 103 can be specified based on the time information of the audio information associated with the peripheral image information, but as described above, it is recorded in the peripheral image information itself. Information indicating the time of time may be recorded in association with each other.

  In step S103, an icon 200 indicating audio information associated with the image identification information of the peripheral image information of the peripheral image is displayed on the peripheral image displayed on the display 106. At this time, the icon 200 is displayed at a position on the peripheral image specified by the position information (x coordinate, y coordinate) recorded in the flash memory 103 in association with the corresponding audio information.

  Next, when the user performs an operation of selecting a peripheral image (layer) on the touch panel 105, that is, an operation of selecting one of the sticky notes displayed on the display 106 (an operation of clicking (tapping) the sticky note). The CPU 100 determines in step S105 that a layer has been selected.

  In step S107, only the selected peripheral image (layer) is displayed in three dimensions on the display 106, and the audio information icon 200 added to the peripheral image selected in step S109, that is, the selected peripheral image is displayed. The audio information icon 200 recorded in association with the image identification information is displayed on the peripheral image displayed on the display 106. Also at this time, the icon 200 is displayed at a position based on the position information recorded in the flash memory 103 in association with the corresponding audio information.

  As described above, when the user performs an operation of clicking (tapping) the icon 200 on the touch panel 105 in a state where the peripheral image and the icon 200 are displayed on the display 106, the CPU 100 displays the icon in step S111. It is determined that 200 is clicked (step S111: YES), and the audio information corresponding to the clicked icon 200 is reproduced in step S113.

  When the user performs an operation of dragging the icon 200 on the touch panel 105 and changing its position, the CPU 100 determines that the drag operation of the icon 200 has been performed in step S115 (step S115: YES). In step S117, the position of the icon 200 on the peripheral image is changed according to the user's drag operation. At this time, the position information (x coordinate, y coordinate) recorded in the flash memory 103 is changed according to the changed position of the icon 200.

  Further, when the user performs an operation of dragging the surrounding image on the touch panel 105 and changing the direction thereof, the CPU 100 determines that the dragging operation of the surrounding image has been performed in step S119 (step S119: YES), and is used. The direction of the peripheral image displayed on the display 106 is changed according to the drag operation of the person.

When the user performs an operation to end browsing of the audio information on the touch panel 105, the CPU 100 determines in step S123 that the browsing end operation has been performed, and ends the process.
<Fourth Embodiment>
In the audio recording / reproducing apparatus 10a of the third embodiment, the surrounding image to which the audio information is added is updated when the environment around the user changes. However, in the audio recording / reproducing apparatus 10a of the fourth embodiment, The peripheral image to which the audio information is added is also updated when a predetermined time has elapsed. Note that the configuration of the audio recording / reproducing apparatus 10a is the same as that of the third embodiment.

Hereinafter, processing executed by the CPU 100 of the PDA 11a when audio is recorded by the audio recording / reproducing apparatus 10a will be described with reference to a flowchart shown in FIG. Note that when the processing shown in the flowchart of FIG. 18 is executed and the voice uttered by the user is recorded on the PDA 11a, the flash memory 103 stores one piece of voice information for image identification as shown in FIG. Information, audio information, position information, and time information are recorded. Separately from these, image identification information and peripheral image information are recorded in the flash memory 103. Furthermore, the processing shown in the flowchart of FIG. 18 is an example, and the order of the processing steps may be changed as long as the invention can be implemented even if the order is changed.

  First, the CPU 100 of the PDA 11a determines whether or not there is an audio input in step S131 of FIG. The determination as to whether or not there is an audio input is made based on an audio signal input from the bone conduction microphone 12 via the audio input I / F 107. In this determination, an attempt is made to cut out a section (voice section) that is a voice signal (sound) from the voice signal input to the audio input I / F 107, and when the voice section is cut out, it is determined that there is a voice input. . Note that the method of segmenting the voice section is the same as in the first embodiment.

  If it is determined that a voice section is cut out from the voice signal and there is voice input (step S131: YES), the CPU 100 then obtains a voice signal of the voice section from the voice signal and generates voice information in step S133. To do. The generated audio information is stored (recorded) in the flash memory 103. This audio information is, for example, an MP3 (MPEG Audio Layer-3) file.

  Next, in step S135, based on the output of the triaxial geomagnetic sensor 14, the direction in which the user's face is facing in the virtual space is specified, and direction information is generated. That is, it is specified how much angle (α) the user's face faces in the horizontal direction from the reference of the virtual space and how much angle (β) faces in the vertical direction. Here, the virtual space is a space indicated by a peripheral image having an angle of view of 180 ° that is captured by the camera 16 having a fisheye lens. The direction information (horizontal angle α and vertical angle β) is recorded in the RAM 102.

  In step S137, an image signal is acquired from the camera 16 via the image signal input I / F 110, peripheral image information in a direction in which the user's body is facing is generated, and this is temporarily recorded in the RAM 102. This peripheral image information is, for example, a JPEG file.

  Next, in step S139, the current time information (at the time of recording the voice information (more precisely, at the start of the voice section)) is acquired from the clock circuit 104, and this corresponds to the voice information stored in step S133. And record it in the flash memory 103.

  In step S141, the time information recorded in the flash memory 103 in step S139 and the time information recorded last in association with the audio information associated with the image identification information in the flash memory 103 (that is, the previous audio information). And the current time (the time indicated by the time information recorded in the flash memory 103 in step S139 (more precisely, the start time of the voice section)) is compared with the previous voice. For example, it is determined whether or not 30 minutes or more have elapsed since the information was recorded in the flash memory 103.

  If it is determined that 30 minutes or more have not passed (step S141: NO), the flash memory is associated with the image information stored in the flash memory 103 at last (previous) in the image identification information stored in step S133. 103. Thus, the last (previous) recorded (existing) peripheral image information and the audio information are associated with each other.

  On the other hand, if it is determined that 30 minutes or more have elapsed (step S141: YES), it is further determined in step S145 whether, for example, 60 minutes or more have elapsed since the previous audio information was recorded in the flash memory 103. To do.

  If it is determined that 60 minutes or more have elapsed (step S145: YES), the peripheral image information once recorded in the RAM 102 is recorded in the flash memory 103 in step S147. At this time, image identification information for identifying the peripheral image information is assigned to the peripheral image information, and the peripheral image information is recorded in the flash memory 103 in association with the image identification information.

  Next, in step S149, the peripheral image information newly recorded by recording the image identification information of the peripheral image information recorded in the flash memory 103 in step S147 in the flash memory 103 in association with the audio information stored in step S133. Correlate information with audio information.

  On the other hand, if it is determined that 60 minutes or more have not elapsed (step S145: NO), in step S151, the peripheral image information generated in step S137 and once recorded in the RAM 102 and finally (previous) recorded in the flash memory 103 are recorded. The surrounding image information is compared to determine whether there is a change in the environment around the user.

  If it is determined that there is a change in the environment around the user (step S151: YES), the above-described processing is executed in steps S147 and S149.

  On the other hand, if it is determined that there is no change in the surrounding environment of the user (step S151: NO), the peripheral image information recorded in the flash memory 103 at the end (one peripheral image used for the determination in step S151). (Information) image identification information is recorded in the flash memory 103 in association with the audio information stored in step S133, so that the last (previous) recorded (existing) peripheral image information and audio information are associated with each other.

  In step S155, the horizontal angle α and the vertical angle β, which are the direction information generated in step S135 and recorded in the RAM 102, are associated with the audio information in step S143, step S149, or step S153. Then, it is converted into coordinates (x, y) in the peripheral image (information), and the positional information (coordinates (x, y)) is recorded in the flash memory 103 in association with the previously stored audio information.

Note that the processing executed by the CPU 100 of the PDA 11a when the audio information is visually browsed and reproduced by the audio recording / reproducing apparatus 10a has been described with reference to the flowchart of FIG. 16 in the third embodiment. Since it is the same as the process, a duplicate description is omitted.
<Other embodiments>
As other embodiments, layers defined by the layer identification information (number of layers) in the second embodiment and the distance from the user in the virtual space with time (referred to as “time layer”), the third and third A mode in which a layer defined by a peripheral image (referred to as an “image layer”) in the fourth embodiment can be considered. In this case, it is preferable to store the peripheral image information in association with time information (time information) and layer identification information (number of layers) at the time of shooting. In this way, the number of layers that are serial numbers can be assigned to a set of layers composed of a time layer and an image layer according to the time indicated by the time information, and can be handled as a series of layers.

  Also, in this case, the audio information associated with the peripheral image information is recorded in association with the information on the user's face direction (direction information) at the time of recording the sound, and the sound source is based on this direction information at the time of reproduction. It is better to play it so that it can be localized. Furthermore, when the audio information is visually browsed as in the third and fourth embodiments, a pseudo peripheral image prepared in advance corresponding to the time layer is displayed on the display 106 and the audio is displayed thereon. An icon 200 corresponding to the information may be displayed.

  The embodiment described above may be variously modified within the scope of the technical idea of the present invention. For example, in the above-described embodiment, the orientation of the user's face is detected only by the triaxial geomagnetic sensor, but may be detected by two of the triaxial geomagnetic sensor and the triaxial acceleration sensor. Further, the orientation of the face may be detected by processing an image obtained by photographing the user's face.

  Furthermore, the direction in the virtual space is designated by the orientation of the user's face, but instead, the direction may be designated by the orientation of the user's line of sight or pointing. To identify the user's line of sight, H. Yamazoe, A. Utsumi, T. Yonezawa, S. Abe. “Remote Gaze Direction Estimation with a Single Camera Based on Facial-Feature Tracking,” ETRA2008, pp. The method proposed in 245-250, 2008 can be used.

Further, although the bone conduction microphone 12 is used for voice input by the user, a NAM (non-audible tweeting) microphone may be used instead. The NAM microphone is a microphone that can collect unvoiced exhalation sounds that are not accompanied by vocal cord vibrations by acoustic filter characteristics due to the movement of the speech organs and that are conducted mainly through the soft tissue of the human head. By using this NAM microphone, it is possible to input voice to the voice recording / reproducing apparatus 10, 10a without hesitation even in places such as trains where people are concerned.

  In order to input voice without hesitation even in places where people are worried, a throat microphone, which is a system that is attached to the throat and transmits the vibration of the vocal cords as voice, may be employed.

  In the above-described embodiment, the layer is an arc or a hemisphere, and the audio information belongs to a layer that is far from the user as time passes. However, the layer is not necessarily centered on the user. It does not have to be a circular arc or hemispherical surface, and the distance from the user for each layer does not have to be far from a concentric circle. In other words, the layer may be a plane, and may be configured to be equidistant around the user. Furthermore, the layer has no spatial spread and is a conceptual distinction unit that simply summarizes the audio information to which it belongs. Also good.

It is an illustration figure which shows an example of the utilization condition of the audio | voice recording / reproducing apparatus of this invention. It is a block diagram which shows an example of a structure of the portable information terminal which comprises the audio | voice recording / reproducing apparatus of this invention. It is a flowchart which shows an example of the process which CPU of a portable information terminal performs at the time of recording of an audio | voice. It is a flowchart which shows an example of the process which CPU of a portable information terminal performs at the time of reproduction | regeneration of an audio | voice. It is an illustration figure explaining the direction where the user's face is facing in virtual space. It is an illustration figure which shows an example of a structure of the data recorded on the flash memory of a portable information terminal. It is an illustration figure explaining the sound source localization of audio | voice information. It is an illustration figure explaining reproduction | regeneration of audio | voice information. It is an illustration figure explaining reproduction | regeneration of audio | voice information. It is a flowchart which shows an example of the process which CPU of a portable information terminal performs at the time of recording of an audio | voice. It is a flowchart which shows an example of the process which CPU of a portable information terminal performs at the time of reproduction | regeneration of an audio | voice. It is an illustration figure explaining the direction where the user's face is facing in virtual space. It is an illustration figure which shows an example of the utilization condition of the audio | voice recording / reproducing apparatus of this invention. It is a block diagram which shows an example of a structure of the portable information terminal which comprises the audio | voice recording / reproducing apparatus of this invention. It is a flowchart which shows an example of the process which CPU of a portable information terminal performs at the time of recording of an audio | voice. It is a flowchart which shows an example of the process which CPU of a portable information terminal performs at the time of browsing and reproduction | regeneration of audio | voice information. It is an illustration figure which shows an example of the display in the display of a portable information terminal. It is a flowchart which shows an example of the process which CPU of a portable information terminal performs at the time of recording of an audio | voice.

Explanation of symbols

10: Audio recording / reproducing device 11: PDA
DESCRIPTION OF SYMBOLS 12 ... Bone conduction microphone 13 ... Stereo headphone 14 ... Triaxial geomagnetic sensor 16 ... Camera 100 ... CPU
103 ... Flash memory 104 ... Clock circuit

Claims (16)

An audio recording / reproducing device for recording audio information in a recording device and reproducing the audio information from the recording device,
Voice information generating means for generating voice information based on the voice emitted by the user;
Recording direction specifying means for specifying the recording direction indicated by the user using the body during recording,
Recording means for recording the sound information generated by the sound information generating means in the recording device in association with the direction information indicating the recording direction;
Playback direction specifying means for specifying the playback direction indicated by the user using the body during playback, and the voice recorded in the recording device in association with the direction information indicating the recording direction corresponding to the playback direction An audio recording / reproducing apparatus comprising first audio reproducing means for reproducing information from the recording apparatus.   The audio recording / reproducing apparatus according to claim 1, wherein the first audio reproducing unit reproduces the audio information so that sound source localization is possible based on the direction information and distance information indicating a distance from the user to the audio information. Voice icon recording means for recording a symbolized voice icon indicating the presence of voice information in the recording device in association with the voice information and the direction information, and direction information indicating a recording time direction not corresponding to the playback time direction The audio recording / reproducing apparatus according to claim 1, further comprising audio icon reproducing means for reproducing the audio icon recorded in the recording apparatus in association with the audio information recorded in the recording apparatus in association with .   4. The audio recording / reproducing apparatus according to claim 3, wherein the audio icon reproducing means generates the audio icon so that sound source localization is possible based on the direction information and distance information indicating a distance from the user to the audio information. Before Kiki recording means, the two or more different layers of depth audio information, further recording and in association, with the layer designation accepting means further accepts a designation of the layer and the layer identification information indicating the layer,
The first sound reproducing means is a layer indicating the layer received by the layer designation receiving means in the sound information recorded in the recording device in association with the direction information indicating the recording time direction corresponding to the reproduction time direction. The audio recording / reproducing apparatus according to claim 1, wherein the audio information associated with the identification information is reproduced.   The audio recording / reproducing apparatus according to claim 5, wherein the distance from the user to the audio information is determined based on a layer depth indicated by the layer identification information.   The reproduction direction specifying means specifies any one of a direction indicated by a user's face direction, a direction indicated by a user's gaze direction, and a direction indicated by a user's pointing direction. The audio recording / reproducing apparatus according to any one of the above.   7. The recording time specifying means specifies any one of a direction indicated by a user's face direction, a direction indicated by a user's line of sight, and a direction indicated by a user's pointing direction. The audio recording / reproducing apparatus according to any one of the above. A program executed by a voice recording / reproducing computer for recording voice information in a recording device and reproducing the voice information from the recording device, wherein the program
Voice information generating means for generating voice information based on the voice emitted by the user;
Recording direction specifying means for specifying the recording direction indicated by the user using the body during recording,
Recording means for recording the sound information generated by the sound information generating means in the recording device in association with the direction information indicating the recording direction;
Playback direction specifying means for specifying the playback direction indicated by the user using the body during playback,
An audio recording / reproducing program for causing audio information recorded in the recording device to function as audio reproducing means for reproducing from the recording device in association with direction information indicating a recording time direction corresponding to the reproducing time direction. An audio recording / playback computer-executed method for recording audio information on a recording device and reproducing the audio information from the recording device,
A voice information generation step for generating voice information based on the voice emitted by the user;
Recording direction specifying step for specifying the recording direction indicated by the user using the body during recording,
A recording step of recording the audio information generated in the audio information generation step in the recording device in association with the direction information indicating the recording direction;
A playback direction specifying step for specifying the playback direction indicated by the user using the body during playback, and the voice recorded in the recording device in association with the direction information indicating the recording direction corresponding to the playback direction. An audio recording / reproducing method including an audio reproducing step of reproducing information from the recording device. An imaging unit that captures a peripheral image of the user at the time of recording; and a conversion unit that converts the recording direction to a position on the peripheral image captured by the imaging unit,
It said recording means in association records and the peripheral image and the position information of the audio information, the audio recording and reproducing apparatus according to any one of claims 1 to 8. A first environmental change for determining whether or not the environment of the user has changed by comparing the peripheral image captured by the imaging unit with the peripheral image associated with the sound information recorded last in the recording device A judgment means,
The recording means records audio information in association with a peripheral image taken by the photographing means when the first environment change judging means judges that the environment has changed, and the first environment changing means 12. The audio recording / reproducing apparatus according to claim 11, wherein the audio information is recorded in association with a peripheral image associated with the audio information last recorded in the recording apparatus when it is determined that the information has not changed. Time elapse judging means for judging whether or not a predetermined time has elapsed since the last recording of the audio information in the recording device;
A second environment change determination for determining whether or not the environment of the user has changed by comparing the peripheral image captured by the image capturing unit with the peripheral image associated with the audio information recorded last in the recording device. Further comprising means,
The recording unit records audio information in association with a peripheral image captured by the imaging unit based on a determination result of at least one of the time passage determination unit and the second environment change determination unit, or audio The audio recording / reproducing apparatus according to claim 11, wherein the information is recorded in association with a peripheral image to which audio information recorded last in the recording apparatus is associated. A display for displaying a peripheral image recorded in the recording device and displaying an icon indicating audio information recorded in association with the peripheral image at a position on the peripheral image based on the position information of the audio information means,
Icon designation means for accepting designation of an icon displayed on the display means;
The audio recording / reproducing apparatus according to claim 11, further comprising second audio reproducing means for reproducing audio information recorded in the recording apparatus indicated by the icon that has been designated by the icon designating means. The display means displays a plurality of peripheral images recorded in the recording means,
Further comprising image designation means for accepting designation of a peripheral image displayed on the display means,
15. The audio recording / reproducing apparatus according to claim 14, wherein the icon designating unit accepts designation of an icon displayed on a peripheral image for which designation is accepted by the image designating unit. Position designation means for accepting designation of an icon change position on a peripheral image displayed on the display means;
The voice recording / reproducing according to claim 14 or 15, further comprising changing means for changing the position information recorded in the recording device in association with the voice information of the icon based on the changed position received by the position specifying means. apparatus.

Images