JP2009122687A - Audio regeneration device and audio regeneration method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an audio regeneration device capable of controlling the regeneration of an audio data, while matched with an operation of a user. <P>SOLUTION: A feature extracting part 12 detects a rhythm of the audio data stored in a memory 11, and generates time information to be output to a main control part 13. The main control part 13 detects an operation of a casing of the audio regeneration device 1, based on a detection result from a sensor 17, when regenerating the audio data, and generates detection information, based on a detection result therein. The main control part 13 compares a time information sequence generated by the feature extracting part 12, with the detection information generated based on the detection result from the sensor 17, and determines the consistency therebetween. A regeneration control part 14 regenerates the audio data from a sound system 15 without changing the audio data, when determined to be consistent, and regenerates the audio data from the sound system 15 after adding a prescribed effect, when determined to be inconsistent. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an audio playback apparatus and an audio playback method for detecting a user operation timing in accordance with playback of audio data.

  2. Description of the Related Art Conventionally, a music playback device with game characteristics that detects an operation performed by a user based on music playback, evaluates the operation timing, and produces sound effects or controls screen display based on the evaluation result Is known (see, for example, Patent Document 1).

In such a music reproducing apparatus, an event data sequence synchronized with music is stored in advance, and event data is displayed while reproducing music. When the user performs an operation at the event data timing, the operation timing is evaluated by detecting a difference between the operation timing and the event data timing.
JP 2001-232058 A

  However, the conventional music playback device has a problem that an authoring process for creating an event data sequence synchronized with music in advance is required to perform predetermined control in accordance with the operation timing of the user. .

  The present invention has been made in view of the above circumstances, and an object of the present invention is to control reproduction of audio data in accordance with a user operation without authoring audio data in an apparatus that reproduces audio data. It is an object of the present invention to provide a possible audio playback apparatus and audio playback method.

  The present invention has been made in order to solve the above-described problems, and is a storage means for storing audio data, a predetermined musical feature on the reproduction time axis is extracted from the audio data, and a time indicating the musical feature is obtained. Feature extraction means for generating an information string, detection means for inputting an operation by a user and detecting an operation mode, playback means for playing back the audio data, and on the time axis of the operation mode detected by the detection means A comparison means comparing the detected information and the time information sequence generated by the feature extracting means, and during the reproduction of the audio data by the reproducing means, the comparison information is a result of the comparison by the comparing means. Control means for performing control for notifying the user that the detection information and the time information string do not match when they do not match the string, That is the audio playback device.

  Also, in the present invention according to the above-mentioned invention, the feature extraction unit extracts time width of time information in the case where there is the musical feature among time information constituting the time information sequence from the audio data. The predetermined time width is longer than the time width of the time information.

  Also, the present invention is characterized in that, in the above invention, the control means changes the reproduction state of the audio data based on the comparison result of the comparison means during reproduction of the audio data.

  In addition, according to the present invention, in the above invention, during the reproduction of the audio data, the control unit pauses reproduction of the audio data or controls the volume or controls the audio data based on the comparison result of the comparison unit. It is characterized by giving the effect of.

  Further, the present invention according to the above invention further comprises display means, and the control means changes the display on the display means based on the comparison result of the comparison means during reproduction of the audio data. It is characterized by.

  In the invention described above, the detection unit is an operation button that detects an operation by a user.

In the present invention according to the present invention, the detection means is an acceleration sensor that detects acceleration.
In the present invention according to the present invention, the detection means is a magnetic sensor for detecting a magnetic field.

According to the present invention, in the audio playback method in the audio playback device, a predetermined musical feature on the playback time axis is extracted from the audio data stored in the storage means, and a time information sequence indicating the musical feature is generated. A first step, a second step of inputting an operation by a user and detecting an operation mode, a third step of reproducing audio data stored in the storage means, and a detection in the second step A fourth step of comparing the detected information on the time axis of the operated mode with the time information sequence generated in the first step, and during the reproduction of the audio data in the third step, If the detection information does not match the time information string as a result of the comparison in step 4, the detection information and the time information string match. A fifth step of performing control for notifying that no user is an audio reproducing method characterized by having a.
Also, in the present invention according to the present invention, in the first step, a time width of time information in the case where there is the musical feature among time information constituting the time information sequence is extracted from the audio data in the first step. The predetermined time width is longer than the time width of the time information.

  According to the audio playback device and audio playback method of the present invention, it is possible to extract the musical features of the audio data by the feature extraction means and generate them as a time information sequence. Audio data can be controlled in accordance with the user's operation.

  In addition, during playback of audio data, the user can control playback such as changing the playback state of the audio data or changing the display content on the display unit based on the operation mode detected by the detecting means. In addition to listening to the audio data, it is possible to obtain an actual feeling that the user is participating in the reproduction of the audio data through an operation input to the detection means.

  In addition, a musical sound generating means for generating a musical sound according to the operation mode detected by the detecting means is provided, and the specific frequency band or musical instrument sound is erased from the audio data and reproduced, so that the erased frequency band or musical instrument sound is reproduced. It is possible to generate and generate a musical sound corresponding to the above in accordance with the user's operation by the musical sound generating means, and the user can participate in the performance of the audio data through the operation input to the detecting means.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a configuration diagram showing the configuration of the audio playback device according to the first embodiment of the present invention.

  In FIG. 1, a memory 11 (storage means) is a memory for storing audio data. Here, the audio data may be compressed by a predetermined compression method such as MPEG or AAC. The feature extraction unit 12 (feature extraction means) extracts musical features (rhythm, melody, etc.) of the audio data stored in the memory 11 and outputs them as a time information string. The detailed operation of the feature extraction unit 12 will be described later with reference to FIGS.

  The main control unit 13 (comparison means) determines the user operation timing based on the data input from the feature extraction unit 12 and the interface 18 and outputs the determination result to the reproduction control unit 14. The main control unit 13 controls the reproduction control unit 14 based on a user operation instruction input from the operation unit 16.

  The reproduction control unit 14 reads music data stored in the memory 11, controls reproduction conditions based on the determination result input from the main control unit 13, and reproduces the sound from the sound system 15. The control unit 141 (control unit) outputs a reproduction condition instruction to each unit in the reproduction control unit 14 based on the determination result input from the main control unit 13.

  The read control unit 142 reads audio data stored in the memory 11 based on an instruction from the control unit 141 and outputs the audio data to the filter unit 143. If the audio data is compressed, the read control unit 142 performs expansion processing.

  The filter unit 143 (erasing unit) performs filter processing such as cutting a predetermined frequency component on the audio data based on an instruction input from the control unit 141, and outputs the result to the effector unit 144. The effector unit 144 performs effect processing on the audio data based on an instruction input from the control unit 141. Examples of the effect processing include volume control, low-pass / high-pass filter, distortion, chorus, reverb, and echo.

  The sound system 15 (reproducing means) reproduces audio data output from the effector unit 144. The operation unit 16 includes a push button (operation button) for inputting an operation instruction by the user, detects a press on the push button, and outputs a detection result (operation instruction) to the main control unit 13.

  The sensor 17 (detection means) is an acceleration sensor that detects the operation of the casing of the audio playback device 1. The interface 18 inputs the detection output of the sensor 17 and outputs it to the main control unit 13.

Next, the detailed operation of the feature extraction unit 12 of FIG. 1 will be described with reference to FIGS.
FIG. 2 is a diagram illustrating a functional configuration of the feature extraction unit 12. In FIG. 2, the filter unit 121 performs a filtering process for extracting a predetermined frequency band on the input audio data. In the filter processing, for example, the cut-off frequency is adjusted so that bass sound and bass drum sound can be extracted. Note that the frequency band to be filtered in the filter unit 121 or the musical instrument sound to be extracted may be selectable by the user.

  The envelope generation unit 122 detects the vertex of the waveform from the audio data after the filter processing by the filter unit 121, and generates an envelope by connecting the peak and mountain, and the valley and valley, respectively. It is not always necessary to generate the envelope, but the waveform can be simplified by generating the envelope, and the subsequent processing can be simplified.

  The threshold value comparison unit 123 compares the level of the waveform data of the envelope output from the envelope generation unit 122 with a preset threshold value, and selects a time zone that exceeds the threshold value on the playback time axis of the audio data. Ask. This threshold is set to a value that exceeds, for example, the moment the bass guitar or bass drum is played.

  The threshold value comparison unit 123 outputs a time information string with the time period when the level of the waveform data of the envelope is lower than the threshold value as Low and the time period when the level is higher as High. Thus, in this embodiment, by generating a time information sequence indicating whether or not a predetermined threshold value is exceeded from the waveform data extracted by the filter unit 121, it is possible to detect rhythm of audio data.

  FIG. 3A is an example of a time information sequence output from the threshold comparison unit 123. Time t0 to t1, time t2 to t3, time t4 to t5, and time t6 to t7 on the reproduction time axis of audio data. In FIG. 5, the level of the waveform data of the envelope exceeds the threshold value.

  FIG. 3B shows a time information sequence shown in FIG. 3A in which a high time zone has a constant width Δt. That is, the time zone from time t0−Δt to t1 + Δt, time t2−Δt to t3 + Δt, time t4−Δt to t5 + Δt, time t6−Δt to t7 + Δt becomes High.

  In this way, by giving a predetermined width to the time information string, even if the user's operation timing slightly deviates from the time when the user should originally operate, it can be determined as a match in the comparison in the main control unit 13. is there.

  If the audio data is composed of a single tone melody, it is possible to detect the melody by comparing the waveform level of the audio data with a predetermined threshold without performing filter processing or envelope generation processing, It is possible to create a time information string indicating the timing at which a sound is being produced.

Next, a procedure for controlling the reproduction of audio by detecting a user operation in the audio reproduction apparatus of FIG. 1 will be described with reference to the flowchart of FIG.
In FIG. 4, when the audio reproduction process is started, the main control unit 13 determines whether or not the designation of audio data to be reproduced is input from the operation unit 16 (step S101).

  When the designation of audio data to be reproduced is input from the operation unit 16, the main control unit 13 instructs the feature extraction unit 12 to generate a time information sequence of the audio data. The feature extraction unit 12 reads the audio data instructed by the main control unit 13 from the memory 11 and generates a time information sequence (step S102).

  Note that once the time information sequence is generated, it is stored in the memory in the audio playback device 1, and the procedure of steps S101 to S103 is omitted at the next playback, and the generated time information sequence is used in the processing after step S104. The generation of the time information sequence in step S102 may be performed in real time simultaneously with the reproduction of the audio data.

  When the generation of the time information sequence is completed in the feature extraction unit 12 (step S103: Y), the main control unit 13 temporarily stores the time information sequence and instructs the reproduction control unit 14 to start reproducing the audio data. (Step S104).

  When a reproduction instruction from the main control unit 13 is input, the control unit 141 in the reproduction control unit 14 instructs the read control unit 142 to read audio data to be reproduced from the memory 11. At the start of reproduction, the audio data is reproduced from the sound system 15 without being processed by the filter unit 143 and the effector unit 144.

  When the reproduction of the audio data is started, the main control unit 13 inputs the detection output of the sensor 17 via the interface 18. When the user swings the audio playback device in a predetermined manner (for example, in the vertical direction), the sensor 17 detects acceleration in a predetermined axial direction and outputs the acceleration to the main control unit 13.

  In the present embodiment, an acceleration sensor is used as a sensor for detecting a user operation, but a magnetic sensor may be used. In the case of using a magnetic sensor, when the user swings the audio playback device in a predetermined manner, a change in geomagnetic intensity in a predetermined axial direction is detected.

  Based on the detection result of the sensor 17, the main control unit 13 generates detection information indicating whether or not the acceleration in a predetermined axial direction is a certain value or more. Subsequently, the main control unit 13 compares the information on the current time (time from the start of audio playback) in the time information sequence generated from the audio data with the detection information generated from the sensor output (Step S1). S105), it is determined whether or not they match (step S106). The main control unit 13 outputs this determination result to the control unit 141 in the reproduction control unit 14.

  If it is determined in step S106 that the two match (step S106: N), the control unit 141 causes the sound system 15 to reproduce the audio data without processing.

  On the other hand, when it is determined in step S106 that they do not match (step S106: Y), the control unit 141 instructs the filter unit 143 and the effector unit 144 to perform predetermined processing on the audio data. (Step S107).

  The processing in step S107 is performed, for example, by cutting a predetermined frequency region with the filter unit 143, applying a predetermined effect with the effector unit 144, or reducing the playback volume. For these processes, the type and strength of the process, the number of processes to be applied at the same time, and the like may be determined as appropriate based on the time difference between the operation timing and the time information sequence, the number of times of mismatching, and the like. Thereafter, steps S105 to S108 are repeatedly executed until the reproduction of the audio data is completed.

  In step S107, not only the effect but also the playback may be paused. Further, the reproduction sound is not changed, and the comparison match status in step S105 may be displayed on a display unit (not shown), or the comparison match status may be stored in the memory in the audio playback device 1 only. Good. In addition, although the time information sequence shown in FIG. 3 indicates two values of High and Low by one threshold, it may indicate three or more values by a plurality of thresholds. In that case, the detection result of the sensor 17 may be information of three or more values based on a plurality of threshold values and correspond to each other. The sensor 17 does not necessarily need to detect the operation of the housing of the audio playback device. The sensor 17 may be provided separately from the casing and may detect its own operation.

  Since the audio reproducing apparatus of the present embodiment is configured to generate a time information sequence indicating the musical characteristics of audio data by the feature extraction unit 12, it is not necessary to create an event data sequence synchronized with audio data by authoring in advance. .

  In the present embodiment, when the timing of the time information sequence generated from the audio data does not match the user operation timing, the playback state of the audio data is changed by temporarily stopping playback or applying an effect. Therefore, the user needs to operate the audio playback device 1 at an appropriate timing in accordance with the playback of the audio data.

  For this reason, by using the audio reproducing apparatus of the present embodiment, the user can not only listen to the music to be reproduced, but also can feel that he / she participates in the music through the operation of the audio reproducing apparatus.

Next, a second embodiment of the present invention will be described with reference to FIGS.
FIG. 5 is a block diagram showing the configuration of an audio playback apparatus according to the second embodiment of the present invention. 5, the same reference numerals are given to the same components as those in FIG. 1, and the detailed description thereof is omitted.

  A sound source 145 (musical tone generating means) in the reproduction control unit 14 holds timbre data representing the timbre of various musical instruments, and corresponds to the push button when the user presses the push button detection from the operation unit 16. Musical sound data is generated. The mixer unit 146 mixes the audio data output from the effector unit 144 and the musical sound data output from the sound source 145 and outputs the result to the sound system 15.

  In the second embodiment, the feature extraction unit 12 detects the rhythm of a predetermined instrument sound from the audio data, and the sound source 145 outputs the musical sound data of the instrument sound in accordance with the pressing timing of the push button. Let it play. On the other hand, the reproduction control unit 14 cancels the instrument sound from the audio data by the filter unit 143, and causes the sound system 15 to reproduce the audio data after the cancellation of the instrument sound.

  Note that the filter unit 143 may remove frequency components in a specific band instead of canceling a specific instrument sound. For example, when the filter unit 121 of the feature extraction unit 12 uses an LPF (Low Pass Filter) with a cutoff frequency Hc, the filter unit 143 of the reproduction control unit 14 uses an HPF (High Pass Filter) with a cutoff frequency Hc. .

Next, a procedure for detecting user operation and controlling audio reproduction in the audio reproducing apparatus of FIG. 5 will be described with reference to the flowchart of FIG.
In FIG. 6, when the audio reproduction process is started, the main control unit 13 determines whether designation of audio data to be reproduced is input from the operation unit 16 (step S201).

When the main control unit 13 inputs designation of audio data to be reproduced from the operation unit 16, the main control unit 13 instructs the feature extraction unit 12 to generate a time information sequence of the audio data.
The feature extraction unit 12 reads the audio data instructed by the main control unit 13 from the memory 11 and generates a time information sequence (step S202). In the second embodiment, this time information string indicates the timing at which the musical sound data generated by the sound source 145 should be generated from the sound system 15.

  Note that once the time information sequence is generated, it is stored in the memory in the audio playback device 2, and the procedure of steps S201 to S203 is omitted at the next playback, and the generated time information sequence is used in the processing after step S204.

  When the generation of the time information sequence is completed in the feature extraction unit 12 (step S203: Y), the main control unit 13 temporarily stores the time information sequence and cancels a predetermined instrument sound in the reproduction control unit 14. An instruction to start playback of the audio data is given (step S204). Here, the predetermined instrument sound is an instrument sound that the user performs with a push button, and is an instrument sound that is reproduced in step S206 described later.

  When a reproduction instruction from the main control unit 13 is input, the control unit 141 in the reproduction control unit 14 instructs the read control unit 142 to read audio data to be reproduced from the memory 11. The filter unit 143 performs a filter process on the audio data read by the read control unit 142 to cancel the instrument sound, and outputs it to the effector unit 144.

  When the reproduction of the audio data is started, the main control unit 13 inputs an operation to the operation unit 16 by the user (step S205). When the user depresses the push button of the operation unit 16 (step 205: Yes), the operation unit 16 detects depressing the push button and outputs it to the main control unit 13 and the sound source 145.

  The sound source 145 generates musical tone data of a predetermined musical instrument sound based on the output from the operation unit 16 and outputs it to the mixer unit 146. The mixer unit 146 mixes the audio data input from the effector unit 144 and the musical sound data input from the sound source 145 and reproduces them from the sound system 15 (step S206).

  On the other hand, the main control unit 13 generates detection information based on the time when the push button of the operation unit 16 is pressed based on the detection result of the operation unit 16. Subsequently, the main control unit 13 obtains information on the current time (time since the start of audio playback) in the time information sequence generated from the audio data and detection information generated from the detection result of the operation unit 16. Comparison is made (step S207), and it is determined whether or not they match (step S208). The main control unit 13 outputs this determination result to the control unit 141 in the reproduction control unit 14.

  If it is determined in step S208 that they match (step S208: N), the control unit 141 reproduces the audio data from the sound system 15 without performing any processing other than instrument sound cancellation.

  On the other hand, if it is determined in step S208 that they do not match (step S208: Y), the control unit 141 instructs the filter unit 143 and the effector unit 144 to perform predetermined processing on the audio data. (Step S209).

  The processing in step S209 performs, for example, processing such as cutting a predetermined frequency region by the filter unit 143, applying a predetermined effect by the effector unit 144, or reducing the playback volume. Thereafter, steps S205 to S210 are repeatedly executed until the reproduction of the audio data is completed.

  In step S209, the reproduction may be paused in addition to applying the effect. Further, the reproduction sound is not changed, and the comparison match status in step S207 may be displayed on a display unit (not shown), or the comparison match status may be stored in the memory in the audio playback device 2 alone. Good. Further, in step S204, the instrument sound may not be canceled, and the original audio data and the musical sound data generated by the sound source may be mixed and reproduced. Further, it is not always necessary to use a push button, and a separate MIDI musical instrument may be prepared and connected to the audio playback apparatus 2, and detection information based on the performance (operation) of the MIDI musical instrument may be sent to the main control unit 13.

  As described above, since the user can play a specific musical instrument by pressing the push button of the operation unit 16 with the audio playback device of the present embodiment, the push button is pressed according to the rhythm of the musical instrument. To participate in the playback of audio data.

  As mentioned above, although embodiment of this invention was explained in full detail, the concrete structure is not restricted to this embodiment, The design change etc. of the range which does not deviate from the summary of this invention are included.

  The present invention is suitable for use in an audio playback apparatus that detects user operation timing in accordance with playback of audio data.

It is a block diagram which shows the structure of the audio reproduction apparatus which concerns on 1st embodiment of this invention. It is a figure which shows the detailed structure of the feature extraction part 12 of FIG. It is a figure which shows an example of the time information sequence output from the threshold value comparison part 123 of FIG. FIG. 2 is a flowchart showing a procedure for controlling audio reproduction by detecting a user operation in the audio reproducing apparatus of FIG. 1. FIG. It is a block diagram which shows the structure of the audio reproduction apparatus which concerns on 2nd embodiment of this invention. FIG. 6 is a flowchart illustrating a procedure of controlling audio reproduction by detecting a user operation in the audio reproducing apparatus of FIG. 5.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1, 2 ... Audio reproduction apparatus, 11 ... Memory (storage means), 12 ... Feature extraction part (feature extraction means), 13 ... Main control part, 14 ... Reproduction control part, 15 ... Sound system (reproduction means), 16 ... Operation unit 17 ... sensor (detection unit) 121 ... filter unit 122 ... envelope generation unit 123 ... threshold comparison unit 141 ... control unit (control unit) 142 ... reading control unit 143 ... filter unit (erase) Means), 144 ... Effector section, 145 ... Sound source (musical sound generating means), 146 ... Mixer section

Claims (10)

Storage means for storing audio data;
Feature extraction means for extracting a predetermined musical feature on the playback time axis from the audio data, and generating a time information string indicating the musical feature;
Detection means for inputting an operation by a user and detecting an operation mode;
Playback means for playing back the audio data;
Comparison means for comparing the detection information on the time axis of the operation mode detected by the detection means with the time information sequence generated by the feature extraction means;
During reproduction of the audio data by the reproduction unit, if the detection information does not match the time information sequence as a result of the comparison by the comparison unit, the user is notified that the detection information does not match the time information sequence. Control means for performing control,
An audio playback device comprising:   The feature extraction means has a predetermined time width that is longer than the time width of the time information extracted from the audio data. The audio reproducing apparatus according to claim 1, wherein a time width is used.   3. The audio reproduction apparatus according to claim 1, wherein the control unit changes a reproduction state of the audio data based on a comparison result of the comparison unit during reproduction of the audio data.   The control means pauses reproduction of the audio data or controls the volume or gives an effect to the audio data based on the comparison result of the comparison means during reproduction of the audio data. The audio reproduction device according to claim 3. A display means,
The said control means changes the display to the said display means based on the comparison result of the said comparison means during reproduction | regeneration of the said audio data, The any one of Claims 1-4 characterized by the above-mentioned. Audio playback device.   6. The audio reproducing apparatus according to claim 1, wherein the detecting unit is an operation button that detects an operation by a user.   6. The audio reproducing apparatus according to claim 1, wherein the detecting unit is an acceleration sensor that detects acceleration.   6. The audio reproducing apparatus according to claim 1, wherein the detecting unit is a magnetic sensor that detects a magnetic field. In an audio playback method in an audio playback device,
A first step of extracting a predetermined musical feature on the reproduction time axis from the audio data stored in the storage means, and generating a time information sequence indicating the musical feature;
A second step of inputting an operation by a user and detecting an operation mode;
A third step of reproducing the audio data stored in the storage means;
A fourth step of comparing the detection information on the time axis of the operation mode detected in the second step with the time information sequence generated in the first step;
During reproduction of the audio data in the third step, if the detection information does not match the time information string as a result of comparison in the fourth step, the detection information and the time information string do not match. A fifth step of performing control for notifying the user of,
An audio playback method comprising:   In the first step, of the time information constituting the time information sequence, the time width of the time information in the case where there is the musical feature is longer than the time width of the time information extracted from the audio data. The audio playback method according to claim 9, wherein the time width is as follows.

Images