JP2006106411A - Sound output controller, musical piece reproduction device, sound output control method, program thereof and recording medium with the program recorded thereon - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a musical piece reproduction device which outputs good insertion sound. <P>SOLUTION: When a calculating means 470 of the musical piece reproduction device 400 recognizes a request in which a prescribed musical piece is to be reproduced in a voice corresponding mode, for example, the means starts reproduction of the prescribed musical piece and obtains in-vehicle sound including voice "yea", for example being collected by a microphone 200. The calculating means 470 recognizes a voice BPM of the voice "yea" based on the musical piece BPM of the musical piece being reproduced and recognizes a voice rhythm pattern of the voice corresponding to the voice BPM. Then, the calculating means 470 makes an uttering means 450 output insertion sound corresponding to the voice rhythm pattern i.e., the insertion sound synchronizing the musical piece BPM, in a properly matched manner with the musical piece based on the insertion data related to the rhythm pattern information corresponding to the voice rhythm pattern. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a sound output control device that controls to output a predetermined sound, a music playback device, a sound output control method, a program thereof, and a recording medium on which the program is recorded.

  Conventionally, various configurations such as robots, toys, and video game programs that recognize voice and output voice data have been widely used (see, for example, Patent Document 1). For example, the device described in Patent Document 1 has a configuration applied to a robot, and collects surrounding sounds including a user's utterance with a microphone disposed at a predetermined position of the head unit. Then, based on the obtained audio signal, a synthesized sound is generated in which the prosody in the state information of the model storage unit is controlled based on the value of the emotion model, and is output from the speaker.

JP 2002-304187 A (page 3 right column-page 10 left column)

  By the way, in the conventional structure which outputs a sound according to pronunciation like the patent document 1 mentioned above, when a listener puts a matching hand according to a music, for example, a sound is output according to the utterance of a matching hand. However, the timing of the synthesized sound that is output to the matching hand according to the rhythm of the music is already shifted from the rhythm of the music, which may hinder the listener's appreciation of the music Is given as an example.

  In view of the above, an object of the present invention is to provide a sound output control device, a music playback device, a sound output control method, a program thereof, and a recording medium on which the program is recorded. Is to provide.

  The invention according to claim 1 is a sound output control device that outputs a predetermined sound from the sound output means, and outputs external state information acquisition means for acquiring external state information related to the external state, and outputs music data related to the music. A music characteristic recognition means for recognizing the sound characteristics of the music data output from the music output means, and an output form that synchronizes with the sound characteristics upon recognizing that the external state information relating to the predetermined external state has been acquired. And a sound output control means for controlling the sound output means to output the sound.

  A twenty-first aspect of the present invention is the sound output control device according to any one of the first to twentieth aspects of the present invention, and a music reproduction processing unit that performs a process of reproducing the music data and outputting the music data from the music output unit. , A music reproducing apparatus characterized by comprising:

  According to a twenty-second aspect of the present invention, the output is controlled by the sound output control device according to any one of the first to twentieth aspects, and the sound output control unit of the sound output control device that acquires the music data. Superimposed music data generating means for generating superimposed music data obtained by superimposing the sound to be sounded on the music data; music reproduction processing means for performing processing for reproducing the superimposed music data and outputting it from the music output means; , A music reproducing apparatus characterized by comprising:

  The invention described in claim 23 is a sound output control method for outputting a predetermined sound from the sound output means, which is output from the music output means for acquiring external state information relating to the external state and outputting music data relating to the music. Control to output from the sound output means a sound in an output form tuned to the sound characteristics when recognizing the sound characteristics of the music data and recognizing that the external state information relating to the predetermined external state has been acquired. A sound output control method characterized by:

  According to a twenty-fourth aspect of the present invention, there is provided a sound output control program that causes a computing means to function as the sound output control device according to any one of the first to twentieth aspects.

  A twenty-fifth aspect of the invention is a sound output control program that causes a calculation means to execute the sound output control method according to the twenty-third aspect.

  The invention described in claim 26 is a recording medium on which a sound output control program is recorded, wherein the sound output control program according to claim 24 or claim 25 is recorded so as to be readable by an arithmetic means. .

  Hereinafter, an embodiment according to the present invention will be described with reference to the drawings. In the present embodiment, an apparatus configuration mounted on, for example, a vehicle that is a moving body will be described as an example of a music reproducing apparatus including the sound output control apparatus of the present invention. In addition, as a music reproduction apparatus in this invention, it is applicable also to the structure installed in structures, such as a house, and a portable structure other than the structure mounted in a mobile body. Further, in the present embodiment, as an external sound state that is an external state, a sound state that is sounded by a vehicle driver or a passenger (hereinafter referred to as a user) will be described as an example. Moreover, the state of the user's body movement will be described as an example of the state of the body movement that is an external state. The external sound is not limited to the sound produced by the user, but the user's clapping, nasal or whistling sounds, the sound propagating from the outside of the vehicle to the inside of the vehicle, and the sound of playing an instrument such as a tambourine, castanets or maracas For example, various sounds can be targeted. In addition, the movement of the body can be the movement of an animal inside or outside the car or a person outside the car. Furthermore, as an external state, a movement corresponding to a vehicle of an object inside or outside the vehicle can be targeted. FIG. 1 is a block diagram showing a schematic configuration of a music reproducing system according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing a schematic configuration of music list data. FIG. 3 is a schematic diagram showing a schematic configuration of insertion sound list data. FIG. 4 is a schematic diagram showing a schematic configuration of a calculation means constituting the music reproducing device. FIG. 5 is a schematic diagram showing an example of the output state of the insertion sound in the voice corresponding mode, where (A) is the lyrics of the music, (B) is the voice, and (C) is the insertion sound. FIG. 6 is a schematic diagram showing an example of the output state of the insertion sound in the motion corresponding mode, where (A) is the lyrics of the music, (B) is the operation, and (C) is the insertion sound.

[Configuration of music playback system]
In FIG. 1, reference numeral 100 denotes a music playback system. The music playback system 100 outputs a sound corresponding to a sound produced by a user in a car as a moving body in accordance with the music being played back. In addition, the music playback system 100 outputs a sound corresponding to the user's operation in accordance with the music being played back. In the following, the sound output corresponding to the sound or operation will be described as an insertion sound. Here, the moving body is not limited to a vehicle such as an automobile or a train, and can be applied to any moving body such as an airplane or a ship. The music playback system 100 includes a microphone 200 (hereinafter referred to as a microphone 200), an imaging unit 300, a music playback device 400, and the like.

  The microphone 200 is disposed in, for example, a substantially central portion of the ceiling in the interior space of the vehicle, and acquires sounds that can be recognized in the vehicle such as a user's voice and a sound that propagates from the outside of the vehicle to the inside of the vehicle (hereinafter referred to as in-vehicle sound). Collect sound. Then, the microphone 200 outputs in-vehicle sound information, which is external sound information as external state information corresponding to the collected in-vehicle sound, to the music reproducing device 400.

  The imaging means 300 is disposed, for example, in the front part of the ceiling in the vehicle interior space, and images the user. Then, imaging data that is physical motion information as external state information corresponding to the captured content is output to the music reproducing device 400. For example, the following sensor may be provided instead of the imaging unit 300. In other words, a sensor that recognizes the amount of heat radiated from the user in a predetermined area of the indoor space and outputs external state information corresponding to the amount of heat may be provided. Alternatively, a sensor may be provided that irradiates a predetermined region with a predetermined light beam and outputs external state information corresponding to the reflection state of the light beam. Furthermore, the configuration is not limited to the configuration provided in the vehicle, and a configuration may be provided in which a sensor that is provided on a part of the body and outputs a detection value or measurement value such as a vibration gyroscope or an accelerometer as external state information is provided.

  The music playback device 400 includes music stored in a general-purpose digital format such as MIDI (Musical Instrument Digital Interface) format, WAVE format or MPEG (Moving Picture Coding Experts Group) format, CD (Compact Disc), MD (Mini Disc), etc. Play music recorded on the. In addition, the music playback device 400 outputs an insertion sound corresponding to the user's voice or operation in accordance with the music being played back. The music reproducing device 400 is operated by electric power supplied from a battery (not shown) mounted on the vehicle, for example. The music reproducing device 400 includes a music data storage unit 410, an insertion sound data storage unit 420 as a sound data storage unit, a memory 430, an input unit 440, and a sound output unit 450 as a sound output unit and a music output unit. A display unit 460, a calculation unit 470, and the like.

  The music data storage means 410 stores the music list data 500 as shown in FIG. The music data storage means 410 includes a magnetic disk such as an HD (Hard Disk), an optical disk such as a CD, an MD, or a DVD (Digital Versatile Disc), a magneto-optical disk, and a drive that stores the data in a readable manner. A configuration including a driver and a driver can be exemplified.

  The music list data 500 is data relating to a list of music to be played back. The music list data 500 is configured by associating at least one piece of music individual data 510 as one data structure.

  The music piece individual data 510 is information related to one piece of music piece. The music individual data 510 has a table structure in which music data 511, music related information 512, and the like are configured as one data structure. Note that the individual music data 510 may be composed of only the music data 511.

  The music data 511 is data used when playing music. The music data 511 is recorded in a format capable of reproducing music such as MIDI format, WAVE format, or MPEG format.

  The music related information 512 is information related to the music reproduced by the music data 511. The music related information 512 has a table structure in which music name information 512A, player information 512B, music rhythm information 512C, and the like are configured as one data structure. Note that the music related information 512 may be configured to store at least one of the pieces of information 512A, 512B, and 512C.

  The song name information 512A is data obtained by converting information indicating the name of a song. The player information 512B is obtained by converting information indicating the player of the music into data. The music rhythm information 512C is data obtained by converting information indicating the rhythm of music such as BPM (Beats Per Minute) and time signature. As this music rhythm information 512C, a MIDI message etc. can be illustrated, for example.

  The insertion sound data storage means 420 stores the insertion sound list data 600 as shown in FIG. The inserted sound data storage means 420 can be exemplified by the same configuration as the music data storage means 410.

  The insertion sound list data 600 is data relating to a list of insertion sounds that are appropriately output in accordance with the music being played in response to the user's voice and actions. The insertion sound list data 600 is appropriately changed, added, or deleted by an input operation of the input unit 440 by the user. The inserted sound list data 600 is configured by associating at least one inserted sound individual data 610 as one data structure.

  The inserted sound individual data 610 is information related to one inserted sound. The inserted sound individual data 610 has a table structure in which inserted sound data 611 as sound data, inserted sound related information 612, and the like are configured as one data structure.

  The insertion sound data 611 is data used when reproducing the insertion sound. The insertion sound data 611 is recorded in the MIDI format. As an insertion sound reproduced by the insertion sound data 611, for example, a sound of a musical instrument such as a drum, a Japanese drum, a piano, a cymbal, a guitar, or a combination of a plurality of these musical instruments can be exemplified. Here, the insertion sound data 611 is not limited to the MIDI format, and may be any format capable of reproducing the insertion sound, such as the WAVE format or the MPEG format. In addition, the insertion sound is not limited to the sound of playing a musical instrument. For example, sounds such as “Yay”, “Hyu”, “Hay”, “Yoisho”, sounds of nose and whistle, natural sounds such as waves, winds and water. Any sound such as a sound or an artificial sound such as a horn sound may be used.

  The insertion sound related information 612 is information regarding the insertion sound reproduced from the insertion sound data 611. The inserted sound related information 612 is a table in which rhythm pattern information 612A, target language information 612B as predetermined external state information, target motion information 612C as predetermined external state information, and the like are configured as one data structure. It is a structure.

  The rhythm pattern information 612A is data obtained by converting information indicating the rhythm pattern of the inserted sound reproduced by the inserted sound data 611 into data. Examples of the rhythm pattern include “rock”, “jazz”, and “rhythm and blues”. The target language information 612B is data obtained by converting information indicating a language that is pronounced as speech. Examples of the language indicated by the target language information 612B include “Yay”, “Yayye”, “Hyu”, “Hay”, “Yoshisho”, and the like. The target action information 612C is data obtained by converting information indicating a user's action. Examples of the motion indicated by the target motion information 612C include a motion of shaking the hand, head, or torso, and a motion of pushing up the hand.

  The memory 430 stores various information and various data necessary for reproducing a music piece or outputting an insertion sound so as to be appropriately read out. The memory 430 stores various programs developed on an OS (Operating System) that controls the operation of the entire music playback device 400. As the memory 430, it is desirable to use a memory having a configuration in which memory is retained even when the power is suddenly turned off due to a power failure, for example, a CMOS (Complementary Metal-Oxide Semiconductor) memory. Note that the memory may be configured to include a drive and a driver that are readable and stored in a recording medium such as an HD, a DVD, an optical disk, or a memory card.

  The input unit 440 includes various operation buttons and operation knobs (not shown) that are input. The contents of the input operation of the operation buttons and operation knobs are setting items such as setting of operation contents of the music reproducing device 400, for example. Specifically, the operation settings can be set to play or stop music, adjust the volume, set the mode to output an insertion sound corresponding to audio (hereinafter referred to as audio-compatible mode), and operate. For example, a setting of a mode for outputting an insertion sound (hereinafter referred to as an operation corresponding mode) can be exemplified. Then, the input unit 440 appropriately sets various information as operation signals to the calculation unit 470 by an input operation of setting items. The input unit 440 is not limited to an input operation such as an operation button or an operation knob. For example, various setting items such as an input operation using a touch panel provided on the display unit 460 and an input operation using voice can be set and input. Either configuration is applicable. Further, as the input means 440, for example, remote control light reception that receives various information transmitted via infrared rays from a remote controller (hereinafter referred to as a remote controller) (not shown) and transmits the various information as operation signals to the calculation means 470. It is good also as a structure provided with the part.

  The sound generation means 450 has a speaker (not shown) disposed on, for example, an instrument panel portion, a door portion, a rear dashboard portion, or the like of the vehicle. The sound generation means 450 is controlled by the calculation means 470 and outputs sound signals such as the music data 511 and the insertion sound data 611 from the calculation means 470 from the speaker.

  The display unit 460 is controlled by the calculation unit 470 and displays the image signal of the image data from the calculation unit 470 on the screen. Examples of the image data include image data such as the name of the music being played, performer, performance time, and volume. Examples of the display means 460 include a liquid crystal panel, an organic EL (Electro Luminescence) panel, a PDP (Plasma Display Panel), a CRT (Cathode-Ray Tube), an FED (Field Emission Display), and an electrophoretic display panel.

  The calculation means 470 includes various input / output ports (not shown) such as a microphone port to which the microphone 200 is connected, an imaging port to which the imaging means 300 is connected, a music storage port to which the music data storage means 410 is connected, and an insertion sound data storage means. Insertion sound storage port to which 420 is connected, memory port to which memory 430 is connected, key input port to which input means 440 is connected, sound generation control port to which sound generation means 450 is connected, display control to which display means 460 is connected Has ports etc. As shown in FIG. 4, the calculation means 470 is a music reproduction processing means 471, a music rhythm recognition means 472 as a music characteristic recognition means, an external state information acquisition means, and an external state rhythm recognition means. Voice state recognition means 473, operation state recognition means 474 as external state information acquisition means and external state rhythm recognition means, insertion sound reproduction control means 475 as sound output control means, and the like. The music rhythm recognition unit 472, the voice state recognition unit 473, the operation state recognition unit 474, and the insertion sound reproduction control unit 475 constitute the sound output control device of the present invention.

  The music reproduction processing unit 471 performs processing for reproducing music based on the music data 511 stored in the music data storage unit 410. Specifically, when the music reproduction processing unit 471 acquires an operation signal for reproducing a predetermined music based on the input operation of the input unit 440, the music reproduction processing unit 471 searches the music individual data 510 corresponding to the predetermined music. Further, the music reproduction processing means 471 determines whether or not the music rhythm information 512C is incorporated in the music individual data 510. If it is determined that the music rhythm information 512C is incorporated, the music data 511 is output to the sound generation means 450 as an audio signal at a speed based on the rhythm of the music rhythm information 512C. When the music reproduction processing unit 471 determines that the music rhythm information 512C is not incorporated, the music reproduction processing unit 471 outputs the music data 511 to the sound generation unit 450 as an audio signal at a preset speed. Furthermore, when the music name information 512A and the player information 512B are incorporated in the music individual data 510, the music reproduction processing means 471 includes the name of the music, the performer, and the performance time of the music described in the information 512A and 512B. Image data for displaying various information related to the music being played back is generated. Then, this image data is output to the display means 460 as an image signal.

  The music rhythm recognition means 472 recognizes the rhythm of the music being played back by the music playback processing means 471. Specifically, the music rhythm recognition unit 472 determines whether or not the music rhythm information 512 </ b> C is incorporated in the music individual data 510 including the music data 511 output from the sound generation unit 450. Here, as a method of determining whether or not the music rhythm information 512C is incorporated, for example, a method of determining based on the presence or absence of a predetermined MIDI message can be exemplified. In addition, it is good also as the method of judging based on the presence or absence of the other information regarding the rhythm of a music. When the music rhythm recognition unit 472 determines that the music rhythm information 512C is incorporated, the music rhythm recognition unit 472 acquires the music rhythm information 512C and recognizes the BPM of the music (hereinafter referred to as music BPM). Then, the music BPM information related to the music BPM is stored in the memory 430. When the music rhythm recognition unit 472 determines that the music rhythm information 512C is not incorporated, for example, the music rhythm recognition unit 472 performs the following processing to recognize the music BPM. That is, the music rhythm recognition means 472 acquires the audio signal of the music data 511 and recognizes the output waveform of the music based on this audio signal. Furthermore, the peak interval of this output waveform is recognized, and the music BPM is recognized from this peak interval. Then, the music rhythm recognition unit 472 stores the music BPM information related to the music BPM in the memory 430.

  When the voice state recognition unit 473 recognizes that the voice corresponding mode based on the input operation in the input unit 440 is set, the voice state recognition unit 473 appropriately detects the voice state of the user. The voice state recognition unit 473 includes a voice rhythm recognition unit 473A and a language recognition unit 473B.

  The voice rhythm recognition unit 473A recognizes the voice rhythm. Specifically, the voice rhythm recognition unit 473A causes the microphone 200 to collect in-vehicle sound and output in-vehicle sound information. Then, the voice rhythm recognition unit 473A acquires the in-vehicle sound information from the microphone 200 for a predetermined time set in advance, and recognizes the output waveform of the in-vehicle sound based on the in-vehicle sound information. Further, the voice rhythm recognition unit 473A sets a portion where the output value in the output waveform is equal to or less than a preset reference output value to zero, that is, deletes the sound below the reference volume from the vehicle interior sound. Here, the reference output value is set to a value that is substantially the same as, for example, the output value of the running sound of the vehicle or the output value of the normal conversation of the user. In other words, the reference output value is, for example, “Yay” or “Yayy” that the user consciously pronounced according to the music when the portion below the reference output value in the output waveform of the interior sound is set to zero. It is set so that only the component corresponding to the sound does not become zero.

  When the voice rhythm recognizing unit 473A recognizes that the time length of the sound deleted from the vehicle interior sound is equal to or less than the speech recognition reference time as the preset external sound recognition reference time, for example, "Yay" It is determined that voices consciously pronounced are included. Furthermore, the voice rhythm recognition unit 473A acquires the music BPM information from the memory 430, and outputs the peak of the music BPM of the music BPM information and the output waveform of the in-vehicle sound from which the sound below the reference volume is deleted, that is, the voice output waveform. Based on the interval, an audio BPM based on the music BPM (hereinafter referred to as audio BPM) is calculated and recognized. Then, the voice rhythm recognition unit 473A recognizes a voice rhythm pattern (hereinafter referred to as a voice rhythm pattern) based on the voice BPM, and stores voice rhythm pattern information related to the voice rhythm pattern in the memory 430.

  The language recognizing unit 473B recognizes a language (hereinafter referred to as a “speech language”) that is pronounced by the sound whose sound rhythm pattern is recognized by the sound rhythm recognizing unit 473A based on the in-vehicle sound information. For example, the language recognizing unit 473B converts the in-vehicle sound information acquired by the sound rhythm recognizing unit 473A into a digital signal and recognizes an acoustic feature. Furthermore, a language closest to an acoustic feature based on in-vehicle sound information is searched from a speech model obtained by modeling speech statistical information. Then, the language recognition unit 473B recognizes the searched language as a speech language. Further, when recognizing the speech language, the language recognition unit 473B stores the speech language information related to the speech language in the memory 430.

  When the operation state recognition unit 474 recognizes that the operation corresponding mode based on the input operation in the input unit 440 is set, the operation state recognition unit 474 appropriately recognizes the operation state of the user. The motion state recognition unit 474 includes a motion rhythm recognition unit 474A and a motion content recognition unit 474B.

  The motion rhythm recognition means 474A recognizes the motion rhythm. Specifically, the motion rhythm recognition unit 474A causes the imaging unit 300 to image the user and output the imaging data. Furthermore, the motion rhythm recognition unit 474A acquires the image data from the image capturing unit 300 for a predetermined time, and grasps a predetermined part of the user's body, such as a head or a hand, based on the image data. To do. Then, it is recognized whether or not the grasped predetermined part has moved in a predetermined direction by a predetermined body motion reference amount or more. Here, the body motion reference amount is set to an amount corresponding to a motion other than a conscious motion in accordance with the music, such as a user's driving motion or a window opening / closing motion. In addition, it is good also as a structure which provides the some imaging means 300, makes a user image by these imaging means 300, and recognizes the operation | movement of the predetermined part of a body three-dimensionally.

  The motion rhythm recognition unit 474A moves the predetermined part of the body more than the body motion reference amount, that is, for example, that the hand, head, or torso moves more than the body motion reference amount in the left-right direction, When recognizing that it has moved more than the reference amount, it is determined that the user has consciously moved his body according to the music. Further, the motion rhythm recognition unit 474A acquires the music BPM information from the memory 430, and based on the music BPM of the music BPM information and, for example, the length of time until the hand moves from a predetermined position to another position. The operation BPM (hereinafter referred to as the operation BPM) based on the music BPM is calculated and recognized. The motion rhythm recognition unit 474A recognizes a motion rhythm pattern (hereinafter referred to as a motion rhythm pattern) based on the motion BPM, and stores motion rhythm pattern information related to the motion rhythm pattern in the memory 430.

  The operation content recognition unit 474B recognizes the content of the operation. Specifically, the motion content recognition unit 474B is configured to determine a predetermined part of the body that has moved more than the physical motion reference amount based on the imaging data acquired by the motion rhythm recognition unit 474A, and a direction in which the predetermined part has moved. , Specify. Then, the contents of the operation are recognized based on the specified predetermined part and the moving direction. For example, when the action content recognizing means 474B recognizes that the head is moving when it is specified that the head is moving in the left-right direction, and has specified that the hand is moving up and down, Recognize that the action is to push up the hand. Then, the operation content recognition unit 474B stores the operation content information regarding the operation content in the memory 430.

  The insertion sound reproduction control means 475 controls to reproduce the insertion sound in accordance with the music being reproduced by the music reproduction processing means 471. The insertion sound reproduction control unit 475 includes an insertion position specifying unit 475A, an insertion sound reproduction processing unit 475B, and the like.

  The insertion position specifying means 475A specifies the position where the insertion sound is reproduced, that is, the position where the insertion sound is inserted into the music (hereinafter referred to as the insertion position). Specifically, the insertion position specifying means 475A recognizes the time signature of the music based on the output waveform of the music being played recognized by the music rhythm recognition means 472. Further, based on the time signature of the music, the break of the music bar is recognized. If the voice recognition mode is set, the insertion position specifying unit 475A inserts the beginning of the second measure of the music that appears second from the time when the voice whose speech language is recognized by the language recognition unit 473B is emitted. As specified. That is, the insertion position specifying unit 475A specifies the insertion position so that the insertion sound is output within a preset output processing time limit from the time when the sound is emitted.

  For example, the insertion position specifying means 475A is shown in FIG. 5A in a state in which a piece of music in which the beginning of a bar appears at the reproduction time T1, T3, T4, T5, T7, T8, T9 as shown in FIG. When the “Yay” sound uttered at the reproduction time T2 between the reproduction time T1 and the reproduction time T3 as shown in FIG. 5B is recognized, the beginning of the second measure appearing from the reproduction time T2, that is, the reproduction time T4. The beginning of the bar appearing in is specified as the insertion position. Further, when the “Yay” sound uttered at the reproduction time T6 between the reproduction time T5 and the reproduction time T7 is recognized, the beginning of the bar appearing at the reproduction time T8 is specified as the insertion position.

  In addition, when the operation position mode is set, the insertion position specifying unit 475A starts the second musical measure from the time when the operation content is recognized by the operation content recognition unit 474B. Specify the insertion position. That is, the insertion position specifying unit 475A specifies the insertion position so that the insertion sound is output within the output processing time limit from when the operation is performed. For example, the insertion position specifying unit 475A is performed at the reproduction time T2 as shown in FIG. 6B in the state where the same music as that shown in FIG. 5A is being reproduced as shown in FIG. 6A. When recognizing the action of “pushing up the hand”, the beginning of the bar appearing at the reproduction time T4 is specified as the insertion position. Further, when the operation of “shaking the head” performed at the reproduction time T6 is recognized, the beginning of the bar appearing at the reproduction time T8 is specified as the insertion position.

  Here, for example, a configuration is described in which the beginning of the second measure appearing from the time when the sound is emitted is specified as the insertion position, but the present invention is not limited to this, and the first and third from the time when the sound is emitted are exemplified. A configuration may be used in which the beginning of the eye bar, or the end of a predetermined bar, or the start to end of a predetermined bar is specified as the insertion position.

  The insertion sound reproduction processing means 475B performs processing for reproducing the insertion sound corresponding to the voice or the operation in accordance with the music. Specifically, the insertion sound reproduction processing unit 475B acquires the sound rhythm pattern information and the sound language information from the memory 430 when the sound corresponding mode is set. Furthermore, inserted sound individual data 610 in which rhythm pattern information 612A related to a rhythm pattern substantially matching the sound rhythm pattern of the sound rhythm pattern information and target language information 612B related to a language substantially matching the sound language of the sound language information are incorporated. Is searched from the insertion sound list data 600. Then, the insertion sound reproduction processing means 475B acquires the insertion sound data 611 incorporated in the searched insertion sound individual data 610. Further, the insertion sound reproduction processing means 475B sets the setting value of the information related to the BPM of the insertion sound data 611 to a value corresponding to the music BPM of the music BPM information. Then, the insertion sound reproduction processing means 475B outputs the acquired insertion sound data 611 as a sound signal to the sound generation means 450 so that an insertion sound is output at the insertion position specified by the insertion position specification means 475A.

  For example, the inserted sound reproduction processing means 475B inserts “Jakajan” which is a sound of playing a drum based on the audio rhythm pattern information and the audio language information corresponding to the “Yay” audio as shown in FIG. Insert sound data 611 for outputting a sound is acquired. Then, the insertion sound reproduction processing means 475B sets information related to the BPM of the insertion sound data 611 in association with the music BPM, and as shown in FIG. 5C, the insertion position corresponding to the voice of “Yay” In other words, the reproduction of the “Jaca Jean” insertion sound is started at the reproduction time T4 and is output at the reproduction time T5. Also, the insertion sound reproduction processing means 475B acquires the insertion sound data 611 for outputting the insertion sound of “Jakajakajan”, which is a sound of playing a drum, as the insertion sound data 611 corresponding to the “Yay” sound. Then, in response to the voice of “Yay Ya”, the reproduction of the insertion sound of “Jakajakajan” is started at the reproduction time T8, and the insertion sound data 611 is output so as to end at the reproduction time T9. Furthermore, although not shown here, the inserted sound reproduction processing means 475B inserts the inserted sound data for outputting the inserted sound of “Don Don”, which is a sound of a Japanese drum, as the inserted sound data 611 corresponding to the sound of “Hyu”. 611 is acquired and output.

  Further, the insertion sound reproduction processing unit 475B acquires the operation rhythm pattern information and the operation content information from the memory 430 when the operation corresponding mode is set. Furthermore, inserted sound individual data 610 in which rhythm pattern information 612A related to a rhythm pattern that substantially matches the motion rhythm pattern of the motion rhythm pattern information and target motion information 612C related to a motion that substantially matches the motion content of the motion content information are incorporated. Is searched from the insertion sound list data 600. Then, the insertion sound reproduction processing means 475B outputs the insertion sound data 611 incorporated in the searched insertion sound individual data 610 so that the insertion sound is output at the insertion position specified by the insertion position specifying means 475A. The signal is output to the sound generation means 450 as a signal.

  For example, the inserted sound reproduction processing means 475B is a sound that plays a cymbal based on motion rhythm pattern information and motion content information corresponding to the motion of “pushing up a hand” as shown in FIG. The insertion sound data 611 for outputting the insertion sound is acquired. Then, as shown in FIG. 6C, the insertion sound reproduction processing means 475B starts reproduction of the insertion sound of “Jarn” at the reproduction time T4 in response to the operation of “pushing up the hand”. The insertion sound data 611 is output so as to end at T5. Also, the insertion sound reproduction processing means 475B acquires the insertion sound data 611 for outputting the insertion sound of “Pololone” that is a sound of playing the piano, as the insertion sound data 611 corresponding to the action of “shaking the head”. . Then, in response to the operation of “shaking the head”, the reproduction of the insertion sound of “Pololone” is started at the reproduction time T8, and the insertion sound data 611 is output so as to end at the reproduction time T9.

[Operation of music playback system]
Next, the operation of the music playback system 100 will be described with reference to the drawings.

(Inserted sound output processing in voice-compatible mode)
First, as an operation of the music playback system 100, an insertion sound output process in the voice corresponding mode will be described with reference to FIG. Here, a state in which the user has produced a sound as shown in FIG. 5B according to the music as shown in FIG. 5A will be described as an example. FIG. 7 is a flowchart showing an insertion sound output process in the voice corresponding mode.

  First, the user inputs a setting for requesting reproduction of a predetermined music piece in the voice corresponding mode by an input operation of the input unit 440. Then, as shown in FIG. 7, the music playback device 400 of the music playback system 100 recognizes a request for requesting playback of a predetermined music in the audio corresponding mode in the music playback processing unit 471 of the calculation unit 470. The music is reproduced based on the music data 511 corresponding to the predetermined music (step S101). Thereafter, the calculation means 470 recognizes the music BPM of the music being reproduced by the music rhythm recognition means 472 (step S102), and stores the music BPM information related to the music BPM in the memory 430. Here, the calculation means 470 reproduces a song as shown in FIG. 5A and stores the song BPM information corresponding to this song. Further, the calculation means 470 acquires the in-vehicle sound of the in-vehicle sound information output from the microphone 200 for a predetermined time by the audio rhythm recognition means 473A of the audio state recognition means 473 (step S103), and uses the in-vehicle sound as a reference. The sound below the volume is deleted (step S104). Then, the voice rhythm recognition means 473A determines whether or not the time length of the sound deleted from the vehicle interior sound is equal to or shorter than the voice recognition reference time (step S105).

  If it is determined in step S105 that the time is not shorter than the voice recognition reference time, that is, longer than the voice recognition reference time, it is recognized that the in-vehicle sound does not include the voice that the user consciously pronounced according to the music. . Then, the voice rhythm recognition unit 473A returns to step S103 and performs processing for acquiring the in-vehicle sound. On the other hand, if it is determined in step S105 that the voice rhythm recognition unit 473A is equal to or shorter than the voice recognition reference time, the voice rhythm recognition unit 473A recognizes that the in-vehicle sound includes voice consciously pronounced by the user. Here, when the sound rhythm recognition means 473A acquires the in-vehicle sound of the time including the reproduction time T2 and the reproduction time T6, the sound “Yay” or “Yay” uttered by the user consciously in the in-vehicle sound is obtained. Recognize that it is included. Then, based on the music BPM of the music BPM information stored in the memory 430 and the peak interval of the audio output waveform, the audio BPM based on the music BPM is recognized (step S106). Further, the voice rhythm recognition unit 473A recognizes the voice rhythm pattern corresponding to the voice based on the voice BPM (step S107), and stores the voice rhythm pattern information related to the voice rhythm pattern in the memory 430. Further, the voice state recognition unit 473 recognizes the voice language of the voice whose external rhythm pattern is recognized by the voice rhythm recognition unit 473A in the language recognition unit 473B (step S108), and stores the voice language information related to the voice language in the memory. 430 is stored. Here, the language recognizing unit 473B recognizes the language “Yay” or “Yayy” as a sound language when the sound rhythm recognizing unit 473A obtains the in-vehicle sound including the reproduction time T2 and the reproduction time T6. Then, the speech language information related to this speech language is stored.

  Thereafter, the calculation means 470 corresponds to the voice rhythm pattern and the voice language based on the voice rhythm pattern information and the voice language information stored in the memory 430 by the insertion sound reproduction processing means 475B of the insertion sound reproduction control means 475. The inserted sound individual data 610 to be searched is searched. Then, the insertion sound reproduction processing means 475B acquires the insertion sound data 611 of the searched insertion sound individual data 610 (step S109). Further, the insertion sound reproduction control means 475 recognizes the break of the measure of the music being reproduced by the insertion position specifying means 475A, and the music that appears second from the point in time when the sound whose speech language is recognized is emitted. The beginning of the bar is specified as the insertion position (step S110). Then, the insertion sound reproduction processing means 475B reproduces the insertion sound data 611 at a time corresponding to the insertion position specified by the insertion position specifying means 475A (step S111), and outputs an insertion sound that matches the music. Here, as shown in FIG. 5C, the insertion sound reproduction control means 475 reproduces the insertion sound of “Jakajan” corresponding to the sound of “Yay” from the reproduction time T4 to the reproduction time T5, and “Yay”. The inserted sound of “Jakajakajan” corresponding to the voice of “3” is reproduced from the reproduction time T8 to the reproduction time T9. Thereafter, the voice rhythm recognition unit 473A determines whether or not the reproduction of the music has ended (step S112). If it is determined in step S112 that the music has been reproduced, the insertion sound output process in the voice corresponding mode is terminated. On the other hand, when the sound rhythm recognition unit 473A determines that the reproduction of the music has not ended, the sound rhythm recognition unit 473A returns to step S103 to perform a process of acquiring sound.

(Inserted sound output processing in motion-compatible mode)
Next, as an operation of the music reproduction system 100, an insertion sound output process in the operation corresponding mode will be described with reference to FIG. Here, a state in which the user has performed the operation shown in FIG. 6B according to the music shown in FIG. 6A will be described as an example. In addition, description is simplified about the process similar to the process implemented by the insertion sound output process in audio | voice corresponding | compatible mode. FIG. 8 is a flowchart showing the insertion sound output process in the operation support mode.

  First, the user inputs a setting for requesting reproduction of a predetermined music piece in the operation-compatible mode by an input operation of the input unit 440. Then, as shown in FIG. 8, the music playback device 400 of the music playback system 100 recognizes a request for requesting playback of a predetermined music in the operation corresponding mode in the music playback processing unit 471 of the calculation unit 470. The predetermined music is reproduced (step S201). Thereafter, the calculation means 470 causes the music rhythm recognition means 472 to recognize the music BPM of the music being reproduced (step S202) and store the music BPM information in the memory 430. Further, the calculation means 470 acquires the imaging data output from the imaging means 300 for a predetermined time by the motion rhythm recognition means 474A of the motion state recognition means 474 (step S203), and the predetermined part of the user's body is detected. It is determined whether or not the body movement reference amount has been exceeded (step S204).

  In this step S204, when it is determined that the body movement reference amount is not moved, it is recognized that the user has not intentionally moved the body according to the music. Then, the motion rhythm recognition unit 474A returns to step S203 and performs processing for acquiring imaging data. On the other hand, in step S204, when it is determined that the movement rhythm recognition unit 474A is moving more than the physical movement reference amount, the movement rhythm recognition unit 474A recognizes that the user has consciously moved the body according to the music. Here, the motion rhythm recognition means 474A is an operation in which the user consciously “picks up his hand” or “shakes his head” when acquiring imaging data for a time including the playback time T2 and the playback time T6. We recognize that we performed. Then, based on the music BPM of the music BPM information stored in the memory 430 and the length of time until the predetermined part of the body moves from the predetermined position to another position, the operation BPM based on the music BPM is calculated. Recognize (step S205). Furthermore, the motion rhythm recognition unit 474A recognizes the motion rhythm pattern corresponding to the motion based on the motion BPM (step S206), and stores the motion rhythm pattern information regarding the motion rhythm pattern in the memory 430. Further, the motion state recognition unit 474 recognizes the motion content of the motion whose motion rhythm pattern has been recognized by the motion rhythm recognition unit 474A by the motion content recognition unit 474B (step S207), and obtains motion content information regarding the motion content. Store in the memory 430. Here, the action content recognition means 474B is said to “pick up the hand” or “shake the head” when the action rhythm recognition means 474A acquires imaging data for a time including the reproduction time T2 and the reproduction time T6. The operation is recognized as the operation content, and the operation content related to the operation content is stored.

  Thereafter, the calculation unit 470 responds to the operation rhythm pattern and the operation content based on the operation rhythm pattern information and the operation content information stored in the memory 430 by the insertion sound reproduction processing unit 475B of the insertion sound reproduction control unit 475. Insert sound data 611 to be acquired is acquired (step S208). Further, the insertion sound reproduction control means 475 specifies the beginning of the second measure of the music appearing as the insertion position from the point in time when the insertion position specifying means 475A performs the action whose action content is recognized (step). S209). Then, the insertion sound reproduction processing means 475B reproduces the insertion sound data 611 at a time corresponding to the insertion position specified by the insertion position specifying means 475A (step S210), and outputs an insertion sound that matches the music. Here, as shown in FIG. 6C, the insertion sound reproduction control means 475 reproduces the insertion sound of “Jarn” corresponding to the operation of “pick up the hand” from the reproduction time T4 to the reproduction time T5. The insertion sound of “Pololone” corresponding to the action of “shaking the head” is reproduced from the reproduction time T8 to the reproduction time T9. Thereafter, the motion rhythm recognition unit 474A determines whether or not the reproduction of the music has ended (step S211). If it is determined in step S211 that the music has been reproduced, the insertion sound output process in the motion corresponding mode is terminated. On the other hand, when the motion rhythm recognition unit 474A determines that the reproduction of the music has not ended, the operation rhythm recognition unit 474A returns to step S203 to perform processing for acquiring imaging data.

[Effects of music playback system]
As described above, in the above embodiment, when the calculation means 470 of the music playback device 400 recognizes a request to play back a predetermined music in, for example, the voice-compatible mode, the music playback processing means 471 recognizes this predetermined music. Start playing the song. In addition, the calculation unit 470 obtains in-vehicle sound including, for example, “Yay” sound collected by the microphone 200 by the voice rhythm recognition unit 473A of the voice state recognition unit 473. Furthermore, the voice rhythm recognition unit 473A recognizes the voice BPM of the “Yay” voice based on the music BPM of the music being played, and recognizes the voice rhythm pattern of the voice corresponding to the voice BPM. Thereafter, the calculation means 470 acquires the insertion sound data 611 associated with the rhythm pattern information 612A corresponding to the sound rhythm pattern by the insertion sound reproduction processing means 475B of the insertion sound reproduction control means 475. Then, the insertion sound reproduction processing means 475B causes the sound generation means 450 to output an insertion sound corresponding to the sound rhythm pattern based on the insertion sound data 611, that is, an insertion sound that matches the music BPM that is the characteristic of the music. For this reason, since the music reproducing device 400 inserts an insertion sound that synchronizes with the music, it is possible to reduce a sense of discomfort that the user feels when listening to the music. In addition, the user feels that the music is responding to the sound he / she makes, for example, because the insertion sound of “Jakajan” that synchronizes with the characteristics of the music is inserted into the music with respect to the sound of “Yay” You can enjoy listening to music more. Therefore, the music reproducing device 400 can output the insertion sound satisfactorily.

  Furthermore, the insertion sound reproduction processing means 475B outputs an insertion sound corresponding to the sound rhythm pattern of the sound emitted by the user. Specifically, the insertion sound reproduction processing means 475B, for example, outputs an insertion sound of “Jakajan” corresponding to “Yay” and outputs an insertion sound of “Jakajakajan” to “Yayye”. For this reason, the user can feel the sensation that the music is responding more favorably by the insertion sound of the rhythm pattern corresponding to the sound uttered by the user. Therefore, the music reproducing device 400 can output the insertion sound more satisfactorily.

  Then, the insertion sound reproduction processing unit 475B outputs an insertion sound corresponding to the voice language of the voice and the content of the operation. Specifically, the inserted sound reproduction processing means 475B outputs the drum sound of “Jakajan” corresponding to the sound of “Yay”, the Japanese drum sound of “Don Don” corresponding to the sound of “Hyu”, "Jurn" cymbal sound is output in response to the action of "pushing up", and "Pololone" piano sound is output in response to the action of "shaking the head". For this reason, the user can experience a sensation that the music is responding more satisfactorily by using different insertion sounds that are output corresponding to the voice and the content of the operation. Therefore, the music reproducing device 400 can output the insertion sound even better.

  Further, the insertion sound reproduction processing means 475B outputs an insertion sound that matches the music BPM as an insertion sound synchronized with the characteristics of the music. For this reason, the music reproducing device 400 can further reduce the uncomfortable feeling that the user feels when listening to the music, with the insertion sound that matches the music BPM. Therefore, the music reproducing device 400 can output the insertion sound more satisfactorily.

  Further, the insertion position specifying means 475A recognizes, for example, the beginning of the measure of the music appearing second from the time when the speech whose speech language is recognized by the language recognition means 473B is emitted as the insertion position for inserting the insertion sound. To do. For this reason, the music reproducing apparatus 400 can further reduce the uncomfortable feeling that the user feels when listening to the music, by using the insertion sound inserted at the timing matched with the measure. Therefore, the music reproducing device 400 can output the insertion sound even better.

  Further, the insertion position specifying means 475A specifies the insertion position so that the insertion sound is output within the output processing time limit from the time when the sound is emitted, for example. For this reason, the user can feel the sensation that the music is responding more satisfactorily by the insertion sound that is output within the output processing time limit after the sound is emitted. Therefore, the music reproducing device 400 can output the insertion sound even better.

  Further, the insertion sound reproduction processing means 475B outputs an insertion sound based on the insertion sound data 611 stored in the insertion sound data storage means 420. For this reason, the insertion sound reproduction processing means 475B can output the insertion sound by a simple method of acquiring and outputting the insertion sound data 611. Therefore, it is possible to reduce the processing load of the insertion sound reproduction processing means 475B.

  Further, MIDI format data is used as the insertion sound data 611. Then, the insertion sound reproduction processing means 475B sets information related to the BPM of the insertion sound data 611 according to the music BPM and reproduces it as an insertion sound. For this reason, the data amount of the insertion sound data 611 can be minimized. Further, the insertion sound reproduction processing means 475B can change the BPM of the insertion sound only by changing the setting value of the information related to the BPM of the insertion sound data 611, and one insertion sound data 611 can correspond to a plurality of BPMs. Insert sound can be output. Therefore, the capacity of the inserted sound data storage means 420 can be reduced, and the cost of the music reproducing device 400 can be reduced.

  Further, there is provided an insertion sound data storage means 420 for storing the insertion sound data 611 in association with the target language information 612B and the target motion information 612C. Then, the inserted sound reproduction processing means 475B searches the inserted sound data storage means 420 for target language information 612B related to a language that substantially matches the speech language, for example, and inserts the inserted sound data 611 associated with the searched target language information 612B. get. For this reason, the insertion sound reproduction processing means 475B can acquire the insertion sound data 611 only by searching the target language information 612B corresponding to the speech language. Therefore, the insertion sound reproduction processing unit 475B can acquire the insertion sound data 611 more easily. Also, the insertion sound reproduction processing means 475B can acquire the insertion sound data 611 more quickly than the configuration in which the insertion sound data storage means 420 is provided in a server device connected via a wireless medium or a wired medium network.

  Then, the insertion sound reproduction processing means 475B outputs an insertion sound corresponding to the sound collected by the microphone 200 in accordance with the music. For this reason, the user can output the insertion sound by a simple method that only emits a predetermined sound. Moreover, since the user only needs to output a sound, for example, an insertion sound can be output without interrupting driving. Therefore, the user can easily enjoy listening to music.

  In addition, when the voice rhythm recognition unit 473A recognizes that the sound inside the vehicle includes a sound of a reference volume or higher than the reference sound time, the insertion sound reproduction processing unit 475B outputs an insertion sound corresponding to the sound. Let For this reason, the music playback device 400 does not output an insertion sound in response to, for example, a running sound of a vehicle or a normal conversation, and inserts only in response to a voice uttered by a user according to the music. Sound can be output. Therefore, the music reproducing device 400 can output the insertion sound even better.

  Then, the insertion sound reproduction processing unit 475B outputs an insertion sound corresponding to the user's action imaged in the imaging data from the imaging unit 300 in accordance with the music. For this reason, the user can output the insertion sound by a simple method that only performs a predetermined operation. Therefore, the user can enjoy listening to music more easily.

  Furthermore, when the motion rhythm recognition unit 474A recognizes that the predetermined part of the body has moved in the predetermined direction by a body motion reference amount or more, the insertion sound reproduction processing unit 475B performs the motion of the predetermined part of the body. The insertion sound corresponding to is output. For this reason, the music reproducing device 400 does not output an insertion sound in response to, for example, a driving operation, and can output an insertion sound only in response to an operation performed by the user in accordance with the music. Therefore, the music reproducing device 400 can output the insertion sound even better.

  And the sound output control apparatus of this invention is applied to the music reproduction apparatus 400 which implements a music reproduction process. For this reason, the insertion sound corresponding to the sound can be output, and the convenience of the music reproducing device 400 can be enhanced.

[Modification of Embodiment]
In addition, this invention is not limited to each embodiment mentioned above, The deformation | transformation shown below is included in the range which can achieve the objective of this invention.

  In other words, the insertion sound reproduction processing means 475B may output an insertion sound that is tuned to at least one of the pitch, strength, and tone color of the music. For example, a configuration in which the volume level of the insertion sound is set in correspondence with the volume level of the music, or a configuration in which the tone color of the insertion sound is set in correspondence with the tone color of the music may be used. With such a configuration, the music playback device 400 does not output an insertion sound having a volume that is significantly higher than the volume of the music or an insertion sound having a tone that is significantly different from the tone of the music, for example. It is possible to reduce the discomfort felt by the user.

  Further, for example, the music rhythm recognition unit 472 recognizes the chord progression of the music being played and predicts the chord progression. The inserted sound reproduction processing means 475B may output an inserted sound corresponding to the predicted chord. With such a configuration, the user can listen to the insertion sound that harmonizes with the music, and can more enjoy listening to the music.

  And it is good also as a structure which makes an insertion sound respond | correspond only to the audio | voice speech language and the content of an operation | movement, without making it respond | correspond to an audio | voice rhythm pattern or an operation | movement rhythm pattern. With such a configuration, it is not necessary to provide each rhythm recognition unit 473A, 474A, and the configuration of the calculation unit 470 can be simplified. Therefore, the cost of the music reproducing device 400 can be reduced.

  Moreover, it is good also as a structure which makes an insertion sound respond | correspond only to an audio | voice rhythm pattern and an operation | movement rhythm pattern, without making it respond | correspond to the audio | voice speech language and the content of operation | movement. With such a configuration, there is no need to provide the language recognition unit 473B and the operation content recognition unit 474B, the configuration of the calculation unit 470 can be simplified, and the cost of the music reproducing device 400 can be reduced.

  Furthermore, it is good also as a structure etc. which select and output the insertion sound matched with the music BPM in the insertion sound reproduction | regeneration processing means 475B, without making an insertion sound respond | correspond to an audio | voice and operation | movement. With such a configuration, it is not necessary to provide the state recognition units 473 and 474, the configuration of the calculation unit 470 can be further simplified, and the cost of the music reproducing device 400 can be further reduced.

  Further, the insertion sound reproduction processing means 475B may output an insertion sound that matches a rhythm such as the time signature or beat of the music instead of the insertion sound that matches the music BPM. For example, in the case where the “Jaca Jean” insertion sound is output twice for a music with a 4-beat rhythm as shown in FIG. 9A, the “Jaca Jean” insertion sound is output as shown in FIG. 9B. It is good also as a structure which outputs in 2 beats, or a structure which outputs in 1 beat as shown in FIG.9 (C). Further, when the insertion sound of “Jakajakajan” is output twice, the insertion sound of “Jakajakajan” is output in 3 beats as shown in FIG. 9D, or as shown in FIG. 9E. It is good also as a structure which outputs in 1 beat. Even with such a configuration, the music reproducing device 400 inserts an insertion sound that is in tune with the music, so that it is possible to reduce a sense of incongruity felt by the user when listening to the music.

  Then, the insertion sound reproduction processing means 475B may be configured to output insertion sounds that are respectively tuned to the beginning and end of the music, and further to the so-called rust portion that is the rising portion of the music. For example, in the insertion sound reproduction processing means 475B, an insertion sound that is loose and gentle corresponding to the beginning of the music is inserted into a flashy feeling corresponding to the sub part, for example, “Jakajakajakajakajakajakajan” For example, the drum sound may be configured to output a drum sound of “Dekedeke Jaka Jaka Jaka Jaka Jane”, which is reminiscent of the end corresponding to the end. With such a configuration, the user can listen to the insertion sound that synchronizes with the tune of the music, and can further enjoy listening to the music.

  Further, the inserted sound reproduction processing means 475B may output the inserted sound not at the timing of matching the music bar, but at the timing of the music, for example, the beat or beat of the music or the rhythm such as BPM. Even with such a configuration, the music reproducing device 400 can further reduce the uncomfortable feeling felt by the user when listening to the music by the insertion sound inserted at a timing that matches the rhythm, and can output the insertion sound more satisfactorily. .

  In addition, the configuration in which the insertion position specifying unit 475A specifies the insertion position in accordance with the measure has been exemplified, but the present invention is not limited to this, and the output processing start time is from the time when the sound is emitted or the operation is performed. For example, it is good also as a structure which specifies the time of 3 second passing as an insertion position. With such a configuration, the insertion position specifying unit 475A can easily specify the insertion position without recognizing the measure of the music. Therefore, it is possible to reduce the processing load on the insertion position specifying means 475A. Furthermore, the configuration of the insertion position specifying means 475A can be simplified.

  In the insertion position specifying unit 475A, for example, the configuration for specifying the insertion position so that the insertion sound is output within the output processing time limit from the time when the sound is emitted is exemplified. It is good also as such a structure. That is, the insertion position specifying unit 475A may be configured to appropriately specify the insertion position so that the insertion sound is output after the output processing time limit has elapsed. Even with such a configuration, the music reproducing device 400 inserts an insertion sound that is in tune with the music, so that it is possible to reduce a sense of incongruity felt by the user when listening to the music.

  Further, although the insertion sound reproduction processing unit 475B has exemplified the configuration in which the insertion sound is output based on the insertion sound data 611 acquired from the insertion sound data storage unit 420, the present invention is not limited thereto, and the insertion sound reproduction processing unit 475B It is good also as a structure which produces | generates insertion sound data and outputs insertion sound. With such a configuration, it is not necessary to provide the insertion sound data storage means 420 in the music playback device 400, and the weight of the music playback device 400 can be reduced. Moreover, the cost of the music reproducing device 400 can be reduced.

  Further, the insertion sound data storage unit 420 may be provided in a server device connected to the music playback device 400 via a network of a wireless medium or a wired medium. With such a configuration, the weight of the music reproducing device 400 can be reduced. Further, the insertion sound data 611 can be shared with other music reproducing devices 400 connected to the server device.

  Although the MIDI format data is used as the insertion sound data 611, data in another format such as the WAVE format or the MPEG format may be used. With such a configuration, the music reproducing device 400 can appropriately output, for example, natural sounds or human voices that cannot be reproduced in the MIDI format as insertion sounds. Therefore, the music reproducing device 400 can make the user further enjoy listening to the music.

  Furthermore, although the voice rhythm recognition unit 473A has been illustrated with respect to the configuration in which the function of determining whether or not the sound within the vehicle includes the voice of the reference volume or higher is higher than the voice recognition reference time, the configuration in which such a function is not provided It is good. In addition, although the operation rhythm recognition unit 474A is illustrated with the configuration provided with the function of determining whether or not the predetermined part of the body has moved more than the physical motion reference amount in the predetermined direction, the configuration may not be provided with such a function. . With such a configuration, the processing in step S105 and step S204 can be omitted, and the processing load on each rhythm recognition means 473A, 474A can be reduced. Moreover, the structure of each rhythm recognition means 473A, 474A can be simplified, and the cost of the music reproducing device 400 can be reduced.

  And although the composition which provided the function which outputs the insertion sound corresponding to an audio | voice and the insertion sound corresponding to operation | movement was illustrated in the music reproduction apparatus 400, it is good also as a structure which does not provide any one function. With such a configuration, it is not necessary to provide any one of the state recognition units 473 and 474, the configuration of the calculation unit 470 can be simplified, and the cost of the music reproducing device 400 can be reduced.

  In addition, the configuration for outputting the insertion sound in response to the sound generated by the user is illustrated, but not limited to this, the user's hand clapping, nose singing, whistling sound, and tambourine, castanets, maracas and other musical instruments It is good also as a structure which outputs an insertion sound corresponding to the sound to play. Here, the musical instrument may or may not be connected to the music playback device 400 via a wired or wireless network. With such a configuration, the user can experience the feeling that the music is responding to the various sounds generated by himself or the instrument sounds played by the insertion sound corresponding to the various sounds described above. You can enjoy listening more.

  Further, for example, the music reproduction processing means 471 is caused to function as the superimposed music data generation means of the present invention. Then, the music reproduction processing unit 471 generates superimposed music data in which the insertion sound of the insertion sound data 611 acquired by the insertion sound reproduction processing unit 475B is superimposed on the music of the music data 511 being reproduced. Thereafter, the music reproduction processing means 471 may reproduce the superimposed music data and output the music in which the insertion sound is inserted. With such a configuration, it is possible to output the music and the insertion sound without causing the insertion sound reproduction processing means 475B to perform the insertion sound output process. Therefore, the processing load of the computing means 470 at the time of outputting the insertion sound can be reduced as compared with the above embodiment in which the music and the insertion sound are output by the processing of each reproduction processing means 471 and 475B.

  Further, for example, the configuration of a music piece being reproduced may be predicted, and an insertion sound based on this configuration may be output. That is, for example, the waveform of music is analyzed to detect a high and low part of phoneme attack density. Furthermore, chord progression prediction is performed based on each detected part, and a music composition is predicted by detecting a pattern from the chord progression prediction. For example, at the beginning or end of a song, a configuration may be adopted in which an insertion sound using a pattern simulating voice or rhythm as a basic pattern is output. In addition, before the rust of the music, an insertion sound of an extended pattern in which a decorative note is added to the key pattern may be output. Furthermore, for example, after a so-called solo interlude where an interlude is played with a single instrument, an insertion sound of an extended pattern with an increased proportion of decoration compared to before the chorus may be output at the most exciting part of the music. .

  Here, as a configuration for generating the insertion sound of the extended pattern as described above, for example, the following configuration can be exemplified. That is, for example, the minimum resolution of the decoration additional sound is determined by the minimum beat feeling of the music. A configuration that generates an insertion sound in which a decorative note is added to a portion other than the accent at the determined minimum resolution, using the keynote pattern as an accent, for example, a decorative pattern in the front part, the rear part, and the front and rear parts of the keynote pattern. It is good also as a structure which adds a simple note. For example, for a percussion instrument, a configuration may be adopted in which the number of timbres used as the insertion sound is increased, such as adding a tom or cymbal to the insertion sound corresponding to the sound of the snare drum.

  Moreover, although the composition which applied the music reproduction apparatus 400 of this invention to what is called a vehicle-mounted car audio was illustrated, you may apply to what is called a karaoke apparatus etc. which output only the accompaniment music of a personal computer, a robot, and a music. Further, the sound output control device of the present invention is not limited to the configuration in which the sound output control device is applied to the music playback device 400, but the music rhythm recognition unit 472, the sound state recognition unit 473, the operation state recognition unit 474, and the insertion sound playback control unit 475 are independent. It is good also as the structure made to do. And the structure which made each means 472-475 independent may be provided in various electronic devices and electric devices, such as a personal computer, a robot, and a mobile telephone. Here, as music for recognizing rhythm by music rhythm recognizing means 472 in which each means 472 to 475 is made independent, music output from a television receiver or radio receiver, mobile phone or stationary telephone incoming call Examples include sound and music played at a public concert.

  Each function described above is constructed as a program, but may be configured by hardware such as a circuit board or an element such as a single integrated circuit (IC), and can be used in any form. In addition, by adopting a configuration that allows reading from a program or a separate recording medium, handling is easy, and usage can be easily expanded.

  In addition, the specific structure and procedure for carrying out the present invention can be changed as appropriate to other structures and the like within the scope of achieving the object of the present invention.

[Effect of the embodiment]
As described above, in the above-described embodiment, when the calculation unit 470 of the music playback device 400 recognizes a request to play back a predetermined music in, for example, the voice corresponding mode, the playback of the predetermined music starts. The in-vehicle sound including, for example, “Yay” sound collected by the microphone 200 is acquired. Further, the calculation means 470 recognizes the voice BPM of the “Yay” voice based on the music BPM of the music being reproduced, and recognizes the voice rhythm pattern of the voice corresponding to the voice BPM. Thereafter, the calculation means 470 matches the insertion sound corresponding to the voice rhythm pattern, that is, the music BPM that is the characteristic of the music, based on the insertion sound data 611 associated with the rhythm pattern information 612A corresponding to the voice rhythm pattern. The insertion sound is output from the sound generation means 450 according to the music. For this reason, since the music reproducing device 400 inserts an insertion sound that synchronizes with the music, it is possible to reduce a sense of discomfort that the user feels when listening to the music. Therefore, the music reproducing device 400 can output the insertion sound satisfactorily.

It is a block diagram which shows schematic structure of the music reproduction system which concerns on one embodiment of this invention. It is a schematic diagram which shows schematic structure of the music list data in the said embodiment. It is a schematic diagram which shows schematic structure of the insertion sound list data in the said embodiment. It is a schematic diagram which shows schematic structure of the calculating means which comprises the music reproduction apparatus in the said embodiment. It is a schematic diagram which shows an example of the output state of the insertion sound in the audio | voice corresponding | compatible mode in the said embodiment, (A) is the lyric of a music, (B) is an audio | voice, (C) is an insertion sound. FIG. 5 is a schematic diagram illustrating an example of an output state of an insertion sound in the motion corresponding mode in the embodiment, where (A) is a song lyrics, (B) is an operation, and (C) is an insertion sound. It is a flowchart which shows the insertion sound output process in the audio | voice corresponding | compatible mode in the said embodiment. It is a flowchart which shows the insertion sound output process in the operation | movement corresponding mode in the said embodiment. FIG. 6 is a schematic diagram illustrating an example of an output state of an insertion sound according to another embodiment of the present invention, where (A) is a rhythm of a song, (B) is an output pattern 1 of an insertion sound of “Jakajan”, and (C ) Is an output pattern 2 of an insertion sound of “Jakajajan”, (D) is an output pattern 1 of an insertion sound of “Jakajakajan”, and (E) is an output pattern 2 of an insertion sound of “Jakajakajan”.

Explanation of symbols

400 Music Reproducing Device 420 Inserted Sound Data Storage Means as Sound Data Storage Means 450 Sound Output Means and Sound Generation Means as Music Output Means 470 Arithmetic Means 471 Music Reproduction Processing Means that Can Function as Superposed Music Data Generation Means 472 Sound Output Control Music rhythm recognizing means as a music characteristic recognizing means constituting the apparatus 473 External state information acquiring means constituting the sound output control apparatus and voice state recognizing means as the external state rhythm recognizing means 474 External constituting the sound output control apparatus Operating state recognition means as state information acquisition means and external state rhythm recognition means 475 Inserted sound reproduction control means as sound output control means constituting the sound output control device 511 Music data 611 Inserted sound data as sound data 612B Predetermined external state Target language information as information 612C Predetermined Target operation information as external status information

Claims (26)

A sound output control device for outputting a predetermined sound from the sound output means,
An external state information acquisition means for acquiring external state information related to the external state;
Music characteristic recognition means for recognizing the sound characteristics of the music data output from the music output means for outputting music data relating to the music;
Recognizing that the external state information related to the predetermined external state has been acquired, a sound output control means for controlling the sound output means to output a sound in an output form synchronized with the characteristics of the sound;
A sound output control device comprising: The sound output control device according to claim 1,
The sound output control means controls to output a sound in an output form that is tuned to at least one of the pitch, strength, and tone color of the music. apparatus. The sound output control device according to claim 1 or 2,
The sound output control means controls to output a sound in an output form synchronized with the chord of the music. The sound output control device according to any one of claims 1 to 3,
Comprising external state rhythm recognition means for recognizing the rhythm of the predetermined external state;
The sound output control device is characterized in that the sound output control means controls to output a sound in an output form synchronized with the rhythm of the predetermined external state. The sound output control device according to any one of claims 1 to 4,
The sound output control device is characterized in that the sound output control means controls to output a sound in an output form corresponding to the content of the predetermined external state. The sound output control device according to any one of claims 1 to 5,
The sound output control means controls to output a sound in an output form synchronized with the rhythm of the music. The sound output control device according to claim 6,
The sound output control means controls to output a sound in an output form in accordance with BPM (Beats Per Minute) of the music. The sound output control device according to claim 6 or 7,
The sound output control means controls to output a sound in an output form synchronized with the tune of the music. The sound output control device according to any one of claims 1 to 5,
The sound output control device controls the output of the sound at a timing according to the rhythm of the music. The sound output control device according to claim 9,
The sound output control device is characterized in that the sound output control means controls to output the sound at a timing according to a measure of the music. The sound output control device according to any one of claims 1 to 10,
The sound output control means performs control to output the sound before an output processing deadline time that is a predetermined time after the external state information relating to the predetermined external state is acquired. Sound output control device. The sound output control device according to any one of claims 1 to 8,
The sound output control means performs control to output the sound when recognizing that an elapsed time from the acquisition of the external state information relating to the predetermined external state has reached an output processing start time which is a predetermined time. Sound output control device characterized by. The sound output control device according to any one of claims 1 to 12,
The sound output control device, wherein the sound output control means acquires sound data related to the sound output in the output form, and controls to output the sound in the output form based on the sound data. The sound output control device according to claim 13,
The sound data conforms to the MIDI (Musical Instrument Digital Interface) standard,
The sound output control device, wherein the sound output control means performs control to change a set value of information relating to an output form in the sound data to output a sound of the output form. The sound output control device according to claim 13 or 14,
Sound data storage means for storing the sound data in association with predetermined external state information relating to the predetermined external state;
The sound output control means retrieves the predetermined external state information related to the predetermined external state of the acquired external state information from the sound data storage means, and the sound associated with the retrieved predetermined external state information A sound output control device characterized by acquiring data. The sound output control device according to any one of claims 1 to 15,
The external state information acquisition means acquires external sound information related to an external sound as the external state information,
When the sound output control means recognizes that the external sound information related to the predetermined external sound has been acquired, the sound output control means controls to output the sound. The sound output control device according to claim 16,
The sound output control unit is configured to output the sound when recognizing that a time length of the external sound equal to or less than a reference volume in the external sound of the external sound information is equal to or less than an external sound recognition reference time that is a predetermined time. A sound output control device characterized by The sound output control device according to claim 16 or 17,
The sound output control device, wherein the external sound is a sound. The sound output control device according to any one of claims 1 to 18,
The external state information acquisition means acquires body motion information related to body motion as the external state information,
When the sound output control means recognizes that the body motion information related to a predetermined body motion has been acquired, the sound output control means controls to output the sound. The sound output control device according to claim 19,
The sound output control means performs control to output the sound when recognizing that the amount of movement of the body in the body movement of the body movement information is equal to or greater than a body movement recognition reference amount that is a predetermined movement amount. Sound output control device characterized by. The sound output control device according to any one of claims 1 to 20,
Music reproduction processing means for performing processing for reproducing the music data and outputting the music data from the music output means;
A music playback device characterized by comprising: The sound output control device according to any one of claims 1 to 20,
Superimposed music data generating means for acquiring the music data and generating superimposed music data in which the sound whose output is controlled by the sound output control means of the sound output control device is superimposed on the sound of the music data;
Music reproduction processing means for performing processing for reproducing the superimposed music data and outputting the music from the music output means;
A music playback device characterized by comprising: A sound output control method for outputting a predetermined sound from sound output means,
Get external state information about the external state,
Recognizing the sound characteristics of the music data output from the music output means for outputting music data related to the music,
When recognizing that the external state information relating to a predetermined external state has been acquired, a sound output control method is provided for controlling the sound output means to output a sound in an output form that is tuned to the sound characteristics. 21. A sound output control program that causes a computing means to function as the sound output control device according to claim 1. A sound output control program for causing a calculation means to execute the sound output control method according to claim 23. 26. A recording medium on which a sound output control program according to claim 24 or 25 is recorded so as to be readable by an arithmetic means.

Images