WO2021038833A1

WO2021038833A1 - Acoustic space creation apparatus

Info

Publication number: WO2021038833A1
Application number: PCT/JP2019/034122
Authority: WO
Inventors: 潤耶及川
Original assignee: ソニフィデア合同会社
Priority date: 2019-08-30
Filing date: 2019-08-30
Publication date: 2021-03-04
Also published as: EP4024391A4; EP4024391A1; JP6710428B1; JPWO2021038833A1

Abstract

[Problem] To provide an apparatus capable of more easily providing an improvised performance based on a user motion. [Solution] An acoustic space creation apparatus provided with: a camera 20 for imaging an object; a light source 30 for irradiating the object with light; an input unit 61 for acquiring an image from the camera; a processing unit 62 that selects a sound to be generated in accordance with the frame of a video; and a plurality of speakers 50 that outputs the sound selected by the processing unit. The processing unit compares the quantity of light in a frame with the quantity of light in the immediately preceding frame to determine whether the object is in motion or stationary on the basis of a predetermined threshold for the light quantity. When determining that the object is in motion, the processing unit: generates a motion detection signal; selects a musical pitch to be output in accordance with a pre-stored musical structure program; allocates one of the plurality of speakers to each sound of the selected musical pitch in accordance with a pre-stored spatial structure program; and causes the allocated speaker to output a sound. When a state of motion occurs consecutively a first predetermined number of times during a first unit time and the number of times of the consecutive occurrences for the first predetermined number of times reaches a second predetermined number of times, and the processing unit determines that the object is in motion after being stationary during a second unit time or greater, with a specific speaker among the plurality of speakers allocated to a predetermined sound, the processing unit causes the specific speaker to generate a sine wave in accordance with the musical structure program. When a state of motion occurs consecutively the first predetermined number of times during the first unit time and the number of times of consecutive occurrences for the first predetermined number of times reaches a third predetermined number of times, and the processing unit determines that the object is in motion after being stationary during the second unit time or greater, with a specific speaker allocated to a selected sound, the processing unit causes the specific speaker to output a first reverberation in accordance with the musical structure program. When a state of motion occurs consecutively a fourth predetermined number of times during the first unit time, the processing unit causes a speaker to output a second reverberation in a predetermined order in accordance with the musical structure program.

Description

Acoustic space generator

The present invention relates to an acoustic space generator.

There is known a musical tone generator that can automatically create musical tone data according to the motion of a person in front of the camera and play it. For example, Patent Document 1 below discloses a configuration of a musical tone generator that creates musical tone data based on the position and amount of change of the motion of a person who is the subject of the camera and outputs the music as sound. Has been done. According to this music generator, the performer does not need to acquire the knowledge and skills to operate the device that automatically plays music and to play the musical instrument, and the performer can improvise by simply exercising in front of the camera. The performance can be easily performed.

Japanese Patent No. 3643829

As described above, a device that outputs a sound that reacts with the movement of the subject is known, but there is a device that can recognize the movement of the subject from a viewpoint different from the conventional technology and play a new musical tone generated by the recognition. For example, the musical sound generated according to the movement of the subject can be further enjoyed. In view of these points, in the present invention, the state of movement of the object is different from that of the prior art so that the musical sound generated according to the state of movement of the object (for example, the movement of the performer) can be further enjoyed. The main purpose is to provide a device that recognizes from different viewpoints and plays a new musical tone generated by this.

Further, in the musical sound generator described in Patent Document 1, it is necessary to both specify the position of the movement of the camera on the subject and calculate the amount of change in the movement of the subject. Therefore, the musical tone generator has a possibility that the burden of data processing at the time of playing becomes large. In view of these points, it is also an object of the present invention to provide an apparatus capable of more easily performing improvisational performance based on the state of movement of an object.

The present invention focuses on simple movements of "moving" and "stopping", and temporal sensations (intervals) of various movements such as physical expression, breathing method, "pause", "pause", and ON / OFF of things. ) Was conceived for the purpose of making it a sound or a syllable. Then, in the present invention, a device that mainly focuses on the repetition of "movement (movement)" and "stationary" of an object, and generates and organizes a melody, a syllable, and an acoustic space according to a specific scale based on this. I will provide a.

That is, the present invention comprises a camera for photographing an object, a light source for irradiating the object, an input unit for acquiring an image from the camera, a processing unit for selecting generated sound according to a frame of an image, and a processing unit. A plurality of speakers that output a selected sound are provided, and the processing unit compares the light amount of the frame with the light amount of the frame immediately before the frame, and the object is based on a preset light amount threshold. It determines whether it is in a moving state or a stationary state, and when it is determined that it is in a moving state, it generates a motion detection signal, selects the pitch to be output by the pre-stored music structure program, and stores it in advance. According to the spatial structure program, one of a plurality of speakers is assigned to each sound of the selected pitch, the sound is output from the assigned speaker, and the state of movement is the first in the first unit time. It occurs in a predetermined number of times in a row, and after the number of times in which the first predetermined number of times has occurred reaches the second predetermined number of times and the state is stationary for the second unit time or more, it is determined that the state is in motion. When a specific speaker is assigned to a predetermined sound from a plurality of speakers, a sine wave is generated from the specific speaker by the music structure program, and the state of movement is performed in the first unit time. Is generated in the first predetermined number of times in a row, and when the number of times in which the first predetermined number of times is continuously reached reaches the third predetermined number of times, the state of movement is after being in a stationary state for a second unit time or more. When it is determined that the sound is, and when a specific speaker is assigned to the selected sound, the music structure program causes the specific speaker to output the first reverberant sound, and the first unit time. This is an acoustic space generation device that outputs a second reverberant sound from a speaker in a predetermined order by a music structure program when a fourth predetermined number of consecutive motion states are generated.

Further, the present invention outputs a sensor capable of detecting the state of an object, an input unit that acquires a signal from the sensor, a processing unit that selects a sound to be generated according to the signal, and a sound selected by the processing unit. A plurality of speakers are provided, and the processing unit compares a signal with a signal immediately preceding the signal, and determines whether the object is in a moving state or a stationary state based on a preset signal threshold. When it is determined that the sound is in a motion state, a motion detection signal is generated, the pitch to be output is selected by the pre-stored music structure program, and the sound is selected by the pre-stored spatial structure program. One of a plurality of speakers is assigned to each sound of the pitch, the sound is output from the assigned speaker, and the state of movement occurs continuously in the first predetermined number of times in the first unit time, and the first When the number of times of occurrence in a predetermined number of times reaches the second predetermined number of times, the sound is stationary for the second unit time or more, and then it is determined that the sound is in motion, and the sound is set to a predetermined sound. On the other hand, when a specific speaker is assigned among a plurality of speakers, a sine wave is generated from the specific speaker by the music structure program, and a state of movement is generated for the first predetermined number of times in a row in the first unit time. , When the number of times of occurrence in the first predetermined number of times reaches the third predetermined number of times, when it is determined to be in a moving state after being in a stationary state for a second unit time or more, and When a specific speaker is assigned to the selected sound, the music structure program outputs the first reverberant sound from the specific sound, and the state of movement is the fourth predetermined in the first unit time. This is an acoustic space generator that outputs a second reverberant sound from a speaker in a predetermined order by a music structure program when it occurs continuously a number of times.

In the acoustic space generator of the present invention, each state of motion and rest is determined based on the movement of an object (for example, a subject), a sound is selected according to the frequency, period, number of times, etc. of each state, and the selected sound. Is output as a musical sound. Therefore, according to the present invention, it is possible to easily play a musical sound according to the mode of movement of the object, and it is possible to easily provide an acoustic space for playing such a musical sound. Further, according to the present invention, it is possible to play the musical sound or sound of a new syllable generated based on the repetitive motion of the motion and the rest of the object.

It is a schematic diagram which shows an example of the acoustic space generation apparatus which concerns on embodiment, (a) is a view seen from the front side, (b) is a view seen from above. It is a block diagram which shows the main part of the acoustic space generation apparatus of FIG. It is a figure which shows an example of the use state of the acoustic space generation apparatus of FIG. It is a flowchart which shows an example of the operation of the acoustic space generation apparatus of FIG. It is a flowchart which shows the processing of the processing part at the time of outputting the sound of a predetermined pitch. It is a figure which shows the audio file of the pitch. It is a figure for demonstrating an example of operation of a processing part. It is a flowchart which shows the generation process of a sine wave. It is a figure for demonstrating an example of operation of a processing part. It is a flowchart which shows the output processing of the 1st reverberation sound. It is a figure for demonstrating an example of operation of a processing part. It is a flowchart which shows the output processing of the 2nd reverberation sound. It is a figure for demonstrating an example of operation of a processing part.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to this. Further, in the drawings, in order to explain the embodiment, the scale is appropriately changed and expressed, such as drawing a part in a large or emphasized manner. 1A and 1B are views showing an outline of an example of the configuration of the acoustic space generator 100 according to the embodiment, FIG. 1A is a view seen from the front side, and FIG. 1B is a view seen from above. Note that FIG. 1A shows a state in which the wall portion 11b on the front side is transmitted, and FIG. 1B shows a state in which the ceiling portion 11c is transmitted.

The acoustic space generation device 100 according to the embodiment is a device that generates a melody or syllable according to a specific scale and generates an acoustic space 10 in which the melody or syllable is played. In the embodiment, the acoustic space 10 is a space for playing music.

In the embodiment, the acoustic space 10 is the internal space of the structure 11. The structure 11 is a container-like structure, and is formed to be hollow and movable. As shown in FIG. 1, for example, the structure 11 has a substantially square floor surface 11a in a plan view, a wall surface 11b rising from each side of the floor surface 11a, and above the floor surface 11a so as to block the internal space. It is configured to include a ceiling surface 11c arranged in. The acoustic space 10 is a space defined by the floor surface 11a, the wall surface 11b, and the ceiling surface 11c of the structure 11. The acoustic space 10 is not limited to the internal space of a container-like structure such as the structure 11 described above, and any space capable of propagating sound can be applied. That is, the acoustic space 10 may be, for example, an internal space of a building such as a concert hall or an event hall, or an underground space. Further, the acoustic space 10 is not limited to a closed space, and may be, for example, a space on an outdoor stage, a space on the ground surface, or the like. In this case, the camera 20, the light source 30, the stand 40, the plurality of speakers 50, and the like, which will be described later, are installed on, for example, an outdoor stage or the surface of the earth.

The acoustic space generation device 100 includes a camera 20, a light source 30, a stand 40, a plurality of speakers 50, and an information processing device 60 (see FIG. 2). The camera 20, the light source 30, and the base 40 are installed inside the acoustic space 10, but may be installed outside the acoustic space 10. For example, the camera 20 and the like may be installed at a place away from the acoustic space 10.

The camera 20 is a device capable of photographing the movement of the subject (object) (that is, the hand of the user U) H (see FIG. 3). The camera 20 is installed in the vicinity of the ceiling portion 11c so as to face downward. The camera 20 captures the subject H from above and captures a moving image of the subject H. The camera 20 is arranged in the central portion of the acoustic space 10 in a plan view. The camera 20 is supported in the air, and is supported by, for example, a support metal fitting (not shown) extending in the horizontal direction from the wall portion 11b. The camera 20 is not limited to the above configuration, and the orientation of the camera 20, the installation location, the installation method, and the like can be arbitrarily set. For example, the camera 20 is installed facing upward or horizontally. It may be installed in a state of being fixed to the ceiling portion 11c, or may be installed in a state of being suspended from the ceiling portion 11c.

The camera 20 continuously captures the subject H and transmits the captured image to the information processing device 60. Specifically, the camera 20 captures the subject H at a preset frame rate, and inputs each frame as image data to the input unit 61 (see FIG. 2) of the information processing device 60. The camera 20 is connected to the information processing device 60 so as to be capable of data communication, and the image data is input from the camera 20 to the input unit 61 via wired (for example, USB or Ethernet (registered trademark)) or wireless (for example, various radio communication or the Internet). Will be sent to. The frame rate of the camera 20 is set according to the speed of movement of the subject H, the performance of the information processing device 60, and the like. In the embodiment, the frame rate of the camera 20 is set to 40 fps (frames per second). In this case, the camera 20 photographs the subject H at a frequency of 40 times per second, and transmits the acquired frame images at a frequency of 40 images per second to the information processing device 60. The frame rate of the camera 20 is not limited to 40 fps, and may be set to, for example, 25 fps, 30 fps, 50 fps, 60 fps, or the like.

The light source 30 is an instrument or device that emits light that irradiates the subject H. The light source 30 is arranged in the vicinity of the ceiling portion 11c in the central portion of the plan view inside the acoustic space 10. The light source 30 is, for example, an LED spotlight. The spotlight is an illumination that intensively illuminates a part of the acoustic space 10. The light source 30 is installed so as to face downward, and emits light L directly downward. As a result, the light L is projected onto a part of the region R (see FIG. 3) of the upper surface 41 of the table 40. The light source 30 is provided integrally with the camera 20. The light source 30 is not limited to the above configuration, and may be, for example, an arc lamp, an incandescent lamp, a fluorescent lamp, sunlight, or the like, or one that uniformly illuminates a wide range including the periphery of the table 40. You may. Further, the light source 30 can be arbitrarily set with respect to the direction of the light L to be emitted, the installation location, the installation method, and the like. For example, the light source 30 may be installed so as to emit the light L in the upward or horizontal direction. It may be installed separately from the camera 20.

The stand 40 is installed on the floor portion 11a below the light source 30. The height of the table 40 is set to a height corresponding to the position of the waist of the user (performer) U in an upright posture, for example (see FIG. 3). The base 40 is installed substantially in the center of the surface of the floor portion 11a, and has a rectangular parallelepiped shape with a height of 90 cm (cm) and a length and width of 45 cm. The upper surface 41 of the table 40 is flat and is arranged so as to include the entire region R to which the emitted light L of the light source 30 is irradiated. The table 40 is not limited to the above configuration, and its shape, size, arrangement, and the like can be appropriately changed. Further, it is arbitrary whether or not such a stand 40 is installed in the acoustic space generation device 100.

As described above, the light L is emitted from the light source 30 directly below. When the light L emitted from the light source 30 goes downward as it is, it irradiates, for example, a circular region R on the upper surface 41 of the table 40. At this time, the substantially conical space having the light source 30 as the apex and the upper surface 41 of the base 40 as the bottom surface is partially illuminated by the light L in the acoustic space 10 which is set to be dark as a whole. Such a substantially conical space illuminated by the light L of the light source 30 is referred to as a light irradiation space S (see FIG. 3).

Five speakers 50 are installed in the acoustic space 10. These five speakers 50 output sound based on the music data generated by the information processing device 60. The first speaker 51, the second speaker 52, the third speaker 53, the fourth speaker 54, and the fifth speaker 55 each have

sound emitting units

51a, 52a, 55a that emit sound in a predetermined direction.

The fifth speaker 55 is arranged in the central part and the upper side in the plan view in the acoustic space 10. The fifth speaker 55 is provided on the upper surface of the camera 20, and is installed with the sound emitting portion 55a facing upward.

On the other hand, the first speaker 51, the second speaker 52, the third speaker 53, and the fourth speaker 54 are arranged at the bottom in the acoustic space 10 and are evenly spaced on the same circumference centered on the fifth speaker 55. It is arranged so as to be. The first to fourth speakers 51 to 54 are installed at four corners of a substantially square floor surface 11a, respectively. Further, each of the first to fourth speakers 51 to 54 is in a state where the

sound emitting portions

51a, 52a, 55a are directed toward the center side of the acoustic space 10, and is horizontally so as to emit sound slightly upward. On the other hand, it is installed in a state of being tilted upward by about 5 ° to 25 °.

The plurality of speakers 50 included in the acoustic space generator 100 are not limited to the configurations of the first to fifth speakers 51 to 55 described above. That is, the number of speakers 51 and the like installed in the acoustic space 10, the arrangement of the speakers 51 and the like, the orientation of the sound emitting portion 51a and the like, and the like can be appropriately changed. Specifically, for example, the number of speakers included in the acoustic space generator 100 may be 2 or more, 4 or less, or 6 or more. Further, for example, all the speakers 51 to 55 constituting the five speakers 50 may be arranged at the upper portion, the bottom portion, or the central portion in the acoustic space 10. You may. Further, for example, the fifth speaker 55 may be installed away from the camera 20 with the sound emitting portion 55a facing downward so as to emit sound downward, and the first to fourth speakers 51 and the like may be installed. It may be installed so as to emit sound in the horizontal direction. Further, although the first to fifth speakers 51 and the like are all speakers having the same configuration, some or all of them may be speakers having different configurations.

The information processing device 60 is composed of, for example, a computer. The information processing device 60 is communicably connected to each of the first to fifth speakers 51 and the like and the camera 20 via wired or wireless communication. The information processing device 60 acquires the image of the camera 20 and outputs a predetermined sound from the first to fifth speakers 51 and the like. The information processing device 60 is installed outside the acoustic space 10, but may be installed inside the acoustic space 10. FIG. 2 is a block diagram showing a main part of the acoustic space generation device 100. As shown in FIG. 2, the information processing device 60 includes an input unit 61, a processing unit 62, and a storage unit 63.

The input unit 61 acquires the image of the camera 20. The input unit 61 acquires a plurality of frames captured by the camera 20 as image data. The processing unit 62 performs predetermined processing based on the input image data, selects a pitch according to the operation state of the subject H, and outputs each sound of the selected pitch from the speaker 50. The data processing and the like in the processing unit 62 will be described later. Further, the processing unit 62 is realized by a configuration including, for example, a CPU. The storage unit 63 stores image data input from the camera 20, data generated by the processing unit 62, and the like. In addition, the storage unit 63 also stores a program for executing the processing of the processing unit 62, and a music structure program and a spatial structure program described later. The storage unit 63 is realized by, for example, a memory or a hard disk.

Next, how to use the acoustic space generator 100 will be described. FIG. 3 is a diagram showing an example of a usage state of the acoustic space generation device 100. As shown in FIG. 3, the user U stands on the back side of the table 40 and performs an operation of moving or stopping the hand H with the hand H in the light irradiation space S. The acoustic space generator 100 is used in this way. At this time, the user U puts both the left and right hand H or one hand H in the light irradiation space S, and moves, rotates, or moves the finger in the vertical direction or the horizontal direction, for example. , Exercises such as expanding and closing the palm of the hand H, and temporarily stopping the movement of the hand H. Such a series of movements of the hand H is photographed by the camera 20. Then, the sound generated according to the operation of the hand H is output from the speaker 50.

Next, the operation of the acoustic space generator 100 will be described. FIG. 4 is a flowchart showing an example of the operation of the acoustic space generation device 100. Hereinafter, the operation of the acoustic space generation device 100 will be described with reference to the flowchart of FIG.

First, the camera 20 captures a moving image of the subject (object) H (step S01). Then, the image data captured by the camera 20 is transmitted to the information processing device 60 (step S02). For example, in the usage state shown in FIG. 3, the camera 20 continuously photographs the hand H in the light irradiation space S at a preset frame rate, and inputs the information processing device 60 as image data for each frame. Enter in 61.

When the image data (frame image) for each frame is input to the input unit 61, the processing unit 62 executes a predetermined process and a predetermined operation based on the image data (step S03). Then, a musical sound or the like is output from the speaker 50 (step S04). The sound output from the speaker 50 has a melody or a syllable composed of sounds of a predetermined pitch selected by the processing unit 62.

The sound output from the speaker 50 includes a sine wave (sine wave) sound and a reverberation sound in addition to such a melody or syllable sound if a predetermined condition is satisfied. The conditions under which the sine wave sound is output from the speaker 50 and the conditions under which the reverberation sound is output will be described later.

By the operations of steps S01 to S04 of the acoustic space generation device 100, a musical sound automatically composed according to the operation of the subject H is played from the speaker 50, whereby the acoustic space 10 is generated.

Subsequently, the content of the processing of the processing unit 62 in step S03 will be specifically described. FIG. 5 is a flowchart showing the processing of the processing unit 62 when outputting a sound having a predetermined pitch. The processing of the processing unit 62 is automatically executed based on, for example, the program stored in the storage unit 63.

As shown in FIG. 5, the processing unit 62 first acquires a frame image (step S11). Next, the processing unit 62 compares the light amount of the image data of the frame with the light amount of the image data of the frame acquired immediately before the frame (step S12). In step S12, the amount of light in the entire area of the frame image is compared, but instead, the amount of light in a part of a predetermined area in the frame image may be compared.

Subsequently, it is determined whether or not the difference in the amount of light is equal to or greater than the threshold value (step S13). In step S13, the processing unit 62 determines whether the difference between the light amount of the image data of the frame and the light amount of the image data of the frame acquired immediately before the frame is equal to or more than the threshold value or less than the threshold value. The threshold value is a value of a predetermined amount of light, is set in advance, and is stored in the storage unit 63.

When the difference in the amount of light described above is equal to or greater than the threshold value (when “YES” in step S13), the processing unit 62 determines that the subject H is in a “motion” state (step S14). Then, when the processing unit 62 determines that the subject H is in a moving state, it generates one motion sensing signal (step S15).

On the other hand, when the processing unit 62 determines that the difference in the amount of light described above is less than the threshold value (when "NO" in step S13), the processing unit 62 determines that the subject H is in the "stop" state (step S24). When the processing unit 62 determines that the subject H is in a stationary state, the processing unit 62 may generate one pause / lingering signal (step S25).

When one motion detection signal is generated in step S15, the processing unit 62 selects one output sound pitch based on the preset and stored music structure program (step S16). In step S16, for example, the pitch of the sound to be output from the file having the data shown in FIG. 6 is selected based on the music structure program. In this case, one pitch is selected from the 42 pitch data (A1 to A42) shown in FIG. The pitch selected in step S16 may be selected from a recording material such as acoustic noise instead of the 42 pitch data (A1 to A42) shown in FIG. Further, in step S16, a plurality of pitches may be selected at the same time from the data (A1 to A42) of the 42 pitches shown in FIG.

Following step S16, the processing unit 62 assigns a speaker to a sound of a selected pitch by a preset and stored spatial structure program (step S17). In step S17, any one of the five speakers 51 to 55 is assigned to each sound of the selected pitch based on the spatial structure program. When a plurality of pitches are selected at the same time in step S16, any one of the five speakers 51 to 55 is assigned to each of the selected pitches in step S17.

FIG. 6 is a diagram showing an audio file of pitch. FIG. 6 shows, for example, data on 42 pitches. In FIG. 6, A1 to A42 are serial numbers of 42 pitches. These 42 pitches are the pitches that can be selected in step S16. The MIDI number is also called a note number or a note number, and is a numerical value indicating the pitch and range of MIDI (Musical Instrument Digital Interface). The polyphony is the number of sounds that can be output at the same time, and is also a value corresponding to the number of layers. Further, the appearance frequency of each pitch when the motion sensing signal is generated 127 times indicates how many times each pitch is selected when the motion sensing signal is generated 127 times. The output frequency of SP1 / SP2 / SP3 / SP4 / SP5 when the motion sensing signal is generated 20 times means that when the motion sensing signal is generated 20 times and a sound having a predetermined pitch is selected 20 times. How many times are assigned to each of the first speaker 51 (SP1), the second speaker 52 (SP2), the third speaker 53 (SP3), the fourth speaker 54 (SP4), and the fifth speaker 55 (SP5)? Is shown.

Numerical values related to the appearance frequency of each pitch and the allocation frequency of each speaker for the sound of each pitch are set in advance. In step S16 and step S17, the pitch is automatically selected and the speaker 50 is assigned according to the numerical values relating to the appearance frequency and the allocation frequency.

Returning to FIG. 5, the processing unit 62 outputs a sound having a predetermined pitch from the assigned speaker 50 (step S18). In step S18, the sound of the selected pitch is output from the assigned speaker 50. When the speaker 50 outputs continuous sound, the speaker 51 or the like assigned to each sound constituting the continuous sound is changed (for example, a different speaker 51 or the like is used for each sound constituting the continuous sound). (Assignment, etc.), the sound may be output from the speaker 50 so as to move up, down, left and right in the acoustic space 10.

FIG. 7 is a diagram for explaining an example of the operation of the processing unit 62. As shown in FIG. 7, by the above-described processing of the processing unit 62, when the subject H is in a moving state, the sound of the pitch selected based on the music structure program is output from the speaker 50. On the other hand, when the subject H is in a stationary state, the speaker 50 is temporarily in a paused state in which no sound is output, and as a result, the acoustic space 10 is in a state of silence or lingering sound. Therefore, when the subject H keeps moving continuously, the sound of a predetermined pitch is continuously output, so that the sound as a texture is generated. On the other hand, when the subject H appropriately sandwiches a stationary state in the moving state, the sound expresses between the movements and the lingering sound can be heard. As a result, an acoustic space 10 in which a melody or a syllable plays is generated.

Further, in the process of step S03 described above, the processing unit 62 generates a sine wave from the speaker 50 under a predetermined condition. This sine wave is output in a state of being synthesized with respect to the waveform of the sound having the predetermined pitch described above. Therefore, the processing of the processing unit 62 regarding the generation of such a sine wave will be described.

FIG. 8 is a flowchart showing a sine wave generation process in the processing unit 62. The processing unit 62 executes the following processing. As shown in FIG. 8, it is determined whether or not the motion sensing signal is continuously generated a predetermined number of times in the first unit time (step S31). That is, it is determined whether or not the state of movement has occurred continuously in the first predetermined number of times in the first unit time. When it is determined that the occurrence has occurred (in the case of "YES" in step S31), it is determined whether the number of occurrences in the first predetermined number of times in the step S31 has reached the second predetermined number of times (step S32). When it is determined that the signal has been reached (in the case of "YES" in step S32), it is determined whether or not the motion detection signal is generated after the rest state for the second unit time or more (step S33). Then, when it is determined that the speaker is in a moving state after being in a stationary state for the second unit time or more (in the case of “YES” in step S33), further, the plurality of speakers 50 for a predetermined sound. When a specific speaker 51 or the like is assigned (when “YES” in step S34), a sine wave is generated from the specific speaker 51 or the like (step S35). In the above-mentioned sine wave generation process, the determination of the stationary state may be recognized by not generating the motion detection signal, or may be recognized by detecting the pause / lingering signal.

Specifically, for example, it is as follows. FIG. 9 is a diagram for explaining an example of the operation of the processing unit 62. The first unit time is set to, for example, 180 ms, the first predetermined number of times is set to, for example, 1 to 30, the second unit time is set to, for example, 500 ms, and the second predetermined number of times is set to, for example, 10 times. To. These set values are stored in the storage unit 63. As shown in FIG. 9, the processing unit 62 continuously generates motion sensing signals 1 to 30 times in a period of 180 ms or less, the total of the cycles reaches 10 times, and the processing unit 62 is in a stationary state for 500 ms or more. When the motion detection signal is detected, for example, the fifth speaker 55 is assigned, and the first sine wave is generated from the fifth speaker 55 for 60 to 90 seconds. Further, when the sound of the pitch selected by the generation of the motion detection signal in step S33 is the sound of any of the pitches A19 to A25 among the data of the 42 pitches of FIG. Generates a second sine wave in addition to the first sine wave. The second sine wave is a combination of the third sine wave and the fourth sine wave, which will be described later, by a volume curve of 10 seconds. Here, the third sine wave is a sine wave having a frequency two octaves higher than the pitch selected by the generation of the motion sensing signal in step S33. The fourth sine wave is a sine wave in which a frequency of 0.5 to 11 Hz is added to the third sine wave. The second sine wave is output from, for example, the fifth speaker 55 for 10 seconds. In the 4th sine wave, when the numerical value is changed, the reached value is reached by using 1500 to 4000 ms time complementation.

Further, in the process of step S03 described above, the processing unit 62 outputs the first reverberation sound from the speaker 50 under a predetermined condition. Therefore, subsequently, the processing of the processing unit 62 regarding the output of the first reverberation sound will be described.

FIG. 10 is a flowchart showing the output processing of the first reverberation sound. The processing unit 62 executes the following processing. As shown in FIG. 10, it is determined whether or not the motion sensing signal is continuously generated a predetermined number of times in the first unit time (step S41). When it is determined that the occurrence has occurred (in the case of "YES" in step S31), it is determined whether the number of occurrences in the first predetermined number of times in the step S41 has reached the third predetermined number of times (step S42). When it is determined that the object has been reached (in the case of "YES" in step S42), it is subsequently determined whether or not the motion detection signal is generated after the rest state for the second unit time or more (step S43). Then, when it is determined that the speaker is in a moving state after being in a stationary state for the second unit time or more (in the case of “YES” in step S43), the plurality of speakers 50 are further used for the selected sound. When a specific speaker is assigned (when “YES” in step S44), the first reverberation sound is output from the specific speaker 50 (step S45).

Specifically, for example, it is as follows. FIG. 11 is a diagram for explaining an example of the operation of the processing unit 62. The first unit time, the first predetermined number of times, and the second unit time are set in advance, for example, to the above values. The third predetermined number of times is set to, for example, 3 to 5 times. These set values are stored in the storage unit 63. As shown in FIG. 11, the processing unit 62 counts the period in which the motion sensing signal is continuously generated 1 to 30 times in a period of 180 ms or less as 1, and when the 3rd to 5th count is reached, further 500 ms or more. When the motion detection signal is detected after being in a stationary state, for example, the fifth speaker 55 is assigned, and the first reverberation sound is output from the fifth speaker 55 for 60 to 90 seconds. At this time, when the fifth speaker 55 is the speaker assigned to the sound of the pitch selected based on the motion detection signal after being stationary for 500 ms or more, the first reverberation sound is from the fifth speaker 55 to the above. It is combined with the pitch sound and output.

Furthermore, in the process of step S03 described above, the processing unit 62 outputs the second reverberation sound from the speaker 50 under a predetermined condition. Therefore, subsequently, the processing of the processing unit 62 regarding the output of the second reverberation sound will be described.

FIG. 12 is a flowchart showing the output processing of the second reverberation sound. The processing unit 62 executes the following processing. As shown in FIG. 12, it is determined whether or not the motion sensing signal is continuously generated a fourth predetermined number of times in the first unit time (step S51). When it is determined that the sound has occurred (when “YES” in step S51), and when a specific speaker among the plurality of speakers 50 is assigned to the selected sound (when “YES” in step S52). , The second reverberation sound is output from the specific speaker 50 (step S53).

Specifically, for example, it is as follows. FIG. 13 is a diagram for explaining an example of the operation of the processing unit 62. In advance, the first unit time is set to, for example, the above value. Further, the fourth predetermined number of times is set to, for example, 30 times. These set values are stored in the storage unit 63. As shown in FIG. 13, the processing unit 62 outputs the second reverberation sound from the speaker 50 when the motion detection signal is continuously generated 30 times in a period of 180 ms or less. The processing unit 62 synthesizes the second reverberation sound with, for example, the sounds of the pitches A1 to A18 and A70 to A79 out of the data of the 42 pitches shown in FIG. 6, and outputs the second reverberation sound for 3 to 10 seconds. Let me. Further, the processing unit 62 outputs the second reverberation sound from, for example, the first speaker 51 and the second speaker 52, and also outputs the second reverberation sound from the third speaker 53 and the fourth speaker 54 after 1 second of acoustic movement.

As described above, the processing unit 62 outputs a sound having a predetermined pitch from the speaker 50 by the processing of steps S11 to S18. Further, the processing unit 62 generates a sine wave from the speaker 50 by the processing of steps S31 to S35. Further, the processing unit 62 outputs the first reverberation sound from the speaker 50 by the processing of steps S41 to S45. Furthermore, the processing unit 62 outputs the second reverberation sound from the speaker 50 by the processing of steps S51 to S53.

As described above, in the acoustic space generation device 100, the sound space generation device 100 is stored by the motion sensing signal emitted from the input unit 61 (the processing unit 62 that determines the repetition (movement / rest time variation) of "movement / rest" through the sensor input). The acoustic space 10 is generated via the music structure program and the space structure program implemented in the unit 63. Then, according to the acoustic space generation device 100, each state of the motion and the rest is determined based on the motion of the subject H. Then, a sound is selected according to the frequency, cycle, number of times, etc. of each state, and the selected sound is output as a musical sound. Therefore, according to the acoustic space generator 100, the musical sound according to the mode of movement of the object is easily performed. It is possible to easily provide an acoustic space 10 that can play such a musical sound. Further, according to the acoustic space generator 100, a new sound is generated based on a repetitive motion of an object moving and resting. It is possible to play musical sounds and sounds of syllables. Further, under a predetermined condition, an acoustic space 10 that plays a unique musical sound including a sound generated by synthesizing a sine wave, a first reverberation sound, a second reverberation sound, and the like. Can be provided.

Further, in the acoustic space generation device 100, since the moving subject H is irradiated with the light L, the user U plays improvisational music while observing the movement of the subject H illuminated by the light L. It is possible to appreciate it. Thereby, according to the acoustic space generation device 100, it is possible to provide the acoustic space 10 having a new entertainment property that allows the user U to enjoy improvised music with visual changes. Then, according to the acoustic space generator 100, the user U is made to experience or hear the sounds related to the body's breathing such as physical expression, yoga, and welfare, which are generated from the state of movement and rest. Therefore, it is possible not only to experience or listen to music, but also to contribute to the improvement of its spirit.

Subsequently, a modification of the acoustic space generator 100 according to the above embodiment will be described. The acoustic space generation device 100 includes a camera 20 for photographing a subject and a light source 30 for irradiating the subject H. However, the state of an object can be detected instead of both the camera 20 and the light source 30. It may be configured to include various sensors. Therefore, in the following, the configuration of the acoustic space generator according to the modified example will be specifically described.

In the configuration of the acoustic space generator according to the above-described modification, as the sensor capable of detecting the state of the object, for example, a distance sensor capable of detecting the distance to the object can be applied. The camera 20 is also one of the sensors capable of detecting the movement of an object.

The operations of the input unit and the processing unit of the acoustic space generator according to such a modification are the same as the operations of the input unit 61 and the processing unit 62 described above. However, the input unit according to the modified example continuously acquires signals from the sensor according to the distance from the sensor to the object, for example. Further, the processing unit of the modified example compares the acquired signal with the signal acquired immediately before, and if the difference is equal to or greater than the threshold value, determines that the state of movement of the object. The threshold value is a value related to the signal and is set in advance. The subsequent processing of the processing unit is the same as that of the processing unit 62 described above. That is, in the modified example, a sensor capable of detecting the state of an object, an input unit that acquires a signal from the sensor, a processing unit that selects a sound to be generated according to the signal, and a sound selected by the processing unit are used. A plurality of speakers 50 for output are provided, and the processing unit compares the signal with the signal immediately before the signal, and the object is in a moving state or a stationary state based on the above threshold value of the preset signal. When it is determined that the sound is in motion, a motion detection signal is generated, the pitch to be output is selected by the pre-stored music structure program, and the pre-stored spatial structure program is used. One of a plurality of speakers 50 is assigned to each sound of the selected pitch, and the sound is output from the assigned speaker. Other configurations of the acoustic space generator according to the modified example are the same as those of the acoustic space generator 100 according to the above-described embodiment. In addition to the sound of the above pitch, the processing unit of the acoustic space generator according to the modified example outputs a sine wave, a first reverberation sound, and a second reverberation sound to a plurality of speakers according to the frequency, period, number of times, etc. of each state. Output from 50.

According to such a modification, the acoustic space generator can be realized with a simpler configuration. Therefore, for example, it is possible to configure the audio space generator with a small PC having a moving image camera function. As a result, it becomes easy to apply the acoustic space generator to music teaching materials for children.

In addition to the above modification, the acoustic space generator 100 can also have the following configuration. Specifically, for example, the processing unit determines in advance whether the cell in which the sensor is mounted is stepped on and the state in which the foot is taken off from the cell, or whether the buzzer switch is ON / OFF. It may be determined whether the object is in a moving state or a stationary state based on the set signal threshold value. That is, for example, using a sensor capable of detecting contact with the foot, different signals are used when the foot is in contact with the square and when the foot is separated (for example, when the foot is in contact, the signal is separated by 1. When it is, 0) is output from the sensor to the processing unit, or a sensor that can detect the ON state of the buzzer is used, and different signals (for example, ON) are used when the buzzer switch is ON and OFF. When it is 1, it outputs 0) when it is OFF, and it outputs it from the sensor to the processing unit. Then, the processing unit compares the signal acquired from the sensor with the signal acquired immediately before, and determines whether the difference is equal to or greater than the threshold value (for example, 1 or more). As a result, the processing unit determines whether the object is in a moving state (a state in which the movement is changed) or a stationary state (a state in which the movement is unchanged) (for example, when the above difference is 1, the moving unit moves. If the above difference is 0, it is determined to be a stationary state). Then, when the processing unit determines that it is in a motion state, it generates a motion sensing signal, selects a pitch to be output by the music structure program stored in advance, and is selected by the spatial structure program stored in advance. One of a plurality of speakers is assigned to each sound of the pitch, and the sound is output from the assigned speaker. Further, in addition to the sound of the above pitch, a sine wave, a first reverberation sound, and a second reverberation sound are output from the plurality of speakers 50 according to the frequency, cycle, number of times, etc. of each state.

Further, in the acoustic space generation device 100, the processing unit attacks (that is, the sound is resolutely strong) based on a preset signal threshold in the input voice information (for example, the voice input to the microphone). It may be determined whether or not the object is in a moving state or a stationary state by determining the presence or absence of (exit, sound output, and adult rise). That is, for example, the acoustic space generation device 100 includes a microphone that is a voice recognition device for picking up the voice of an object, and the processing unit compares the voice signal acquired from the microphone that is the sensor with the voice signal acquired immediately before. , It may be determined whether the difference (for example, the difference in volume) is equal to or more than a preset threshold value, and whether the object is in a moving state or a stationary state.

Although the embodiments and modifications of the present invention have been described above, the technical scope of the present invention is not limited to the above embodiments. For example, an object whose movement is detected by a sensor such as a camera is not limited to the hand H, and may be, for example, a person's foot or the whole body, or an object held by a person (for example, a stick). It may be the shadow of a leaf, a wave, or the like. Further, the plurality of speakers 50 included in the acoustic space generator 100 may be composed of a plurality of stationary speakers 51 or the like as in the embodiment, or a pair of speakers that directly output sound to both ears of the user U. Headphones or binaural earphones equipped with speakers may be used. Further, the acoustic space 10 generated by the acoustic space generation device 100 is a real space, but may be a virtual space. That is, for example, when headphones or binaural earphones are applied as the speaker 50 of the acoustic space generator 100, the acoustic space generator 100 provides a virtual acoustic space to the user U who wears the speaker 50. To do. Further, in the above-described embodiment and modification, processing such as discriminating between a motion state (motion) and a stationary state (stop) of the state of the object by using a sensor capable of detecting the movement of the object (subject) is performed. The sound is generated according to the result, but here, the movement of the object (subject) is not limited to the movement of the actual object. That is, the acoustic space generator of the present invention is, for example, a motion state for the motion of an object in a specific fictitious place in virtual reality (VR) or in a game (such as on OpenGL® drawing). A process such as discriminating between and a stationary state (stop) may be performed to generate a sound corresponding to the process. Further, the acoustic space generation device 100 stores a plurality of music structure programs having algorithms in which different conditions (threshold values and the like) are set in the storage unit 63, and the processing of the above step S16 is performed by the plurality of music structure programs. It may be appropriately selected from the above and executed, or a plurality of music structure programs may be executed at the same time. In this case, the plurality of music structure programs are stored in the storage unit 63 in a state of being divided into different layers, for example. As a result, the music structure program can be appropriately selected according to the usage environment of the acoustic space generation device 100 and the number of mounting locations, and the music structure program can be easily improved or added.

Further, the acoustic space generator of the present invention has, for example, a system configuration in which a large number of speakers 51 or the like are randomly installed on a tree branch in a natural environment, or a flat surface (for example, a painting, a book, or a wall surface (for example, vertical). (Square wall surfaces with horizontal lengths of 10 cm and 5 m each)), a product configuration in which multiple ultra-small speakers 51 and sensors are studded, and sensors and speakers 51 and the like mounted on multiple street lights that are also light sources 30. It may be realized by the configuration of the environmental system. Further, the acoustic space generation device of the present invention may constitute a complex improvisational ensemble device / musical instrument. The acoustic space generator in this case is, for example, changed / assigned to another recording material for 42 numbers (A1 to A42) to which each pitch file shown in FIG. 6 is assigned, or mounted in the storage unit 63. It may be configured to perform processing such as associating the operations of other music structure programs (layers) that have been performed. That is, for example, when the pitch number "A2" is selected in step S16, the noise recording material and the sound collection / output from the microphone are played in real time in the present acoustic space, and then the pitch number "A2" and the like are transmitted. When the number of is selected once or a plurality of times, the sound collection output of the microphone may be turned off, and at the same time, another music structure program / layer may be operated. Although the acoustic space generation device 100 of the above embodiment includes a plurality of speakers 50, the number of speakers 51 and the like may be one.

H Subject, hand (object)
20 camera (sensor)
30 light source
50, 51, 52, 53, 54, 55 Speaker 61 Input unit 62 Processing unit 100 Acoustic space generator

Claims

A camera that shoots objects and
The light source that irradiates the object and
An input unit that acquires an image from the camera and
A processing unit that selects the generated sound according to the frame of the video, and
A plurality of speakers that output the sound selected by the processing unit are provided.
The processing unit
The light amount of the frame is compared with the light amount of the frame immediately before the frame, and it is determined whether the object is in a moving state or a stationary state based on a preset threshold value of the light amount.
When it is determined to be in a motion state, a motion detection signal is generated, a pitch to be output is selected by a pre-stored music structure program, and each sound of the selected pitch is selected by a pre-stored spatial structure program. On the other hand, one of the plurality of speakers is assigned, and the sound is output from the assigned speaker.
In the first unit time, the state of movement occurs in the first predetermined number of times in a row, and the number of times in which the first predetermined number of times occurs in succession reaches the second predetermined number of times, and is equal to or longer than the second unit time. When it is determined to be in a moving state after being in a stationary state, and when a specific speaker among the plurality of speakers is assigned to a predetermined sound, the specific speaker is specified by the music structure program. Generate a sine wave from the speaker of
When the state of movement occurs continuously in the first predetermined number of times in the first unit time and the number of times generated in the first predetermined number of times reaches a third predetermined number of times, the second unit When it is determined that the sound is in motion after being stationary for an hour or longer, and when the specific speaker is assigned to the selected sound, the music structure program determines the specific sound. Output the first reverberation sound from the speaker,
An acoustic space generator that outputs a second reverberation sound from the speaker in a predetermined order by the music structure program when the motion state occurs continuously in the fourth predetermined number of times in the first unit time. ..
A sensor that can detect the state of an object and
An input unit that acquires a signal from the sensor and
A processing unit that selects the sound to be generated according to the signal, and
A plurality of speakers that output the sound selected by the processing unit are provided.
The processing unit
The signal is compared with the signal immediately preceding the signal, and based on a preset signal threshold, it is determined whether the object is in a moving state or a stationary state.
When it is determined to be in a motion state, a motion detection signal is generated, a pitch to be output is selected by a pre-stored music structure program, and each sound of the selected pitch is selected by a pre-stored spatial structure program. On the other hand, one of the plurality of speakers is assigned, and the sound is output from the assigned speaker.
In the first unit time, the state of movement occurs in the first predetermined number of times in a row, and the number of times in which the first predetermined number of times occurs in succession reaches the second predetermined number of times, and is equal to or longer than the second unit time. When it is determined to be in a moving state after being in a stationary state, and when a specific speaker among the plurality of speakers is assigned to a predetermined sound, the specific speaker is specified by the music structure program. Generate a sine wave from the speaker of
When the state of movement occurs continuously in the first predetermined number of times in the first unit time and the number of times generated in the first predetermined number of times reaches a third predetermined number of times, the second unit When it is determined that the sound is in motion after being in a stationary state for an hour or more, and when the specific speaker is assigned to the selected sound, the music structure program determines the specific sound. Output the first reverberation sound from the speaker,
An acoustic space generator that outputs a second reverberation sound from the speaker in a predetermined order by the music structure program when the state of movement occurs continuously in the fourth predetermined number of times in the first unit time. ..