WO2021033593A1 - 信号処理装置および方法、並びにプログラム - Google Patents

信号処理装置および方法、並びにプログラム Download PDF

Info

Publication number
WO2021033593A1
WO2021033593A1 PCT/JP2020/030560 JP2020030560W WO2021033593A1 WO 2021033593 A1 WO2021033593 A1 WO 2021033593A1 JP 2020030560 W JP2020030560 W JP 2020030560W WO 2021033593 A1 WO2021033593 A1 WO 2021033593A1
Authority
WO
WIPO (PCT)
Prior art keywords
acoustic
user
sound
sensing value
signal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/030560
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
稀淳 金
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Priority to US17/635,073 priority Critical patent/US20220293073A1/en
Priority to JP2021540738A priority patent/JP7574797B2/ja
Priority to CN202080058671.7A priority patent/CN114258565B/zh
Publication of WO2021033593A1 publication Critical patent/WO2021033593A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/02—Means for controlling the tone frequencies, e.g. attack or decay; Means for producing special musical effects, e.g. vibratos or glissandos
    • G10H1/04—Means for controlling the tone frequencies, e.g. attack or decay; Means for producing special musical effects, e.g. vibratos or glissandos by additional modulation
    • G10H1/053—Means for controlling the tone frequencies, e.g. attack or decay; Means for producing special musical effects, e.g. vibratos or glissandos by additional modulation during execution only
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0008—Associated control or indicating means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0033—Recording/reproducing or transmission of music for electrophonic musical instruments
    • G10H1/0041—Recording/reproducing or transmission of music for electrophonic musical instruments in coded form
    • G10H1/0058—Transmission between separate instruments or between individual components of a musical system
    • G10H1/0066—Transmission between separate instruments or between individual components of a musical system using a MIDI interface
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0033—Recording/reproducing or transmission of music for electrophonic musical instruments
    • G10H1/0083—Recording/reproducing or transmission of music for electrophonic musical instruments using wireless transmission, e.g. radio, light, infrared
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0091—Means for obtaining special acoustic effects
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/46—Volume control
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/155—Musical effects
    • G10H2210/195—Modulation effects, i.e. smooth non-discontinuous variations over a time interval, e.g. within a note, melody or musical transition, of any sound parameter, e.g. amplitude, pitch, spectral response or playback speed
    • G10H2210/201—Vibrato, i.e. rapid, repetitive and smooth variation of amplitude, pitch or timbre within a note or chord
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/155—Musical effects
    • G10H2210/195—Modulation effects, i.e. smooth non-discontinuous variations over a time interval, e.g. within a note, melody or musical transition, of any sound parameter, e.g. amplitude, pitch, spectral response or playback speed
    • G10H2210/221—Glissando, i.e. pitch smoothly sliding from one note to another, e.g. gliss, glide, slide, bend, smear or sweep
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/155—Musical effects
    • G10H2210/195—Modulation effects, i.e. smooth non-discontinuous variations over a time interval, e.g. within a note, melody or musical transition, of any sound parameter, e.g. amplitude, pitch, spectral response or playback speed
    • G10H2210/241—Scratch effects, i.e. emulating playback velocity or pitch manipulation effects normally obtained by a disc-jockey manually rotating a LP record forward and backward
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/091—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith
    • G10H2220/101—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith for graphical creation, edition or control of musical data or parameters
    • G10H2220/116—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith for graphical creation, edition or control of musical data or parameters for graphical editing of sound parameters or waveforms, e.g. by graphical interactive control of timbre, partials or envelope
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/161—User input interfaces for electrophonic musical instruments with 2D or x/y surface coordinates sensing
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/201—User input interfaces for electrophonic musical instruments for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/321—Garment sensors, i.e. musical control means with trigger surfaces or joint angle sensors, worn as a garment by the player, e.g. bracelet, intelligent clothing
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/391—Angle sensing for musical purposes, using data from a gyroscope, gyrometer or other angular velocity or angular movement sensing device
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/395—Acceleration sensing or accelerometer use, e.g. 3D movement computation by integration of accelerometer data, angle sensing with respect to the vertical, i.e. gravity sensing
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/441—Image sensing, i.e. capturing images or optical patterns for musical purposes or musical control purposes
    • G10H2220/455—Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2230/00—General physical, ergonomic or hardware implementation of electrophonic musical tools or instruments, e.g. shape or architecture
    • G10H2230/005—Device type or category
    • G10H2230/015—PDA [personal digital assistant] or palmtop computing devices used for musical purposes, e.g. portable music players, tablet computers, e-readers or smart phones in which mobile telephony functions need not be used

Definitions

  • the present technology relates to signal processing devices and methods, and programs, and to signal processing devices, methods, and programs that enable intuitive operation of sound.
  • Patent Document 1 a technique for manipulating sound according to the movement of the user's body has been proposed (see, for example, Patent Document 1).
  • Patent Document 1 since the effect processing is executed based on the output waveform of the sensor mounted on the user, when the user moves the mounting portion of the sensor, the sound reproduced changes according to the movement.
  • the DJ can move the arm up and down to change the volume of the sound being reproduced, that is, to operate the sound.
  • This technology was made in view of such a situation, and makes it possible to intuitively operate the sound.
  • the signal processing device of one aspect of the present technology performs a non-linear acoustic processing on an acoustic signal according to an acquisition unit that acquires a sensing value indicating the movement of a predetermined part of the user's body or an instrument and the sensing value. It is equipped with a control unit.
  • the signal processing method or program of one aspect of the present technology is a step of acquiring a sensing value indicating the movement of a predetermined part of the user's body or an instrument and performing non-linear acoustic processing on the acoustic signal according to the sensing value. including.
  • a sensing value indicating the movement of a predetermined part of the user's body or an instrument is acquired, and non-linear acoustic processing is performed on the acoustic signal according to the sensing value.
  • the arm is most frequently and fastest in the upward range when viewed from the DJ, for example, in the range where the arm is pushed forward (horizontal state) and the angle at which the arm is moved upward is 45 degrees or more. Often moved.
  • the DJ should be able to intuitively operate the sound. is there.
  • the DJ's arm is at the top or the bottom.
  • the sound changes linearly with the change in the position (height) of the DJ's arm regardless of whether or not it is in. Then, the change in the sound imagined when the DJ moves his arm and the change in the actual sound will be different, and intuitive operation will be difficult.
  • non-linear acoustic processing is applied to the acoustic signal to be reproduced according to the movement of the user.
  • a specific curve or polygonal line function that inputs the sensing value of the user's movement and outputs the sensitivity of the sound corresponding to the sensing value at the time of operation is obtained in advance by interpolation processing. Sound processing is performed with the parameters corresponding to the output value of the function.
  • the degree of change in the sound to be operated that is, the sensitivity of the sound operation
  • the degree of change in the sound to be operated can be determined according to the magnitude of the user's movement such as the angle, position, speed and strength of the user's body part. It changes dynamically, and the user can intuitively operate the sound. In other words, the user can easily reflect his or her intention when manipulating the sound.
  • An audio reproduction system to which the present technology is applied includes, for example, a musical instrument 11 played by a user as shown in FIG. 1, a wearable device 12, an information terminal device 13, a speaker 14, and an audio interface 15 mounted on a predetermined portion of the user. Have.
  • the musical instrument 11, the information terminal device 13, and the speaker 14 are connected by the audio interface 15, and when the user plays the musical instrument 11, the sound corresponding to the performance is reproduced by the speaker 14. At this time, the reproduced performance sound changes according to the movement of the user.
  • the musical instrument 11 may be any musical instrument such as a keyboard instrument such as a piano or a keyboard, a stringed instrument such as a guitar or a violin, a percussion instrument such as a drum, a wind instrument, or an electronic musical instrument such as a track pad.
  • a keyboard instrument such as a piano or a keyboard
  • a stringed instrument such as a guitar or a violin
  • a percussion instrument such as a drum
  • a wind instrument or an electronic musical instrument such as a track pad.
  • the wearable device 12 is a device that can be attached to any part such as the user's arm, and includes various sensors such as an acceleration sensor, a gyro sensor, a microphone, a myoelectric meter, a pressure sensor, and a bending sensor.
  • various sensors such as an acceleration sensor, a gyro sensor, a microphone, a myoelectric meter, a pressure sensor, and a bending sensor.
  • the wearable device 12 detects the movement of the user, more specifically, the movement of the wearing portion of the wearable device 12 of the user by a sensor, and supplies a sensing value indicating the detection result to the information terminal device 13 by wireless or wired communication. ..
  • the present invention is not limited to this, and the movement of the user may be detected by a sensor arranged around the user without being attached to the user, such as a camera or an infrared sensor, and such a sensor may be used in the musical instrument 11. It may be provided.
  • the wearable device 12 may be combined with a sensor arranged around such a user to detect the movement of the user.
  • the information terminal device 13 is a signal processing device such as a smart phone or a tablet. Not limited to this, the information terminal device 13 may be any signal processing device such as a personal computer.
  • a user plays a musical instrument 11 with a wearable device 12 attached, and a desired motion (operation) for realizing a change in the sound that he / she wants to express according to the performance. )I do.
  • the motion referred to here is, for example, a movement such as raising or lowering an arm or waving a hand.
  • the acoustic signal for reproducing the performance sound is supplied from the musical instrument 11 to the information terminal device 13 via the audio interface 15.
  • the audio interface 15 will be described as being a normal audio interface for inputting and outputting acoustic signals for reproducing the performance sound.
  • the audio interface 15 may be a MIDI interface or the like that inputs / outputs a MIDI signal indicating the pitch of the performance sound.
  • the wearable device 12 the movement of the user during performance is detected, and the sensing value obtained as a result is supplied to the information terminal device 13.
  • the information terminal device 13 calculates the acoustic parameters of the acoustic processing applied to the acoustic signal based on the sensing value supplied from the wearable device 12 and the conversion function representing the sensitivity curve prepared in advance. This acoustic parameter changes non-linearly with respect to the sensing value.
  • the information terminal device 13 performs acoustic processing on the acoustic signal supplied from the instrument 11 via the audio interface 15 based on the obtained acoustic parameters, and the reproduced signal obtained as a result is used as the audio interface 15. It is supplied to the speaker 14 via.
  • the speaker 14 outputs sound based on the reproduction signal supplied from the information terminal device 13 via the audio interface 15. As a result, a sound in which an acoustic effect such as an effect according to the movement of the user is added to the performance sound of the musical instrument 11 is reproduced.
  • the sensitivity curve is a non-linear curve or a polygonal line that shows the sensitivity characteristics when operating the performance sound by the movement of the user, that is, adding a sound effect
  • the function representing the sensitivity curve is the conversion function
  • the sensing value indicating the detection result of the user's movement is assigned to the conversion function and the calculation is performed.
  • the degree of the strength (magnitude) of the acoustic effect added to the movement of the user that is, the value indicating the sensitivity is determined. can get.
  • an acoustic parameter is calculated based on the function output value, and an acoustic process for adding an acoustic effect is performed based on the obtained acoustic parameter.
  • the acoustic effects added to an acoustic signal are various effects such as delay, pitch bend, panning, and volume change due to gain correction.
  • the acoustic parameter is a value indicating the shift amount of the pitch (pitch) at the time of pitch bend.
  • nonlinear acoustic processing can be realized by using acoustic parameters obtained from the function output value of the conversion function that represents the nonlinear sensitivity curve. That is, the sensitivity can be dynamically changed according to the movement of the user's body.
  • the intention of the user can be sufficiently reflected, and the user can intuitively operate the sound, that is, add a sound effect while playing the musical instrument 11.
  • the conversion function may be prepared in advance, or the user may be able to create a desired motion and a conversion function for adding a new sound effect corresponding to the motion. ..
  • the information terminal device 13 downloads a desired conversion function prepared in advance from a server or the like via a wired or wireless network, or obtains a conversion function created by the user and information indicating motion.
  • the associated one may be uploaded to a server or the like.
  • the sound reproduction system to which this technology is applied may have, for example, the configuration shown in FIG. In FIG. 2, the parts corresponding to those in FIG. 1 are designated by the same reference numerals, and the description thereof will be omitted as appropriate.
  • the musical instrument 11 and the information terminal device 13 are connected wirelessly or by wire such as an audio interface or a MIDI interface, and the information terminal device 13 and the wearable device 12 are connected wirelessly or by wire. ..
  • the information terminal device 13 receives a supply of an acoustic signal from the instrument 11, and performs acoustic processing on the acoustic signal based on an acoustic parameter obtained from a sensing value supplied from the wearable device 12. Generate a playback signal. Then, the information terminal device 13 reproduces the sound based on the generated reproduction signal.
  • the sound may be reproduced on the instrument 11 side.
  • the information terminal device 13 may supply a MIDI signal corresponding to the reproduction signal to the musical instrument 11 to reproduce the sound, or the information terminal device 13 may set a sensing value, an acoustic parameter, or the like to the musical instrument 11.
  • the sound processing may be performed on the instrument 11 side.
  • the information terminal device 13 receives the supply of the acoustic signal from the musical instrument 11 and reproduces the sound in the information terminal device 13 based on the reproduced signal.
  • the information terminal device 13 is configured as shown in FIG. 3, for example.
  • the information terminal device 13 shown in FIG. 3 has a data acquisition unit 21, a sensing value acquisition unit 22, a control unit 23, an input unit 24, a display unit 25, and a speaker 26.
  • the data acquisition unit 21 connects to the musical instrument 11 by wire or wirelessly, acquires the acoustic signal output from the musical instrument 11, and supplies it to the control unit 23.
  • the acoustic signal to be reproduced is the performance sound of the musical instrument 11
  • the present invention is not limited to this, and the acoustic signal of an arbitrary sound is acquired by the data acquisition unit 21 as the reproduction target. You may do so.
  • an acoustic signal such as a predetermined musical piece recorded in advance is acquired by the data acquisition unit 21
  • an acoustic process for adding an acoustic effect to the acoustic signal is performed, and the acoustic effect is added. Music etc. is played.
  • the acoustic signal to be reproduced may be the sound of the acoustic effect, that is, the signal of the sound effect (effect sound) itself, and the degree of the effect in the sound effect may change according to the movement of the user. Further, along with the performance sound of the musical instrument 11, a sound effect whose strength of the effect (effect) changes according to the movement of the user may be reproduced.
  • the sensing value acquisition unit 22 is connected to the wearable device 12 by wire or wirelessly, acquires a sensing value indicating the movement of the wearing portion of the wearable device 12 by the user from the wearable device 12, and supplies the sensing value to the control unit 23.
  • the sensing value acquisition unit 22 acquires a sensing value indicating the movement of the instrument, in other words, the movement of the user who handles the instrument, from a sensor provided on the instrument such as the musical instrument 11 played by the user. Good.
  • the control unit 23 controls the overall operation of the information terminal device 13. Further, the control unit 23 has a parameter calculation unit 31.
  • the parameter calculation unit 31 calculates the acoustic parameter based on the sensing value supplied from the sensing value acquisition unit 22 and the conversion function held in advance.
  • the control unit 23 performs non-linear acoustic processing based on the acoustic parameters calculated by the parameter calculation unit 31 on the acoustic signal supplied from the data acquisition unit 21, and supplies the reproduced signal obtained as a result to the speaker 26. To do.
  • the input unit 24 includes, for example, a touch panel, buttons, switches, etc. superimposed on the display unit 25, and supplies a signal according to the user's operation to the control unit 23.
  • the display unit 25 includes, for example, a liquid crystal display panel, and displays various images under the control of the control unit 23.
  • the speaker 26 reproduces sound based on the reproduction signal supplied from the control unit 23.
  • the sensitivity curve is a non-linear curve as shown in FIG.
  • the horizontal axis represents the user's movement, that is, the sensing value
  • the vertical axis represents the sensitivity, that is, the function output value.
  • the change in sensitivity to the change in the sensing value is large in the range where the sensing value is small and the range where the sensing value is large, and the conversion function is a non-linear function.
  • the function output value obtained by substituting the sensing value into the conversion function is set to be a value between 0 and 1.
  • a sensitivity curve for example, two or more combinations of a predetermined point, that is, a sensing value and a sensitivity (function output value) corresponding to the sensing value are specified, and interpolation is performed based on the specified point and a specific Bezier curve. It can be obtained by performing processing. That is, the sensitivity curve is obtained by interpolating between two or more points determined with respect to the specified point based on the Bezier curve.
  • the acoustic parameters change non-linearly along this sensitivity curve. That is, the amount of change in the playing sound of the musical instrument 11 can be dynamically changed along the sensitivity curve according to the movement of the user.
  • the sensitivity can be seamlessly changed by connecting a range in which the sensitivity of the change in sound with respect to the user's movement is desired to be lowered and a range in which the sensitivity is desired to be increased.
  • the sound can be changed non-linearly and continuously, unlike the case where the sound is changed discretely by the threshold processing, so that the range of musical expression of the user can be expanded.
  • This reproduction process is started when the user wearing the wearable device 12 plays the musical instrument 11 while appropriately performing a desired motion.
  • step S11 the data acquisition unit 21 acquires the acoustic signal output from the musical instrument 11 and supplies it to the control unit 23.
  • step S12 the sensing value acquisition unit 22 acquires the sensing value indicating the user's movement (motion) by receiving the sensing value from the wearable device 12 by wireless communication or the like, and supplies the sensing value to the control unit 23.
  • step S13 the parameter calculation unit 31 substitutes the sensing value supplied from the sensing value acquisition unit 22 into the conversion function held in advance to perform the calculation, and obtains the function output value.
  • the parameter calculation unit 31 holds a conversion function corresponding to each of a plurality of motions by the user, and in step S13, the conversion function corresponding to the motion indicated by the sensing value is used. Good.
  • a conversion function selected by a user or the like in advance by operating the input unit 24 may be used to obtain a function output value.
  • step S14 the parameter calculation unit 31 calculates the acoustic parameter based on the function output value obtained in step S13.
  • the parameter calculation unit 31 calculates the acoustic parameter by scale-converting the function output value to the scale of the acoustic parameter. Therefore, the acoustic parameters change non-linearly according to the sensing value.
  • the function output value is a normalized acoustic parameter, so the conversion function takes the user's movement (movement amount) as an input and uses the amount of change in sound due to the sound effect, that is, the acoustic parameter. It can be said that it is a function to be output.
  • step S15 the control unit 23 generates a reproduction signal by performing non-linear acoustic processing on the acoustic signal acquired in step S11 and supplied from the data acquisition unit 21 based on the acoustic parameters obtained in step S14. To do.
  • step S16 the control unit 23 supplies the reproduction signal obtained in step S15 to the speaker 26 to reproduce the sound, and the reproduction process ends.
  • the information terminal device 13 calculates acoustic parameters based on the sensing value and the conversion function representing the non-linear sensitivity curve, and performs non-linear acoustic processing on the acoustic signal based on the acoustic parameters.
  • the sensitivity of the sound operation can be dynamically changed, and the user can intuitively perform the operation on the sound.
  • the sensitivity curve represented by the conversion function is not limited to the example shown in FIG. 4, and may be any other non-linear curve or polygonal line.
  • the sensitivity curve can be an exponential curve as shown in FIG.
  • the horizontal axis represents the movement of the user's body, that is, the sensing value
  • the vertical axis represents the sensitivity, that is, the function output value.
  • the sensitivity curve shown in FIG. 6 can be obtained by interpolation processing based on the Bezier curve, as in the example shown in FIG. 4, and in this example, the conversion function representing the sensitivity curve is an exponential function. ..
  • the sensitivity that is, the function output value decreases as the user's movement decreases, and conversely, the function output value increases as the user's movement increases.
  • the movement of the user's body that is, the sensing value input to the conversion function is, for example, the acceleration in the direction of each of the user's x-axis, y-axis, and z-axis in the three-dimensional xyz space, and the synthesis of those accelerations. It can be the acceleration, the degree of movement of the user, the rotation angle (tilt) of the user whose rotation axes are the x-axis, the y-axis, and the z-axis.
  • the sensing value is the sound pressure level and each frequency component of the aerodynamic sound generated by the movement of the user, the main frequency of the aerodynamic sound, the moving distance of the user, the contraction state of the muscle measured by the myoelectric meter, the user's keyboard, etc. It can be the pressure when pressing.
  • the curves shown in FIGS. 7 and 8 may be used.
  • each curve represents a sensitivity curve, and the name of the curve to be the sensitivity curve is written on the lower side in the figure of the sensitivity curve. Further, in each sensitivity curve, the horizontal direction (horizontal axis) indicates the movement of the user, and the vertical direction (vertical axis) indicates the sensitivity.
  • the amount of change in the playing sound can be changed in a curved (non-linear) manner according to the movement of the user.
  • easeIn which includes "easeIn” in the name of the curve
  • easeIn the amount of change in sound decreases as the movement of the user's body decreases, and the movement of the user's body increases. As the sound changes, the amount of change increases.
  • easeOut that includes "easeOut” in the name
  • easeOut the amount of change in sound increases as the movement of the user's body decreases, and the sound increases as the movement of the user's body increases.
  • the amount of change in is small.
  • the amount of change in sound is small in the range where the movement of the user's body is small, and the amount of change in sound is rapidly large when the movement of the user's body is medium.
  • the amount of change in sound is small in the range where the movement of the user's body is large.
  • any non-linear curve or polygonal line such as the polygonal line or curve shown in FIG. 9 can be used as the sensitivity curve.
  • the horizontal axis shows the user's movement, that is, the sensing value
  • the vertical axis shows the sensitivity, that is, the function output value
  • the sensitivity curve is a triangular wavy polygonal line
  • the sensitivity curve is a rectangular corrugated polygonal line
  • the sensitivity curve is a sinusoidal periodic curve.
  • the sound based on the acoustic signal is changed. Can be done.
  • the angle when the user moves the arm can be detected (measured) by, for example, a gyro sensor provided in the wearable device 12.
  • the acoustic effect applied to the acoustic signal can be a delay effect called a Yamabiko effect realized by a delay filter, a filter effect realized by a low frequency cut using a cutoff filter, or the like.
  • control unit 23 performs a filtering process by a delay filter or a cutoff filter as a non-linear acoustic process.
  • the change in sound may increase as the angle of the user's arm decreases, and the change in sound may decrease as the angle of the user's arm increases.
  • an effect of panning the sound image position of the sound based on the acoustic signal to the left and right may be added as an acoustic effect according to the position of the user's arm in the left-right direction.
  • the sound may be panned louder.
  • effects such as reverb, distortion, and pitch bend, that is, an acoustic effect may be added to the acoustic signal. Good.
  • the snap operation by the user can be detected by sensing the vibration applied to the wearable device 12 worn by the user on the wrist or the like during the snap operation, that is, the jerk.
  • acoustic processing such as filtering processing for adding an effect is performed so that the amount of change in the effect (sound effect) such as reverb changes based on the sensing value of jerk.
  • an acceleration sensor or the like provided on the wearable device 12 mounted on the user's wrist or the like detects the movement of swinging the arm in the left-right direction, and the acceleration value as the sensing value obtained as the detection result is obtained. A sound effect is added based on.
  • the amount of pitch shift in pitch bend as an acoustic effect increases, and conversely, as the value of acceleration decreases, the amount of pitch shift decreases. can do.
  • the amount of pitch shift in pitch bend is used as the acoustic parameter.
  • the lateral swing of the user's arm (finger) as a motion is a pressure sensor provided in each keyboard portion such as the piano keyboard KY11 portion as the musical instrument 11 as shown in FIG. It may be detected by.
  • the left-right sway of the user's arm (finger) as a motion is, for example, as shown in FIG. 15, a camera or an infrared sensor provided on the front part of the piano as the musical instrument 11. It can also be detected by the sensor CA11 such as.
  • the musical instrument 11 side or the sensing value acquisition unit 22 obtains the magnitude of the user's left-right shaking from a moving image taken by the camera, and the shaking is obtained.
  • a value indicating the magnitude of is used as a sensing value.
  • pitch bend is added as a sound effect.
  • the pitch bend shifts the playing sound of the instrument 11 to a high tone, and conversely, it is indicated by arrow W52.
  • pitch bend causes the playing sound to shift to the bass.
  • the left / right rotation angle of the user's arm is detected as a sensing value, and an effect such as pitch bend is added to the performance sound as a sound effect according to the rotation angle.
  • the sensing value acquisition unit 22 may acquire a sensing value indicating the movement of the head portion of the guitar or the like from a sensor provided on the guitar or the like as the musical instrument 11, or is output from the wearable device 12.
  • the sensing value may be acquired as a sensing value indicating the movement of the head portion.
  • the movement of the user pressing the pad (keyboard) of the track pad as the musical instrument 11 or the keyboard of a keyboard instrument such as a piano particularly the strength (pressure) of pressing the pad or the keyboard as a motion. Sound effects may be added depending on the detected pressure.
  • the movement of the user is detected by the pressure sensor provided on the pad (keyboard) portion of the musical instrument 11 instead of the wearable device 12. Therefore, for example, when the user shakes his / her hand while pressing the pad portion, the pressure applied to the pad portion changes according to the shaking, so that the strength of the added sound effect also changes.
  • the strength (pressure) when a percussion instrument such as a drum is struck as an instrument 11 is detected by a pressure sensor or the like provided on the percussion instrument, and an effect is obtained according to the detection result. (Sound effect) may be added to the performance sound of a drum or the like.
  • the performance sound of a drum or the like can be collected by a microphone, and the resulting acoustic signal can be acquired by the data acquisition unit 21. Then, the control unit 23 can perform non-linear acoustic processing based on the acoustic parameters on the acoustic signal of the performance sound of the drum or the like. It should be noted that the performance sound of the drum or the like may not be picked up, and the sound effect sound having the strength of the effect according to the acoustic parameter may be reproduced from the speaker 26 together with the performance sound.
  • the movement of the user tilting the wind instrument as the musical instrument 11 in the direction indicated by the arrow W81 is detected as a motion, and the acoustic signal of the performance sound of the musical instrument 11 is detected according to the degree of tilt.
  • a sound effect may be added.
  • the performance sound of the wind instrument can be obtained by collecting the sound with a microphone.
  • the movement of tilting the stringed instrument can be detected as a motion.
  • a method of using the sensitivity curve preset by default a method of selecting by the user from multiple sensitivity curves, a method of using the sensitivity curve according to the type of motion, etc. Can be considered.
  • the parameter calculation unit 31 receives the supply of the sensing value corresponding to the motion from the sensing value acquisition unit 22.
  • the parameter calculation unit 31 calculates the acoustic parameter based on the conversion function representing the sensitivity curve that is predetermined, that is, preset for the motion performed by the user, and the supplied sensing value.
  • the performance sound of the musical instrument 11 automatically changes along the preset sensitivity curve from the user's point of view.
  • the sensitivity is low when the swing of the arm is small, and the sensitivity is automatically increased as the swing of the arm is large, and the change in sound is large.
  • step S41 the control unit 23 reads image data from a memory (not shown) and supplies it to the display unit 25 to display a selection screen which is a GUI (Graphical User Interface) based on the image data.
  • a GUI Graphic User Interface
  • the display unit 25 displays, for example, the sensitivity curve (conversion function) selection screen shown in FIG. 24.
  • the selection screen is displayed on the display unit 25, and the selection screen lists a plurality of sensitivity curves previously held in the parameter calculation unit 31 and the names of the sensitivity curves. It is displayed.
  • the user specifies (selects) a desired sensitivity curve by touching it with a finger from among the plurality of sensitivity curves displayed in the list in this way.
  • a touch panel as an input unit 24 is superimposed on the display unit 25, and when the user performs a touch operation on the area where the sensitivity curve is displayed, a signal corresponding to the touch operation is output from the input unit 24 to the control unit. It is supplied to 23.
  • the user may be able to select the sensitivity curve for each motion.
  • step S42 the control unit 23 selects the sensitivity curve specified by the user from among the plurality of sensitivity curves displayed on the selection screen based on the signal supplied from the input unit 24.
  • the conversion function to be represented is selected as the conversion function used to calculate the acoustic parameters.
  • step S13 of the reproduction process of FIG. 5 to be performed later the conversion function selected in step S42 of FIG. 23 is used to obtain the function output value. Be done.
  • the information terminal device 13 displays the selection screen and selects the conversion function according to the user's instruction. By doing so, not only can the conversion function be switched according to the user's preference and application, but also the sound effect can be added along the sensitivity curve desired by the user.
  • the selection process performed by the information terminal device 13 will be described with reference to the flowchart of FIG. 25.
  • the selection process described with reference to FIG. 25 is started when the sensing value is acquired in step S12 of the reproduction process described with reference to FIG.
  • step S71 the parameter calculation unit 31 specifies the type of user's movement (motion) based on the sensing value supplied from the sensing value acquisition unit 22.
  • the type of motion is specified based on the temporal change of the sensing value, the information supplied from the wearable device 12 together with the sensing value, and the information indicating the type of the sensor used to obtain the sensing value.
  • step S72 the parameter calculation unit 31 selects and selects a sensitivity curve conversion function determined for the type of motion specified in step S71 from among the plurality of sensitivity curve conversion functions held in advance. The process ends.
  • step S13 of the reproduction process of FIG. 5 the conversion function selected in step S72 is used to obtain the function output value.
  • the information terminal device 13 specifies the type of user's movement from the sensing value and the like, and selects a sensitivity curve (conversion function) according to the specific result. By doing so, it is possible to add a sound effect with an appropriate sensitivity for each type of movement.
  • a curve conversion function called "easeInExpo” is selected as the sensitivity curve in step S72.
  • easeInExponential function is selected as the conversion function.
  • step S72 of the selection process of FIG. 25, which is newly performed a curve conversion function called "easeOutExpo" is selected as the sensitivity curve.
  • easeOutExpo the easeOutExponential function
  • the sensitivity curve and the sound effect are also selected according to the type of the musical instrument 11 and the type (genre) of the music. You may do so.
  • the type of the musical instrument 11 may be specified by the control unit 23 connecting to the musical instrument 11 via the data acquisition unit 21 and acquiring information indicating the type (type) of the musical instrument 11 from the musical instrument 11. Good. Further, for example, the type of the musical instrument 11 may be specified by the control unit 23 specifying the movement of the user's musical instrument 11 during performance from the sensing value supplied from the sensing value acquisition unit 22.
  • the sound based on the acoustic signal to be reproduced is specified by the control unit 23 performing various analysis processes on the acoustic signal supplied from the data acquisition unit 21.
  • the control unit 23 may specify the metadata of the acoustic signal or the like.
  • the information terminal device 13 performs the drawing process shown in FIG. 27.
  • the drawing process by the information terminal device 13 will be described with reference to the flowchart of FIG. 27.
  • step S101 the control unit 23 controls the display unit 25 to display a sensitivity curve input screen for inputting the sensitivity curve to the display unit 25.
  • the sensitivity curve input screen shown in FIG. 28, for example, is displayed on the display unit 25.
  • the user traces the sensitivity curve input screen with a finger or the like to draw a sensitivity curve with the horizontal axis as motion and the vertical axis as sensitivity, thereby drawing an arbitrary sensitivity curve. It is designed so that it can be specified.
  • a touch panel as an input unit 24 is superimposed on the display unit 25, and the user inputs a desired sensitivity curve such as a non-linear curve or a polygonal line by performing an operation of tracing the sensitivity curve input screen with a finger or the like. To do.
  • the sensitivity curve input method is not limited to this, and may be any method. Further, for example, a preset sensitivity curve is displayed on the sensitivity curve input screen, and the user may input the desired sensitivity curve by deforming the sensitivity curve by a touch operation or the like.
  • the parameter calculation unit 31 represents a conversion representing the sensitivity curve input by the user based on the signal supplied from the input unit 24 in response to the drawing operation of the sensitivity curve of the user. Generate and record a function. When the conversion function of the sensitivity curve drawn by the user is recorded, the drawing process ends.
  • the information terminal device 13 generates and records a conversion function representing a sensitivity curve freely drawn by the user.
  • the user can finely adjust or customize the sensitivity when manipulating the sound according to his / her movement, and specify the sensitivity curve as he / she intended, and more intuitively operate the sound. Will be able to do.
  • a specific movement (motion) when a user makes a specific movement (motion), an animation effect is added as a sound effect to the sound to be reproduced over a certain period of time according to the type of the movement. May be good.
  • a specific movement (motion) of the user will be referred to as a gesture in particular.
  • the animation effect is an acoustic effect that adds an effect to the sound to be reproduced for a certain period of time along the animation curve obtained by interpolation processing based on the Bezier curve, for example.
  • the animation curve can be, for example, a curve as shown in FIG. 29.
  • the vertical axis represents the change in sound
  • the horizontal axis represents time.
  • the function representing the animation curve will be referred to as the animation function. Therefore, the value on the vertical axis of the animation curve, that is, the value indicating the change in sound is the output value of the animation function (hereinafter, referred to as the function output value).
  • the animation effect changes the volume level of the reproduced sound
  • the animation effect when the animation effect is added to the reproduced sound along the animation curve shown in FIG. 29, the volume level of the reproduced sound is added. Will decrease over time.
  • the sensing value acquisition unit 22 detects the swing of the user's arm in the left-right direction or the up-down direction as a gesture based on the sensing value, and when the gesture is detected, the gesture, more specifically, the type of gesture.
  • a predetermined sound source sound hereinafter, also referred to as a gesture sound
  • an animation effect is added so that the volume level of the gesture sound gradually decreases with time, for example, along the animation curve shown in FIG.
  • the vertical axis represents the change in sound, that is, the function output value of the animation function
  • the horizontal axis represents time.
  • control unit 23 can make the animation curve and sound processing, that is, the animation effect, selected according to the detected gesture.
  • the parameter calculation unit 31 calculates the gain value as an acoustic parameter at each time based on the function output value at each time.
  • the function output value is scale-converted to the scale of the acoustic parameter and used as the acoustic parameter.
  • the gain value as an acoustic parameter becomes smaller as it is at a later time (future time).
  • control unit 23 applies gain correction to the acoustic signal of the gesture sound as acoustic processing based on the acoustic parameters of that time at each time, and reproduces the signal. Is generated.
  • a movement (gesture) in which the user plays the musical instrument 11 a movement of pressing a keyboard, a movement of ringing a string, or the like is detected, and the performance sound of the musical instrument 11 is along an animation curve according to the movement of the user.
  • the animation effect may be added for a predetermined time.
  • the performance sound of the musical instrument 11 may be played as it is, and the sound effect to which the animation effect is added according to the movement of the user may be reproduced together with the performance sound.
  • the sensing value acquisition unit 22 sequentially detects the peak value of the time waveform of the sensed value based on the acquired sensing value indicating the movement of the user at each time, and responds to the detected peak value.
  • the initial value of the acoustic parameter may be determined.
  • the information terminal device 13 performs the reproduction process shown in FIG. 31, for example.
  • the reproduction process by the information terminal device 13 will be described with reference to the flowchart of FIG.
  • step S131 the sensing value acquisition unit 22 acquires the sensing value indicating the movement (motion) of the user by receiving the sensing value from the wearable device 12 by wireless communication or the like.
  • step S132 the sensing value acquisition unit 22 detects whether or not a specific gesture has been performed by the user based on the sensing values acquired so far.
  • step S133 the sensing value acquisition unit 22 determines whether or not a gesture is detected as a result of the detection in step S132.
  • step S133 If it is determined in step S133 that no gesture is detected, the process returns to step S131, and the above-mentioned process is repeated.
  • the sensing value acquisition unit 22 in step S134 peaks the waveform of the sensing value based on the sensing values acquired so far in the latest predetermined period. Detect the value.
  • the sensing value acquisition unit 22 supplies the parameter calculation unit 31 with information indicating the gesture and the peak value detected in this way.
  • step S135 the parameter calculation unit 31 determines the animation effect, that is, the animation curve and the acoustic processing, based on the information indicating the detected gesture and the peak value supplied from the sensing value acquisition unit 22.
  • the parameter calculation unit 31 selects a predetermined animation effect for the detected gesture as an animation effect to be added to the gesture sound.
  • control unit 23 controls the data acquisition unit 21 to acquire a predetermined gesture sound acoustic signal for the detected gesture.
  • the animation effect is added to an arbitrary sound such as the performance sound of the musical instrument 11. Can be done.
  • step S136 the parameter calculation unit 31 calculates the acoustic parameter based on the information indicating the detected gesture and the peak value supplied from the sensing value acquisition unit 22.
  • the parameter calculation unit 31 calculates the initial value of the acoustic parameter by scale-converting the peak value of the sensing value to the scale of the acoustic parameter.
  • the initial value of the acoustic parameter referred to here is the value of the acoustic parameter at the start of the animation effect added to the gesture sound.
  • the parameter calculation unit 31 determines the acoustic parameter at each time within the period for adding the animation effect to the gesture sound based on the initial value of the acoustic parameter and the animation curve for realizing the animation effect determined in step S135. Is calculated.
  • the value of the acoustic parameter is based on the initial value of the acoustic parameter and the function output value of the animation function representing the animation curve at each time so that the value of the acoustic parameter gradually changes from the initial value along the animation curve.
  • the value of the acoustic parameter at each time is calculated.
  • the period during which the animation effect is added will also be referred to as the animation period.
  • step S137 the control unit 23 generates a reproduction signal by performing acoustic processing for adding an animation effect to the acoustic signal of the gesture sound based on the acoustic parameter at each time calculated in step S136.
  • control unit 23 generates a reproduced signal by performing acoustic processing based on the acoustic parameter on the acoustic signal of the gesture sound while gradually changing the value of the acoustic parameter from the initial value along the animation curve. ..
  • step S138 the control unit 23 supplies the reproduction signal obtained in step S137 to the speaker 26 to reproduce the sound, and the reproduction process ends.
  • the speaker 26 reproduces the gesture sound to which the animation effect corresponding to the gesture is added.
  • the information terminal device 13 calculates acoustic parameters based on the peak value of the sensing value, and performs non-linear acoustic processing on the acoustic signal based on the acoustic parameters.
  • the user can add a desired animation effect to the gesture sound simply by performing a predetermined gesture. Therefore, the user can intuitively operate the sound.
  • the volume of the gesture sound gradually decreases with respect to the gesture sound, and a Bounce animation of an animation curve as shown in FIG. 32 is performed. It is possible to add it.
  • the vertical axis shows the change in sound, that is, the function output value of the animation function
  • the horizontal axis shows time
  • the animation curve shown in FIG. 32 is a curve in which the sound gradually decreases with time while changing up and down.
  • the sensing value acquisition unit 22 detects the peak value of the jerk waveform as the sensing value.
  • the gain value as an acoustic parameter that is, the initial value of the volume at the time of reproducing the gesture sound is determined based on the peak value of the jerk, and the acoustic parameter is the animation curve shown in FIG. 32.
  • the acoustic parameters at each time are determined to vary along.
  • control unit 23 performs gain correction as acoustic processing on the acoustic signal of the gesture sound based on the determined acoustic parameter at each time, that is, the gain value, and as a result, the gesture sound is corrected. Bounce animation effect is added.
  • the Bounce animation effect reproduces the user's gesture, that is, the gesture sound in which the sound generated in response to the swing of the arm bounces off the object, changes to bounce, and gradually decreases in volume over time. Will be done.
  • FIG. 33 the vertical axis represents the change in sound, that is, the function output value of the animation function, and the horizontal axis represents time.
  • the acceleration indicating the vibration when the percussion instrument as the musical instrument 11 is struck is acquired as the sensing value, and the peak value of the sensing value indicating the vibration waveform is used to perform reverb, delay, etc. in the same manner as in the above example. You may animate various effects of.
  • the degree of application of acoustic effects such as reverb and delay added to the playing sound of the musical instrument 11 changes with time along the animation curve.
  • a gesture sound may be generated according to the movement (gesture) of the user, and an animation effect may be added to the gesture sound, that is, the waveform of the sound.
  • an acceleration indicating a user's movement is detected as a sensing value, and a sound wave-shaped acoustic signal having a specific frequency such as a sine wave is generated as a gesture sound signal according to the sensing value.
  • the initial value of the acoustic parameter is determined in the same manner as in the above example, and an animation effect is added to the gesture sound so that the degree of effect changes with time along a predetermined animation curve. Can be considered.
  • the sound pressure of aerodynamic sound is detected as a sensing value
  • the initial value of the acoustic parameter is determined based on the peak value of the waveform of the sensing value
  • the acoustic signal of the aerodynamic sound obtained by collecting the sound. Is subjected to acoustic processing based on the acoustic parameters of each time.
  • ⁇ Modification 2 of the second embodiment> ⁇ Addition of animation effect> Further, when the animation effect is added according to the movement of the user, the animation effect may be added again when a new large movement of the user is detected before the end of the animation.
  • the initial value of the acoustic parameter is determined according to the peak value of the sensing value indicating the user's motion, and an animation effect that changes the degree of effect applied based on the initial value and the animation curve is added to the acoustic signal. To be done.
  • the sound based on the acoustic signal may be any sound such as the performance sound of the musical instrument 11 or the sound effect defined for the motion of the user, but here, the performance sound of the musical instrument 11 is reproduced. It shall be done.
  • the initial value of the acoustic parameter is determined based on the peak value of the acceleration.
  • the value of the acoustic parameter at each time thereafter is determined so that the value of the acoustic parameter changes along the animation curve determined for the user's motion or the like.
  • the acoustic processing for the acoustic signal to be reproduced is performed based on the acoustic parameters at each time, and the reproduced signal is generated. Then, when the sound is reproduced based on the reproduction signal thus obtained, the animation effect for a certain period of time is added to the performance sound of the musical instrument 11 and the sound is reproduced.
  • the sound obtained for the peak value of the acceleration (sensing value) indicating the user's motion exceeds the acoustic parameter at the current time before the end of the animation period, the sound obtained for the peak value The parameter is the new initial value.
  • the acoustic parameter obtained from the peak value at an arbitrary time within the animation period becomes larger than the actual acoustic parameter at that time
  • the acoustic parameter obtained for the peak value at that time is the new acoustic parameter.
  • an animation effect is newly added to the performance sound of the musical instrument 11.
  • step S161 the data acquisition unit 21 acquires the acoustic signal output from the musical instrument 11 and supplies it to the control unit 23.
  • step S162 the sensing value acquisition unit 22 acquires the sensing value indicating the movement (motion) of the user by receiving the sensing value from the wearable device 12 by wireless communication or the like.
  • step S163 the sensing value acquisition unit 22 detects the peak value of the waveform of the sensing value based on the sensing value acquired so far in the latest predetermined period.
  • the sensing value acquisition unit 22 supplies the peak value of the sensing value detected in this way to the parameter calculation unit 31.
  • step S164 the parameter calculation unit 31 calculates the acoustic parameter based on the peak value supplied from the sensing value acquisition unit 22.
  • the parameter calculation unit 31 calculates the initial value of the acoustic parameter by scale-converting the peak value of the sensing value to the scale of the acoustic parameter.
  • step S165 the parameter calculation unit 31 determines whether or not the initial value of the acoustic parameter calculated in step S164 is larger than the acoustic parameter at the current time.
  • a predetermined animation effect is added to the performance sound of the musical instrument 11 for the motion.
  • step S165 it is determined in step S165 that it is larger than the acoustic parameter at the current time.
  • step S164 if the initial value of the acoustic parameter obtained in step S164 is larger than the acoustic parameter at the current time actually used for adding the animation effect, the acoustic parameter at the current time is obtained in step S165. Is determined to be greater than.
  • step S165 If it is determined in step S165 that the acoustic parameter is not larger than the acoustic parameter at the current time, the processes of steps S166 to S168 are not performed, and then the process proceeds to step S169.
  • the control unit 23 supplies the sound effect, that is, the sound signal to which the animation effect is not added to the speaker 26 as a reproduction signal as it is, and reproduces the performance sound of the musical instrument 11.
  • the acoustic signal is subjected to acoustic processing based on the acoustic parameter at the current time, and the sound is reproduced by the speaker 26 based on the obtained reproduction signal. In this case, the performance sound to which the animation effect is added is reproduced.
  • step S165 determines whether it is larger than the acoustic parameter at the current time. If it is determined in step S165 that it is larger than the acoustic parameter at the current time, the process proceeds to step S166 thereafter.
  • step S166 the parameter calculation unit 31 calculates the acoustic parameters at each time within the animation period based on the initial values of the acoustic parameters calculated in step S164 and the animation curve determined for the user's motion and the like.
  • the value of the acoustic parameter is based on the initial value of the acoustic parameter and the function output value of the animation function representing the animation curve at each time so that the value of the acoustic parameter gradually changes from the initial value along the animation curve.
  • the value of the acoustic parameter is calculated.
  • step S167 the control unit 23 outputs a reproduced signal by performing acoustic processing for adding an animation effect to the acoustic signal acquired by the data acquisition unit 21 based on the acoustic parameters at each time calculated in step S166. Generate.
  • control unit 23 generates a reproduced signal by performing acoustic processing based on the acoustic parameter on the acoustic signal while gradually changing the value of the acoustic parameter from the initial value along the animation curve.
  • step S168 the control unit 23 supplies the reproduction signal obtained in step S167 to the speaker 26 to reproduce the sound. As a result, a new animation period is started, and the animation effect is added to the performance sound of the musical instrument 11 and reproduced.
  • step S168 determines in step S169 whether or not to end the reproduction of the sound based on the acoustic signal. To do.
  • step S169 when the user finishes playing the musical instrument 11, it is determined that the playback is finished.
  • step S169 If it is determined in step S169 that the reproduction is not finished yet, the process returns to step S161, and the above-described process is repeated.
  • step S169 when it is determined in step S169 to end the reproduction, each part of the information terminal device 13 stops the processing being performed, and the reproduction processing ends.
  • the information terminal device 13 calculates an acoustic parameter based on the peak value of the sensing value, and performs acoustic processing on the acoustic signal based on the acoustic parameter.
  • the information terminal device 13 when the information terminal device 13 has a movement of the user such that the value of the acoustic parameter becomes larger than the acoustic parameter at the current time during the animation period, the information terminal device 13 responds to the performance sound of the musical instrument 11 according to the movement. Add a new animation effect.
  • the user can add a desired animation effect according to his / her own movement. Therefore, the user can intuitively operate the sound.
  • the series of processes described above can be executed by hardware or software.
  • the programs that make up the software are installed on the computer.
  • the computer includes a computer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.
  • FIG. 35 is a block diagram showing a configuration example of computer hardware that executes the above-mentioned series of processes programmatically.
  • the CPU Central Processing Unit
  • the ROM ReadOnly Memory
  • the RAM RandomAccessMemory
  • An input / output interface 505 is further connected to the bus 504.
  • An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.
  • the input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, and the like.
  • the output unit 507 includes a display, a speaker, and the like.
  • the recording unit 508 includes a hard disk, a non-volatile memory, and the like.
  • the communication unit 509 includes a network interface and the like.
  • the drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
  • the CPU 501 loads the program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executes the above-described series. Is processed.
  • the program executed by the computer (CPU501) can be recorded and provided on a removable recording medium 511 as a package medium or the like, for example. Programs can also be provided via wired or wireless transmission media such as local area networks, the Internet, and digital satellite broadcasting.
  • the program can be installed in the recording unit 508 via the input / output interface 505 by mounting the removable recording medium 511 in the drive 510. Further, the program can be received by the communication unit 509 and installed in the recording unit 508 via a wired or wireless transmission medium. In addition, the program can be pre-installed in the ROM 502 or the recording unit 508.
  • the program executed by the computer may be a program that is processed in chronological order in the order described in this specification, or may be a program that is processed in parallel or at a necessary timing such as when a call is made. It may be a program in which processing is performed.
  • the embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.
  • this technology can have a cloud computing configuration in which one function is shared by a plurality of devices via a network and processed jointly.
  • each step described in the above flowchart can be executed by one device or shared by a plurality of devices.
  • one step includes a plurality of processes
  • the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices.
  • this technology can also have the following configurations.
  • An acquisition unit that acquires a sensing value indicating the movement of a predetermined part of the user's body or an instrument
  • a signal processing device including a control unit that performs non-linear acoustic processing on an acoustic signal according to the sensing value.
  • the control unit performs the acoustic processing based on a parameter that changes non-linearly according to the sensing value.
  • the control unit calculates the parameter according to the sensing value based on a non-linear curve or polygonal line conversion function input by the user.
  • the control unit obtains the initial value of the parameter of the acoustic processing based on the peak value of the waveform of the sensing value, and performs the acoustic processing while changing the parameter from the initial value to obtain the acoustic signal.
  • the signal processing device according to (6) or (7), which adds the animation effect.
  • the control unit When the parameter corresponding to the peak value at the time becomes larger than the actual parameter at the time at an arbitrary time within the animation period in which the animation effect is applied, the control unit said.
  • the signal processing apparatus according to (8) wherein the acoustic processing is performed so that the animation effect is newly added to the acoustic signal based on the initial value obtained based on the peak value at the time.
  • the signal processing device (10) The signal processing device according to any one of (1) to (9), wherein the acoustic signal is a signal of a performance sound of a musical instrument played by a user. (11) The signal processing device according to any one of (1) to (9), wherein the acoustic signal is a signal defined for the type of motion. (12) The signal processing device Acquires a sensing value that indicates the movement of a predetermined part of the user's body or an instrument, A signal processing method for performing non-linear acoustic processing on an acoustic signal according to the sensing value.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Electrophonic Musical Instruments (AREA)
PCT/JP2020/030560 2019-08-22 2020-08-11 信号処理装置および方法、並びにプログラム Ceased WO2021033593A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US17/635,073 US20220293073A1 (en) 2019-08-22 2020-08-11 Signal processing device, signal processing method, and program
JP2021540738A JP7574797B2 (ja) 2019-08-22 2020-08-11 信号処理装置および方法、並びにプログラム
CN202080058671.7A CN114258565B (zh) 2019-08-22 2020-08-11 信号处理设备、信号处理方法和程序产品

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019152123 2019-08-22
JP2019-152123 2019-08-22

Publications (1)

Publication Number Publication Date
WO2021033593A1 true WO2021033593A1 (ja) 2021-02-25

Family

ID=74661115

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/030560 Ceased WO2021033593A1 (ja) 2019-08-22 2020-08-11 信号処理装置および方法、並びにプログラム

Country Status (4)

Country Link
US (1) US20220293073A1 (https=)
JP (1) JP7574797B2 (https=)
CN (1) CN114258565B (https=)
WO (1) WO2021033593A1 (https=)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019202718A1 (ja) * 2018-04-19 2019-10-24 ローランド株式会社 電気楽器システム
JP2021107843A (ja) * 2018-04-25 2021-07-29 ローランド株式会社 電子楽器システムおよび楽器用コントローラ
US20220180854A1 (en) * 2020-11-28 2022-06-09 Sony Interactive Entertainment LLC Sound effects based on footfall

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH04125693A (ja) * 1990-09-18 1992-04-27 Yamaha Corp 電子楽器
JPH05108064A (ja) * 1991-10-18 1993-04-30 Yamaha Corp 楽音制御装置
JPH0667661A (ja) * 1991-07-12 1994-03-11 Yamaha Corp 楽音制御装置
JPH0683347A (ja) * 1992-09-02 1994-03-25 Yamaha Corp 電子楽器
JPH09237087A (ja) * 1996-02-29 1997-09-09 Kawai Musical Instr Mfg Co Ltd 電子楽器
JPH1097245A (ja) * 1996-09-20 1998-04-14 Yamaha Corp 楽音制御装置
JP2011237662A (ja) * 2010-05-12 2011-11-24 Casio Comput Co Ltd 電子楽器
JP2013213745A (ja) * 2012-04-02 2013-10-17 Casio Comput Co Ltd 姿勢検出装置、方法及びプログラム

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS5131223A (https=) * 1974-09-11 1976-03-17 Matsushita Electric Industrial Co Ltd
JP2998602B2 (ja) * 1995-08-11 2000-01-11 ヤマハ株式会社 楽音制御装置
US7183480B2 (en) * 2000-01-11 2007-02-27 Yamaha Corporation Apparatus and method for detecting performer's motion to interactively control performance of music or the like
JP3646600B2 (ja) * 2000-01-11 2005-05-11 ヤマハ株式会社 演奏インターフェイス
JP3901098B2 (ja) * 2003-01-17 2007-04-04 ヤマハ株式会社 楽音編集システム
JP2006267226A (ja) 2005-03-22 2006-10-05 Yamaha Corp 楽音制御装置
JP5598490B2 (ja) * 2012-03-19 2014-10-01 カシオ計算機株式会社 演奏装置、方法及びプログラム
CN104834642B (zh) * 2014-02-11 2019-06-18 北京三星通信技术研究有限公司 改变音乐演绎风格的方法、装置及设备
US10705620B2 (en) 2015-10-09 2020-07-07 Sony Ccorporation Signal processing apparatus and signal processing method
JP2017097214A (ja) * 2015-11-26 2017-06-01 ソニー株式会社 信号処理装置、信号処理方法及びコンピュータプログラム。

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH04125693A (ja) * 1990-09-18 1992-04-27 Yamaha Corp 電子楽器
JPH0667661A (ja) * 1991-07-12 1994-03-11 Yamaha Corp 楽音制御装置
JPH05108064A (ja) * 1991-10-18 1993-04-30 Yamaha Corp 楽音制御装置
JPH0683347A (ja) * 1992-09-02 1994-03-25 Yamaha Corp 電子楽器
JPH09237087A (ja) * 1996-02-29 1997-09-09 Kawai Musical Instr Mfg Co Ltd 電子楽器
JPH1097245A (ja) * 1996-09-20 1998-04-14 Yamaha Corp 楽音制御装置
JP2011237662A (ja) * 2010-05-12 2011-11-24 Casio Comput Co Ltd 電子楽器
JP2013213745A (ja) * 2012-04-02 2013-10-17 Casio Comput Co Ltd 姿勢検出装置、方法及びプログラム

Also Published As

Publication number Publication date
JPWO2021033593A1 (https=) 2021-02-25
JP7574797B2 (ja) 2024-10-29
CN114258565A (zh) 2022-03-29
US20220293073A1 (en) 2022-09-15
CN114258565B (zh) 2025-10-28

Similar Documents

Publication Publication Date Title
JP7263957B2 (ja) 情報装置、自動設定方法及び自動設定プログラム
US10388122B2 (en) Systems and methods for generating haptic effects associated with audio signals
EP2772903B1 (en) Electroacoustic signal emitter device and electroacoustic signal emitter method
WO2021033593A1 (ja) 信号処理装置および方法、並びにプログラム
EP2945152A1 (en) Musical instrument and method of controlling the instrument and accessories using control surface
CN109375767A (zh) 用于生成触觉效果的系统和方法
WO2020224322A1 (zh) 音乐文件的处理方法、装置、终端及存储介质
JP5333517B2 (ja) データ処理装置およびプログラム
JP6805422B2 (ja) 装置、プログラム及び情報処理方法
WO2017028686A1 (zh) 一种信息处理方法、终端设备及计算机存储介质
WO2020059245A1 (ja) 情報処理装置、情報処理方法および情報処理プログラム
US20150123897A1 (en) Gesture detection system, gesture detection apparatus, and mobile communication terminal
JP5742163B2 (ja) 情報処理端末および設定制御システム
JP2018049052A (ja) 仮想楽器演奏プログラム、仮想楽器演奏装置および仮想楽器演奏方法
CN111782865B (zh) 音频信息的处理方法、装置及存储介质
CN115494953A (zh) 音频调节方法、装置、设备及计算机可读存储介质
CN108337367B (zh) 一种基于移动终端的乐器演奏方法及装置
JP6028489B2 (ja) 動画再生装置、動画再生方法、及びプログラム
JP5742472B2 (ja) データ検索装置およびプログラム
KR20170068795A (ko) 소리의 시각화 방법 및 이를 이용한 모바일 단말
JP2023068240A (ja) 情報処理プログラム、情報処理装置および情報処理方法
Patrício MuDI-Multimedia Digital Instrument for Composing and Performing Digital Music for Films in Real-time
JP2015026021A (ja) 楽曲再生装置、および楽曲再生システム
WO2012111045A1 (ja) 動作制御装置、動作制御プログラム、動作制御方法、再生装置および記録再生装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20854454

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021540738

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20854454

Country of ref document: EP

Kind code of ref document: A1

WWG Wipo information: grant in national office

Ref document number: 202080058671.7

Country of ref document: CN