WO2009007512A1 - Système de synthèse de musique commandé par geste - Google Patents

Système de synthèse de musique commandé par geste Download PDF

Info

Publication number
WO2009007512A1
WO2009007512A1 PCT/FI2008/050421 FI2008050421W WO2009007512A1 WO 2009007512 A1 WO2009007512 A1 WO 2009007512A1 FI 2008050421 W FI2008050421 W FI 2008050421W WO 2009007512 A1 WO2009007512 A1 WO 2009007512A1
Authority
WO
WIPO (PCT)
Prior art keywords
player
cha
musical
computer program
pose
Prior art date
Application number
PCT/FI2008/050421
Other languages
English (en)
Inventor
Perttu HÄMÄLÄINEN
Original Assignee
Virtual Air Guitar Company Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Virtual Air Guitar Company Oy filed Critical Virtual Air Guitar Company Oy
Publication of WO2009007512A1 publication Critical patent/WO2009007512A1/fr

Links

Classifications

    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63H—TOYS, e.g. TOPS, DOLLS, HOOPS OR BUILDING BLOCKS
    • A63H5/00—Musical or noise- producing devices for additional toy effects other than acoustical
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63H—TOYS, e.g. TOPS, DOLLS, HOOPS OR BUILDING BLOCKS
    • A63H33/00—Other toys
    • A63H33/30—Imitations of miscellaneous apparatus not otherwise provided for, e.g. telephones, weighing-machines, cash-registers
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0008—Associated control or indicating means
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/46—Volume control
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/10—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals
    • A63F2300/1087—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals comprising photodetecting means, e.g. a camera
    • A63F2300/1093—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals comprising photodetecting means, e.g. a camera using visible light
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/80—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game specially adapted for executing a specific type of game
    • A63F2300/8047—Music games
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/201—User input interfaces for electrophonic musical instruments for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/441—Image sensing, i.e. capturing images or optical patterns for musical purposes or musical control purposes
    • G10H2220/455—Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data

Definitions

  • the present invention is related to gesture control, user interfaces and music synthesis.
  • the present invention discloses a method for creating synthesized music in response to a player's gestures.
  • the method is characterized in that at first a current player state is recognized. After that differences are calculated between the recognized player state and at least one pre-stored player state, wherein the pre-stored player states are each associated with a musical passage. Finally, playback volumes of the associated musical passages are adjusted according to the calculated differences so that a small difference yields a high volume and a large difference yields a low volume.
  • the player state comprises a pose.
  • the player state comprises playing speed.
  • the playing speed is determined in relation to the frequency of consecutive trigger gestures performed by the player.
  • such trigger gestures are recognized that mimic the strumming of the strings of a guitar.
  • the player state comprises the distance between the hands of the player .
  • the method further comprises the step of producing a plurality of pre-stored player states and associated musical passages so that the player states with a relatively high distance between the hands of the player are associated with musical passages that have notes with relatively low pitches .
  • the method further comprises producing the musical passages so that each passage has a tempo that is an integer multiple of a base tempo, and playing the musical passages in sync with each other.
  • the method further comprises the step of playing back a musical accompaniment track that has the same musical key as the musical passages and a tempo that is an integer multiple of the base tempo.
  • the playback of an associated musical passage is stopped if its playback volume is below a threshold value, and a stopped musical passage whose playback volume is above a threshold value, is restarted at a playback position where the playback position would be in case the musical passage had not been stopped.
  • the lengths of musical passages are defined as integer multiples of a base length, and the musical passages are looped consecutively and continuously.
  • a threshold speed for the playing is defined, and playback of a single non-synchronized sound is started with each trigger gesture when playing at slower speed than the threshold speed.
  • the playback of single non-synchronized sounds is started when predefined special poses and gestures are recognized.
  • a pose is recognized among a predetermined discrete set of poses, and only the musical passages, which are associated with the pre-stored player state comprising the same discrete pose as the recognized player state, are played audibly.
  • playing speed is recognized among a predetermined discrete set of playing speed values, and only the musical passages, which are associated with the pre-stored player state comprising the same discrete playing speed as the recognized player state, are played audibly.
  • instruction images representing predefined poses are displayed for giving movement instructions to the player.
  • the instruction images are highlighted as a function of the distance between the pose of the displayed image and the pose of the player.
  • visual selection objects are provided for allowing the player to choose a desired pose by virtually touching the visual selection objects on screen.
  • a level of matching is calculating between the instruction images and the player' s current visual representation, and the desired pose is chosen as the best match among the instruction images.
  • player' s hands and feet are included in said matching calculation.
  • a limited subset of instruction images are provided at a time which show the best matches according to player' s current visual representation.
  • the method steps described above are implemented in a form of a computer program.
  • the computer program comprises program code configured to control a data- processing device to perform the previously disclosed method steps while applicable.
  • the computer program is embodied on a computer readable medium.
  • the parameters and additional features mentioned regarding different embodiments of the method are also applied to the program code of the computer program and to the system according to the invention.
  • a system for creating synthesized music in response to a player's gestures comprises a motion tracking tool and data processing means configured to recognize a current player state.
  • the system further comprises a memory configured to store a plurality of musical passages which each comprise an association to pre-stored player states.
  • the system comprises a calculating means configured to calculate differences between the recognized player state and at least one pre-stored player state stored in the memory.
  • the system also comprises a control unit configured to adjust playback volumes of the associated musical passages according to the calculated differences so that a small difference yields a high volume and a large difference yields a low volume.
  • the system comprises a sound production device configured to produce the synthesized music according to the control unit output .
  • a further embodiment of the invention is a system comprising a computing device controlled by said computer program.
  • the system further comprises location indicators worn by the player in order to define the locations of player' s hands and possibly feet, and means for capturing the location data with the motion tracking tool.
  • the system further comprises a camera in the motion tracking tool for taking plurality of pictures of the player, and means for using the picture data for the player state recognition.
  • the system further comprises a screen and graphics rendering means configured to show the picture of the player composited inside three-dimensional computer graphics as a billboard texture that stays facing the virtual camera if the virtual camera moves.
  • a screen and graphics rendering means are configured to display instruction images representing predefined poses for giving movement instructions to the player.
  • visual selection objects are provided for allowing the player to choose a desired pose by virtually touching the visual selection objects on screen.
  • the calculating means is configured to calculate a level of matching between the instruction images and the player' s current visual representation, and the control unit is configured to choose the desired pose as the best match among the instruction images.
  • the device is a useful virtual musical instrument which provides a new manner of transforming player' s gestures and postures into a realistic audio experience.
  • the invention is useful e.g. for creating virtual air guitar playing system and corresponding guitar sound creation.
  • Fig. 1 is a simple block diagram of the architecture according to an embodiment of the invention.
  • Fig. 2 is a flow chart of the procedure with needed apparatus in an embodiment of the invention
  • Fig. 3 is a flow chart of a more detailed embodiment of the procedure according to the invention
  • Fig. 4 is a flow chart of another more detailed embodiment of the procedure according to the invention.
  • Fig. 5 is an example of computer graphics rendered by a software based embodiment of the invention.
  • Fig. 6 is a second example of computer graphics rendered by a software based embodiment of the invention.
  • Fig. 7 is a third example of computer graphics rendered by a software based embodiment of the invention.
  • the present invention discloses a method for synthesizing music by playing a so-called air guitar.
  • the system according to the invention is able to capture the gestures and convert them to different licks, that is, musical passages like the real guitar would produce.
  • the musical passages may be stored in computer memory or on a computer readable medium, for example, as MIDI or as sound waveform data.
  • the present invention allows the user to control a virtual musical instrument without any physical user interface of an actual musical instrument.
  • the physical user interface is exchanged into that of a virtual full- body interface, i.e. controlling the created sound with postures assumed by the user' s body, here referred to as poses, and movements of the body, here referred to as gestures.
  • the user may use three controls to select the sound being played. These controls may be for instance the pose, playing speed defined by the frequency of performed trigger gestures, and distance between the left and right hands of the player.
  • player's state is observed with a suitable apparatus.
  • the player state may comprise gesture data of the player and the posture of the player.
  • Gesture data comprises at least the aforementioned distance between the hands and the playing speed.
  • Each combination of the three controls is mapped to a specific musical passage, called a lick, which can be stored in a database, for instance.
  • the database with the lick data can be stored in a memory, which can form a part of a computer or it can be a separate memory unit. For example, standing in a certain pose, playing at slow speed (where the frequency of the player' s trigger gestures is low) , and holding the hands at a far mutual distance corresponds to one lick. Keeping the other parameters the same but playing at a high speed changes the lick to a different one, as does moving the hands closer to each other. In one embodiment of the invention, changing the pose opens up a new collection of licks for each combination of playing speed and the mutual distance of hands.
  • each pose can be seen to contain a subset of licks, where each lick is activated by changing playing speed and the hand distance.
  • the reason for this categorization is because changing poses is subjectively a major change, while the hand distance and playing speed are minor changes.
  • the control parameters may be categorized into discrete parameter sets, but the user's movements and gestures are continuous by nature.
  • licks are selected based on the combination of control parameters which the user is the nearest to. For example, if the user's position is between the two defined poses, the lick is selected based on which pose the user is closer to.
  • selection is performed which means that the volumes of the other licks are decreased either completely or partially.
  • the volumes may be normalized so that the overall loudness perceived by the player stays constant even though the relative volumes of the licks change.
  • the pose may be described as a plurality of features, such as the locations of the player's hand and feet, body joint angles, or image features computed from an image acquired by a camera or from data provided by some other motion tracking tool, such as the control devices of Nintendo Wii game console.
  • the pose may be determined as one of a predetermined discrete set of reference poses using pattern recognition and classification techniques, such as selecting the predetermined pose with a feature vector closest to the feature vector computed based on the motion tracking data.
  • the predetermined poses may include, for example, "low grinding”, “basic playing", or "intense solo”.
  • Each predetermined pose may have an associated a plurality of input image reference areas, and pose may be determined by computing the overlap of the player' s body parts and the reference areas .
  • the poses represent the emotional and expressive qualities of the licks (e.g., “relaxed”, “anxious”, “heroic”) , whereas hand distance and playing speed are more directly mapped to musical parameters, such as the amount and pitch of notes in the lick.
  • Learning to express emotion through playing a real instrument requires considerably more training and fine- motor skill than simply producing the correct notes.
  • the benefit of the present invention is that the player can produce professional sounding music by expressing emotion with full body and controlling the essential musical parameters with simple gestures.
  • frantic solo licks may be played by dropping on one's knees and arching one's back intensely, whereas confident, low, and crunchy licks may be played by assuming a stabile low stance with legs widely spread apart but keeping upper body relaxed.
  • the player states may be categorized into discrete sets, but the user's movement is continuous by nature.
  • licks are selected based on player state that the user is the nearest to. For example, if the user is between two poses, the lick is selected based on which pose the user is the closest to.
  • selection means that the volumes of the other licks are decreased either completely or partially.
  • the playing speed parameter is achieved by detecting the movements of one hand in a top-down motion similar to strumming the strings of a real guitar.
  • the motion tracking tool 11 may be, for example, an ordinary digital camera that is capable of providing images at a certain desired resolution and rate.
  • the motion tracking tool 11 may comprise computing means configured to perform data analysis, such as detecting a pose or filtering the signals.
  • the motion tracking tool 11 may also comprise a number of devices held by or attached to the user 10 that are capable of sensing their location and/or orientation in space.
  • these devices may be gloves which are worn by the virtual guitar player.
  • the gloves can be connected to the computing unit 12 with wires or wirelessly via radio transmission, or the gloves can simply act as passive markers that can be recognized from a picture by computer vision software executed by the computing unit.
  • the computing unit 12 may be, for example, an ordinary computer having sufficient computing power to provide the result at desired quality level. Furthermore, the computing unit 12 includes common means, such as a processor and a memory, in order to execute a computer program or a computer-implemented method according to the present invention. Furthermore, the computing device includes storage capacity for storing the recorded music pieces that are played back in the usage of the present invention.
  • the sound production device 13 may be, for example, an ordinary loudspeaker capable of generating sound waves or a device that is capable of recording audio signals into digital or analog format on a storage device .
  • Figure 2 discloses a flow chart disclosing different functional units in one embodiment of the invention.
  • synthesized musical passages are chosen based on the poses and gestures of the user.
  • the motion tracking tool 21 monitors the user 20 and transmits data concerning the user's body and movements to a player state analysis system 22.
  • the player state analysis system 22 analyses the data and determines the posture of the user and/or gestures which the user is performing.
  • the player state analysis system 22 outputs data that corresponds to the control parameters. These parameters can include for example a pose, speed or rate of playing, the distance between player's hands, the angle between the player's hands, and/or the locations of the player's hand and/or feet.
  • This data is transmitted to a mixing unit 25, which may be simply a multiplexer, but also, for example, a volume controller.
  • the mixing unit 25 reads the licks 24 (in this example, N pieces of choosable licks) from a lick database 23 and modifies their playback states and playback volumes so that the lick corresponding to the control parameters is being played back.
  • the mixed audio is sent to the sound production device 26 for audio output .
  • Figure 3 discloses a flow chart of a more detailed embodiment according to the invention.
  • the motion tracking tool 31 monitors the player 30, and transmits data to a pose analysis unit 32 and a musical parameter analysis unit 33.
  • the pose analysis unit 32 outputs pose data, such as a pose identifier.
  • the musical parameter analysis unit 33 outputs musical gesture data, such as the hand distance and playing speed.
  • the group selector 36 selects a lick group from the database 35 based on the pose data.
  • the group selector 36 outputs the individual licks that belong to the selected group.
  • the lick mixer 37 selects a lick based on the musical gesture data.
  • the lick mixer 37 outputs the selected lick for audio output 34.
  • Figure 4 discloses a flow chart of yet another embodiment of the invention where the selection of licks proceeds hierarchically.
  • the motion tracking tool 41 monitors the player 40, and transmits data to the pose analysis unit 42, playing speed analysis unit 43 and hand distance analysis unit 44.
  • the pose based selector 47 selects and outputs the licks from the lick database 46 that correspond to the detected pose.
  • the speed based selector 48 selects and outputs the licks that correspond to the detected playing speed.
  • the hand distance based selector 49 selects and outputs the lick that corresponds to the detected hand distance.
  • Units 47, 48 and 49 may also be placed to any other mutual order than the one shown in Figure 4.
  • the selected lick is directed to the audio output device 45.
  • the content of the pre-recorded licks can correspond to the combinations of pose, speed and distance, for example. This means that each combination can be mapped to a certain lick and any chosen lick corresponds to one specific combination of the playing gesture parameters.
  • For each playing speed there are licks containing passages, where the recorded instrument is played at different speeds. These speeds may be, for example, musical 4 th , 8 th and 16 th notes, corresponding to slow, medium and fast playing speeds, respectively. More accurately, the passages contain the perception of being played with 4 th , 8 th and 16 th notes where the actual sounds may contain syncopations or deviations in timing to make them musically rich and interesting while still maintaining the perception of slow, medium or fast playing speed.
  • the same or similar lick exists with a certain pitch.
  • broad grip large distance of the hands
  • narrow grip results to a higher pitch of the same lick.
  • Playing a certain lick at, for example, medium speed and then moving the hands from near to far distance changes the lick from high to low pitch.
  • the licks within one subset of poses are typically fairly similar to each other compared to the licks with a different pose.
  • an accompaniment track can be played throughout the performance.
  • the user' s perception is that they are playing the lead instrument in an orchestra or band.
  • the method of selecting discrete licks based on continuous movements is prone to fluctuations.
  • ways of mitigating these fluctuations can be incorporated into the system.
  • the system knows the user's Mistance' to each defined pose at all times.
  • the same theory applies to hand distance and playing speed as well.
  • the second way of the mitigation process is performing a cross-fade between two licks.
  • all licks in the current subset of the pose are being played back, but only one of them is at full volume while the other licks are at zero volume.
  • the playing volume of the two licks related to that control parameter' s extreme values changes as a function of the parameter change. For example, moving from a near hand distance towards a far hand distance, the volume of the lick corresponding to the near distance is reduced, and the volume of the lick corresponding to the far distance is increased correspondingly. Since the user can not be assumed to be proficient with any instrument, the system must ensure that the result sounds musical. This is accomplished by adhering to certain musical rules both in the recording of the musical content as well as in the real-time control.
  • each lick in the database is recorded with a tempo (beats per minute) that is an integer multiple of a base tempo.
  • a tempo beats per minute
  • a lick with 8 th is played twice as fast, containing approximately 240 notes, and a lick with 16 th notes is four times as fast (480) .
  • the length of each lick is an integer multiplier of a base length, e.g. a musical measure, so that memory and disk capacity can be saved by looping the licks (restarting playback once the lick has reached its end) so that a steady tempo is maintained.
  • all licks have also been recorded in the same musical key and overall style so that they fit together well.
  • the licks must be synchronized in time. Usually, this is accomplished by playing back all licks at the same time and controlling the volumes of each lick. However, for the sake of efficient calculation, licks to be played at zero volume may be stopped entirely and restarted, when the user picks up one of them. In this case, the playback must be started at a position that is synchronized to the global time.
  • the new lick must start playing from the position of 1.317 seconds, not the beginning, and not the position it was previously stopped at. This ensures that each lick is always synchronized to a global tempo, thus ensuring that they fit into the accompaniment track.
  • the user may also trigger single, non- synchronized sounds called long notes by playing at a speed even slower than the slowest licks.
  • each playing gesture is interpreted as a trigger for a long note.
  • the recorded content of long notes may be, for example, individual notes or chords that are let ring for a long time.
  • the long notes may also be miscellaneous, non-musical sounds such as explosions or similar kinds of special sounds.
  • triggering a long note immediately mutes any lick being played and begins the playback of a long note from the beginning.
  • the note will continue playing until the user plays another note or chooses to mute the currently playing note with a special gesture.
  • the user may mute the lick by lifting the right hand back up above the centerline, as if on top of an imaginary guitar's strings.
  • An exception to this might be a case where the user moves the right hand horizontally away to the side of the imaginary guitar and only then lifts the hand back up, in which case the note would continue to be played.
  • the pose and hand distance affect which long note is chosen. Typically, there may be 2-6 different hand distances which produce a different note, each of which is perceptually higher or lower in pitch corresponding to the distance.
  • the user may also trigger single, non-synchronized sounds called special effects with predefined trigger gestures or poses. For example, assuming a certain pose may trigger the sound of an explosion, or moving the right hand in a circular motion may trigger the sound of hitting a drum.
  • An embodiment of the invention may be a gaming system that comprises a screen configured to display the picture of the user captured by the motion tracking tool.
  • the background may be removed from the picture using computer vision software, and the picture may be rendered as a billboard texture inside 3d graphics.
  • the 3d graphics are displayed from the point of view of a virtual camera that can move, and the billboard texture is rotated so that it always faces the camera to maintain the illusion and not reveal the 2d nature of the texture.
  • the present invention may comprise also instruction methods and visualizations that solve the problem of communicating to the player what player states
  • a plurality of instruction images 51, 52, 53, one for each recognized player pose are visualized on screen in addition to the user 50.
  • feedback provided by the system to the player about the pose analysis is important. The feedback helps the player determine how to move for allowing the system to detect a desired pose.
  • the instruction images may be modified depending on their difference from the current pose of the player. For example, the instruction image depicting a pose closest to the player may be made larger than the other instruction images.
  • all instruction images may be scaled according to the distances between the poses corresponding to the instruction images and the player's current pose. Additionally, the instruction image depicting a pose closest to the player may be highlighted, e.g., using a glow effect.
  • the pose analysis may comprise the player' s visual representation 60 interacting on screen with visual elements, e.g., by touching at least one of a plurality of visual selection objects 61, 62, 63 to define the current pose.
  • the user experience may become similar to the player playing a virtual guitar and selecting a lick group by touching virtual foot pedals on screen, and then modifying the volumes of the licks within the group with other player state information, such as the distance between hands and the playing speed.
  • the pose analysis may comprise displaying a plurality of instruction images 71, 72, 73, 74, each corresponding to a pre-defined pose, and determining the recognized pose by measuring how well the player's current visual representation 70 on screen matches the instruction images.
  • the instruction image 74 the instruction image 74
  • the match may be measured by computing the distance between the on screen locations of the player' s body parts and the corresponding on screen locations of the body parts of the instruction images.
  • the body parts used in the distance computation may comprise hands and feet.
  • some of the instruction images may be hidden part of the time based on, e.g., the player's location on screen or on a virtual stage 75 so that the player may move around to search for the instruction images.
  • the instruction images may move on screen so that they stay close to the player, and to minimize visual clutter, only a subset of the instruction images are visible at a given moment. The visible subset may be determined by computing which instruction images are matching the player's current pose best.

Abstract

L'invention concerne un procédé, un logiciel et un système pour synthèse de musique commandée par geste. L'état de l'utilisateur est reconnu et comparé à des états pré-stockés, chacun étant associé à un passage musical. En fonction de données d'états, telles qu'une pose, une fréquence de gestes de déclenchement (par exemple, l'action de gratter les cordes d'une guitare imaginaire), et la distance entre les mains de l'utilisateur, des passages appropriés sont choisis pour la lecture. L'invention laisse l'utilisateur générer une musique de haute qualité sans entraînement musical. De façon caractéristique, l'invention est utilisée pour créer l'illusion du fait qu'un utilisateur joue de la guitare en solo dans un groupe de rock, en particulier dans des modes de réalisation qui comprennent l'affichage d'une image de l'utilisateur sur une scène virtuelle devant un public virtuel. L'utilisateur peut commander des paramètres musicaux simples, tels que la hauteur et la quantité de notes, à l'aide de gestes simples, et utiliser des poses de tout le corps pour commander des aspects émotionnels de la sortie musicale.
PCT/FI2008/050421 2007-07-09 2008-07-09 Système de synthèse de musique commandé par geste WO2009007512A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FI20075530 2007-07-09
FI20075530A FI20075530A0 (fi) 2007-07-09 2007-07-09 Eleohjattu musiikkisynteesijärjestelmä

Publications (1)

Publication Number Publication Date
WO2009007512A1 true WO2009007512A1 (fr) 2009-01-15

Family

ID=38331618

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/FI2008/050421 WO2009007512A1 (fr) 2007-07-09 2008-07-09 Système de synthèse de musique commandé par geste

Country Status (2)

Country Link
FI (1) FI20075530A0 (fr)
WO (1) WO2009007512A1 (fr)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012069614A1 (fr) * 2010-11-25 2012-05-31 Institut für Rundfunktechnik GmbH Procédé et ensemble pour présentation par signal audio améliorée de sons durant un enregistrement vidéo
GB2496521A (en) * 2011-11-11 2013-05-15 Fictitious Capital Ltd Computerised musical instrument using motion capture and analysis
US8618405B2 (en) 2010-12-09 2013-12-31 Microsoft Corp. Free-space gesture musical instrument digital interface (MIDI) controller
US9573049B2 (en) 2013-01-07 2017-02-21 Mibblio, Inc. Strum pad
WO2017048713A1 (fr) * 2015-09-16 2017-03-23 Magic Leap, Inc. Mélange de fichiers audio en fonction de poses de tête
US10188957B2 (en) 2016-10-18 2019-01-29 Mattel, Inc. Toy with proximity-based interactive features
WO2020007179A1 (fr) * 2018-07-05 2020-01-09 腾讯科技（深圳）有限公司 Procédé et dispositif de réglage de posture, support de stockage et dispositif électronique
US10839778B1 (en) * 2019-06-13 2020-11-17 Everett Reid Circumambient musical sensor pods system
GB2585060A (en) * 2019-06-27 2020-12-30 Sony Interactive Entertainment Inc Audio generation system and method
CN113299256A (zh) * 2021-05-14 2021-08-24 上海锣钹信息科技有限公司 Midi数字音乐演奏交互方法
CN116504205A (zh) * 2023-03-01 2023-07-28 广州感音科技有限公司 一种音乐演奏控制方法、系统、介质及计算机

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10357714B2 (en) 2009-10-27 2019-07-23 Harmonix Music Systems, Inc. Gesture-based user interface for navigating a menu

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4658427A (en) * 1982-12-10 1987-04-14 Etat Francais Represente Per Le Ministre Des Ptt (Centre National D'etudes Des Telecommunications) Sound production device
WO1993022762A1 (fr) * 1992-04-24 1993-11-11 The Walt Disney Company Appareil et procede pour suivre un mouvement afin de generer un signal de commande
US6506969B1 (en) * 1998-09-24 2003-01-14 Medal Sarl Automatic music generating method and device
WO2005094958A1 (fr) * 2004-03-23 2005-10-13 Harmonix Music Systems, Inc. Procede et appareil pour commander un personnage en trois dimensions dans un environnement de jeu en trois dimensions

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4658427A (en) * 1982-12-10 1987-04-14 Etat Francais Represente Per Le Ministre Des Ptt (Centre National D'etudes Des Telecommunications) Sound production device
WO1993022762A1 (fr) * 1992-04-24 1993-11-11 The Walt Disney Company Appareil et procede pour suivre un mouvement afin de generer un signal de commande
US6506969B1 (en) * 1998-09-24 2003-01-14 Medal Sarl Automatic music generating method and device
WO2005094958A1 (fr) * 2004-03-23 2005-10-13 Harmonix Music Systems, Inc. Procede et appareil pour commander un personnage en trois dimensions dans un environnement de jeu en trois dimensions

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
KARJALAINEN M. ET AL.: "Virtual Air Guitar", AUDIO ENGINEERING SOCIETY, CONVENTION PAPER, SAN FRANCISCO, 28 October 2004 (2004-10-28) - 31 October 2004 (2004-10-31) *

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103329145A (zh) * 2010-11-25 2013-09-25 无线电广播技术研究所有限公司 用于视频记录期间的声音的改善音频信号呈现的方法和组件
US9240213B2 (en) 2010-11-25 2016-01-19 Institut Fur Rundfunktechnik Gmbh Method and assembly for improved audio signal presentation of sounds during a video recording
WO2012069614A1 (fr) * 2010-11-25 2012-05-31 Institut für Rundfunktechnik GmbH Procédé et ensemble pour présentation par signal audio améliorée de sons durant un enregistrement vidéo
US8618405B2 (en) 2010-12-09 2013-12-31 Microsoft Corp. Free-space gesture musical instrument digital interface (MIDI) controller
GB2496521B (en) * 2011-11-11 2019-01-16 Fictitious Capital Ltd Computerised percussion instrument
GB2496521A (en) * 2011-11-11 2013-05-15 Fictitious Capital Ltd Computerised musical instrument using motion capture and analysis
US9224377B2 (en) 2011-11-11 2015-12-29 Fictitious Capital Limited Computerized percussion instrument
US9573049B2 (en) 2013-01-07 2017-02-21 Mibblio, Inc. Strum pad
WO2017048713A1 (fr) * 2015-09-16 2017-03-23 Magic Leap, Inc. Mélange de fichiers audio en fonction de poses de tête
US11039267B2 (en) 2015-09-16 2021-06-15 Magic Leap, Inc. Head pose mixing of audio files
US10681489B2 (en) 2015-09-16 2020-06-09 Magic Leap, Inc. Head pose mixing of audio files
US11778412B2 (en) 2015-09-16 2023-10-03 Magic Leap, Inc. Head pose mixing of audio files
US11438724B2 (en) 2015-09-16 2022-09-06 Magic Leap, Inc. Head pose mixing of audio files
US10188957B2 (en) 2016-10-18 2019-01-29 Mattel, Inc. Toy with proximity-based interactive features
WO2020007179A1 (fr) * 2018-07-05 2020-01-09 腾讯科技（深圳）有限公司 Procédé et dispositif de réglage de posture, support de stockage et dispositif électronique
US10839778B1 (en) * 2019-06-13 2020-11-17 Everett Reid Circumambient musical sensor pods system
GB2585060A (en) * 2019-06-27 2020-12-30 Sony Interactive Entertainment Inc Audio generation system and method
CN113299256A (zh) * 2021-05-14 2021-08-24 上海锣钹信息科技有限公司 Midi数字音乐演奏交互方法
CN116504205A (zh) * 2023-03-01 2023-07-28 广州感音科技有限公司 一种音乐演奏控制方法、系统、介质及计算机
CN116504205B (zh) * 2023-03-01 2023-11-24 广州感音科技有限公司 一种音乐演奏控制方法、系统、介质及计算机

Also Published As

Publication number Publication date
FI20075530A0 (fi) 2007-07-09

Similar Documents

Publication Publication Date Title
WO2009007512A1 (fr) Système de synthèse de musique commandé par geste
US9981193B2 (en) Movement based recognition and evaluation
US8444464B2 (en) Prompting a player of a dance game
US7589727B2 (en) Method and apparatus for generating visual images based on musical compositions
US9358456B1 (en) Dance competition game
US7893337B2 (en) System and method for learning music in a computer game
US6541692B2 (en) Dynamically adjustable network enabled method for playing along with music
JP5198766B2 (ja) テンポの変化に対応するメトロノーム及び拍の間隔の記録方法
JP6137935B2 (ja) 身体動作評価装置、カラオケシステム、及びプログラム
US20150103019A1 (en) Methods and Devices and Systems for Positioning Input Devices and Creating Control
US6878869B2 (en) Audio signal outputting method and BGM generation method
Petersen et al. Musical-based interaction system for the Waseda Flutist Robot: Implementation of the visual tracking interaction module
WO2012020242A2 (fr) Système de réalité augmentée
Baba et al. " VirtualPhilharmony": A Conducting System with Heuristics of Conducting an Orchestra.
Fels et al. Musikalscope: A graphical musical instrument
WO2015194509A1 (fr) Dispositif de génération de vidéo, procédé de génération de vidéo, programme, et support de stockage d'informations
JP2001215963A (ja) 音楽演奏装置、音楽演奏ゲーム装置及び記録媒体
Siegel Dancing the music: Interactive dance and music
JP4131279B2 (ja) 合奏パラメータ表示装置
Hoang et al. Multimodal Metronome—Rhythm game for musical instruments
JP2005321514A (ja) ゲーム装置および音程付効果音生成プログラムならびに方法
Sourin Music in the air with leap motion controller
Feitsch et al. Tangible and body-related interaction techniques for a singing voice synthesis installation
Hansen An Introduction to Interactive Music for Percussion and Computers
JP2021140065A (ja) 処理システム、音響システム及びプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 08787698

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 08787698

Country of ref document: EP

Kind code of ref document: A1