FR3034887A1 - DEVICE AND METHOD FOR REAL-TIME GESTURAL CONTROL OF AUDIO SIGNAL - Google Patents

Abstract

The invention relates to a device and a method for real-time gestural control of an audio signal, which, as a function of gestural data (311), makes it possible to modify reading parameters (on / off, reading speed) and to apply effects sound to an incoming audio signal (321). This incoming audio signal (321) is continuously generated from a working audio signal (361) corresponding to the original audio signal (11) extracted as a read operation of a file (1) audio or audio-video. The gestural data is generated from successive positions of at least a portion (91a, 91b) of a human body (9) detected in a capture field (21) by a sensor (2).

Description

FIELD OF THE INVENTION The present invention relates to the field of human-machine interactions and real-time systems. More specifically, the present invention relates to a device and method for real-time gesture control of audio signal. STATE OF THE PRIOR ART In the prior art, real-time gesture control devices for audio signals are known.

For example, the "iSymphony" system makes it possible to modify, in real time, parameters such as the playback speed or the volume of a digital audio recording via the capture of the movements of a stick. This system is endowed with an intelligence that typically makes it possible to recognize different types of gestures associated with trajectories of this stick in a capture field (physical space) and to modify, according to the type of gesture recognized, such parameters. This system also makes it possible to perform a temporal stretching operation, called "time-stretching" in English. A disadvantage of such a system is that it involves performing a prior reading of the digital audio recording, in particular to implement the time stretching operation. A prior reading of the file must be performed, at the nominal reading speed, in order to identify in advance the characteristic points of the digital audio recording. Another known system is the "TimeFlux" system which makes it possible in particular to modify the playback speed of an audio recording, for example by means of a computer mouse or an optical sensor (for example a sensor "Leap Motion") able to detect movements of one or two hands in a capture field (physical space).

A disadvantage of such a system is that it does not allow a temporal stretching operation to be carried out in real time, that is to say without pretreatment of the file, and without changing the tone. 3034887 - 2 - Another disadvantage is that this system requires a multiplicity of plug-ins (or software plugins), not allowing it to constitute a turnkey system. Another disadvantage of this system is that it does not allow customizing effects to be applied to the audio recording according to a user's gestures. Another disadvantage is that this system is not open to any sound or musical file: this system is constrained by a pretreated and limited sound bank.

Another disadvantage of such a system is that it requires advanced computer-aided music (CAM) knowledge and must be used within a home studio environment (or home). studio "in English).

An object of the invention is to overcome all or part of the disadvantages of control devices of the state of the art which aim to influence the reproduction of audio signals by the gesture. A particular object of the invention is to propose a device for real-time gestural control of the reproduction of an audio signal which makes it possible to overcome any preprocessing of this audio signal, prior to its reproduction. In this case, the invention aims to allow interaction on the reproduction of an audio signal encoded in a digital file, without prior decoding of the entire digital file.

SUMMARY OF THE INVENTION This object is achieved with a real-time gesture control device for reproducing an audio signal, which device is arranged to receive a file, preferably digital, audio or audio-video encoding a signal. original audio, this device comprising: a sensor, for example the Leap Motion® system or a smartphone, arranged to detect in a multidimensional capture field, preferably three-dimensional, a position of at least a part of a human body preferably, at least one hand, a computer processing system comprising: a gesture data management module arranged to generate gestural data from successive positions of the at least a portion of said least one human body detected by the sensor, o an audio data management module arranged to continuously generate an incoming audio signal from the audio signal of original origin it progressively audio or audio file, o a signal processing module arranged to apply sound effects to the incoming audio signal according to the gestural data. Sound effect means any alteration of the signal, especially in terms of volume or sound intensity, frequency equalization, time-stretching, granular synthesis, fade, or reverberation. Sound effects can alter signal dynamics, ie, influence the differences between the lowest signal levels and its highest levels. The sound effects can also consist of temporal effects, modifying the temporal reproduction (delays, phase shifts, etc.) of all or part of the signal. Finally, sound effects can act on the frequency spectrum of the signal, for example by filtering frequency bands, shifting frequencies, or generating harmonics. According to the invention, the original audio signal is extracted progressively from the audio or audio-video file. This means that the generation of the incoming audio signal from a given sample of the original audio signal requires the prior extraction only of that given sample. In this case, the samples upstream of this given sample (located backwards in time) are no longer needed for the generation of the incoming audio signal, and the samples located downstream of this given sample (located further in time) are not necessary. Of course, the original audio signal sample required for the generation of the incoming audio signal evolves over time. According to a particular embodiment, the control device further comprises: an audio data management module arranged to perform a reading operation in which the original audio signal is extracted progressively from the audio file; or audio-video according to reading parameters, o a reading parameter management module arranged to modify at least one of the following reading parameters according to the gestural data: a reading mode that can take an "active" state Wherein the read operation is performed and an "idle" state in which the read operation is not performed, a read speed, the read speed is the read speed of the read original audio signal during the read operation. Such a device then makes it possible to modify, by the gesture: - reading parameters (of the type play, pause, stop, playback speed) of the audio or audio-video file received by the device, 20 and / or - sound parameters (of the type of sound effects mentioned above) of the incoming audio signal. The audio data management module and the reading parameter management module are preferably integrated in the computer processing system of the control device. The audio data management module may further be arranged to: store in a buffer a work audio signal corresponding to the original audio signal progressively extracted from the audio or audio-video file, and generate continuously the incoming audio signal from this working audio signal. It should be noted that the working audio signal corresponds to the original audio signal extracted as the audio or audio-video file is read.

Such a device is particularly advantageous because it makes it possible to detect gestures of a non-paired user, who can in particular have his hands free of any control device of the stick, remote control or other type of device enabling motion capture. In addition, the computer processing system makes it possible to process any type of digital audio or audio file in real time, without prior reading of the file. Preferably, the computer processing system further comprises a sound pitch management module arranged to maintain a sound pitch of the incoming audio signal when the playback speed is changed. The pitch of a sound is the frequency of vibration of that sound. When the sound is composed of several frequencies, the pitch is defined at least by the fundamental frequency of this sound.

For example, the sound level management module is integrated in the computer processing system of the control device. The at least part of the human body can be at least one hand. The device then allows a user to interact on the reproduction of the sound signal in the manner of a conductor. According to an advantageous characteristic, the sensor and the gesture data management module can also be arranged to generate gestural data from a degree of closure of the at least one hand.

According to another advantageous characteristic, the sensor and the gesture data management module may also be arranged to generate gestural data from a degree of inclination of the at least one hand. The invention also relates to a method for real-time gesture control of the reproduction of an audio signal comprising: a detection in a multidimensional, preferably three-dimensional, capture field of a position of at least a part of a human body, preferably at least one hand, a generation of gestural data from successive positions of the at least a portion of said human body, a read operation in which an original audio signal is progressively extracted from an audio or audio-video file according to reading parameters, a storage of a working audio signal corresponding to the original audio signal extracted as the read operation, a generation in a continuous stream of an incoming audio signal from the working audio signal, a modification of at least one of, preferably all of, the following playback parameters according to the gesture data: e of read that can take an "active" state in which the read operation is implemented and an "inactive" state in which the read operation is not implemented, and / or o a speed of reading, the reading speed being the speed of extraction of the original audio signal during the reading operation, an application of sound effects to the incoming audio signal according to the gesture data. Preferably, the sound effects applied to the incoming audio signal include a change in loudness.

Advantageously, the sound intensity of the incoming audio signal can be modified when the generated gestural data translate a displacement of the at least part of said human body between two successive positions located in a vertical plane of the capture field, the plane vertical preferably comprising the earth gravity vector. According to an advantageous characteristic, the sound intensity of the arrival audio signal can be increased when a first position among said at least two successive positions is located at an altitude lower than a second position, and the sound intensity. the arrival audio signal can be decreased when the first position is at an altitude greater than the second position. According to another advantageous feature, the change in loudness may include frequency equalization processing. Preferably, the higher the loudness, the higher the frequency equalization processing is in opening, that is to say the more the frequency spectrum is fully restored. For example, at maximum loudness (100%), all frequencies are restored; when the sound intensity deviates from this maximum intensity, i.e. away from 100% and approaches 0%, a high-pass filter is typically applied which blocks mid-high frequencies. According to a particular embodiment, the modification of the sound intensity comprises a fading effect in which the sound intensity progressively changes from a first sound intensity value to a second sound intensity value, and then gradually returns to the first loudness value. The first loudness value is preferably the loudness applied just before the fade effect is triggered. The second loudness value is, for example, less than the first loudness value, typically 10 to 30% lower. The fade effect can have a predetermined duration, for example of the order of a few seconds. The return to the first intensity value is then performed independently of the gestural data.

Advantageously, the reading mode remains in the "active" state when the loudness is changed. According to yet another advantageous feature, the at least a portion of said human body may be a hand, and the loudness may progressively change from the first loudness value to the second loudness value. when the gestural data generated translate a movement of supination or pronation of the hand.

The second value may advantageously be less than the first value when the gestural data generated translate a supination movement of the hand. Preferably, the change in loudness is accompanied by reverberation processing, preferably of the "short tail" type and moderate intensity. Advantageously, the sound effects applied to the incoming audio signal may include a time stretching operation during which: the read operation is interrupted, the incoming audio signal is generated by combining one or more samples the working audio signal. By sample is meant a part of the audio signal considered between two distinct instants, typically separated from each other by a duration of between a few tenths of seconds and a few seconds. According to an advantageous characteristic, the at least a portion of said human body is preferably a hand, and the temporal stretching operation is preferably carried out during the entire period when the generated gestural data translate a hand gripping motion. during which fingers of the hand are closed on the palm of the hand. By gripping movement, is meant a movement by which one or more fingers of the hand are closed, or these fingers being directed towards the palm of the hand. According to another advantageous characteristic, the temporal stretching operation comprises a granular synthesis treatment. According to yet another advantageous characteristic, the temporal stretching operation comprises a reverberation treatment. Preferably, the reading speed is changed according to a difference between, on the one hand, a first time interval separating a first and a second position taken by the at least a portion of said human body and, secondly, a second time interval separating a third position from a fourth successive position taken by the at least part of said human body. The third position may correspond to the first position and the fourth position may correspond to the second position. Typically, when the movement is periodic, the third and fourth positions respectively correspond to the first and second positions. Preferably, the second time interval succeeds in time at the first time interval. Advantageously, the reading speed can be slowed down when the second time interval is shorter than the first time interval, and the reading speed can be accelerated when the second time interval is longer than the first time interval. DESCRIPTION OF THE FIGURES AND EMBODIMENTS Other advantages and particularities of the invention will appear on reading the detailed description of implementations and non-limiting embodiments, and FIG. 1 representing a device according to the invention. receiving an audio or audio-video file, as well as an individual placed in a capture field. The embodiments described below being in no way limiting, it will be possible to consider variants of the invention comprising only a selection of characteristics described, isolated from the other characteristics described (even if this selection is isolated within a sentence including these other features), if this selection of features is sufficient to confer a technical advantage or to differentiate the invention from the state of the art. This selection comprises at least one characteristic, preferably functional without structural details, or with only a part of the structural details if this part alone is sufficient to confer a technical advantage or to differentiate the invention from the state. of the prior art. An embodiment of the invention is illustrated in FIG. 1 showing a gesture control device in real time for reproducing an audio signal according to the invention.

This device is arranged to receive a file, preferably digital, audio or audio-video file 1. This file 1 typically encodes an original audio signal 11 according to any type of known format, for example MPEG, MP3, WAV, FLAC, etc.

The device of the invention typically comprises a sensor 2 adapted to detect a position of at least a portion of a human body 9. The human body 9 is also referred to herein as the user. In a preferred embodiment, the at least part of the user 9 corresponds to one or both hands 91a, 91b.

The sensor 2 makes it possible to detect the hand or hands 91a, 91b within a multidimensional capture field 21. This capture field 21 may be all or part of the three-dimensional physical space in which the user 9 is located. The sensor 2 is for example an optical sensor. In order to capture two-dimensional motions, the sensor comprises for example a photosensitive element formed of photosensitive points arranged in the form of a two-dimensional matrix. In order to capture three-dimensional movements, and / or gripping movements of a hand, the sensor may comprise at least two photosensitive elements each formed of a two-dimensional matrix of photosensitive dots. Insofar as the device according to the invention aims to capture movements of a human body, the sensor advantageously operates in the infrared range, that is to say in the wavelength range substantially between 750 nm. and 0.1 mm. Nevertheless, the sensor could also operate in the visible spectrum. By way of non-limiting example, the sensor is, for example, the Leap Motion® system, a smartphone optical sensor, or any system equipped in particular with one or more cameras. The device of the invention comprises a computer processing system 3. This computer processing system 3 notably makes it possible to collect data detected by the sensor 2. In the embodiment described here, these data correspond to manual gestures. of the user 9.

The collection of the data detected by the sensor 2 is carried out by a gesture data management module 31 of the computer processing system 3. This module 31 makes it possible to generate gestural data 311 from the data detected by the sensor 2, in accordance with FIG. occurrence of successive positions of one or both hands 91a, 91b of the user 9. As illustrated in FIGURE 1, the computer processing system 3 can be arranged to: - compare the generated gestural data 311 with one or 20 sets of reference gesture data 351, 352, 353, 354 stored in a library 35, and - generating control data according to the result of the comparison. The comparison of the gestural data generated 311 with reference gesture data series 351, 352, 353, 354 makes it possible to identify types of movement made by the user 9. Let us take the example of a series of gestural data of reference 351 corresponding to a gesture of a hand 91a successively occupying two distinct positions A, B in the capture field 21, this gesture corresponding for example to a vertical linear movement of the hand 91a from bottom to top relative to the body of the user 9 placed in a neutral position (for example while standing). In such an example, the sensor 2 detects the hand 91a in a first position A at a time TA and then in a second position B at a time TB, the instant TB being later than the instant TA, and the module 31 generates gesture data 3034887 - 12 - 311 translating this movement. This gestural data 311 is then compared by the computer processing system 3 to the reference data series 351, 352, 353, 354. In this example, the computer processing system 3 then associates the gestural data 311 to the series of data. reference gesture data 351, and generates control data that can be used by other modules 33, 34 described below to control one or more actions associated with such a gesture (for example, increasing the loudness an audio signal 321).

The computer processing system 3 also includes an audio data management module 32 arranged to continuously generate an incoming audio signal 321 from the original audio signal 11 progressively extracted from the audio or audio-video file 1.

To do this, in the example of FIG. 1, the audio data management module 32 is arranged to: perform a read operation in which the original audio signal 11 is extracted progressively from the audio or audio file; video 1 as a function of reading parameters, 20 - storing in a buffer 36 a working audio signal 361 corresponding to the original audio signal extracted as the reading operation proceeds, - continuously generating the audio signal of arrival 321 from the working audio signal 361.

The storage of the working audio signal 361 in a buffer memory 36 does not mean that it is necessary to perform a pre-reading of the audio or audio-video file 1 to apply reading parameters or any other type of signal processing, including a time stretching operation (see below).

The buffer memory 36 may simply consist of any medium capable of carrying digital or signal information, which is likely to cause a delay between the reading of the audio or audio file, and the reproduction of the incoming audio signal. In addition, the working audio signal 361 may consist of a truncated or partial portion of the original audio signal 11 being extracted. Preferably, the sensor 2 and the gesture data management module 31 are arranged to generate gestural data: from a degree of closure of the at least one hand 91a, 91b, 5 and / or or - from a degree of inclination of the at least one hand 91a, 91b. The computer processing system 3 may also include a read parameter management module 33 arranged to modify at least one of the following reading parameters (preferably all) as a function of the gesture or control data: read that can take an "active" state in which the read operation is implemented and an "inactive" state in which the read operation is not implemented, - a read speed, the speed of reading is the speed of extraction of the original audio signal 11 during the read operation. As illustrated in FIGURE 1, the computer processing system 3 further comprises a signal processing module 34 arranged to apply sound effects to the incoming audio signal 321 or work according to the gesture or control data. In distinction with the read parameter management module 33, the signal processing module 34 acts on the incoming audio signal 321 or the work signal and not simply on the reading of the original audio signal 11. Such a device associates, with each type of gesture performed by the user 9, in particular by moving one or both of his hands 91a, 91b, one or more read actions of the original audio signal 11 and / or processing the incoming audio signal 321 or work. Several types of nonlimiting association are described below as examples. In a first type of association, the sound intensity of the incoming audio signal 321 is modified when the generated gestural data 311 translate a displacement of the at least part of said human body 9 between two successive positions located in a vertical plane of the capture field 21, the vertical plane preferably comprising the earth gravity vector. Preferably, in this first type of association, the sound intensity of the incoming audio signal 321 is increased when a first position among said at least two successive positions is situated at an altitude lower than a second position (this example corresponds to the example already described above), and the sound intensity of the incoming audio signal 321 is decreased when the first position is located at an altitude greater than the second position. In one embodiment, the capture field 21 comprises three distinct areas separated by two horizontal planes. Each zone is assigned a sound intensity value or a sound intensity attenuation value with respect to a reference sound intensity. For example, the loudness is attenuated by 3 dB in a first lower zone, 0 dB in a second intermediate zone and is increased by 3 dB in a third upper zone. The change in loudness may include a fading effect, wherein the sound intensity progressively changes from a first loudness value to a second loudness value. Preferably, during such a change in loudness, the playback mode remains in the "active" state. Preferably, the change in loudness includes frequency equalization processing. Preferably, the lower the sound intensity, the less the field (or spectrum) of the frequencies of the equalization is wide.

In a second type of association, the loudness gradually changes from the first loudness value to the second loudness value when the generated gestural data 311 translate a supination or pronation movement of the hand. Preferably, the second value is less than the first value when the generated gestural data 311 translate a supination movement of the hand. The first value is, for example, 100% at the end of the pronation movement (for example, palm pointing downwards), and the second value of 0% at the end of the supination movement (for example, palm facing upwards). . In the second type of association, the change in loudness is preferably accompanied by a short-tail reverberation treatment of moderate intensity.

The sound effects applied to the incoming audio signal 321 may include a time stretching operation during which: the read operation is interrupted; the incoming audio signal 321 is generated by combining one or 15 In a third type of association, the time stretching operation is performed during the entire time that the generated gesture data 311 translate a gripping movement of the hand during which fingers of the hand are closed on the palm of the hand. Preferably, the time stretching operation comprises granular synthesis processing. To do this, it is possible, for example, to perpetrate a duration effect, to generate a complex wave established from micro-grains (or short samples) extracted from the file 1, modified and reassembled in a different order from the first one. order in which these micro-grains are assembled in the file 1. Preferably, the temporal stretching operation comprises a reverberation treatment, called "long tail" and maximum intensity (it is called "wet" effect ").

In a fourth type of association, the reading speed is changed according to a difference between, on the one hand, a first time interval separating a first and a second position taken by the at least a part of said body. 9 and secondly, a second time interval separating a third position from a fourth successive position taken by the at least a portion of said human body 9. Preferably, the reading speed is slowed when the second time interval is shorter than the first time interval, and it is accelerated when the second time interval is longer than the first time interval. It should be noted that the control device performing real-time operations, a change in the read speed, ie the extraction speed of the original audio signal, normally results in a change in the "speed". felt "at which the output audio signal is reproduced. A notable exception to the correspondence between these two speeds exists in the case of a time stretching operation, where the output audio signal is generated from the last extracted original audio signal sample, i.e. without extraction of a new signal sample.

Of course, the invention is not limited to the examples which have just been described and many adjustments can be made to these examples without departing from the scope of the invention. In addition, the various features, shapes, variants and embodiments of the invention can be associated with each other in various combinations in that they are not incompatible or exclusive of each other.

Claims (19)

REVENDICATIONS1. A device for real-time gesture control of the reproduction of an audio signal, this device being arranged to receive an audio or audio-video file (1) encoding an original audio signal (11), this device comprising: a sensor ( 2) arranged to detect in a multidimensional capture field (21) a position of at least a portion (91a, 91b) of a human body (9), a computer processing system (3) comprising: gesture data management (31) arranged to generate gestural data (311) from successive positions of the at least part of said at least one human body (9) detected by the sensor (2), o a management module audio data arrangement arranged to continuously generate an incoming audio signal from the original audio signal progressively extracted from the audio or audio-video file, o a signal processing module (34) arranged to apply sound effects to the signal Incoming audio (321) in function we have gestural data. 2. Device according to claim 1, characterized in that the computer processing system (3) further comprises a sound pitch management module (37) arranged to maintain a sound pitch of the incoming audio signal (321) when the reading speed is changed. 3. Device according to claim 1 or 2, characterized in that the at least one part (91a, 91b) of the human body (9) is at least one hand (91a, 91b), and in that the sensor (2 ) and the gesture data management module (31) are furthermore arranged to generate gestural data from a degree of closure of the at least one hand (91a, 91b). 4. Device according to one of claims 1 to 3, characterized in that the at least a portion of the human body (9) is at least one hand (91a, 91b), and 3034887 - 18 - in that the sensor (2) and the gesture data management module (31) are further arranged to generate gestural data from a degree of inclination of the at least one hand (91a, 91b). 5 A method for real-time gesture control of reproduction of an audio signal comprising: a detection in a multidimensional capture field (21) of a position of at least a portion (91a, 91b) of a human body ( 9), a generation of gestural data (311) from successive positions of the at least a portion of said human body (9), a read operation in which an original audio signal (11) is progressively extracted from an audio or audio-video file (1) according to read parameters, a storage of a working audio signal (361) corresponding to the original audio signal extracted as the read operation proceeds , continuously generating an incoming audio signal (321) from the working audio signal (361), changing at least one of the following reading parameters according to the gesture data: reading mode that can take an "active" state in which the read operation is implemented and an "idle" state in which the read operation is not implemented, o a read speed, the read speed being the extraction speed of the original audio signal (11 ) during the read operation, an application of sound effects to the incoming audio signal (321) in accordance with the gesture data. The method of claim 5, characterized in that the sound effects applied to the incoming audio signal (321) include a change in a loudness. 3034887 - 19 - 7. Method according to claim 6, characterized in that the sound intensity of the incoming audio signal (321) is modified when the generated gestural data (311) translate a displacement of the at least part of said human body (9). ) between two successive positions located in a vertical plane of the capture field (21). The method of claim 7, characterized in that the sound intensity of the incoming audio signal (321) is increased when a first one of said at least two successive positions is at an altitude lower than that of a second position, and in that the sound intensity of the incoming audio signal (321) is decreased when the first position is at an altitude higher than that of the second position. 9. Method according to one of claims 6 to 8, characterized in that the modification of the loudness comprises a frequency equalization process. 10. Method according to one of claims 6 to 9, characterized in that the modification of the sound intensity comprises a fading effect in which the sound intensity passes gradually from a first sound intensity value to a second one. sound intensity value. 11. The method of claim 10, characterized in that the at least a portion of said human body (9) is a hand, and in that the sound intensity 25 passes gradually from the first sound intensity value to the second sound intensity value when the generated gestural data (311) translate a movement of supination or pronation of the hand. 12. A method according to claim 11, characterized in that the second value is less than the first value when the gesture data generated (311) translate a supination movement of the hand. 13. Method according to one of claims 10 to 12, characterized in that the modification of the sound intensity is accompanied by a reverberation treatment. 3034887 - 20 - The method according to one of claims 5 to 13, characterized in that the sound effects applied to the incoming audio signal (321) comprise a time stretching operation during which: the read operation is interrupted the incoming audio signal (321) is generated by combining one or more samples of the working audio signal (361). 15. Method according to claim 14, characterized in that the at least part of said human body (9) is a hand, and in that the temporal stretching operation is carried out during the entire duration of the gestural data. generated (311) translate a gripping movement of the hand during which fingers of the hand are closed on the palm of the hand. 15 16. The method of claim 14 or 15, characterized in that the time stretching operation comprises a granular synthesis treatment. 20 17. Method according to one of claims 14 to 16, characterized in that the time stretching operation comprises a reverberation treatment. 18. Method according to one of claims 5 to 17, characterized in that the reading speed is changed according to a difference between, on the one hand, a first time interval separating a first and a second position taken by the at least a portion of said human body (9) and secondly a second time interval separating a third position from a fourth successive position taken by the at least a portion of said human body (9). 30 19. The method of claim 18, characterized in that the reading speed is slowed down when the second time interval is shorter than the first time interval, and in that the reading speed is accelerated when the second time interval. is longer than the first time interval.