TWI743472B - Virtual electronic instrument system and operating method thereof - Google Patents

Abstract

The present invention provides a virtual electronic instrument system, comprising at least one sensor, a server and a speaker. The at least one sensor, which is disposed on a user or mounted on at least one instrument component, detects a gesture of the user or a moving track of the at least one instrument component, and generate a track data. The server which is connected to the at least one sensor and receives the track data, comprising a processor. The processor analyzes the track data by an algorithm and generates an audio signal corresponding to the track data. The speaker is connected to the processor and outputs the audio signal.

Description

Virtual electronic musical instrument system and its operating method

The present invention provides a virtual electronic musical instrument system and an operating method thereof, in particular to a virtual electronic musical instrument system that detects a user's gesture trajectory or a movement trajectory of a musical instrument component through a sensor to output corresponding audio signals or visual signals, and How it works.

At present, most of the existing musical instruments are divided into two types: (1) Traditional musical instruments, such as percussion instruments such as jazz drums, stringed instruments such as guitars and various violins, or pianos; (2) Traditional musical instruments and electronic musical instruments, such as electronic drums, Electric guitar or keyboard, etc. However, the aforementioned musical instruments usually have disadvantages such as large size and complex structure. Many manpower and trucks or larger vehicles are required to transport the musical instruments, and the disassembly and assembly steps are also complicated, often requiring a lot of manpower and time costs. In addition, during the rehearsal process of the orchestra, the above sets of instruments are needed to perform the rehearsal. Whether it is renting a rehearsal studio or venue, or buying all the above-mentioned instrument combinations, these are often obstacles to the composition of the orchestra.

In view of this, manufacturers have successively begun to study virtual musical instruments, that is, the effect of musical instruments can be produced without the need for physical musical instruments. For example, US Patent Publication No. US9224377B2 proposes an electronic percussion instrument, which can produce the same sound as a percussion instrument without hitting the surface of the physical object. The image recognition technology calculates the meaning of the image and then emits the corresponding sound effect; however, the disadvantage of detecting the user's strike action through the camera is that it may be blocked by external objects during the shooting or recording process (such as the user himself). The problem of inoperability in special environments (such as darkrooms or vehicles in action). Therefore, there is an urgent need for a virtual electronic musical instrument system and its operating method that can be used in any situation to solve the problems mentioned in the prior art.

In order to solve the problem mentioned in the previous technology, how to let users experience the effect of performance without playing a physical instrument, and at the same time solve the problem of being blocked by external objects during the traditional use of one or more cameras to capture images For the problem of sight (such as the user’s own hands) or the inability to operate in the dark, the present invention proposes a virtual electronic musical instrument system, which includes: at least one sensor arranged on a user or installed on at least one musical instrument component, The at least one sensor detects the user's gesture trajectory or the movement trajectory of the at least one musical instrument component and generates a trajectory data; a system server is connected to the sensor and receives the trajectory data, the system servo The device includes: a processor that analyzes the trajectory data with an algorithm and generates an audio signal corresponding to the trajectory data; and a speaker connected to the processor, and the speaker outputs the audio signal.

In addition, the present invention further provides an operating method of a virtual electronic musical instrument system, including the following steps: (A) at least one sensor arranged on a user or installed on at least one musical instrument component detects the user's gesture trajectory; Or the movement trajectory of the at least one musical instrument component and generate trajectory data; (B) a system server analyzes the trajectory data with an algorithm and generates an audio signal corresponding to the trajectory data; and (C) a speaker outputs the Audio signal.

Furthermore, in step (B), the system server can also generate a visualization signal corresponding to the trajectory data, and the operation method of the virtual electronic musical instrument system further includes a step (D), a display device outputs the A visual signal, and the visual signal and the audio signal are displayed synchronously.

The above brief description of the present invention aims to provide a basic description of several aspects and technical features of the present invention. The brief description of the invention is not a detailed description of the invention. Therefore, its purpose is not to specifically enumerate the key or important elements of the invention, nor to define the scope of the invention. It merely presents several concepts of the invention in a concise manner.

In order to understand the technical features and practical effects of the present invention, and implement it in accordance with the content of the specification, the preferred embodiment shown in the figure is further described in detail as follows:

The present invention provides a virtual electronic musical instrument system and its operating method. The user can use sensors installed on musical instrument components (such as drum sticks, guitar picks or bows, etc.), or directly through sensors equipped with them Wearable devices (such as smart watches, smart bracelets or rings, etc.) to detect the user’s motion (gesture) trajectory when playing the instrument, then use algorithms to analyze the motion (gesture) trajectory and output the corresponding motion (gesture) trajectory Audio signal or/and visual signal to realize the experience of traditional musical instrument.

In addition, the virtual electronic musical instrument system of the present invention can also be combined with a virtual reality (Virtual Reality, VR) or augmented reality (Augmented Reality, AR) system to combine the visual signal with a virtual reality or augmented reality Ways to present virtual instruments (such as drum kits, guitars, various types of violin or piano, etc.) to increase the musical experience of the performer or audience.

First, please refer to the first A and the first B. The first A is a virtual electronic musical instrument system according to a preferred embodiment of the present invention, and the first B is a virtual electronic musical instrument system according to another preferred embodiment of the present invention. As shown in Figure 1A, the virtual electronic musical instrument system 10 includes at least one sensor 100, a system server 200 wired or wirelessly connected to the at least one sensor 100, and wired or wirelessly connected to the system server 200 The speaker 300 is connected, and the display device 400 is wired or wirelessly connected to the system server 200; wherein, the speaker 300 is not only an external speaker 300 connected to the system server 200, but also can be embedded in the system server 200 Embedded speaker 300, the present invention should not be limited to this.

The aforementioned at least one sensor 100 can be installed on musical instrument components such as drum sticks, guitar picks, or bows, and can also be configured in wearable devices such as smart watches, smart bracelets or rings. When the user directly equips them With the aforementioned wearable device, even musical instrument components (such as drum sticks, guitar picks, or bows and other trigger components) can be played without using them; furthermore, each sensor 100 is also configured correspondingly There is an electronic circuit 110 (refer to Figure 1B), and the electronic circuit 110 may include a transceiver unit (transceiver), a processing unit 111 (processor), a storage unit 112 (memory), a big data analysis unit 113, or/and System on a Chip (System on a Chip, SoC). The function of the at least one sensor 100 is to detect the gesture trajectory of the user or performer when playing a virtual musical instrument, or to detect the movement trajectory of the at least one musical instrument component when playing, so as to generate a trajectory data and send a message to an external device . Wherein, the trajectory data includes horizontal displacement data, vertical displacement data, horizontal angle data, vertical angle data, horizontal direction data, vertical direction data, horizontal velocity data, vertical velocity data, horizontal acceleration data, vertical acceleration data, or a combination thereof.

The sensor 100 may include a gyroscope, a magnetometer, an accelerometer, or a combination thereof, etc.; the sensor 100 may also be a nine-axis attitude sensor or an inertial measurement unit (IMU). The nine-axis attitude sensor has a three-axis gyroscope, a three-axis magnetometer, and a three-axis accelerometer. The inertial measurement unit is composed of three accelerometers and three gyroscopes. The purpose of the two is to detect the angle of the object. , Position, displacement, rotation and speed. The sensor 100 can detect the initial position, end position, relative position, displacement, three-axis rotation matrix, angle (direction angle or pitch angle), speed and acceleration data of the user's gesture or musical instrument component, so as to determine the use When the player is playing a musical instrument, his gestures or the trajectory of the musical instrument components.

The system server 200, which is wired or wirelessly connected to the at least one sensor 100, includes a processor 210, which can receive the trajectory data transmitted by each sensor 100 in real time, and analyze and identify it through the built-in algorithm of the system. The audio signal or visualization signal corresponding to the track data is output (played) in real time via the speaker 300 connected to the processor 210, or output (displayed) in real time via the display device 400 connected to the processor 210 The visual signal, and the visual signal and the audio signal are output synchronously, so as to improve the visual and auditory experience of the user. Furthermore, the aforementioned speaker 300 and display device 400 can be external speakers 300 and display device 400 that are wired or wirelessly connected to the system server (such as Bluetooth devices, Wi-Fi network devices, etc.), or can be internal The embedded speaker 300 and the embedded display device 400 embedded in the system server should not be limited to the present invention.

In addition, the system server also includes a database 220 connected to the processor 210, and the database 220 can store the historical track data, the audio signal, and the visualization signal. When the user wants to play historical audio signals or display historical visual signals, the data in the database 220 can be mobilized for output. Furthermore, the system server 200 also includes a large data analysis module 230, which is respectively connected to the database 220 and the processor 210, which can collect all the trajectory data in the database 220 in real time, and target each trajectory data in real time. Analyze and find out the passages (syllables) that are more prone to errors when the user is playing, such as the position of the guitar plucking or fingering, the position and strength of the drum kit or the position of the piano keys, etc., and display them on the display device 400 The gesture trajectory or movement trajectory required for the correct performance method allows the user not to have a physical instrument and can independently practice various types of music.

It is worth noting that, in addition to sending the detected trajectory data to the system server 200 for analysis and calculation, the sensor 100 can also send the trajectory data to the corresponding electronic circuit 110 included in the transceiver. A unit (transceiver), a processing unit 111 (processor), a storage unit 112 (memory) and a big data analysis unit 113, and the processing unit 111 is equivalent to the processor 210 in the aforementioned system server 200, which can analyze the trajectory using algorithms Data and output the audio signal or/and visualization signal corresponding to the trajectory data, and transmit the audio signal to the speaker 300 through the transceiver unit to output (play) the audio signal, and transmit the visualization signal to the display device 400 through the transceiver unit. Output (display) a visual signal; the storage unit 112 is equivalent to the database 220 in the aforementioned system server 200, and can store the historical track data, the audio signal, and the visual signal; the big data analysis unit 113 is equivalent to The big data analysis module 230 in the aforementioned system server 200 can collect all the trajectory data in the storage unit 112 in real time, analyze each trajectory data in real time, and find out the passages that are more prone to errors when the user is playing. In this way, the virtual electronic musical instrument system 10' of another embodiment can further eliminate the components of the system server 200, and only through the sensor configured on the user (the wearable device) or installed on at least one musical instrument component The device module (including at least one sensor 100 and the corresponding electronic circuit 110) can detect the user’s gesture trajectory or the movement trajectory of the at least one musical instrument component, and transmit it through the speaker according to the trajectories 300 outputs the corresponding audio signal, or/and outputs the corresponding visualization signal through the display device 400, and the visualization signal is output synchronously with the audio signal.

The above-mentioned system server 200 may be a personal computer (PC), a tablet computer, a set-top box (STB), a personal digital assistant (PDA), a mobile device, or any device capable of executing machine instructions for arithmetic processing. The speaker 300 can be a headset, a stereo, a speaker, or any device capable of playing audio signals. The display device 400 can be a general display device such as a projector, a screen, or a display panel, and can also be a virtual reality (Virtual Reality, VR) or augmented reality (Augmented Reality, AR) display device for displaying virtual reality signals Or to enhance the visual signal of the real signal.

For continuation, please refer to the second figure, which is a flowchart of the operation method of the virtual electronic musical instrument system according to the preferred embodiment of the present invention. As shown in the second figure, the operating method of the virtual electronic musical instrument system of the present invention includes the following steps: (A) At least one sensor 100 disposed on a user or installed on at least one musical instrument component detects the user’s Gesture trajectory, or the movement trajectory of the at least one musical instrument component, and generate trajectory data; (B) a system server 200 (or electronic circuit 110) analyzes the trajectory data with an algorithm and generates an audio corresponding to the trajectory data Signal; and (C) a speaker 300 outputs the audio signal. In step (B), the system server 200 can also generate a visualization signal corresponding to the trajectory data.

Furthermore, in the above-mentioned operation method flow, a step (D) is included. A display device 400 outputs the visual signal, and the visual signal and the audio signal are displayed synchronously. Wherein, the output mode of the visual signal includes a virtual reality (Virtual Reality, VR) or augmented reality (Augmented Reality, AR) signal.

Hereinafter, the calculation method of how to recognize the user's gesture trajectory or/and the movement trajectory of at least one musical instrument component and convert the trajectory data generated by the gesture trajectory or/and the movement trajectory into audio signals and visualization signals will be further explained in the present invention. .

First of all, please refer to the third, fourth and fifth figures at the same time. The third figure shows the at least one musical instrument component with the sensor installed in the preferred embodiment of the present invention. The fourth and fifth figures show the preferred embodiments of the present invention. Schematic diagram of the virtual drum kit. As shown in the third figure, in this embodiment at least one sensor 100A/100B is installed on a pair of drumsticks 500 that can produce audio or visual effects, and the pair of drumsticks includes a first drumstick 500A ( Hold with the left hand) and a second drumstick 500B (hold with the right hand), and the first drumstick 500A and the second drumstick 500B are respectively installed with at least one sensor 100A/100B, and the sensor 100A/ An electronic circuit 110A/110B corresponding to 100B is configured, and the electronic circuit may include (refer to the first B diagram at the same time) a transceiver unit (transceiver), a processing unit (processor), a storage unit (memory) or/and System on a Chip (System on a Chip, SoC). In this embodiment, the first drumstick 500A may be a parent drumstick, and the second drumstick 500B may be a child drumstick. The parent drumstick may receive data from the sensor 100B on the child drumstick, and The sensor data of the child drumstick and the sensor data of the parent drumstick itself (that is, the trajectory data) are sent to the system server 200 (such as computers, computers, and mobile devices) for further processing, or through After the electronic circuit 110A/110B analyzes and calculates, it directly sends the audio signal to the speaker 300 or directly sends the visualization signal to the display device 400; in other possible embodiments, the first drum stick 500A and the second drum stick 500B can be independent Ground sends corresponding sensor data, audio signals or visualization signals to the system server 200, the speaker 300, or the display device 400.

Further, the first drum stick 500A can be assigned to the user’s left hand holding, and the second drum stick 500B can be assigned to the user’s right hand holding; when the user uses the pair of drum sticks 500 to perform, the left hand holds The at least one sensor 100A on the first drum stick 500A held by it detects the movement trajectory of the first drum stick 500A to generate first trajectory data, and is configured to output the first audio signal or the first visualization signal; At least one sensor 100B on the held second drumstick 500B detects the movement trajectory of the second drumstick 500B to generate second trajectory data, and is configured to output a second audio signal or a second visualization signal. Specifically, the drumsticks can enable the virtual drum set 600 (refer to the fourth figure) to output different audio signals (drum sounds) based on the different pairs of tracks of the first drumstick 500A and the second drumstick 500B. For example, when The movement trajectory of the first drumstick 500A is percussion on the bass drum, and the first audio signal produced by it is the drum sound of the bass drum; when the movement trajectory of the second drumstick 500B is percussion on the high-pitched cymbal, the second audio signal produced by it is It is the drum sound of the high-pitched cymbal. By separately detecting the movement tracks of the first drumstick 500A and the second drumstick 500B, different audio signals are generated in coordination with each other to achieve the sound effect of simulating a real drum set.

Continuing, as shown in the fourth and fifth figures, the pair of drum sticks 500 (instrumental components) in the third figure in this embodiment are the triggering components of the virtual drum set 600 (instrument), and the virtual drum set 600 can be divided into The three planes include: a first plane 61, a second plane 62, and a third plane 63. Furthermore, the first plane 61 is a horizontal plane, and has four drumheads from left to right, namely a snare drum 610 and a second middle drum (Tom-Tom) 620. /630 and a Floor Tom head 640. The diameter of the snare head 610 is 12-14 inches, and the diameter of the middle drum head 620/630 is 12-16 inches. The diameter is 14-18 inches; the second plane 62 is a plane with an upward elevation angle W of about 30 to 90 degrees (preferably 30 to 40 degrees) relative to the first plane 61, and has two drumheads From left to right are the Crash Cymbal drumhead 650 and the Ride Cymbal drumhead 660. The diameter of the Crash Cymbal drumhead 650 is 16-20 inches. The diameter of the 660 is 20-24 inches; the third plane 63 is a plane that is downward and perpendicular to the first plane 61 relative to the first plane 61, and the third plane 63 has a Bass Drum drum surface 670. The diameter of the drum head 670 is 18-26 inches. In other possible embodiments, the number of drum heads in each of the above-mentioned planes and their placement positions can be adjusted according to the needs of the user. The present invention This should not be the limit.

Next, the method of judging the movement trajectory of the drumsticks when the user is playing will be explained. First of all, the most basic way of judging is that when the trajectory of the drumstick 500 held by the user hits the first plane 61 or the second plane 62 from top to bottom, the audio signal (that is, the drum sound) or/and the visual signal ( Corresponding drum head vibration), or the trajectory of the drumstick 500 hitting the third plane 63 from left to right or from right to left also outputs audio signals or/and visualization signals. It is worth noting that if the trajectory of the drumstick 500 hits any plane from bottom to top, no audio signal or/and visualization signal will be output; and when the left hand holds the first drumstick 500A, the trajectory is caused by The movement trajectory of hitting the third plane from right to left and holding the second drum stick 500B with the right hand is hitting the third plane from left to right, which represents the inertial return of the movement trajectory of normal hitting the third plane 63, so neither Output audio signal or/and visual signal.

In addition, whether the trajectory of the drumstick 500 hitting from top to bottom actually touches the first, second or third plane 61/62/63 is determined by the pitch angle of each plane; the drumstick 500 hits The timbre of the output audio signal after hitting is determined by the direction angle of each plane. The volume of the audio signal is determined by the linear acceleration or angular acceleration of the movement track of the drumstick 500. When the value of the linear acceleration or angular acceleration is greater (that is, the greater the percussive force of the user holding the drumstick 500) ), the volume of the audio signal is also greater; conversely, if the value of linear acceleration or angular acceleration is smaller (that is, the lower the percussion force of the user holding the drumstick 500), the volume of the audio signal is also lower.

Please refer to the sixth and seventh figures below. The sixth and seventh figures will further explain the definition of the trajectory data of the first plane, the second plane and the third plane.

First, as shown in the sixth figure, the timbre of the audio signal in the first plane 61 is determined by the direction angle. Specifically, the drum heads from left to right in the first plane 61 are a snare drum head 610, a second middle drum head 620/630, and a floor drum head 640. The audio signal is judged by the motion track. The horizontal range value (H) of the effective strike is determined. When the user holds the drumstick 500 and strikes the first plane 61, the horizontal direction is 0-45 degrees (that is, the horizontal range value H is 0-45) , The audio signal corresponding to the trajectory data is the audio signal of the snare drum 610, and the visual signal is the vibration of the snare drum; The range value H is 45-90), the audio signal corresponding to the trajectory data is the audio signal of the first middle drum 620, and the visual signal is the first middle drum drum head vibration; when the movement trajectory of the drum stick 500 hits the first middle drum The horizontal direction in the plane 61 is at an angle of 90-135 degrees (that is, the horizontal range value H is 90-135), then the audio signal corresponding to the trajectory data is the audio signal of the second middle drum, and the visual signal is the second middle drum 630 drum Surface vibration; when the trajectory of the drumstick 500 hits the first plane 61 at an angle of 135-180 degrees in the horizontal direction (that is, the horizontal range value H is 135-180), the audio signal corresponding to the trajectory data is the floor drum 640 The visual signal is the vibration of the floor drum surface. For example, when the trajectory data of the user's movement trajectory is hitting the first plane 61 and the horizontal range value H is 120, the corresponding output audio signal is the audio signal of the second middle drum 630, and the visualization signal is the second middle drum 630. The drum head vibration can be achieved through the above methods when the user taps on the same plane in different horizontal directions to output audio signals of different timbres, as well as the visual effects of different drum head vibrations, so as to achieve a real drum set musical experience.

On the other hand, whether it actually touches the drum surface of the first plane 61 to output audio signals is determined by the pitch angle. Specifically, when the trajectory of the drumstick 500 is the position where the user holds the drumstick 500, it is located above the first plane and the pitch angle (R) is greater than or equal to 5 degrees (preferably the pitch angle is between 10- 30 degrees), and when hitting D from top to bottom (toward the first plane 61) at an angle greater than or equal to 5 degrees (preferably between 10-30 degrees), the drum head 610/620 will be triggered A hit of /630/640 causes the system server 200 to output the corresponding audio signal or visual signal. Conversely, when the user holds the drum stick 500 with respect to the pitch angle above the first plane 61 is less than 5 degrees, the angle at which the drum stick strikes D from top to bottom is less than 5 degrees, and the drum stick 500 is relative to the first plane 61. The data generated by hitting U from bottom to top will not trigger the system server 200 to output audio signals and visualization signals.

Furthermore, when the processor 210 of the system server 200 receives the track data detected by the at least one sensor 100 that it actually touches the drum surface of the first plane 61, it can also send an actuation signal to the drum surface. The actuation unit (not shown) on the stick 500, or an actuation unit connected to the at least one sensor 100 (the actuation unit is also wired or wirelessly connected to the processor 210 to receive the actuation signal), wherein the actuation signal The drum stick 500 (instrument component) can be made to vibrate or a reaction force is provided in the opposite direction of the force applied to the drum head, and the drum stick vibration or reaction force can be realized through the actuation unit, so as to let the user know that it has been confirmed. Hit the drumhead of the first plane 61 and simulate a more realistic hitting effect. And this concept is not only applied to percussion instruments. For example, when the guitar picks are used to actually play the (simulated) strings, the vibration of the picks is caused, or when the bow is used to actually play the (simulated) strings, the vibrations of the bow are both Within the protection scope of the present invention.

Continuing, as shown in the seventh figure, the timbre of the audio signal in the second plane 62 is also determined by the direction angle. Specifically, the drum heads from left to right in the second plane 62 are a Crash Cymbal drum head 650 and a Ride Cymbal drum head 660 in sequence, and the audio signal is determined by the motion track. The horizontal range value (H) of the effective strike is determined. When the user holds the drumstick 500 and strikes the second plane 62, the horizontal direction is 0-90 degrees (that is, the horizontal range value H is 0-90) , The audio signal corresponding to the trajectory data is the audio signal of the broken sound cymbal 650, and the visual signal is the broken sound cymbal drum surface vibration; when the movement trajectory of the drumstick 500 hits the second plane 62, the horizontal direction is 90-180 degrees Angle (that is, the horizontal range value H is 90-180), the audio signal corresponding to the trajectory data is the audio signal of the double cymbal 660, and the visual signal is the drum surface vibration of the double cymbal. For example, when the trajectory data of the user's motion trajectory is hitting the second plane 62 and the horizontal range value H is 120, the corresponding output audio signal is the audio signal of the cymbal 660, and the visualization signal is the cymbal of the cymbal. Drum head vibration can be achieved through the above methods when the user taps on the same plane in different horizontal directions to output audio effects of different timbres, as well as visual effects of different drum head vibrations, so as to achieve a real drum set musical experience.

On the other hand, whether it actually touches the drum surface of the second plane 62 to output audio signals is determined by the pitch angle. Specifically, when the trajectory of the drumstick 500 is the position where the user holds the drumstick 500, it is located above the second plane 62 and the pitch angle (R) is greater than or equal to 30 degrees (preferably the pitch angle is between 30 degrees). -90 degrees), and when hitting D from top to bottom (to the second plane) at an angle greater than or equal to 30 degrees (preferably between 30-90 degrees), the drum head 650/660 will be triggered One strike of the system server 200 causes the system server 200 to output the corresponding audio signal or visualization signal. Conversely, when the user holds the drum stick 500 with respect to the pitch angle above the second plane 62 is less than 30 degrees, the angle of the drum stick hitting D from top to bottom is less than 30 degrees, and the drum stick 500 is relative to the second plane 62. The data generated by hitting U from bottom to top will not trigger the system server to output audio signals and visualization signals.

Furthermore, when the processor 210 of the system server 200 receives that the track data detected by the at least one sensor 100 is indeed touching the drum surface of the second (third) plane 62/63, it can also send an action The signal is sent to an actuation unit (not shown) installed on the drum stick 500, or an actuation unit connected to the at least one sensor 100 (the actuation unit is also wired or wirelessly connected to the processor 210 to receive the actuation signal ), where the actuation signal can be used to vibrate the drumstick 500 (instrumental component) or provide a reaction force in the opposite direction of the force applied to the drum head, and realize the drumstick vibration or reaction force through the actuation unit, thereby allowing The user is sure that he has hit the 62/63 head of the second (third) plane and simulates a more realistic hitting effect.

As for the above-mentioned first and second plane 61/62 effective strike horizontal range value (H), as well as the first and second plane 61/62 above the pitch angle (R) and other parameters, can be based on the user's Habits adjust by themselves, and the present invention should not be limited accordingly.

Finally, the third plane 63 is located below the first plane 61 and perpendicular to the first plane 61. When the user holds the first drumstick 500A (left hand), the pairing trajectory is swiping from left to right, or the second drumstick The movement track of 500B (right hand) is swiping from right to left, which can make the system server 200 output the corresponding audio signal (bass drum sound) or visual signal (bass drum surface vibration). Theoretically, if there are multiple drums located on the third plane 63, the pitch angle of the third plane 63 should be used to determine the position of the drum head; What is the pitch angle? Only the correct trajectory on the third plane 63 is required (the first drumstick 500A swipes from left to right or the second drumstick 500B swipes from right to left), which can make the system server 200 output audio signal and visualization signal.

The following will further explain the multiple application modes of the virtual electronic musical instrument system 10/10' of the present invention.

The first is the most basic free-strike mode. The user can hold a drumstick 500 equipped with at least one sensor 100, or directly wear a wearable device equipped with at least one sensor 100 to hit a virtual instrument (here Are the first, second and third planes 61/62/63 of the virtual drum set 600), and after receiving the trajectory data of the movement (gesture) trajectory of the performance process through the system server 200 or the electronic circuit 110, through the speaker 300 or The display device 400 outputs the corresponding audio signal or visualization signal.

In the accompaniment mode, the user can select a random or preset music melody through the system server 200 or the electronic circuit 110 in advance, and output the music melody through the speaker 300. At this time, the user can play along with the music melody. The user can also hold the drumstick 500 equipped with at least one sensor 100, or directly wear a wearable device equipped with at least one sensor 100 to hit the first part of the virtual instrument (here, the virtual drum set 600). , The second and third planes 61/62/63, and after receiving the trajectory data of the movement (gesture) trajectory of the performance process through the system server 200 or the electronic circuit 110, the corresponding audio signal is output through the speaker 300 or the display device 400 Or a visual signal, and the audio signal can be used to accompany the musical melody.

In the above-mentioned two modes, the big data analysis module 230 (or the big data analysis unit 113) can also be used to make real-time parameter settings and adjustments in response to situations that may occur during the user's performance. For example, when the user of the performance is a beginner or the performance is on the road (informal performance occasion), which leads to more errors in the performance, such as the sensor 100 The pitch angle of the motion (gesture) trajectory is too small to trigger the swipe and no audio signal and visual signal are generated, or the horizontal angle of the motion (gesture) trajectory of the sensor 100 is wrong, resulting in an audio signal of the wrong sound and Visualize the signal and other issues. The big data analysis module 230 (or the big data analysis unit 113) of the present invention can collect the information in real time and determine the appropriate parameters for the user. For example, if the user frequently exercises (gesture) trajectory If the pitch angle is too small and no audio signal is generated, you can adjust and lower the pitch angle setting threshold in the system server 200 in real time (for example, the pitch angle above the second horizontal plane 62 is reduced from 60 degrees to 30 degrees) to improve sensing The detection range of the device 100; if the user frequently hits the wrong drumhead due to the wrong horizontal angle and produces the wrong sound, he can adjust and enlarge the area of each drumhead or a specific drumhead in real time (such as floor drum head 640 The diameter is enlarged from 14 inches to 18 inches) to reduce the chance of users hitting errors.

The visualization signal in all the above embodiments can be realized by a display device 400 (such as VR glasses) with virtual reality (VR) or augmented reality (AR) effects. The user can wear virtual reality or augmented reality glasses to present the real drum set. When the movement trajectory generated by the held drumstick 500 is detected as hitting a certain drum head in the drum set, the The display device 400 receives the corresponding visualization signal, and causes the beaten drum head in the display screen to produce a visualization effect of vibration. At the same time, the speaker 300 synchronously outputs an audio effect corresponding to the visualization signal, thereby realizing a real drum hit. Group experience.

The at least one sensor 100 described above is installed on the drumstick 500 and detects the movement track of the drumstick when the user is playing. However, in other possible embodiments, the at least one sensor 100 can also be directly configured On a user, such as a wearable device such as a smart watch, smart bracelet or ring, and detect the user's gesture trajectory to generate trajectory data, and output audio signals or visual signals based on the trajectory data; therefore, there is something about the above The description of "motion trajectory" can be replaced with "gesture trajectory". In addition, the virtual electronic musical instrument system 10/10' of the present invention not only receives track data through the system server 200 and outputs corresponding audio signals or visual signals, but also directly through the electronic circuit configured corresponding to the sensor 100 110 (such as the embodiment in Figure 1B) receives the trajectory data and outputs the corresponding audio signal or visualization signal, so as to further save the components of the system server 200; therefore, the description of the above "system server 200" is also Can be replaced with "Electronic Circuit 100".

In addition, although this embodiment only gives a detailed description of the drum kit and its corresponding trigger sound drum sticks (instrumental components), the virtual instrument can also be a guitar, piano, or various types of violin, and its corresponding trigger sound The musical instrument component can also be a guitar pick or bow. The sensor of the present invention can also detect the movement track of the guitar pick or finger when the user plays the guitar, and output the corresponding audio signal (guitar sound) and visualization Signal (guitar string vibration); or detect the movement of the finger when the user is playing the piano, and output the corresponding audio signal (piano sound) and visual signal (vertical displacement of the key); or detect the user playing the violin When the movement track of the bow and fingers, the corresponding audio signal (violin sound) and visual signal (violin string vibration) are output. Therefore, any detection of the user's gesture trajectory or the movement trajectory of a musical instrument component through a sensor, and outputting the corresponding audio signal or visual signal belongs to the scope of protection of the present invention, that is, according to the scope and description of the patent application of the present invention The replacement of musical instruments made in the content is still within the scope of the present invention.

However, the above are only the preferred embodiments of the present invention, and should not be used to limit the scope of implementation of the present invention, that is, simple changes and modifications made in accordance with the scope of the patent application and the description of the present invention still belong to the present invention. Covered in the scope.

10: Virtual electronic musical instrument system 10’: Virtual Electronic Musical Instrument System 100: sensor 100A: the first sensor 100B: second sensor 110: Electronic Circuit 111: Processing Unit 112: storage unit 113: Big Data Analysis Unit 200: system server 210: processor 220: database 230: Big Data Analysis Module 300: speaker 400: display device 500: Drumsticks 500A: The first drum stick 500B: second drum stick 600: drum kit 610: Snare Drum Head 620: The first middle drum head 630: First Middle Drum Head 640: Floor Drum Head 650: Broken Sound Cymbal Drum Head 660: Stacked Cymbal Drum Head 670: Bass Drum Head 61: First plane 62: second plane 63: third plane W: pitch angle between the first plane and the second plane D: Strike from top to bottom U: Hit from bottom to top (A)-(D): Steps

The first A is a virtual electronic musical instrument system according to a preferred embodiment of the present invention.

The first figure B is a virtual electronic musical instrument system according to another preferred embodiment of the present invention.

The second figure is a flowchart of the operation method of the virtual electronic musical instrument system according to the preferred embodiment of the present invention.

The third figure shows at least one musical instrument component with a sensor installed in a preferred embodiment of the present invention.

The fourth figure is a schematic diagram of a virtual drum set according to a preferred embodiment of the present invention.

The fifth figure is a schematic diagram of the operation method of the virtual drum set according to the preferred embodiment of the present invention.

The sixth figure is a schematic diagram of the first plane trajectory data definition according to the preferred embodiment of the present invention.

The seventh figure is a schematic diagram of the second plane trajectory data definition according to the preferred embodiment of the present invention.

10: Virtual electronic musical instrument system

100: sensor

200: system server

210: processor

220: database

230: Big Data Analysis Module

300: speaker

400: display device

Claims (12)

A virtual electronic musical instrument system includes: at least one sensor configured to detect a user or at least one musical instrument element in one of a predefined first plane, a second plane, and/or a third plane based on The trajectory generates a trajectory data, the first plane, the second plane, and the third plane are virtual planes; a system server is configured to receive the trajectory data and includes a processor and a speaker, the The processor is connected to the speaker for analyzing the trajectory data according to an algorithm; and generating and outputting an audio signal to the speaker for playback; wherein, based on the first plane, the second plane is located on the The position on the first plane with an elevation angle of 30 to 90 degrees, and the third plane is a position with a downward angle of 90 degrees on the first plane; wherein the trajectory is the user's gesture trajectory or the at least one musical instrument The trajectory of the component. The virtual electronic musical instrument system according to claim 1, wherein the processor can also generate a visualization signal corresponding to the trajectory data. The virtual electronic musical instrument system described in claim 2 further includes a display device connected to the system server, the display device outputs the visualization signal, and the visualization signal and the audio signal are displayed synchronously. The virtual electronic musical instrument system according to claim 2, wherein the system server further includes a database connected to the processor, and the database stores the trajectory data, the audio signal, and the visualization signal. The virtual electronic musical instrument system according to claim 1, wherein the at least one sensor includes an inertial measurement unit (IMU). The virtual electronic musical instrument system according to claim 1, wherein the at least one sensor includes a gyroscope, a magnetometer, an accelerometer, or a combination thereof. The virtual electronic musical instrument system according to claim 3, wherein the visualization signal is a Virtual Reality (VR) signal or an Augmented Reality (AR) signal. The virtual electronic musical instrument system according to claim 1, wherein the trajectory data includes horizontal displacement data, vertical displacement data, horizontal angle data, vertical angle data, horizontal direction data, vertical direction data, horizontal speed data, vertical speed data, horizontal Acceleration data, vertical acceleration data, or a combination thereof. An operating method of a virtual electronic musical instrument system, comprising: (A) providing the virtual electronic musical instrument system as described in claim 1; (B) the system server receiving the at least one provided by the user or the at least one musical instrument component A sensor generates the trajectory data based on detecting the trajectory of the user or the at least one musical instrument component on the first plane, the second plane and/or the third plane; (C) the processor is based on the calculation Method to analyze and identify the audio signal corresponding to the track data; and (D) play the audio signal through the speaker. The operating method of the virtual electronic musical instrument system according to claim 9, wherein in step (B), a visualization signal corresponding to the trajectory data can also be generated. The operating method of the virtual electronic musical instrument system according to claim 10 further includes a step (E): a display device outputs the visualization signal, and the visualization signal and the audio signal are displayed in synchronization. The operating method of the virtual electronic musical instrument system according to claim 11, wherein the output mode of the visual signal includes virtual reality (VR) or augmented reality (AR).

Images