WO2022237362A1

WO2022237362A1 - Method for detecting user action on basis of music beats, and device

Info

Publication number: WO2022237362A1
Application number: PCT/CN2022/083752
Authority: WO
Inventors: 肖志
Original assignee: 华为技术有限公司
Priority date: 2021-05-14
Filing date: 2022-03-29
Publication date: 2022-11-17
Also published as: CN115346553A

Abstract

A method for detecting a user action on the basis of music beats, and a wearable device, a terminal device and a computer storage medium, which relate to the technical field of wearable devices, and can detect whether a user keeps up with beats. The method comprises: determining beats corresponding to a target rhythm to be detected and a first unit time occupied by a beat (201); acquiring a target feature that corresponds to each beat in the target rhythm, wherein the target feature comprises a first feature, and the first feature is used for representing corresponding time information of when a user performs an action within the first unit time (202); and on the basis of a beat category corresponding to each beat, comparing each target feature with a predetermined reference feature, so as to obtain detection data (203).

Description

Method and device for detecting user actions based on music tempo

technical field

The present application relates to the field of wearable devices, in particular to a method and device for detecting user actions based on music beats.

Background technique

At present, in many scenes such as music learning, musical instrument performance, dance training, etc., it is necessary to cultivate the user's sense of rhythm. The electronic metronome can generate vibrations of corresponding intensity according to the strength of the beat, so that the user can feel when wearing the electronic metronome. The strength of the beat changes regularly to assist in training the user's sense of rhythm. However, during the training process, the actual actions made by the user may not keep up with the beat accurately, such as snap shots, drop shots, etc., or the strength of the retake is weak, and the strength of the weak beat is relatively strong. The correction depends on the user's active perception. In most cases, the user may not be able to actively realize the existence of the problem, and will not correct it in a targeted manner. The training effect of the sense of rhythm needs to be improved. How to detect whether the user keeps up with the beat has become an urgent problem to be solved.

Contents of the invention

The present application provides a method and device for detecting a user's action based on a music beat, which can detect whether the user follows the beat.

In the first aspect, this technical solution provides a method for detecting user actions based on music beats, including: determining the beat corresponding to the target rhythm to be detected and the first unit time value occupied by one beat; target feature, the target feature includes the first feature, the first feature is used to represent the corresponding time information when the user makes an action within the first unit time value; based on the beat category corresponding to each beat, each target feature and the predetermined reference feature Compare to obtain detection data; wherein, the reference features include time reference intervals corresponding to each beat category; the detection data is used to at least give feedback to the user whether they keep up with the beat in the time dimension.

In a possible implementation manner, the first feature includes a target time interval; the target time interval is used to represent the difference between the first moment specified by the user in the process of making an action within the first unit time value and the first starting moment. The time difference between, wherein, the first starting moment is the starting moment of the first unit time value.

In a possible implementation manner, the first moment is determined based on the following method: within the first unit time value, any moment when the acceleration is detected as zero is taken as the first moment.

In a possible implementation manner, the first moment is determined based on the following method: within the first unit time value, detect the duration from the start of the vibration to the end of the vibration as the first duration, and use any specified value in the first duration A moment as the first moment.

In a possible implementation manner, the first moment is determined based on the following method: within the first unit time value, detect at least one moment when the acceleration is zero, and detect the duration from the start of the vibration to the end of the vibration as the first duration , taking the moment when the acceleration closest to the first duration or falling within the first duration is zero as the first moment.

In a possible implementation manner, before determining the beat corresponding to the target rhythm to be detected and the first unit time value occupied by one beat, it also includes: determining the user's personal reference features.

In a possible implementation manner, determining the user-specific reference features includes: obtaining the single time intervals corresponding to multiple actions of any beat category performed by the user, and determining any The time reference interval corresponding to the beat category is used as a user-specific reference feature; among them, the single time interval is determined based on the following method: determine the second moment specified in the process of the user making a single action, and compare the second moment with the first The time difference between the two starting moments is taken as a single time interval; wherein, the second starting moment is the starting moment of collecting a single action.

In a possible implementation manner, based on multiple single time intervals, determining the time reference interval corresponding to any beat category includes: calculating the average value of multiple single time intervals; based on the average value, calculating multiple single time intervals The mean square error corresponding to the time interval; based on the mean square error and the average value, determine the time reference interval corresponding to any beat category.

In a possible implementation, based on the mean square error and the average value, determine the time reference interval corresponding to any beat category, including: the interval obtained by adding or subtracting N times the mean square error from the average value, as the time reference interval, 1≤ N≤3.

In a possible implementation manner, the target feature also includes a second feature, and the second feature is used to characterize the corresponding force information when the user makes an action within the first unit time; the reference feature also includes the corresponding Strength reference interval; based on the beat category corresponding to each beat, compare each target feature with a predetermined reference feature, and also include: based on the beat category corresponding to each beat, compare the second feature with the strength reference interval corresponding to the corresponding beat category ; The detection data is also used to feed back to the user whether the action strength of each beat conforms to the strength of the beat.

In a possible implementation manner, before obtaining the target features respectively corresponding to each beat in the target rhythm, it further includes: determining a user-specific dynamic reference interval.

In a possible implementation manner, determining the user-specific strength reference interval includes: obtaining the strength information corresponding to multiple actions of any type of beat by the user, and determining the corresponding strength information of any beat category based on multiple pieces of strength information. The strength reference interval of is used as the user's personal strength reference interval; wherein, the strength information includes the corresponding acceleration information when the user makes an action of any beat type.

In a possible implementation manner, before acquiring the target features corresponding to each beat in the target rhythm, it also includes: acquiring the target speed when the user makes an action; comparing the target speed with a predetermined speed threshold to determine the target When the value of the speed is lower than the speed threshold, the step of acquiring the target features corresponding to each beat in the target rhythm is executed.

In a possible implementation manner, the target velocity includes a linear velocity or an angular velocity.

In a possible implementation manner, before comparing each target feature with a predetermined reference feature based on the beat category corresponding to each beat, it further includes: reading the user's personal reference feature according to the predetermined first path; When the feature reading fails, read the spare second reference feature according to the predetermined second path; the second reference feature is a pre-stored reference feature applicable to each user.

In a possible implementation manner, after determining the beat corresponding to the target rhythm to be detected and the first unit time value occupied by one beat, it also includes: according to the strength rule corresponding to the beat, within each first unit time value, respectively Prompting with corresponding intensity, the mode of prompting includes at least one of vibration, sound playing and light emitting.

In a possible implementation manner, after obtaining the detection data, it further includes: according to the detection data, at least giving feedback to the user whether they keep up with the beat in the time dimension, and the feedback method includes at least one of display, vibration, sound playback, and light emission.

In a possible implementation manner, the beat subcategories include at least a strong beat, a second strong beat and an upbeat beat.

In a second aspect, the technical solution also provides a wearable device, including: a display screen; at least one sensor; one or more processors; a memory; at least one application program; and one or more computer programs, wherein one or more One or more computer programs are stored in the memory, and one or more computer programs include instructions. When the instructions are executed by the device, the device performs the following steps: determine the beat corresponding to the target rhythm to be detected and the first unit time value occupied by one beat. ; Obtain the target feature corresponding to each beat in the target rhythm, the target feature includes the first feature, and the first feature is used to represent the corresponding time information when the user makes an action within the first unit time value; based on the beat category corresponding to each beat , comparing each target feature with a predetermined reference feature to obtain detection data; wherein, the reference feature includes a time reference interval corresponding to each beat category; the detection data is used to at least give feedback to the user whether to keep up with the beat in the time dimension.

In the third aspect, the technical solution also provides a terminal device, including: a display screen; at least one sensor; one or more processors; memory; at least one application program; and one or more computer programs, wherein one or more The computer program is stored in the memory, and one or more computer programs include instructions. When the instructions are executed by the terminal device, the terminal device performs the following steps: determine the beat corresponding to the target rhythm to be detected and the first unit occupied by one beat value; obtain the target features corresponding to each beat in the target rhythm, the target features include the first feature, and the first feature is used to represent the corresponding time information when the user makes an action within the first unit time value; based on the beats corresponding to each beat category, each target feature is compared with a predetermined reference feature to obtain detection data; wherein, the reference feature includes the time reference interval corresponding to each beat category; the detection data is used to at least give feedback to the user whether to keep up with the beat in the time dimension.

In a fourth aspect, the technical solution further provides a computer storage medium, including computer instructions, which, when the computer instructions are run on an electronic device, cause the electronic device to execute the method described in any one of the above first aspects.

The method and device provided in the embodiments of the present application detect whether the user is accurately following the beat by detecting the target feature of each beat in the target rhythm and comparing it with the reference feature, and then feed back the wrong beat point to the user, so that the user has a targeted It can be adjusted systematically, which improves the efficiency of training users' sense of rhythm.

Description of drawings

FIG. 1 is a schematic diagram of an example application scenario of a wearable device provided by an embodiment of the present application;

FIG. 2 is a schematic flowchart of an embodiment of a method for detecting user actions based on music beats provided in the embodiment of the present application;

Fig. 3 is a schematic diagram of the trajectory of a drumming action in the scene example provided by the embodiment of the present application;

Fig. 4 is the example of a group of target rhythms of the embodiment of the present application;

FIG. 5 is a schematic flow diagram of another embodiment of the method for detecting user actions based on music tempo provided by the embodiment of the present application;

Fig. 6 is a schematic diagram of a set of display interface examples of a wearable device provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a group of display interface examples of a terminal device provided by an embodiment of the present application;

Fig. 8 is a schematic diagram of another display interface provided by the embodiment of the present application;

Fig. 9 is a schematic diagram of comparing the target time interval with the time reference interval in the method provided by the embodiment of the present application;

FIG. 10 is a schematic diagram of an example interface for displaying a feedback report provided by an embodiment of the present application;

Fig. 11 is a schematic diagram of the hardware structure of a wearable device in an embodiment provided by the embodiment of the present application;

Fig. 12 is a schematic diagram of a hardware structure of a wearable device in another embodiment provided by the embodiment of the present application.

Detailed ways

The technical solution in this application will be described below with reference to the accompanying drawings.

In order to facilitate understanding of the method or device for detecting a user's performance action based on the music tempo provided by the embodiment of the present application, a number of technical terms involved in the embodiment of the present application are firstly introduced.

Beat and rhythm are an important part of a complete piece of music. The cultivation of rhythm is the basis of music learning, which relies on brain inertia and muscle memory after a lot of practice. Among them, the beat is the combination of the fixed unit time value and the law of strength and weakness in the music, and it is the unit to measure the rhythm. Beat training is the basis for cultivating the sense of rhythm. The meaning of rhythm includes broad meaning and narrow meaning. The broad meaning of rhythm is a kind of regular change that goes hand in hand with rhythm in nature, society and human activities. Various changing factors are organized in the form of repetition and correspondence. , forming a coherent and orderly whole, and the narrow meaning of rhythm specifically refers to the rhythmic characteristics of music. In the embodiment of the present application, rhythm can be understood as a combination of beats.

As more and more people learn music, it is necessary to have a suitable metronome to cultivate the user's sense of rhythm in the early stage of learning. A metronome is a mechanical, electric or electronic device that can send out a stable beat in various performance speeds. The metronome in the related art can be divided into a mechanical metronome and an electronic metronome. The electronic metronome can be installed on An APP with a metronome function on a terminal device, such as a metronome APP installed on a mobile phone. It should be noted that, in the embodiment of this application, the performance speed refers to beats per minute, Beats Per Minute, BPM.

The mechanical metronome is made according to the principle of the pendulum, with a built-in mainspring drive. The upstream code of the pendulum can be adjusted up and down to adjust the speed, and compared with the scale on the panel, the number of swings back and forth per minute can be known. There are many types of mechanical metronomes. For example, a metronome has a pyramid shape, a clock structure inside, gears and springs, and drives a pendulum. The pendulum makes a sharp "dida" at the end of each swing. The speed of these ticking sounds can be adjusted by moving the pendulum up and down according to the vernier scale engraved on the pendulum, and the speed is 40-210 beats per minute.

The metronome APP is an application program with the function of an electronic metronome. It is originally a beat tool for music lovers, which can truly simulate the beat track of a metronome. Define the beat type as 2/4 beat, 3/4 beat, 3/8 beat, etc. This metronome APP supports users to select the performance speed by setting the tempo, so as to realize the rapid positioning of the performance tempo (that is, the number of beats per minute BPM).

An electronic metronome in the related art is a wearable music device, which can be worn on the wrist of the human body and other parts, and supports the user to customize the performance speed (BPM) and the time signature of the beat through the touch display screen. The set beat time signature and speed will give corresponding strong and weak regular vibration prompts.

Common beat time signatures are 1/4, 2/4, 3/4, 4/4, 3/8, 6/8, 7/8, 9/8, 12/8, etc., the length of each measure It is fixed, where the denominator indicates the note type, that is, which note is taken as a beat, and the numerator indicates how many beats are in each measure. For example, a denominator of 4 means a quarter note is a beat, and a denominator of 8 means an eighth note. One beat, 2/4 beat means that a quarter note is a beat, and each measure has 2 beats. The types of notes also include whole notes, half notes, sixteenth notes, rests, and the like. Among them, "beat" is the unit for timing at a fixed frequency, and the performance speed BPM is the number of beats per minute. When the BPM is determined, the corresponding unit time value of one beat is determined. The unit time value, that is, the duration of a "beat", will not change with the note type. For example, when the playing speed is determined to be 120BPM, the note types in four-three beats or eight-three beats are different, but one The duration of shooting is 0.5s.

The method or device for detecting the user's performance based on the music tempo provided by the embodiment of the present application can be applied to various scenes that require the cultivation of a sense of rhythm, such as musical instrument performance or learning, band performance, square dance, dance performance, dance practice, military training, etc. The musical instrument playing learning or practicing scenes include playing learning or playing scenes of various musical instruments such as percussion instruments, stringed instruments, and keyboard instruments. For example, as one of the typical application scenarios, as shown in FIG. 1 , in the drumming scenario, the electronic metronome 10 provided by the embodiment of the present application is worn on the user's wrist.

In the military training scene, the uniform step, forward step and running, as well as the footsteps such as stepping and stepping in the marching of the army have a certain sense of rhythm. One step can be regarded as one beat. 1, 2, 1" have obvious rules of strength and weakness. Generally, the slogan "1" is an accent, while the slogan "2" is a weak sound, showing the law of strong, weak, strong and weak, which is equivalent to the 2 in the music beat. /4 beats, the uniformity of the army's movements can be achieved by cultivating the sense of rhythm of all members. The method or equipment provided in the embodiments of the present application can assist in military training. For example, each member wears the equipment provided in the embodiments of the present application. The device synchronously gives strong and weak prompts (such as vibration, light or display), and detects the actions of each member, and provides personal feedback to the instructor, officer or soldier whether the member is keeping up with the beat detection data, and adjusts the wrong beat point in a targeted manner. It can assist in the detection of wrong actions of members who cannot be detected only by the instructor's vision or hearing, and improve the efficiency of military training.

In the square dance scene, the beat of the corresponding music is generally fixed, for example, the waltz is mostly in 3/4 time, and each member of the dance team can adjust the beat to the corresponding music by wearing the equipment provided by the embodiment of the present application. Beat, to accurately perceive the correct dance beat, and use the device to detect whether the dance movements of each member keep up with the beat, and then feedback the detection results to the user. The detection results can be communicated to customers through voice broadcasts, display screens, and lighting. Let the user know at which tempo point a drop or snap shot occurs, and develop a sense of rhythm faster.

In the scene of musical instrument performance or practice, especially in the scene of band concerto, the device provided by the embodiment of the present application is used to synchronously detect whether the actions of each member keep up with the beat, and specifically prompt the user if there is a rush or drop in the performance action The beat points to cultivate the members' sense of rhythm.

The device for detecting the user's performance based on the music beat provided by the embodiment of the present application may be a wearable device, for example, it may be a wearable electronic metronome, which can be worn on the user as an independent wearable device For example, the electronic metronome can be a head-mounted device, a wristband device, a smart bracelet, a smart ankle ring, electronic glasses, an electronic collar, etc., or it can be added to an existing wearable device as an additional function In terms of products, for example, a smart watch that supports video calls can also integrate hardware and software for implementing the method provided by the embodiment of the present application, so that while supporting video and audio calls, it also has the function of a metronome.

It should be noted that the device provided in the embodiment of the present application may also be a terminal device installed with a Metronome APP, and based on the terminal device, the application program can implement the method provided in the embodiment of the present application. Optionally, the metronome APP installed on the terminal device can also communicate with the physical product of the electronic metronome as a wearable device, so as to realize the synchronization between the physical product and the APP. For example, the electronic metronome is a smart bracelet, and the user who holds the smart bracelet can install the metronome APP corresponding to the smart bracelet product on the mobile phone, and the metronome APP and the smart bracelet are connected via Bluetooth (bluetooth), near field communication (Near Field Communication, NFC), wifi and other short-distance wireless communication methods.

The metronome in the related art focuses on conveying accurate music beats to the user. When the user is playing or playing, the electronic metronome can be worn on the arm, ankle, wrist, etc. according to the user's wishes, so that the user can perceive the rhythm of the beat. . The vibration of the metronome corresponds to the BPM. For example, under the following combinations of different notes, beats, and BPM, the corresponding vibration conditions are shown in Table 1 below.

Table 1

Different from the electronic metronome in the related art, in the embodiment of the present application, the electronic metronome provided can also realize the detection of the user's action while conveying the accurate music tempo to the user. The electronic metronome provided by the embodiment of the present application determines the wearing position of the electronic metronome on the principle of making it easier to detect relevant actions of the user in various application scenarios. For example, in the application scenario of percussion music, in most cases, the user uses the wrist to drive the percussion components (such as drumsticks, etc.) or directly beats the instrument directly with the hand, so in this application scenario, the embodiment of the present application provides The electronic metronome can be worn on the user's wrist to better sense wrist movements. In other scenarios, such as military training scenarios, the electronic metronome provided by the embodiment of the present application can be worn on the user's ankle to more accurately perceive the user's leg or foot movements in various footwork training.

It should be noted that in different application scenarios, the electronic metronome provided by the embodiment of the present application can also be equipped with an expansion device, which is used to tie the wearable device to other parts of the user's body, for example, to improve the versatility of the product , the electronic metronome can be realized in the product form of a unified smart bracelet, and when the user needs to wear the electronic metronome on other parts, he can realize the firm wearing of other parts by expanding the device, for example, it can be attached with an armband or headband, ankle Bands, etc., to replace the wrist strap or detachably connect with the wrist strap.

For example, in the scene of band performance or square dance learning, the electronic metronome provided by the embodiment of the present application can be worn on the user's leg or arm by extending the device to more accurately perceive the leg or arm movement.

In another embodiment, the expansion device may also include auxiliary sensors and other devices for assisting in collecting user actions. For example, in a piano performance scene, most actions are fingers hitting keys on the piano, and only the movement of the wrist is sensed. The action error will be relatively large, so in this scenario, an accessory can be added to the electronic metronome main device. The accessory can be an additional sensor to sense the movement of each finger, for example, it can specifically include a vibration sensor and/or acceleration The patch of the sensor (attached to the finger by magnetic attraction or other means) or the ring, etc. In the playing scene of a stringed instrument, the performance action is mainly fingers plucking the strings, and an expansion device can also be installed on the finger to sense the finger movement, so as to assist the main device worn on the wrist to detect the user's movement.

The method and device for detecting a user's performance action based on the music tempo provided by the embodiments of the present application are respectively described below.

Specifically, the following is an example of a method for detecting a user's performance based on the music tempo provided by this application. For ease of understanding, the various steps in this example are described in chronological order, but this chronological order is only one of them. The feasible chronological order cannot be regarded as a necessary limitation for the method provided in the embodiment of the present application. Specifically, as shown in Figure 2, the method may include the following steps:

Step 201, determine the beat corresponding to the target rhythm to be detected and the first unit time value occupied by one beat.

In a practical application scenario, before using the device provided by the embodiment of the application for performance training or other activities, the user can at least perform basic settings on the two parameters of tempo and performance speed (BPM), and can also set performance duration, number of bars, etc. parameters, as well as setting the playing track, etc. Determination of the tempo and the first unit time value may be determined based on user input. That is, through the design of the UX (User Experience) layer, the user can set and change the tempo and BPM. Once the BPM is determined, the first unit time value is determined.

A section of rhythm that the user is about to play or has already played can be used as the detection object of the detection method of the present application, that is, the target rhythm. In this embodiment of the application, rhythm refers to a combination of beats. In a practicable manner, mixed beats, compound beats, transformed beats, etc. appear in the target rhythm, for example, the mixed beats in which 3/4 and 2/4 are combined to form 5/4.

Wherein, determining the beat includes determining the time signature or type of the beat, and the time signature includes the time signatures of various commonly used music beats, such as 1/4, 2/4, 3/4, 4/4, 3/8, 6/ 8, etc., the user can realize the input of the time signature by selecting the number of the numerator and denominator of the time signature. In some embodiments, the electronic metronome can also support the input of beats of a custom type, for example, multiple unconventional candidates for beats of a custom type can be prestored in the electronic metronome, and the strong The combination of weak patterns can be more complex or diverse, such as strong, weak, second-strongest, second-strongest, weaker, weaker, etc.; this custom type can also support user-defined, and then create user-specific original beat types.

Determine the first unit time value corresponding to one beat, that is, determine the performance speed, that is, determine the beat number BPM per minute. After the BPM is determined, the unit time value corresponding to one beat is determined. For example, when the performance speed is determined to be 100BPM, the unit time value occupied by one beat is 0.6s. For clarity of description, in the embodiment of the present application, the unit time value corresponding to the target rhythm to be detected is defined as the first unit time value.

In an achievable manner, the unit time value can be used as the detection time unit, and the time length corresponding to the target rhythm to be detected can be divided into multiple unit time values on average, and the user's actions within the unit time value can be detected respectively. Action target feature data.

It should be noted that, in the embodiment of this application, whether the user keeps up with the beat or whether the beat is accurate includes whether the user is stuck in the time dimension and whether the strength meets the strength requirements of the beat in the strength dimension. Specifically, whether the user keeps up with the beat The beat evaluation factors may include at least one of the following situations: snapping, missing beat, insufficient retake, too strong weak beat, too strong or too weak sub-beat. Among them, the downbeat and downbeat are evaluated from the time dimension, and whether the strength is appropriate is judged from another dimension. When at least one of the above situations occurs in the user, it is considered that the user fails to keep up with the beat.

In an achievable manner, after the beat and the unit time value occupied by one beat are determined, during the user's performance, the corresponding intensity can be prompted in each unit time value according to the strength law corresponding to the beat, The prompting method includes at least one of vibration, sound playing and light emitting. For example, when the user selects the beat as 2/4 beat, then according to the law of strength corresponding to 2/4 beat, strong, weak, strong and weak vibrations are performed, or different colors of light are emitted during strong beats and weak beats, for example , the strong beat emits red light, the weak beat emits green light, the next strong beat emits yellow light, etc.

It should be noted that various tempo parameters and device operating parameters are preset in the device provided in the embodiment of the present application. For example, the tempo parameter is at least the combination rule of strong and weak sounds corresponding to the beat, and the device operating parameters may include vibrations corresponding to strong and weak sounds. Intensity parameters, strong and weak sounds, sound types, light-emitting colors, etc., such as whether the played sound is electronic or human voice, etc., the light-emitting color and light-emitting mode are flashing, etc. These device operating parameters can also be set by the user.

Step 202, acquiring target features corresponding to each beat in the target rhythm.

Acquiring target features within each first unit time value means collecting feature data of a single action made by the user within each first unit time value. Whether the user drops or grabs a shot is determined by the time when the user makes a single action (that is, the action corresponding to one shot). Therefore, in the time dimension, the feature data to be collected may include the time information of the user's action ( The first feature) to judge whether the single action made by the user is too early or too late in time. For example, the time of the user's action is too late for a dropped shot, which is later than the correct action time. The snap shot is due to the action time Too early, earlier than the correct action time.

In a practicable manner, the time dimension data to be detected may be the time interval between a moment specified by the user during a single action and the start moment (first start moment) of the first unit time value . The time corresponding to the process of the user making a single action is an interval, which is composed of multiple time points. One of the time points is designated as the first moment, and the time difference between the first moment and the starting moment of the unit time value It is the time information of the action corresponding to the beat. Wherein, in theory, any moment in the action process can be used as the designated first moment, but in the current technical implementation, only detectable time points can be used as the designated first moment.

Specifically, the first moment is detected in the following ways:

As one of the implementable ways, within a unit time value, the moment when the acceleration is zero is detected as the first moment. For example, referring to Fig. 3, in the drumming scene, the motion track of a drumming action can be approximated as shown in Fig. 3 as shown in dotted line a→b→c, wherein, point a is when the user holds the drumstick and starts to move downward The starting point of the action, point b is the point where the drumstick touches the drum surface, and point c is the highest point reached by the user lifting the drumstick. In the process of this trajectory, during the process from point a to point b, the user holds the drumstick and exerts downward force. In addition to the gravity, the drumstick also receives the downward force from the user. After reaching point b, the drumstick contacts the drum surface , the drum surface will give the drumstick a rebound force, which will offset the gravity and the user's force, so that there is a moment when the acceleration of the drumstick and the wrist is 0, and the acceleration sensor can be used to collect the moment when the acceleration is 0, as Represents the first moment in time when the action occurred. It should be noted that, in a drumming work, during the process from point b to point c, the user will give the drumstick an upward force, and at a certain moment, the user's upward force will offset the gravity, In the trajectory of b→c, there will be a moment when the acceleration is also 0, and the moment when the acceleration is 0 can also be used as the designated first moment.

As another practicable manner, within a unit time value, detect the duration from the start of the vibration to the end of the vibration as the first duration, and use any specified time in the first duration as the first moment. For example, the start time, midpoint time or end time of the first duration can be used as the designated first time. For example, as shown in Figure 3, in the drumming scene, when the drumstick comes into contact with the drum surface, the drumstick will drive the wrist to vibrate due to the rebound force of the drum surface. At this time, the vibration sensor can detect that the drumstick drives the wrist. The generated vibration is regarded as the moment when the vibration starts, until the end of the vibration is detected, the moment when the vibration ends is subtracted from the moment when the vibration starts, that is, the duration t ₀ of the vibration process is determined, and the starting moment of the time interval t ₀ for which the vibration process lasts , the end time or the midpoint time can be used as the designated first time. It should be noted that point b represents the spatial point where the drumstick and the drumhead touch, and cannot be interpreted as a certain moment in _t0 .

As yet another achievable way, the acceleration detection and vibration detection methods are combined, within a unit time value, at least one moment when the acceleration is zero is detected, and the duration from the start of the vibration to the end of the vibration is detected as the first duration, The moment when the acceleration closest to the first duration or falling within the first duration is zero is taken as the first moment. For example, referring to Figure 3, it has been mentioned above that in a drumming action, the moment when the acceleration is 0 is not unique. The moment, the first moment, more accurately reflects when the drumming action occurred. Limited by the detection accuracy of the vibration sensor and the acceleration sensor, the acceleration sensor may detect the moment when the acceleration is 0 before the vibration sensor. In this case, the moment when the acceleration is 0 will not fall within the first duration corresponding to the vibration process. At this time, the moment when the acceleration closest to the first duration on the time axis is 0 may be taken as the first moment. In other cases, the acceleration sensor may be delayed. The vibration sensor detects the vibration first, and then the acceleration sensor detects the moment when the acceleration is 0. In this case, the moment when the acceleration is 0 will fall into the first time corresponding to the vibration process. In the duration, the moment when the acceleration is 0 can also be used as the first moment. In other words, this implementation may also be the first time when the acceleration closest to the start time, end time or midpoint time of the first duration is detected to be 0 on the time axis. The way of determining the first moment can be expressed in various ways, but on the premise that the technical essence does not change, all of them should fall within the scope of protection of the present application.

After the first moment is determined, the time difference between the first moment and the starting moment of the unit time value can be used as the feature of the time dimension in the target feature.

In another achievable way, whether the user keeps up with the beat can also be detected from the strength dimension data. The target features to be detected can include the strength dimension data (second feature). The strength dimension data means that the user is within the unit time value. The force information when making this single action. The beat is composed of multiple beats according to certain rules of strength and weakness. In general, the heavy beat in the beat needs more strength than the weak beat to produce the accent. For example, in the scene of drumming, the heavy beat requires the user to use more Use the drumsticks with great force to hit the drum head, while the weak beat should be hit with relatively light strength, and the strength of the sub-strong beat should be between the accent and weak sound. If you hit it, it will also disrupt the rhythm, and you will not be able to play the exact beat.

Specifically, the strength information is collected by an acceleration sensor, at least including information on the strength of the user's action within a unit time value, through which the acceleration sensor at least collects the acceleration corresponding to the user's action, or the direction of the acceleration can also be collected to obtain a more comprehensive strength information. For example, the acceleration is collected every 10 microseconds in a unit time value, and then the magnitudes of the multiple accelerations collected are averaged, and the average value is used as the collected strength information of the user's action.

Step 203 : Based on the beat category corresponding to each beat, each target feature is compared with a predetermined reference feature to obtain detection data.

After the target features corresponding to each beat are determined, the target features are compared with predetermined reference features. Wherein, the reference feature is a reference standard for judging whether the action of the user to be detected keeps up with the beat. In the time dimension, the reference feature can be a time interval, which is used to represent the time period corresponding to the correct single action within the unit time value, that is, if actions such as hitting are made within this time period, it is considered that the beat is not lost. Not snapped.

When the user performs actions corresponding to different types of beats, the reference features presented are different. For example, the time reference interval corresponding to the retake action and the time reference interval corresponding to the downbeat work are quite different. The beats of the category are compared with the same reference feature, and the error will be relatively large, which makes the detection result less accurate.

In the embodiment of the present application, it is proposed to compare the target feature and the reference feature based on the beat category, that is, to select the target feature of any unit time value according to the beat category corresponding to the unit time value, and select the reference corresponding to the beat of the corresponding category. feature to compare to improve detection accuracy.

Wherein, the beat category is a category divided according to the strength and weakness of the beat. In a practicable manner, the beat category includes strong beat (accent), second strong beat (subforte) and weak beat (weak tone). In another possible way, the beat category can also include the case of additional dotted notes. Because the beat of the added dotted note is delayed, the actions in the delayed beat will also match the actions of the undelayed beat. There are some differences, therefore, in this implementation, the beats with additional dots are regarded as a category alone, that is, the categories of beats include not only strong beats, sub-strong beats, and weak beats, but also further include strong beats with dotted notes. Multiple categories such as beat, second strong beat, weak beat, etc., as well as categories such as second strong beat and weak beat that cause the time value to be shortened due to the addition of dot notes.

For example, referring to shown in Fig. 4, Fig. 4 shows a plurality of notes arranged in the same sequence, processed according to different beats, 3 groups of rhythms obtained respectively, wherein 1. the beat of the group is a 3/4 beat, The beat of group ② is 4/4 beat, the beat of group ③ is still 3/4 beat, and dotted notes are added after the rebeat of each measure.

The three groups of rhythms are taken as an example to illustrate how to compare the target features with the reference features. The law of strength of 3/4 beat is strong, weak, weak, the first beat (note "1") in group ① is a strong beat (that is, a strong beat), and the second beat (note "2") is a weak beat 1. The third beat (the note "3") is a weak beat. After the target feature of the first beat is determined, choose to read the reference feature corresponding to the predetermined strong beat (ie, the accent). For example, in the time dimension, the strong beat The target feature corresponding to the beat is the time difference between the user's actual action and the starting moment of the beat, and the reference feature is the time reference interval corresponding to the strong beat; for the third beat (note "3"), select the time reference corresponding to the weak beat Because of the different beats, 4/4 beats correspond to strong, weak, second-strongest and weakest beats. In group ②, the beat corresponding to the third beat is the second-strongest beat. In the group rhythm, the third beat (the note "3") should select the reference feature corresponding to the sub-downbeat, and compare it with the determined target feature. In group ③ rhythm, due to the addition of dotted notes, the beat category corresponding to the first beat (note "1") is a strong beat with added dotted notes, and the unit time value occupied by the beat is changed from the original 1 The unit time value is extended to 3/2 unit time value. At this time, when performing feature comparison, the reference feature corresponding to the corresponding category of downbeats with added dotted notes should be selected.

How to determine the reference feature is explained below. As an implementable manner, the reference feature may include a reference feature, and the reference feature may be predetermined according to a personalized action feature of the user. The data can be initialized and calibrated according to each user action, and a user-specific reference feature can be established, such as a user-specific time reference interval. In this implementation mode, the electronic metronome can be set with the link of user personal data initialization, that is, the electronic metronome provided by the embodiment of the application supports user personal data initialization, and when the user turns on or logs in the electronic metronome for the first time, the user is prompted to enter personal data. Initialize the data, when the user makes multiple actions corresponding to any category such as strong beat, second strong beat, weak beat, etc., detect multiple single actions under the category of one beat, and determine the type based on multiple single actions The reference feature corresponding to the category beat. For example, the user is prompted to perform a single action of multiple downbeats, and according to the features corresponding to the single actions of multiple downbeats, the reference features corresponding to the user-specific downbeats are determined.

Specifically, corresponding to the target feature, the reference feature can also be determined from the time dimension and the force dimension. In the time dimension, based on multiple single actions made by the user, the corresponding time reference interval can be determined. By the following way:

Multiple collections of the first feature corresponding to when the user makes a single action corresponding to any type of beat within a unit time value, the first feature can be obtained based on time dimension data, for example, the first feature can be a single time interval, a single The time interval is the time it takes for the user to make a single action.

As an achievable way, the duration occupied by a single user action is the time difference between the specified time point (second moment) corresponding to the user's single action and the starting moment (second starting moment). When performing a single action, first determine the starting time (second starting time) of collecting each single action, and take any one specified time in the single action process of the user as the second time, and the second time is the same as the second time The time difference between the starting moments is taken as a single time interval. The manner of determining the second moment may refer to the manner of determining the first moment, which will not be repeated here.

It should be noted that the determination method of the first moment and the second moment should be consistent, that is, the second moment used to determine the single time interval in the reference feature determination process is the same as the first moment used to determine the target time interval in the target feature. The methods of determining the moments should correspond to each other. For example, if the first moment is the first moment when the acceleration is 0 during the action, the second moment should also be the first moment when the acceleration is 0 during the work process; if the first moment is the end moment of the continuous vibration process from the start of the vibration to the end of the vibration, then the second moment should also be the end time of the continuous vibration process when a single action of the user is detected; or, the first moment is the beginning or middle of the continuous vibration process Point moment, then correspondingly, the second moment should also be the start or midpoint moment in the vibration continuation process.

It should be noted that, in a practicable manner, the second starting moment is the time point at which the single action collection process starts to count. After the voice is played, the device emits a beep sound. At the same time as the beep sound, the device starts timing. The second time minus the start time of timing is the single time interval, which completes the collection process of a single action. Repeating the acquisition process of the single action can realize multiple action acquisitions. In this implementation, the single time interval corresponding to a single action is based on the time when the timing of the single action is collected as the starting time, and the obtained single time interval has nothing to do with the unit time value of the beat, so there is no Affected by the unit time value of the beat in the target rhythm to be detected, that is to say, the time reference interval determined by the single time interval collected in this way is suitable for various performance speeds, and does not need to be specific to various performance speeds Collect user actions separately. Moreover, the single time interval includes the user's personal reaction time, which reduces the impact of the difference in the user's personal reaction time on the accuracy of the detection result.

In another practicable way, in the reference feature determination stage, such as the user personal data initialization stage, when collecting multiple single actions made by the user, the collection can also be performed based on a fixed BPM speed, that is, at a fixed unit time value (to avoid confusion, hereinafter referred to as the second unit time value) as the detection unit, prompting the user to continuously perform multiple single actions corresponding to a type of beat at a certain playing speed. For example, if the BPM is set to 60, then The corresponding second unit time value is 1s, prompting the user to perform continuous reshooting actions at a speed of 60 beats per minute, and then record the second moment of the user's single action within each unit time value of 1s. A single time interval is obtained by subtracting the start moment (second start moment) of the second unit time value with a duration of 1 s from the second moment. It should be noted that in this implementation, when the second unit time value of the reference feature determination stage is consistent with the first unit time value of the target feature acquisition stage, the detection accuracy will be higher, and if the user inputs The BPM of the user's personal data is inconsistent with the BPM when the user's personal data is initialized, which will affect the accuracy of the detection to a certain extent. The greater the difference between the first unit time value and the second unit time value, the greater the error.

After determining the multiple single time intervals corresponding to each beat category, for any one of the beat categories, the average value of multiple single time intervals is calculated, and based on the average value, the average value corresponding to multiple single time intervals is calculated. Variance (or standard deviation); based on the mean square deviation (or standard deviation) and the mean value, determine the time reference interval corresponding to any type of beat. For example, the interval obtained by adding or subtracting N times the mean square error from the average value is used as a time reference interval, where 1≤N≤3.

It should be noted that when comparing the target time interval in the target rhythm to be detected with the time reference interval, the time reference interval is transplanted into the corresponding first unit time value, and the first starting moment of the first unit time value As the starting moment of the time reference interval, for example, the time reference interval corresponding to the calculated retake (i.e. downbeat) is 0.35-0.55s, which means that the first moment corresponding to the retake action falls into the first unit time value In the 0.35s to 0.55s within the time, it is regarded as keeping up with the beat, otherwise it is regarded as a drop or rush.

In the strength dimension, corresponding to the target feature, based on multiple single actions made by the user, the corresponding strength reference interval is determined. The reference range of the force dimension for a single action may be determined based on the force information corresponding to multiple single actions made by the user. Specifically, it is possible to collect the strength information corresponding to a single action corresponding to any type of beat within a unit time value multiple times, and further extract or determine the second feature based on the strength information. Based on multiple second features, determine The velocity reference interval corresponding to any type of beat. The force information may include the corresponding acceleration information when the user makes a single action within a unit time value, such as acceleration magnitude, acceleration direction, and the like. For example, in a drumming scene, detecting the acceleration of the wrist when the user is holding a drumstick and performing a knocking action can accurately reflect the degree of force of the user. Specifically, it is possible to collect multiple acceleration values within a unit time value, calculate the average value within the unit time value as the acceleration value of the word action corresponding to the unit time, and then calculate the acceleration values corresponding to the multiple word actions respectively. Average, and calculate the mean square error, the mean value plus or minus N times the mean square error can obtain the strength reference interval.

Based on the above method, the user-specific reference features are determined. The reference features determined based on the user's personal actions include the user's personal action characteristics and reaction time, which can be used as a reference feature to compare with the single action in the user's actual performance, which can reduce the user's individualized actions. errors due to features. In this way, individual differences are fully considered, and different users establish their own personalized standard data, so that the judgment result has higher accuracy.

It should be noted that, as an auxiliary implementable manner, a spare second reference feature may also be determined in advance based on music samples or multiple action samples as a general reference standard applicable to all users. The spare second reference feature can be obtained by performing motion analysis on music samples, or can be determined from the motion samples of professionals whose performance levels meet the level requirements. When the target user to be detected does not cooperate with the initialization of personal data, or other circumstances make it impossible to read the reference features of the user's personal initialization data extraction, the general (standby) data can be read from the pre-designated storage address. The second reference feature of . Specifically, before step 203 is performed, the user-specific reference feature can be read from the specified storage address according to the predetermined first path, and when the reading of the reference feature fails, the spare second path is read according to the predetermined second path. Two reference features.

When the determined reference features include both the time reference interval and the strength reference interval, based on the categories of the beats corresponding to the unit time value, the target features are compared with the reference features corresponding to the beats of the corresponding category to obtain the detection data. It is used to feed back whether the user’s actions are rushed or lost in time, and is also used to feedback whether the strength of a single action made by the user in a unit time value conforms to the law of the strength of the beat, that is, to feedback whether the strength of the user’s action is strong or not. The beat force is too small, the weak beat force is too strong, etc.

In a practicable manner, the reference feature may only include the time reference interval but not the strength reference interval, or only include the strength reference interval but not the time reference interval; correspondingly, the target feature may only include the first feature but not the The second feature, or only including the second feature but not including the first feature, that is, the method provided in the embodiment of the present application, can be detected in at least one of the time dimension and the strength dimension.

It should be noted that the sequence of the steps described above is not used as a limitation on the execution sequence of the corresponding steps. In terms of execution sequence, the step of determining or reading the reference feature needs to be prior to step 203 , and may not be prior to step 201 or step 202 .

As an achievable way, before step 202, it may also include the step of collecting the target speed of the user when making an action, the target speed may be the linear speed or angular speed when the user makes a performance action, etc., and the target speed may be passed Linear velocity sensor or angular velocity sensor acquisition. It should be noted that the speed and performance speed here are different technical concepts, and the performance speed is the number of beats per minute (BPM), and the two should not be confused.

Compare the collected target speed with the predetermined speed threshold corresponding to the user. When the target speed of the user's current action exceeds the speed threshold, it indicates that the current user's action speed is very fast. At this time, the detection error of whether to keep up with the beat will be is larger, so when it is determined that the target speed exceeds the threshold, the subsequent detection steps are not performed. When it is determined that the speed value corresponding to the target speed is lower than the speed threshold, step 202 and other subsequent steps are performed.

As an implementable mode, in the initialization stage of the user's personal data, the user can also be prompted to select the corresponding scene, for example, select a large category of scenes first, and display multiple alternative options, such as musical instrument playing scenes, military training scenes, dance training, etc. Alternatives, after the user selects one of the alternatives, continue to prompt the user to select a small category of scenes, for example, in the musical instrument performance scene, further prompt the user to select the corresponding musical instrument, percussion? keyboard instrument? stringed instrument? The division of specific scenes can also improve the accuracy of detection.

After the detection data is obtained, a feedback report is generated according to the detection data. The feedback report can convey the wrong tempo point to the user through a variety of prompts, such as text display, or voice broadcast, or the music that the user has just played Replay, light up or vibrate at the wrong beat of the user to inform the user that there is a mistake here, the prompt method can be more detailed, and different types of mistakes are indicated by emitting different colors of light, for example, red light means off beat, Green light means rushing to shoot, yellow light means insufficient force, purple light means too much force, etc., or vibrations of different intensities can prompt users to make different types of mistakes. The above prompting methods can be combined arbitrarily.

In order to further understand the method provided by the embodiment of the present application, an embodiment is listed below according to a feasible step execution sequence.

Referring to shown in Figure 5, this embodiment comprises the following steps:

Step 501, after determining that the device is turned on or the user opens the login interface, jump to step 502; in one implementable manner, the subsequent steps are performed only when the user starts the device for the first time or logs in to the device for the first time, and in another embodiment, every At predetermined intervals, when the user turns on the device or logs in to the device, the subsequent steps are executed.

In step 502, the user is prompted to initialize personal data. For example, taking the actual product as a smart watch as an example, refer to the operation interface U10 in FIG. Action data?", and set two options of "Yes" and "No", and text prompts the user to "tap once to select "Yes", tap twice to indicate "No", and confirm that a tap action from the user is sensed After that, enter step 503, otherwise, return to step 501, monitor the user's next startup or login behavior, and wait for the user to ask whether to perform personal data initialization again when the user starts or logs in next time. Or, personal data initialization is only performed when the user first Prompt when turning on the machine for the first time, if the user selects No in step 502, the collection process of the user's personal data can also be finished, and check whether a spare general reference feature is stored to the designated storage address. The storage path of the spare general reference feature ( That is, the second predetermined path) can be pre-agreed and written into the code of the application program. Or, as another possible way, the user can also directly click the "Yes" or "No" button to make a selection. Specifically, you can design A variety of user interaction modes are not listed in the embodiments of the present application. It should be noted that the circular interface shown in Fig. 6 is only an example, and the display interface of the electronic metronome can be designed in various shapes. Do not list them one by one.

In addition, it should be noted that the electronic metronome can install a corresponding APP on a terminal device (such as a mobile phone), that is, an application program with a metronome function can be installed on a terminal device such as a mobile phone. The rooms can be connected through wireless communication methods such as bluetooth or wifi, and the above-mentioned user interaction process can also be completed based on terminal devices such as mobile phones. The corresponding user operation interface is shown in FIG. 7 .

Step 503, collecting multiple single actions of the user. Specifically, the user is prompted to make multiple single actions corresponding to each category of beats. The prompt method can be a combination of text and voice to prompt the user to start inputting actions, and during the process of inputting multiple single actions, voice broadcast, At least one of text display, vibration, luminescence and other methods prompts the user with the beat corresponding to each action. For example, as shown in interface U11 in Figure 6, while the text prompts on the operation interface, a voice broadcast "Please make an accent action according to the vibration prompt after hearing the prompt tone", at this time, the device will emit a beep as a prompt sound, and the device gives a vibration prompt, and counts at the moment when the vibration starts. The user performs a tapping action after hearing the prompt sound and sensing the vibration. The device records the precise time at the second moment according to the motion data in the action; Then vibrate again, start timing again and record a second moment, and so on for many times. The motion data may be information such as vibration and/or acceleration generated by the user's motion collected by the sensor, and time information or strength information such as the second moment is further obtained based on the vibration and/or acceleration information and other information. Specifically, how to determine the action time and action strength based on the data collected by the sensor refers to the description in the aforementioned step 202 , which will not be repeated here.

After confirming that the required data (such as time information and acceleration information, etc.) has been collected, the user will be notified in text or voice: "The strong beat action has been recorded successfully, please enter the weak beat action after hearing the prompt tone", and then perform the weak beat action. The collection of beat motion data, and so on, at least obtain the motion data of typical types of beats such as strong beat, weak beat, and sub-strong beat. For example, in the percussion scene, the user wears the smart watch and performs percussion with different strengths according to the instructions, such as heavy percussion, sub-forte percussion, soft percussion, etc., and obtains the belt of three beat categories. Action sample data with user personal characteristics. Correspondingly, refer to FIG. 7 for an operation interface of a terminal device such as a mobile phone.

Step 504, based on the multiple single actions collected in step 503, determine reference features. In an achievable way, the time interval or acceleration value of a single action can be determined first as a single sample, and then multiple samples are averaged and then the mean square error is calculated, and the corresponding time reference interval and Strength reference interval.

For example, in a practicable manner, the time reference interval can be obtained in the following manner:

Calculate the average of multiple single time intervals

The mean square error σ was calculated using the following formula:

Among them, n represents the number of actions corresponding to a category beat made by the user; X _i represents the word time interval corresponding to the i-th action. by average

Adding and subtracting twice the mean square error σ as the time reference interval, that is, the time reference interval is

It should be noted that, taking the time interval or acceleration corresponding to a single action as a single sample, various mathematical algorithms for determining the characteristics of the overall sample based on multiple single samples can be applied to the time reference interval or strength in the embodiment of this application The determination of the reference interval is not limited to the above method of adding/subtracting the mean square error from the mean value, for example, the corresponding sample interval can also be determined based on the mean square value and the mean square error.

Step 505, based on user input, determine beat and unit time value. Determining the unit time value means determining the number of beats per minute (BPM). Determining the beat includes determining the time signature of the beat, and may also include selecting from a user-defined beat type. Specifically, the time signature and unit time value can be adjusted on the interactive interface on the electronic metronome.

Step 506 , detecting the target speed when the user is currently making an action, and the target speed may be a linear velocity (displacement velocity) or an angular velocity. If the linear velocity or angular velocity of the user's action is too high, it means that the user is currently in an abnormal high-speed performance state. At this time, the action frequency will be high, and the detection of whether to keep up with the beat is of little use, and the error will be relatively high, so , when it is determined that the speed of the user's current action exceeds the speed threshold, no subsequent detection is performed, and only when it is determined that the user's speed is lower than the speed threshold, subsequent detection is performed.

Wherein, the speed threshold can also be to prompt the user to play at the fastest speed during the initialization stage of the user's personal data. For example, referring to the interface U12 in FIG. Voice prompt: "Please tap at the fastest speed after hearing the beep", and then the speed sensor senses the user's speed threshold and records it. The speed sensor can be a linear speed sensor or an angular speed sensor.

Step 507, judge whether the target speed is lower than the speed threshold, if yes, execute step 508, otherwise, return to step 506 to continue detecting the target speed.

In a practicable way, the electronic metronome can guide the user to continuously increase the tempo from a slow speed until a small change in the acceleration is detected or no acceleration can be detected, then it can be considered as reaching the user's speed threshold, and record the time speed. It should be noted that the faster the speed tapping action, the greater the BPM at this time, that is, the user's linear velocity or angular velocity is generally positively correlated with the BPM value. Based on the correspondence between the user's target speed and BPM, the The collected linear velocity or angular velocity is converted into a BPM value, that is, the maximum number of notes the user can tap in one minute is used as the unit of measure for the user's limit speed. When the user starts to select the BPM value, if the BPM value selected by the user exceeds the user's speed threshold, the subsequent detection work can be suspended, and the user can be reminded "Your speed limit has been exceeded, please confirm the number of beats".

Step 508, detecting target features. In this embodiment, detecting the feature of the target includes detecting the feature of the time dimension (first feature) and the feature of the force dimension (second feature). For the specific detection method, please refer to the aforementioned step 202, which will not be repeated here.

Step 509, read the reference signature. First read the user-specific reference feature, if the reference feature reading fails, it means that the user has not initialized the personal data, or the initialization data entry is unsuccessful. At this time, it is necessary to read the second reference feature from the execution storage address, that is, execute step 509 . If the reference signature is read successfully, then the reference signature is preferentially used as the reference signature participating in the comparison, and the execution of

step

511 or 512 is continued.

Step 510, read the second reference feature. If the reading is successful, continue to execute

step

511 or 512. If the reading fails, the reference feature cannot be obtained, and the detection process ends (not shown in the figure).

Step 511 , comparing the target time interval in the target feature corresponding to each single action with the time reference interval in the reference feature, and determining whether each single action is a missed shot or snap shot.

For example, referring to FIG. 9 , it is assumed that the performance speed of the target rhythm to be detected is 60 BPM, that is, the unit time value occupied by one beat is 1 s. The beat subcategory corresponding to a unit time value shown in Figure 9 is reshoot (i.e. strong beat), the time reference interval corresponding to the reshoot determined in advance is Tf, the minimum value min of the reshoot reference interval, the maximum value max, if the time difference between the first moment corresponding to the user’s action within the unit time value and the starting time t1 of the unit time value (i.e. the target time interval) is T1, then it is determined that the user is keeping up with this beat If the time length between the first moment and the starting moment t1 of the unit time value is T2, and the time length represented by T2 is higher than (greater than) the maximum value max of Tf, then it is judged The user fell off the beat and did not make an action in the correct interval; if the time length between the first moment and the starting moment t1 of the unit time value is T3, the duration represented by T3 is lower (less than) Tf If the minimum value is min, it is determined that the user snapped a shot at this beat. The beat category corresponding to another unit time value shown in FIG. 9 is the next strong beat, and the predetermined time reference interval is Tmf. If the user makes an action within the other unit time value, the corresponding first moment The time difference from the starting moment t2 of the unit time value is T4, and the duration represented by T4 is higher (greater than) the maximum value of Tmf, then it is determined that the user has dropped a beat in this second strong beat. The specific judgment logic is shown in Table 2 below.

Table 2

Step 512: Compare the strength information in the target feature corresponding to each single movement with the strength reference interval in the reference feature to determine whether the strength of each single movement is too light or too heavy. The force information may be an average value of the acceleration of a single action process. For example, in a possible implementation, a unit time value is divided into M time periods on average, the acceleration collected at the midpoint of each time period is obtained, and the average value of M accelerations is calculated as a single action strength information of the process. Correspondingly, each single sample value can be determined in the same way for the strength reference interval, and then the strength reference interval can be determined by referring to the method of determining the time reference interval based on multiple single time intervals. The judgment logic between the strength reference interval and the strength information can refer to the judgment logic of the time dimension. For details, please refer to Table 3 below and will not repeat it.

table 3

Step 513, output a feedback report.

In a practicable manner, the feedback report is displayed on the interface of the electronic metronome. For example, referring to Fig. 10, in the interface U13, the detection result of the group ② rhythm shown in Fig. 4 is displayed, and below each beat that does not keep up with the beat, an error situation is displayed, wherein the first bar The third beat is dropped, the first beat of the second measure is not strong enough, and the second beat of the third measure is rushed.

As an achievable way, the rhythm can be played again, and prompts such as voice, vibration or light are given at the wrong beat, so that the user can feel the wrong beat point more intuitively.

The embodiment of the present application also provides a wearable device, which includes: a display screen; at least one sensor; a processor; a memory; at least one application program; and one or more computer programs, wherein the one or more The computer programs are stored in memory, one or more computer programs comprising instructions. Wherein, when the instruction is executed, the wearable device may perform the following steps: determine the beat corresponding to the target rhythm to be detected and the first unit time value occupied by one beat; obtain the target rhythm through the at least one sensor Based on the target features corresponding to each beat, based on the beat subcategory corresponding to each beat, each target feature is compared with a predetermined reference feature to obtain detection data.

A possible product hardware architecture of the wearable device 110 provided in the embodiment of the present application is shown in FIG. 11. The hardware architecture of the wearable device 110 may include:

The display screen 111 is configured to display a feedback report to the user, for example, display the interface shown in FIG. 10 . Specifically, the display screen 111 may include a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the wearable device 110 may include at least one display screen 111 .

The sensor group 112 includes at least one sensor, specifically, at least a vibration sensor or an acceleration sensor.

The processor 113 includes one or more processing units. For example: the processor 113 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a digital Signal processor (digital signal processor, DSP), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The memory 114 is used to store data such as the above instruction, reference features and detected target features. The memory 114 may be an external memory independent of the processor 113 , or may be set in the processor 113 . For example, in some embodiments, the memory built into the processor 113 may be a cache memory for storing instructions or data that the processor 113 has just used or recycled. If the processor 113 needs to use the instruction or data again, it can be called directly from the cache memory. Repeated access is avoided, and the waiting time of the processor 113 is reduced.

The memory may be read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (random access memory, RAM), or other types of memory that can store information and instructions A dynamic storage device can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage ( Including Compact Disc, Laser Disc, Optical Disc, Digital Versatile Disc, Blu-ray Disc, etc.), magnetic disk storage medium or other magnetic storage devices, or can be used to carry or store desired program code in the form of instructions or data structures and can Any other media accessed by a computer, etc.

The power management module 115 receives the input of the battery and/or the charging management module, and supplies power to the processor 113 , the memory 114 , the display screen 111 , and the sensor group 112 . The power management module 115 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 115 can also be set in the processor 113 . In some other embodiments, the power management module 115 and the charging management module can also be set in the same device.

It can be understood that the structures shown in the drawings in the embodiments of the present application do not constitute a limitation on the wearable device 110 . In other embodiments of the present application, the wearable device 110 may include more or fewer components than shown in the illustration, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

For example, in an implementable manner, as shown in FIG. 12, the wearable device 110 may further include:

Loudspeaker 116, and audio circuit, described loudspeaker 116 is connected with processor 113 through audio frequency circuit, is used for expressing the strength law of each beat in the beat in sound mode, and takes place in the way of sound (such as voice) to user's feedback or The tempo point for snapping and other issues. For example, while the display screen 111 of the wearable device 110 is displaying the interface shown in FIG. 10 , a voice broadcast is made "Hello, after detection, you fell off the third beat of the first bar...".

The motor 117 provides vibration power for the wearable device 110. As a practical way, the motor of the motor is a 7G ERM motor, and its vibration capability is about 7 times stronger than that of an ordinary smart phone.

The input unit 118 is used for inputting user instructions. For example, both the beat and the first unit time value of the target rhythm to be detected can be determined according to user input, and the input unit is used for the user to input various instructions. Specifically, as a practicable manner, the input unit 118 and the display screen 111 may be integrated into a touch display screen; the input unit 118 may also include a pressure sensor (not shown in FIG. 12 ), through which the pressure generated by the user is felt The signal can convert the pressure signal into an electrical signal to complete user instruction input. In some embodiments, a pressure sensor can be disposed on the display screen 111 . There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. When a touch operation acts on the display screen 111, the wearable device 110 detects the intensity of the touch operation according to the pressure sensor. The wearable device 110 can also calculate the touched position according to the detection signal of the pressure sensor. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the beat icon, an instruction to input a regular beat is executed, for example, the time signature numbers corresponding to a plurality of regular beat alternatives are displayed on the display screen 111, and the user can select Set the beat with the number corresponding to the time signature, for example, select 3 for the numerator and 8 for the denominator, then set the beat of the music to be played to 3/8. When a touch operation whose intensity is greater than or equal to the first pressure threshold acts on the beat icon, an instruction for selecting a custom beat is executed, and an unconventional beat is selected from a pre-stored custom beat database.

The wireless communication unit 119 is configured to perform wireless communication with a metronome application installed on another terminal device. For example, the wearable device 110 is a smart watch with a metronome function, a metronome application program corresponding to the smart watch can be installed on the smart phone terminal, and the smart watch can communicate with the application program wirelessly through the wireless communication unit 119 . Specifically, the wireless communication unit 119 can support applications on the wearable device 110 including wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT) , global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication unit 119 may be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves through the antenna, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 113 . The wireless communication module can also receive the signal to be sent from the processor 113, frequency-modulate it, amplify it, and convert it into electromagnetic wave to radiate out through the antenna.

Based on the wireless communication capabilities of wearable devices, multiple wearable devices can be connected to the same terminal device (such as smartphones, IPADs, laptops, etc.) The application realizes the synchronous collaboration of multiple wearable devices. For example, the wearable devices of multiple members of a band are connected to the metronome APP of the same smartphone, so that multiple people can collaborate and perform at the same rhythm. Alternatively, multiple wearable devices can also use one of the wearable devices as an intermediate station for wireless communication, and realize the cooperation of multiple wearable devices based on one of the wearable devices without resorting to the APP on the terminal device.

The flashlight 120 is used to remind the user of the strength of each beat in the beat in a light-emitting manner, and is also used to feed back to the user the beat point where the user loses a beat or snaps a beat and the strength is uncomfortable. For example, the display screen 111 displays At the same time as the displayed interface, replay the target rhythm played by the user, and then give a flash reminder at the beat point that does not keep up with the beat, to remind the user that the beat does not keep up with the beat, and different types of errors use different colors of flashes, For example, the red light flashes when the shot is missed, the green light flashes when the shot is snapped, the blue light is displayed when the force is not enough, the purple light is displayed when the force is too high, and so on.

In a practicable manner, the sensor group 112 includes a vibration sensor and an acceleration sensor, wherein the vibration sensor is used to sense the vibration or vibration of the wearable device 110. For example, taking a drumming scene as an example, when detecting a user action, it may The vibration sensor senses the vibration generated by the wearable device when the user is hitting the action. When the vibration is detected, it means that the drumstick held by the user is in contact with the drum surface, and vibration is generated under the rebound force of the drum surface, and the vibration continues. The time of includes the contact time between the drumstick and the drum surface, and the specified moment in the duration of the vibration can be used to mark the time when the drumming action occurs. In the military training scene, the vibration sensor can also sense the vibration generated by the user during the stepping process such as kicking, and then identify the user's action time. Specifically, the vibration sensor may be a relative type, an inertial type, or an electric type, a piezoelectric type, an eddy current type, an inductive type, a capacitive type, a resistive type, a photoelectric type, or the like.

The acceleration sensor is used to collect acceleration information of the wearable device 110 . The acceleration sensor can be used to collect acceleration information during the user's action, including the magnitude and/or direction of the acceleration. When the user is making an action, the acceleration information reflects the strength of the user's action on the one hand, and on the other hand, the specific acceleration is 0. The time can also be used as a mark of the user's action time. In some embodiments, the wearable device 110 may only include an acceleration sensor without a vibration sensor. In individual embodiments, only vibration sensors may be included without acceleration sensors.

Optionally, the sensor group 112 also includes a distance sensor. The distance sensor can be used as an accessory to sense the distance between the drumstick and the drumhead, and then determine the time information of the user's action based on the distance.

In one implementable manner, the processor 113 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, general-purpose input/output (general-purpose input/output, GPIO) interface, and/or universal serial bus (universal serial bus, USB) interface, etc.

Wherein, the processor 113 may include multiple sets of I2C buses, and the processor 113 may be respectively coupled to the sensor group 112, the power management module 115, the flashlight 120, etc. through different I2C bus interfaces. For example, the processor 113 may be coupled to the display screen 111 through an I2C interface, so that the processor 113 communicates with the display screen 111 through an I2C bus interface.

In an implementable manner, the processor 113 may include multiple sets of I2S buses, and the I2S interface may be used for audio communication. The processor 113 can be coupled with the audio circuit through the I2S bus, so as to realize the communication between the processor 113 and the audio circuit, and then make the speaker 116 sound under control.

In a practicable manner, the wearable device also has a tuner function, which can be used to tune the musical instrument. Further, the wearable device can also be set with a volume decibel alarm, which is used to monitor the volume throughout the day, and once the volume reaches the dangerous level stipulated by the World Health Organization, an alarm will be issued.

The embodiment of the present application also provides a device, which includes:

display screen; at least one sensor; one or more processors; memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions , when the instruction is executed by the device, the device is made to perform the following steps: determine the beat corresponding to the target rhythm to be detected and the first unit time value occupied by one beat; obtain the time corresponding to each beat in the target rhythm through at least one sensor The target feature, the target feature includes the first feature, and the first feature is used to represent the corresponding time information when the user makes an action within the first unit time value; based on the beat category corresponding to each beat, combine each target feature with a predetermined reference feature Compare to obtain detection data; wherein, the detection data is used to represent whether the user's actions within the first unit time value keep up with the beat; the reference features include the time reference intervals occupied by the user's actions corresponding to each type of beat.

The embodiment of the present application also provides a computer storage medium, including computer instructions, and when the computer instructions are run on an electronic device, the electronic device is made to execute the method shown in FIG. 2 .

To sum up, the method and device provided by the embodiment of the present application initializes the personal user data before playing, collects the time reference interval and speed threshold of percussion stress, weak sound, and subforte, and plays the music repeatedly. Accurate feedback can correct incorrect beats, avoid blind training, achieve the purpose of cultivating users' sense of rhythm faster, and help users learn music more efficiently or keep the team tidy.

It should be understood that in each embodiment of the present application, "first", "second", etc. are only used to refer to different objects, and do not mean that there are other limitations on the referred objects.

It should be understood that the term "unit" in the embodiments of the present application may be implemented in the form of software and/or hardware, which is not specifically limited. For example, a "unit" may be a software program, a hardware circuit or a combination of both to realize the above functions. The hardware circuitry may include application specific integrated circuits (ASICs), electronic circuits, processors (such as shared processors, dedicated processors, or group processors) for executing one or more software or firmware programs. etc.) and memory, incorporating logic, and/or other suitable components to support the described functionality.

Therefore, the units of each example described in the embodiments of the present application can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three kinds of relationships, for example, A and/or B may indicate that A exists alone, A and B exist simultaneously, or B exists alone. Among them, A and B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, and c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, and c may be single or multiple.

Those of ordinary skill in the art can appreciate that each unit and algorithm steps described in the embodiments disclosed herein can be realized by a combination of electronic hardware, computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In several embodiments provided in this application, if any function is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

The foregoing is only a specific implementation of the present application. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should be covered by the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

A method for detecting user actions based on music beats, comprising:

Determine the beat corresponding to the target rhythm to be detected and the first unit time value occupied by one beat;

Obtain the target features corresponding to each beat in the target rhythm, the target features include a first feature, and the first feature is used to represent the corresponding time information when the user performs an action within the first unit time value ;

Based on the beat category corresponding to each beat, each target feature is compared with a predetermined reference feature to obtain detection data; wherein, the reference feature includes a time reference interval corresponding to each beat category; the detection data is used to at least provide information to the user Feedback whether to keep up with the beat in the time dimension.
The method according to claim 1, wherein the first feature comprises a target time interval; the target time interval is used to characterize the process of the user performing an action within the first unit time value The time difference between the specified first moment and the first start moment, wherein the first start moment is the start moment of the first unit time value.
The method according to claim 2, wherein the first moment is determined based on the following method:

Within the first unit time value, any moment when the acceleration is zero is detected as the first moment.
The method according to claim 2, wherein the first moment is determined based on the following method:

Within the first unit time value, the duration from the start of the vibration to the end of the vibration is detected as the first time length, and any specified time in the first time length is used as the first time.
The method according to claim 2, wherein the first moment is determined based on the following method:

Within the first unit time value, detect at least one moment when the acceleration is zero, and detect the duration from the start of the vibration to the end of the vibration, as the first duration, which will be the closest to the first duration or fall within the first The moment when the acceleration in the duration is zero is taken as the first moment.
The method according to claim 1, wherein, before determining the beat corresponding to the target rhythm to be detected and the first unit time value occupied by one beat, further comprising:

A reference characteristic specific to the user is determined.
The method according to claim 6, wherein said determining said user-specific reference features comprises:

Obtain the single time intervals corresponding to multiple actions of any beat category by the user, and determine the time reference interval corresponding to any beat category based on the multiple single time intervals, as the user's personal Reference feature; wherein, the single time interval is determined based on the following method:

Determining the second moment specified in the process of the user making a single action, using the time difference between the second moment and the second starting moment as a single time interval; wherein, the second starting moment is Collect the starting moment of a single action.
The method according to claim 7, wherein said determining a time reference interval corresponding to any beat category based on a plurality of single time intervals comprises:

calculating an average of the plurality of single time intervals;

Based on the average value, calculate the mean square error corresponding to the plurality of single time intervals;

Based on the mean square error and the average value, determine a time reference interval corresponding to any beat category.
The method according to claim 8, wherein said determining the time reference interval corresponding to any beat category based on said mean square error and said average value comprises:

The interval obtained by adding or subtracting N times the mean square error from the average value is used as the time reference interval, and 1≤N≤3.
The method according to claim 1, wherein the target feature further includes a second feature, and the second feature is used to characterize the corresponding strength of the user when making an action within the first unit time value information; the reference feature also includes a strength reference interval corresponding to each beat category;

Based on the beat category corresponding to each beat, each target feature is compared with a predetermined reference feature, which also includes:

Based on the beat subcategory corresponding to each beat, the second feature is compared with the strength reference interval corresponding to the corresponding beat subcategory; the detection data is also used to feed back to the user whether the action strength of each beat conforms to the strength rule of the beat.
The method according to claim 10, wherein, before acquiring the target features corresponding to each beat in the target rhythm, further comprising:

Determine the user-specific strength reference interval.
The method according to claim 11, wherein the determination of the user-specific strength reference interval comprises:

Acquiring dynamic information corresponding to multiple actions of any type of beat by the user, and determining a dynamic reference interval corresponding to any type of beat based on the multiple pieces of dynamic information, as the user's personal dynamic reference interval; Wherein, the strength information includes the corresponding acceleration information when the user makes an action of any beat category.
The method according to claim 1, wherein, before acquiring the target features corresponding to each beat in the target rhythm, further comprising:

Acquiring the target speed of the user when making an action;

The target speed is compared with a predetermined speed threshold, and when it is determined that the value of the target speed is lower than the speed threshold, the step of acquiring target features corresponding to each beat in the target rhythm is executed.
The method according to claim 13, wherein the target speed includes a linear speed or an angular speed.
The method according to claim 6, wherein, before comparing each target feature with a predetermined reference feature based on the beat category corresponding to each beat, further comprising:

Reading the user-specific reference feature according to the predetermined first path;

When the reading of the reference signature fails, a spare second reference signature is read according to a predetermined second path; the second reference signature is a pre-stored reference signature applicable to each user.
The method according to any one of claims 1-15, characterized in that, after determining the beat corresponding to the target rhythm to be detected and the first unit duration occupied by one beat, further comprising:

According to the law of intensity corresponding to the beat, a reminder of the corresponding intensity is carried out within each first unit time value, and the manner of the reminder includes at least one of vibration, sound playback, and light emission.
The method according to any one of claims 1-15, characterized in that, after obtaining the detection data, further comprising:

According to the detection data, at least feedback is given to the user whether they keep up with the beat in the time dimension, and the feedback method includes at least one of display, vibration, sound playback, and light emission.
The method according to any one of claims 1-15, characterized in that, the beat subcategories at least include strong beats, sub-strong beats and weak beats.
A wearable device, characterized in that, comprising:

display screen; at least one sensor; one or more processors; memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more A plurality of computer programs comprising instructions which, when executed by the device, cause the device to perform the following steps:

Determine the beat corresponding to the target rhythm to be detected and the first unit time value occupied by one beat;

Obtain target features corresponding to each beat in the target rhythm, where the target features include a first feature, and the first feature is used to characterize the corresponding time when the user makes an action within the first unit time value information;

Based on the beat category corresponding to each beat, each target feature is compared with a predetermined reference feature to obtain detection data; wherein, the reference feature includes a time reference interval corresponding to each beat category; the detection data is used to at least provide information to the user Feedback whether to keep up with the beat in the time dimension.
A terminal device, characterized in that it includes:

display screen; at least one sensor; one or more processors; memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more A plurality of computer programs include instructions, which, when executed by the terminal device, cause the terminal device to perform the following steps:

Determine the beat corresponding to the target rhythm to be detected and the first unit time value occupied by one beat;

Obtain target features corresponding to each beat in the target rhythm, where the target features include a first feature, and the first feature is used to characterize the corresponding time when the user makes an action within the first unit time value information;

Based on the beat category corresponding to each beat, each target feature is compared with a predetermined reference feature to obtain detection data; wherein, the reference feature includes a time reference interval corresponding to each beat category; the detection data is used to at least provide information to the user Feedback whether to keep up with the beat in the time dimension.
A computer storage medium, characterized by comprising computer instructions, and when the computer instructions are run on an electronic device, the electronic device is made to execute the method according to any one of claims 1 to 18.