CN113327482A - String instrument playing teaching method and system based on video monitoring - Google Patents

String instrument playing teaching method and system based on video monitoring Download PDF

Info

Publication number
CN113327482A
CN113327482A CN202110673924.8A CN202110673924A CN113327482A CN 113327482 A CN113327482 A CN 113327482A CN 202110673924 A CN202110673924 A CN 202110673924A CN 113327482 A CN113327482 A CN 113327482A
Authority
CN
China
Prior art keywords
playing
stringed
stringed instrument
student
instrument
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110673924.8A
Other languages
Chinese (zh)
Inventor
樊星
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Squirrel Classroom Artificial Intelligence Technology Co Ltd
Original Assignee
Shanghai Squirrel Classroom Artificial Intelligence Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Squirrel Classroom Artificial Intelligence Technology Co Ltd filed Critical Shanghai Squirrel Classroom Artificial Intelligence Technology Co Ltd
Priority to CN202110673924.8A priority Critical patent/CN113327482A/en
Publication of CN113327482A publication Critical patent/CN113327482A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B15/00—Teaching music
    • G09B15/001—Boards or like means for providing an indication of chords
    • G09B15/002—Electrically operated systems
    • G09B15/003—Electrically operated systems with indication of the keys or strings to be played on instruments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

The invention provides a string instrument playing teaching method and system based on video monitoring, which can determine whether a student successfully masters the debugging skill of a musical instrument by acquiring and analyzing the debugging audio generated by the student in the process of debugging the string instrument; and the video images of the hands of the students playing the stringed instruments in the process of playing the stringed instruments are shot and analyzed, and whether wrong playing actions exist in each playing chapter is determined from the aspects of the action postures and the action amplitudes of the fingers of the students playing the stringed instruments, so that the reminding message of whether the playing actions are wrong is fed back to the students in time in the follow-up stringed instrument playing exercises, and therefore, comprehensive, automatic and accurate teaching experience can be provided for the students and the stringed instrument playing skills of the students can be improved to the maximum extent.

Description

String instrument playing teaching method and system based on video monitoring
Technical Field
The invention relates to the technical field of intelligent teaching, in particular to a string instrument playing teaching method and system based on video monitoring.
Background
At present, the performance training of string musical instruments such as violins and the like requires a teacher to carry out the in-person teaching of hands and handles, which provides higher requirements for students to learn to play string musical instruments. Due to the high cost of hiring a teacher to teach a stringed instrument, not all students can be charged with the corresponding costs. In addition, the string instrument playing teaching requires a teacher to comprehensively analyze the student's playing skills in the teaching process, and also needs to distinguish the timbre of music in the student's playing process, and not all teachers can comprehensively and carefully teach students to perform string instrument playing training. It is thus clear that the teaching mode of playing string musical instrument of current can't provide comprehensive, automatic and accurate teaching experience for the student, and this is unfavorable for promoting student's string musical instrument and plays the skill.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a string instrument playing teaching method and system based on video monitoring, which collects debugging audio correspondingly generated by students in the process of debugging the string instrument before playing the string instrument; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged; when the stringed instrument is successfully debugged, shooting a video image of a student playing the stringed instrument by hands in the process of playing the stringed instrument; analyzing the video image, and determining the action posture and action amplitude of fingers of the student in the process of playing the stringed instrument; finally, determining the playing chapters corresponding to playing motion errors in the process of playing the stringed instrument by the students according to the motion postures and/or the motion amplitudes; feeding back corresponding reminding information to the student in the subsequent stringed instrument playing practice according to the wrong playing action of the student in the playing performance chapter; therefore, the string instrument playing teaching method and system based on video monitoring can determine whether the student successfully masters the debugging skill of the instrument by acquiring and analyzing the debugging audio generated by the student in the process of debugging the string instrument; and the video images of the hands of the students playing the stringed instruments in the process of playing the stringed instruments are shot and analyzed, and whether wrong playing actions exist in each playing chapter is determined from the aspects of the action postures and the action amplitudes of the fingers of the students playing the stringed instruments, so that the reminding message of whether the playing actions are wrong is fed back to the students in time in the follow-up stringed instrument playing exercises, and therefore, comprehensive, automatic and accurate teaching experience can be provided for the students and the stringed instrument playing skills of the students can be improved to the maximum extent.
The invention provides a stringed instrument playing teaching method based on video monitoring, which is characterized by comprising the following steps:
step S1, collecting debugging audio correspondingly generated by students in the process of debugging stringed instruments before playing the stringed instruments; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged;
step S2, after the stringed instrument is successfully debugged, shooting the video image of the stringed instrument played by the hands of the student in the process of playing the stringed instrument; analyzing the video image, and determining the action posture and action amplitude of fingers of the student in the process of playing the stringed instrument;
step S3, determining that playing chapters corresponding to playing motion errors exist in the process of playing the stringed instrument by the students according to the motion postures and/or the motion amplitudes; feeding back corresponding reminding information to the student in the subsequent stringed instrument playing practice according to the wrong playing action of the student in playing the playing piece;
further, in step S1, acquiring debugging audio correspondingly generated by the student in the process of debugging the stringed instrument before playing the stringed instrument; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged specifically comprises:
step S101, collecting debugging audio correspondingly generated by students in the process of debugging stringed instruments before playing the stringed instruments from a plurality of different azimuth angles by using a microphone array;
step S102, performing Kalman filtering processing on the debugging audio, so as to eliminate background noise existing in the debugging audio;
step S103, extracting corresponding sound waveform information from the debugging audio; analyzing the sound waveform information so as to determine whether the duration time that the actual sound frequency of the stringed instrument is greater than a preset sound frequency threshold value exceeds a preset time threshold value in the debugging process, and if not, determining that the stringed instrument is successfully debugged; if yes, determining that the stringed instrument fails to be debugged, and feeding back a prompt message for continuously debugging the stringed instrument to the student until the stringed instrument is successfully debugged;
further, in the step S2, after determining that the stringed instrument is successfully debugged, capturing a video image of the student playing the stringed instrument with his hand during playing the stringed instrument; analyzing the video image, and determining the action posture and action amplitude of the fingers of the student in the process of playing the stringed instrument specifically comprises the following steps:
step S201, after the stringed instrument is successfully debugged, binocular shooting is carried out on the action of playing the stringed instrument by hands of the student in the process of playing the stringed instrument, so that corresponding binocular video images are obtained;
step S202, determining image parallax corresponding to the binocular video image, and generating a three-dimensional video image corresponding to the action of playing stringed instruments by hands;
step S203, extracting a plurality of frame images of different time periods in the process of playing the stringed instrument by the student from the three-dimensional video image; identifying the finger action state of the student in each frame of image, thereby determining the six-degree-of-freedom action posture of the finger and the action amplitude of the finger in the process of playing the stringed instrument by the student;
further, in the step S3, it is determined that the playing chapter corresponding to the playing motion error exists in the process of playing the stringed instrument by the student according to the motion posture and the motion amplitude; according to the wrong playing action of the student in playing the playing piece, the step of feeding back a corresponding reminding message to the student in the subsequent stringed instrument playing practice specifically comprises the following steps:
step S301, determining fingertip orientation change information of fingers in a three-dimensional space during playing of a stringed instrument by a student according to six-degree-of-freedom action posture of the fingers; determining fingertip action amplitude change information of the fingers in a three-dimensional space in the process of playing the stringed instrument by the student according to the action amplitude of the fingers; determining whether the student has wrong playing actions in the process of playing the stringed musical instrument according to the fingertip orientation change information and the fingertip action amplitude change information;
step S302, determining the stringed instrument playing chapters corresponding to the occurrence of the wrong playing motion according to the time points of the wrong playing motion, and marking the stringed instrument playing chapters as key attention playing chapters;
step S303, in the subsequent string instrument playing practice of students, when the key point is focused on playing chapters, feeding corresponding reminding information back to the students;
further, in step S301, according to the six-degree-of-freedom motion posture of the finger, determining fingertip orientation change information of the finger in a three-dimensional space during playing the stringed instrument by the student; determining fingertip action amplitude change information of the fingers in a three-dimensional space in the process of playing the stringed instrument by the student according to the action amplitude of the fingers; according to the fingertip orientation change information and the fingertip action amplitude change information, determining whether the student has an error playing action in the stringed instrument playing process specifically comprises the following steps:
step S3011, determining the fingertip coordinates of the fingers in a three-dimensional space during playing the stringed instrument by the student according to the six-degree-of-freedom action posture of the fingers by using the following formula (1); the six degrees of freedom of the finger refer to the degree of freedom of the finger moving in the left-right, front-back and up-down directions and the degree of freedom of the finger bending respectively by three joints, wherein the thumb is also regarded as comprising two actual joints and one virtual joint, the degree of freedom of the virtual joint bending is constantly equal to 0, and the distance between the virtual joint and one actual joint adjacent to the virtual joint is constantly equal to 0,
Figure BDA0003119954450000041
in the above formula (1), (X)0(t),Y0(t),Z0(t)) represents the fingertip coordinate of the finger in three-dimensional space at time t, X1(t) represents a moving distance value of the first joint of the finger in the left-right direction at time t, the moving distance value is negative when moving to the left, the moving distance value is positive when moving to the right, and Y is1(t) represents a moving distance value of the first joint of the finger in the front-rear direction at time t, the moving distance value being negative when moving backward and positive when moving forward, Z1(t) represents a moving distance value of the first joint of the finger in the up-down direction at time t, the moving distance value being negative when moving downward and positive when moving upward, L12Representing a value of a distance between a first joint and a second joint，L23Representing a value of the distance, L, between the second joint and the third joint30Representing the distance value between a third joint and the fingertip, wherein the first joint, the second joint and the third joint are close to the fingertip in sequence, alpha12Represents the degree of freedom, beta, of the first joint in the left-right direction with respect to the second joint12Representing a degree of freedom, gamma, of the first joint in a front-rear direction relative to the second joint12Representing a degree of freedom, alpha, of the first joint in the up-down direction with respect to the second joint23Represents the degree of freedom, β, of the second joint in the left-right direction with respect to the third joint23Representing a degree of freedom, gamma, of the second joint in the fore-aft direction relative to the third joint23Representing a degree of freedom, alpha, of the second joint in the up-down direction with respect to the third joint30Represents the degree of freedom, β, of the third joint in the left-right direction with respect to the fingertip30Representing the degree of freedom, gamma, of the third joint in the front-rear direction relative to the fingertips30Representing the degree of freedom of the third joint in the up-down direction with respect to the fingertip;
step S3012, determining the fingertip orientation vector coordinates of the student 'S finger in the three-dimensional space during playing the stringed instrument according to the fingertip coordinates of the student' S finger in the three-dimensional space during playing the stringed instrument by using the following formula (2),
Figure BDA0003119954450000051
in the above formula (2), (x (t), y (t), z (t)) represents the fingertip orientation vector coordinates of the finger in the three-dimensional space at time t;
step S3013, determining whether there is an erroneous playing motion of the student in playing the stringed instrument using the following formula (3),
Figure BDA0003119954450000052
in the above formula (3), η represents a value for evaluating whether there is an erroneous playing motion of the student during playing of the stringed instrument, D (T, T + T) represents a value for a variation in amplitude of a fingertip motion of the student from time T to time T + T during playing of the stringed instrument, D represents a minimum vibration amplitude value of a string in the stringed instrument, T represents a preset unit time, (X, Y, Z) represents a string extending direction vector of the stringed instrument, Φ [ ] represents a judgment function, when the conditional expression in the bracket [ ] is established, a result of the judgment function is 1, and when the conditional expression in the bracket [ ] is not established, a result of the judgment function is 1;
when eta is equal to-1, the student does not have the error of playing direction and plays the action of playing amplitude error in the process of playing the stringed instrument;
when eta is-2, the music instrument indicates that the student plays the string instrument in a wrong playing direction and in a wrong playing action of playing amplitude;
when eta is 1 or eta is 3, the method indicates that the student has no playing direction error and playing amplitude error playing action in the process of playing the stringed instrument;
when η is 2 or 4, it indicates that the student has an error in the playing direction and an error in the playing amplitude during playing the stringed instrument.
The invention also provides a string instrument playing teaching system based on video monitoring, which is characterized by comprising an instrument debugging judgment module, a playing video image acquisition and analysis module, a playing action analysis module and a reminding module; wherein the content of the first and second substances,
the musical instrument debugging judgment module is used for acquiring debugging audio correspondingly generated by students in the process of debugging stringed musical instruments before the students play the stringed musical instruments; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged;
the playing video image acquisition and analysis module is used for shooting video images of hands of students playing stringed instruments in the process of playing stringed instruments after confirming that the stringed instruments are successfully debugged; analyzing the video image, and determining the action posture and action amplitude of fingers of the student in the process of playing the stringed instrument;
the playing motion analysis module is used for determining that playing chapters corresponding to playing motion errors exist in the process of playing the stringed instrument by the students according to the motion postures and/or the motion amplitudes;
the reminding module is used for feeding back corresponding reminding information to the students in the subsequent string instrument playing practice according to the wrong playing action of the students in playing the playing chapters;
furthermore, the instrument debugging judgment module is used for acquiring debugging audio correspondingly generated by a student in the process of debugging the stringed instrument before the student plays the stringed instrument; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged specifically comprises:
acquiring debugging audio correspondingly generated by students in the debugging process of stringed instruments from a plurality of different azimuth angles before the students play the stringed instruments by using a microphone array;
performing Kalman filtering processing on the debugging audio so as to eliminate background noise existing in the debugging audio;
extracting corresponding sound waveform information from the debugging audio; analyzing the sound waveform information so as to determine whether the duration time that the actual sound frequency of the stringed instrument is greater than a preset sound frequency threshold value exceeds a preset time threshold value in the debugging process, and if not, determining that the stringed instrument is successfully debugged; if yes, determining that the stringed instrument fails to be debugged, and feeding back a prompt message for continuously debugging the stringed instrument to the student until the stringed instrument is successfully debugged;
further, the playing video image acquisition and analysis module is used for shooting video images of the string instrument played by hands of the students in the string instrument playing process after the string instrument is successfully debugged; analyzing the video image, and determining the action posture and action amplitude of the fingers of the student in the process of playing the stringed instrument specifically comprises the following steps:
after the fact that the stringed instrument is successfully debugged is determined, binocular shooting is carried out on the action of hands of the students for playing the stringed instrument in the process of playing the stringed instrument, and therefore corresponding binocular video images are obtained;
determining the image parallax corresponding to the binocular video images so as to generate a three-dimensional video image corresponding to the action of playing the stringed instrument by the hands;
extracting a plurality of frame images of different time periods in the process of playing the stringed instrument by the student from the three-dimensional video image; identifying the finger action state of the student in each frame of image, thereby determining the six-degree-of-freedom action posture of the finger and the action amplitude of the finger in the process of playing the stringed instrument by the student;
further, the playing motion analysis module is configured to determine, according to the motion posture and/or the motion amplitude, that a playing chapter corresponding to a playing motion error exists in the process of playing the stringed instrument by the student specifically includes:
determining fingertip orientation change information of fingers in a three-dimensional space in the process of playing the stringed instrument by a student according to the six-degree-of-freedom action posture of the fingers; determining fingertip action amplitude change information of the fingers in a three-dimensional space in the process of playing the stringed instrument by the student according to the action amplitude of the fingers; determining whether the student has wrong playing actions in the process of playing the stringed musical instrument according to the fingertip orientation change information and the fingertip action amplitude change information;
determining the stringed instrument playing chapters corresponding to the error playing action at the moment according to the time points of the error playing action, and marking the stringed instrument playing chapters as key attention playing chapters;
and the number of the first and second groups,
the reminding module is used for feeding back corresponding reminding messages to the students in the subsequent string instrument playing exercises according to the wrong playing actions of the students in playing the playing chapters, and specifically comprises the following steps:
and in the subsequent string instrument playing practice of the students, feeding back corresponding reminding messages to the students when the important attention is paid to the playing chapters.
Compared with the prior art, the string instrument playing teaching method and system based on video monitoring provided by the invention collect debugging audio correspondingly generated by students in the debugging process of the string instrument before playing the string instrument; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged; when the stringed instrument is successfully debugged, shooting a video image of a student playing the stringed instrument by hands in the process of playing the stringed instrument; analyzing the video image, and determining the action posture and action amplitude of fingers of the student in the process of playing the stringed instrument; finally, determining the playing chapters corresponding to playing motion errors in the process of playing the stringed instrument by the students according to the motion postures and/or the motion amplitudes; feeding back corresponding reminding information to the student in the subsequent stringed instrument playing practice according to the wrong playing action of the student in the playing performance chapter; therefore, the string instrument playing teaching method and system based on video monitoring can determine whether the student successfully masters the debugging skill of the instrument by acquiring and analyzing the debugging audio generated by the student in the process of debugging the string instrument; and the video images of the hands of the students playing the stringed instruments in the process of playing the stringed instruments are shot and analyzed, and whether wrong playing actions exist in each playing chapter is determined from the aspects of the action postures and the action amplitudes of the fingers of the students playing the stringed instruments, so that the reminding message of whether the playing actions are wrong is fed back to the students in time in the follow-up stringed instrument playing exercises, and therefore, comprehensive, automatic and accurate teaching experience can be provided for the students and the stringed instrument playing skills of the students can be improved to the maximum extent.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a stringed instrument playing teaching method based on video monitoring according to the present invention.
Fig. 2 is a schematic structural diagram of the string instrument playing teaching system based on video monitoring provided by the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a flow chart of a stringed instrument playing teaching method based on video monitoring according to an embodiment of the present invention is schematically shown. The invention provides a string instrument playing teaching method based on video monitoring, which comprises the following steps:
step S1, collecting debugging audio correspondingly generated by students in the process of debugging stringed instruments before playing the stringed instruments; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged;
step S2, after the stringed instrument is successfully debugged, shooting the video image of the stringed instrument played by the hands of the student in the process of playing the stringed instrument; analyzing the video image, and determining the action posture and action amplitude of the fingers of the student in the process of playing the stringed instrument;
step S3, determining the playing chapters corresponding to playing motion errors in the process of playing the stringed instrument by the student according to the motion posture and/or the motion amplitude; and feeding back corresponding reminding messages to the students in the subsequent string instrument playing practice according to the wrong playing actions of the students in playing the playing piece.
The beneficial effects of the above technical scheme are: the string instrument playing teaching method based on video monitoring is used for determining whether a student successfully masters the debugging skill of a musical instrument by acquiring and analyzing the debugging audio generated by the student in the process of debugging the string instrument; and the video images of the hands of the students playing the stringed instruments in the process of playing the stringed instruments are shot and analyzed, and whether wrong playing actions exist in each playing chapter is determined from the aspects of the action postures and the action amplitudes of the fingers of the students playing the stringed instruments, so that the reminding message of whether the playing actions are wrong is fed back to the students in time in the follow-up stringed instrument playing exercises, and therefore, comprehensive, automatic and accurate teaching experience can be provided for the students and the stringed instrument playing skills of the students can be improved to the maximum extent.
Preferably, in step S1, debugging audio generated by the student during the debugging process of the stringed instrument before playing the stringed instrument is collected; analyzing the tuning audio to determine whether the tuning of the stringed instrument was successful specifically includes:
step S101, collecting debugging audio correspondingly generated by students in the process of debugging stringed instruments before playing the stringed instruments from a plurality of different azimuth angles by using a microphone array;
step S102, Kalman filtering processing is carried out on the debugging audio, so that background noise existing in the debugging audio is removed;
step S103, extracting corresponding sound waveform information from the debugging audio; analyzing the sound waveform information to determine whether the duration of the actual sound frequency of the stringed instrument being greater than a preset sound frequency threshold exceeds a preset time threshold in the debugging process, and if not, determining that the stringed instrument is successfully debugged; if yes, determining that the stringed instrument fails to be debugged, and feeding back a prompt message for continuously debugging the stringed instrument to the student until the stringed instrument is successfully debugged.
The beneficial effects of the above technical scheme are: before being played, stringed instruments can be normally used through debugging operations such as string plucking, string tuning and the like, and if the stringed instruments are directly played without any debugging operation, the played music has the condition of poor tone or running. The method comprises the steps of collecting debugging audio in the string instrument debugging process by using a microphone array at different azimuth angles, carrying out filtering processing and sound waveform analysis processing on the debugging audio, and accurately determining the real-time sound frequency of the string instrument in the debugging process, so that whether the debugged string instrument can be normally played can be quickly determined according to the height and the duration of the real-time sound frequency, wherein when the duration of the actual sound frequency of the string instrument, which is greater than a preset sound frequency threshold, exceeds a preset time threshold in the debugging process, the fact that the music played by the string instrument has a running-tone condition is indicated, and the method is not suitable for formal playing and needs to be continuously debugged. Through the method for analyzing the debugging audio, misjudgment caused by artificial judgment of the stringed instrument debugging result can be effectively avoided, and the accuracy of accuracy judgment of the stringed instrument tone after debugging is greatly improved.
Preferably, in this step S2, after determining that the stringed instrument is successfully fitted, a video image of the student playing the stringed instrument with his hand during playing the stringed instrument is captured; analyzing the video image and determining the action posture and action amplitude of the fingers of the student in the process of playing the stringed instrument specifically comprises the following steps:
step S201, after the stringed instrument is successfully debugged, binocular shooting is carried out on the action of playing the stringed instrument by hands of the student in the process of playing the stringed instrument, so that corresponding binocular video images are obtained;
step S202, determining image parallax corresponding to the binocular video image, and generating a three-dimensional video image corresponding to the action of playing stringed instruments by hands;
step S203, extracting a plurality of frame images of different time periods in the process of playing the stringed instrument by the student from the three-dimensional video image; and identifying the finger action state of the student in each frame of image, thereby determining the six-degree-of-freedom action posture of the finger and the action amplitude of the finger in the process of playing the stringed instrument by the student.
The beneficial effects of the above technical scheme are: when the stringed instrument is successfully debugged, the stringed instrument can be used for normal playing, and the manual skill of the student playing the stringed instrument directly determines the playing quality. Through carrying out binocular shooting on the actions of playing the stringed instrument by hands of the student in the process of playing the stringed instrument, the hand action information of the student when playing the stringed instrument can be accurately and comprehensively obtained. And a plurality of frame images of different time periods in the process of playing the stringed instrument by the student are extracted from the three-dimensional video image and are identified, so that the frame-by-frame detailed analysis can be performed on the hand actions of the stringed instrument played by the student, namely, the action decomposition is performed on the whole set of hand actions of the stringed instrument played by the student, wherein the identification of the finger action state of the student in each frame of image can be realized by a common image pixel identification mode, and detailed description is not repeated here. In addition, the six-degree-of-freedom action posture of the fingers and the action amplitude of the fingers directly influence the vibration frequency and the vibration amplitude of the strings of the stringed instrument during the process of playing the stringed instrument by the student, and then influence the tone of the stringed instrument, so that the six-degree-of-freedom action posture of the fingers and the action amplitude of the fingers during the process of playing the stringed instrument by the student are identified and obtained from each frame of image, and reliable evidence can be provided for subsequently judging whether the action of playing the stringed instrument by the student is standard or not.
Preferably, in the step S3, it is determined that the playing chapter corresponding to the playing motion error exists in the process of playing the stringed instrument by the student according to the motion gesture and the motion amplitude; according to the wrong playing action of the student in playing the playing piece, the step of feeding back a corresponding reminding message to the student in the subsequent stringed instrument playing practice specifically comprises the following steps:
step S301, determining fingertip orientation change information of fingers in a three-dimensional space during playing of a stringed instrument by a student according to six-degree-of-freedom action posture of the fingers; determining fingertip action amplitude change information of the fingers in a three-dimensional space in the process of playing the stringed instrument by the student according to the action amplitude of the fingers; determining whether the student has wrong playing actions in the process of playing the stringed musical instrument according to the fingertip orientation change information and the fingertip action amplitude change information;
step S302, determining the stringed instrument playing chapter corresponding to the occurrence of the wrong playing motion according to the time point of the wrong playing motion, and marking the stringed instrument playing chapter as a key focused playing chapter;
step S303, in the subsequent string instrument playing practice of the students, when the key point is focused on playing the chapters, the corresponding reminding message is fed back to the students.
The beneficial effects of the above technical scheme are: when a specific song is played by using a stringed instrument, the hand movements of the student during playing (such as the orientation of the fingers and the action amplitude of the plucked strings of the fingers) all affect the playing effect. If the student has wrong playing actions in the playing process, the playing timbre and the like of the playing chapters of the corresponding tracks are greatly discounted, accordingly, the playing chapters which are problematic in the playing process of the real tracks can be accurately determined by determining the time point of the wrong playing actions, and the playing chapters of the stringed instrument are marked as key playing chapters which are concerned, so that the playing chapters corresponding to the unskilled playing actions of the student can be effectively identified and marked. Thus, in the subsequent string instrument playing practice of the student, when the important focus is paid to playing the chapter, the corresponding reminding message can be timely fed back to the student, wherein the reminding message can include but is not limited to reminding the student to change at least one of the direction, the strength and the frequency of the string poking of the string instrument by the finger in the string instrument playing process.
Preferably, in step S301, fingertip orientation change information of the finger in a three-dimensional space during playing the stringed instrument by the student is determined according to the six-degree-of-freedom motion posture of the finger; determining fingertip action amplitude change information of the fingers in a three-dimensional space in the process of playing the stringed instrument by the student according to the action amplitude of the fingers; determining whether the wrong playing action exists in the process of playing the stringed musical instrument by the student according to the fingertip orientation change information and the fingertip action amplitude change information specifically comprises the following steps:
step S3011, determining the fingertip coordinates of the fingers in a three-dimensional space during playing the stringed instrument by the student according to the six-degree-of-freedom action posture of the fingers by using the following formula (1); the six degrees of freedom of the finger refer to the degree of freedom of the finger moving in the left-right, front-back and up-down directions and the degree of freedom of the finger bending respectively by three joints, wherein the thumb is also regarded as comprising two actual joints and one virtual joint, the degree of freedom of the virtual joint bending is constantly equal to 0, and the distance between the virtual joint and one actual joint adjacent to the virtual joint is constantly equal to 0,
Figure BDA0003119954450000131
in the above formula (1), (X)0(t),Y0(t),Z0(t)) represents the fingertip coordinate of the finger in three-dimensional space at time t, X1(t) represents a moving distance value of the first joint of the finger in the left-right direction at time t, the moving distance value is negative when moving to the left, the moving distance value is positive when moving to the right, and Y is1(t) represents a moving distance value of the first joint of the finger in the front-rear direction at time t, the moving distance value being negative when moving backward and positive when moving forward, Z1(t) represents a moving distance value of the first joint of the finger in the up-down direction at time t, the moving distance value being negative when moving downward and positive when moving upward, L12Representing a value of the distance, L, between the first joint and the second joint23Representing a value of the distance, L, between the second joint and the third joint30Representing the distance value between a third joint and the fingertip, wherein the first joint, the second joint and the third joint are close to the fingertip in sequence, alpha12Represents the degree of freedom, beta, of the first joint in the left-right direction with respect to the second joint12Representing a degree of freedom, gamma, of the first joint in a front-rear direction relative to the second joint12Representing a degree of freedom, alpha, of the first joint in the up-down direction with respect to the second joint23Represents the degree of freedom, β, of the second joint in the left-right direction with respect to the third joint23Representing a degree of freedom, gamma, of the second joint in the fore-aft direction relative to the third joint23Representing a degree of freedom, alpha, of the second joint in the up-down direction with respect to the third joint30Represents the degree of freedom, β, of the third joint in the left-right direction with respect to the fingertip30Indicating that the third joint is in the front-rear direction relative to the fingertipsDegree of freedom, gamma30Representing the degree of freedom of the third joint in the up-down direction with respect to the fingertip; the degree of freedom refers to an angle capable of freely moving;
step S3012, determining the fingertip orientation vector coordinates of the student 'S finger in the three-dimensional space during playing the stringed instrument according to the fingertip coordinates of the student' S finger in the three-dimensional space during playing the stringed instrument by using the following formula (2),
Figure BDA0003119954450000141
in the above formula (2), (x (t), y (t), z (t)) represents the fingertip orientation vector coordinates of the finger in the three-dimensional space at time t;
step S3013, determining whether there is an erroneous playing motion of the student in playing the stringed instrument using the following formula (3),
Figure BDA0003119954450000142
in the above formula (3), η represents a value for evaluating whether there is an erroneous playing motion of the student during playing of the stringed instrument, D (T, T + T) represents a value for a variation in amplitude of a fingertip motion of the student from time T to time T + T during playing of the stringed instrument, D represents a minimum vibration amplitude value of a string in the stringed instrument, T represents a preset unit time, (X, Y, Z) represents a string extending direction vector of the stringed instrument, Φ [ ] represents a judgment function, when the conditional expression in the bracket [ ] is established, a result of the judgment function is 1, and when the conditional expression in the bracket [ ] is not established, a result of the judgment function is 1;
when eta is equal to-1, the student does not have the error of playing direction and plays the action of playing amplitude error in the process of playing the stringed instrument;
when eta is-2, the music instrument indicates that the student plays the string instrument in a wrong playing direction and in a wrong playing action of playing amplitude;
when eta is 1 or eta is 3, the method indicates that the student has no playing direction error and playing amplitude error playing action in the process of playing the stringed instrument;
when η is 2 or 4, it indicates that the student has an error in the playing direction and an error in the playing amplitude during playing the stringed instrument.
The beneficial effects of the above technical scheme are: determining the fingertip coordinates of the fingers in the three-dimensional space of the student in the string instrument playing process by using the formula (1) according to the six-degree-of-freedom action posture of the fingers, so that the positions of the fingertips in the three-dimensional space can be accurately determined according to the fingertip coordinates of the fingers; then, obtaining a fingertip orientation vector coordinate of the student in the three-dimensional space according to the fingertip coordinate of the student in the three-dimensional space in the process of playing the stringed instrument by using the formula (2), and further judging the moving direction of the student fingers in the process of playing the stringed instrument by the orientation of the fingers; and finally, determining the wrong playing action of the student in the string instrument playing process according to the fingertip orientation vector coordinate of the finger of the student in the three-dimensional space and the fingertip action amplitude in the string instrument playing process by using the formula (3), and further determining the wrong playing action of the student in the string instrument playing process, so that the student can correct the wrong playing action in time.
Fig. 2 is a schematic structural diagram of a string instrument playing teaching system based on video monitoring according to an embodiment of the present invention. The invention provides a string instrument playing teaching system based on video monitoring, which comprises an instrument debugging judgment module, a playing video image acquisition and analysis module, a playing action analysis module and a reminding module; wherein the content of the first and second substances,
the instrument debugging judgment module is used for acquiring debugging audio correspondingly generated by students in the process of debugging stringed instruments before playing the stringed instruments; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged;
the playing video image acquisition and analysis module is used for shooting video images of hands of students playing stringed instruments in the process of playing stringed instruments after confirming that the stringed instruments are successfully debugged; analyzing the video image, and determining the action posture and action amplitude of the fingers of the student in the process of playing the stringed instrument;
the playing motion analysis module is used for determining that playing chapters corresponding to playing motion errors exist in the process of playing the stringed instrument by the students according to the motion postures and/or the motion amplitudes;
the reminding module is used for feeding back corresponding reminding information to students in subsequent string instrument playing exercises according to wrong playing actions of the students in playing the playing chapters.
The beneficial effects of the above technical scheme are: the string instrument playing teaching system based on video monitoring determines whether a student successfully masters the debugging skill of the instrument by acquiring and analyzing the debugging audio generated by the student in the process of debugging the string instrument; and the video images of the hands of the students playing the stringed instruments in the process of playing the stringed instruments are shot and analyzed, and whether wrong playing actions exist in each playing chapter is determined from the aspects of the action postures and the action amplitudes of the fingers of the students playing the stringed instruments, so that the reminding message of whether the playing actions are wrong is fed back to the students in time in the follow-up stringed instrument playing exercises, and therefore, comprehensive, automatic and accurate teaching experience can be provided for the students and the stringed instrument playing skills of the students can be improved to the maximum extent.
Preferably, the instrument debugging judgment module is used for acquiring debugging audio correspondingly generated by a student in the process of debugging the stringed instrument before the student plays the stringed instrument; analyzing the tuning audio to determine whether the tuning of the stringed instrument was successful specifically includes:
acquiring debugging audio correspondingly generated by students in the debugging process of stringed instruments from a plurality of different azimuth angles before the students play the stringed instruments by using a microphone array;
performing Kalman filtering processing on the debugging audio so as to eliminate background noise existing in the debugging audio;
extracting corresponding sound waveform information from the debugging audio; analyzing the sound waveform information to determine whether the duration of the actual sound frequency of the stringed instrument being greater than a preset sound frequency threshold exceeds a preset time threshold in the debugging process, and if not, determining that the stringed instrument is successfully debugged; if yes, determining that the stringed instrument fails to be debugged, and feeding back a prompt message for continuously debugging the stringed instrument to the student until the stringed instrument is successfully debugged.
The beneficial effects of the above technical scheme are: before being played, stringed instruments can be normally used through debugging operations such as string plucking, string tuning and the like, and if the stringed instruments are directly played without any debugging operation, the played music has the condition of poor tone or running. The method comprises the steps of collecting debugging audio in the string instrument debugging process by using a microphone array at different azimuth angles, carrying out filtering processing and sound waveform analysis processing on the debugging audio, and accurately determining the real-time sound frequency of the string instrument in the debugging process, so that whether the debugged string instrument can be normally played can be quickly determined according to the height and the duration of the real-time sound frequency, wherein when the duration of the actual sound frequency of the string instrument, which is greater than a preset sound frequency threshold, exceeds a preset time threshold in the debugging process, the fact that the music played by the string instrument has a running-tone condition is indicated, and the method is not suitable for formal playing and needs to be continuously debugged. Through the method for analyzing the debugging audio, misjudgment caused by artificial judgment of the stringed instrument debugging result can be effectively avoided, and the accuracy of accuracy judgment of the stringed instrument tone after debugging is greatly improved.
Preferably, the playing video image acquisition and analysis module is used for shooting video images of the string instrument played by hands of the student in the string instrument playing process after the string instrument is successfully debugged; analyzing the video image and determining the action posture and action amplitude of the fingers of the student in the process of playing the stringed instrument specifically comprises the following steps:
after the fact that the stringed instrument is successfully debugged is determined, binocular shooting is carried out on the action of hands of the students for playing the stringed instrument in the process of playing the stringed instrument, and therefore corresponding binocular video images are obtained;
determining the image parallax corresponding to the binocular video images so as to generate a three-dimensional video image corresponding to the action of playing the stringed instrument by the hands;
extracting a plurality of frame images of different time periods in the process of playing the stringed instrument by the student from the three-dimensional video image; and identifying the finger action state of the student in each frame of image, thereby determining the six-degree-of-freedom action posture of the finger and the action amplitude of the finger in the process of playing the stringed instrument by the student.
The beneficial effects of the above technical scheme are: when the stringed instrument is successfully debugged, the stringed instrument can be used for normal playing, and the manual skill of the student playing the stringed instrument directly determines the playing quality. Through carrying out binocular shooting on the actions of playing the stringed instrument by hands of the student in the process of playing the stringed instrument, the hand action information of the student when playing the stringed instrument can be accurately and comprehensively obtained. And a plurality of frame images of different time periods in the process of playing the stringed instrument by the student are extracted from the three-dimensional video image and are identified, so that the frame-by-frame detailed analysis can be performed on the hand actions of the stringed instrument played by the student, namely, the action decomposition is performed on the whole set of hand actions of the stringed instrument played by the student, wherein the identification of the finger action state of the student in each frame of image can be realized by a common image pixel identification mode, and detailed description is not repeated here. In addition, the six-degree-of-freedom action posture of the fingers and the action amplitude of the fingers directly influence the vibration frequency and the vibration amplitude of the strings of the stringed instrument during the process of playing the stringed instrument by the student, and then influence the tone of the stringed instrument, so that the six-degree-of-freedom action posture of the fingers and the action amplitude of the fingers during the process of playing the stringed instrument by the student are identified and obtained from each frame of image, and reliable evidence can be provided for subsequently judging whether the action of playing the stringed instrument by the student is standard or not.
Preferably, the playing motion analysis module is configured to determine, according to the motion posture and/or the motion amplitude, that the playing piece corresponding to the playing motion error exists in the process of playing the stringed instrument by the student specifically includes:
determining fingertip orientation change information of fingers in a three-dimensional space in the process of playing the stringed instrument by a student according to the six-degree-of-freedom action posture of the fingers; determining fingertip action amplitude change information of the fingers in a three-dimensional space in the process of playing the stringed instrument by the student according to the action amplitude of the fingers; determining whether the student has wrong playing actions in the process of playing the stringed musical instrument according to the fingertip orientation change information and the fingertip action amplitude change information;
determining the stringed instrument playing chapter corresponding to the error playing action at present according to the time point of the error playing action, and marking the stringed instrument playing chapter as a key focused playing chapter;
and the number of the first and second groups,
the reminding module is used for feeding back corresponding reminding messages to the students in the subsequent string instrument playing practice according to the wrong playing actions of the students in playing the playing piece, and specifically comprises the following steps:
and in the subsequent string instrument playing practice of the students, feeding back corresponding reminding messages to the students when the key points focus on playing the chapters.
The beneficial effects of the above technical scheme are: when a specific song is played by using a stringed instrument, the hand movements of the student during playing (such as the orientation of the fingers and the action amplitude of the plucked strings of the fingers) all affect the playing effect. If the student has wrong playing actions in the playing process, the playing timbre and the like of the playing chapters of the corresponding tracks are greatly discounted, accordingly, the playing chapters which are problematic in the playing process of the real tracks can be accurately determined by determining the time point of the wrong playing actions, and the playing chapters of the stringed instrument are marked as key playing chapters which are concerned, so that the playing chapters corresponding to the unskilled playing actions of the student can be effectively identified and marked. Thus, in the subsequent string instrument playing practice of the student, when the important focus is paid to playing the chapter, the corresponding reminding message can be timely fed back to the student, wherein the reminding message can include but is not limited to reminding the student to change at least one of the direction, the strength and the frequency of the string poking of the string instrument by the finger in the string instrument playing process.
From the content of the above embodiment, the string instrument playing teaching method and system based on video monitoring determines whether the student successfully grasps the debugging skill of the instrument by collecting and analyzing the debugging audio generated by the student in the process of debugging the string instrument; and the video images of the hands of the students playing the stringed instruments in the process of playing the stringed instruments are shot and analyzed, and whether wrong playing actions exist in each playing chapter is determined from the aspects of the action postures and the action amplitudes of the fingers of the students playing the stringed instruments, so that the reminding message of whether the playing actions are wrong is fed back to the students in time in the follow-up stringed instrument playing exercises, and therefore, comprehensive, automatic and accurate teaching experience can be provided for the students and the stringed instrument playing skills of the students can be improved to the maximum extent.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims (9)

1. The string instrument playing teaching method based on video monitoring is characterized by comprising the following steps:
step S1, collecting debugging audio correspondingly generated by students in the process of debugging stringed instruments before playing the stringed instruments; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged;
step S2, after the stringed instrument is successfully debugged, shooting the video image of the stringed instrument played by the hands of the student in the process of playing the stringed instrument; analyzing the video image, and determining the action posture and action amplitude of fingers of the student in the process of playing the stringed instrument;
step S3, determining that playing chapters corresponding to playing motion errors exist in the process of playing the stringed instrument by the students according to the motion postures and/or the motion amplitudes; and feeding back corresponding reminding messages to the students in the subsequent string instrument playing exercises according to the wrong playing actions of the students in playing the playing chapters.
2. The video surveillance-based stringed musical instrument playing teaching method of claim 1, wherein:
in step S1, acquiring debugging audio generated by the student in the process of debugging the stringed instrument before playing the stringed instrument; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged specifically comprises:
step S101, collecting debugging audio correspondingly generated by students in the process of debugging stringed instruments before playing the stringed instruments from a plurality of different azimuth angles by using a microphone array;
step S102, performing Kalman filtering processing on the debugging audio, so as to eliminate background noise existing in the debugging audio;
step S103, extracting corresponding sound waveform information from the debugging audio; analyzing the sound waveform information so as to determine whether the duration time that the actual sound frequency of the stringed instrument is greater than a preset sound frequency threshold value exceeds a preset time threshold value in the debugging process, and if not, determining that the stringed instrument is successfully debugged; if yes, determining that the stringed instrument fails to be debugged, and feeding back a prompt message for continuously debugging the stringed instrument to the student until the stringed instrument is successfully debugged.
3. The video surveillance-based stringed musical instrument playing teaching method of claim 1, wherein:
in step S2, when it is determined that the stringed instrument is successfully fitted, capturing a video image of the student playing the stringed instrument with his hand during playing the stringed instrument; analyzing the video image, and determining the action posture and action amplitude of the fingers of the student in the process of playing the stringed instrument specifically comprises the following steps:
step S201, after the stringed instrument is successfully debugged, binocular shooting is carried out on the action of playing the stringed instrument by hands of the student in the process of playing the stringed instrument, so that corresponding binocular video images are obtained;
step S202, determining image parallax corresponding to the binocular video image, and generating a three-dimensional video image corresponding to the action of playing stringed instruments by hands;
step S203, extracting a plurality of frame images of different time periods in the process of playing the stringed instrument by the student from the three-dimensional video image; and identifying the finger action state of the student in each frame of image, thereby determining the six-degree-of-freedom action posture of the finger and the action amplitude of the finger in the process of playing the stringed instrument by the student.
4. The video surveillance-based stringed musical instrument playing teaching method of claim 3, wherein:
in the step S3, determining that there is a playing chapter corresponding to the playing motion error during the playing of the stringed instrument by the student according to the motion gesture and the motion amplitude; according to the wrong playing action of the student in playing the playing piece, the step of feeding back a corresponding reminding message to the student in the subsequent stringed instrument playing practice specifically comprises the following steps:
step S301, determining fingertip orientation change information of fingers in a three-dimensional space during playing of a stringed instrument by a student according to six-degree-of-freedom action posture of the fingers; determining fingertip action amplitude change information of the fingers in a three-dimensional space in the process of playing the stringed instrument by the student according to the action amplitude of the fingers; determining whether the student has wrong playing actions in the process of playing the stringed musical instrument according to the fingertip orientation change information and the fingertip action amplitude change information;
step S302, determining the stringed instrument playing chapters corresponding to the occurrence of the wrong playing motion according to the time points of the wrong playing motion, and marking the stringed instrument playing chapters as key attention playing chapters;
step S303, in the subsequent string instrument playing practice of the students, when the key point attention playing chapters are played, the corresponding reminding messages are fed back to the students.
5. The video surveillance-based stringed musical instrument playing teaching method of claim 4, wherein:
in step S301, according to the six-degree-of-freedom motion posture of the finger, fingertip orientation change information of the finger in a three-dimensional space during playing of the stringed instrument by the student is determined; determining fingertip action amplitude change information of the fingers in a three-dimensional space in the process of playing the stringed instrument by the student according to the action amplitude of the fingers; according to the fingertip orientation change information and the fingertip action amplitude change information, determining whether the student has an error playing action in the stringed instrument playing process specifically comprises the following steps:
step S3011, determining the fingertip coordinates of the fingers in a three-dimensional space during playing the stringed instrument by the student according to the six-degree-of-freedom action posture of the fingers by using the following formula (1); the six degrees of freedom of the finger refer to the degree of freedom of the finger moving in the left-right, front-back and up-down directions and the degree of freedom of the finger bending respectively by three joints, wherein the thumb is also regarded as comprising two actual joints and one virtual joint, the degree of freedom of the virtual joint bending is constantly equal to 0, and the distance between the virtual joint and one actual joint adjacent to the virtual joint is constantly equal to 0,
Figure FDA0003119954440000031
in the above formula (1), (X)0(t),Y0(t),Z0(t)) represents the fingertip coordinate of the finger in three-dimensional space at time t, X1(t) represents a moving distance value of the first joint of the finger in the left-right direction at time t, the moving distance value is negative when moving to the left, the moving distance value is positive when moving to the right, and Y is1(t) represents a moving distance value of the first joint of the finger in the front-rear direction at time t, the moving distance value being negative when moving backward and positive when moving forward, Z1(t) represents a moving distance value of the first joint of the finger in the up-down direction at time t, the moving distance value being negative when moving downward and positive when moving upward, L12Representing a value of the distance, L, between the first joint and the second joint23Representing a value of the distance, L, between the second joint and the third joint30Is shown asThe distance value between the three joints and the fingertip, wherein the first joint, the second joint and the third joint are close to the fingertip in sequence, alpha12Represents the degree of freedom, beta, of the first joint in the left-right direction with respect to the second joint12Representing a degree of freedom, gamma, of the first joint in a front-rear direction relative to the second joint12Representing a degree of freedom, alpha, of the first joint in the up-down direction with respect to the second joint23Represents the degree of freedom, β, of the second joint in the left-right direction with respect to the third joint23Representing a degree of freedom, gamma, of the second joint in the fore-aft direction relative to the third joint23Representing a degree of freedom, alpha, of the second joint in the up-down direction with respect to the third joint30Represents the degree of freedom, β, of the third joint in the left-right direction with respect to the fingertip30Representing the degree of freedom, gamma, of the third joint in the front-rear direction relative to the fingertips30Representing the degree of freedom of the third joint in the up-down direction with respect to the fingertip;
step S3012, determining the fingertip orientation vector coordinates of the student 'S finger in the three-dimensional space during playing the stringed instrument according to the fingertip coordinates of the student' S finger in the three-dimensional space during playing the stringed instrument by using the following formula (2),
Figure FDA0003119954440000041
in the above formula (2), (x (t), y (t), z (t)) represents the fingertip orientation vector coordinates of the finger in the three-dimensional space at time t;
step S3013, determining whether there is an erroneous playing motion of the student in playing the stringed instrument using the following formula (3),
Figure FDA0003119954440000051
in the above formula (3), η represents a value for evaluating whether there is an erroneous playing motion of the student during playing of the stringed instrument, D (T, T + T) represents a value for a variation in amplitude of a fingertip motion of the student from time T to time T + T during playing of the stringed instrument, D represents a minimum vibration amplitude value of a string in the stringed instrument, T represents a preset unit time, (X, Y, Z) represents a string extending direction vector of the stringed instrument, Φ [ ] represents a judgment function, when the conditional expression in the bracket [ ] is established, a result of the judgment function is 1, and when the conditional expression in the bracket [ ] is not established, a result of the judgment function is 1;
when eta is equal to-1, the student does not have the error of playing direction and plays the action of playing amplitude error in the process of playing the stringed instrument;
when eta is-2, the music instrument indicates that the student plays the string instrument in a wrong playing direction and in a wrong playing action of playing amplitude;
when eta is 1 or eta is 3, the method indicates that the student has no playing direction error and playing amplitude error playing action in the process of playing the stringed instrument;
when η is 2 or 4, it indicates that the student has an error in the playing direction and an error in the playing amplitude during playing the stringed instrument.
6. The string instrument playing teaching system based on video monitoring is characterized by comprising an instrument debugging judgment module, a playing video image acquisition and analysis module, a playing action analysis module and a reminding module; wherein the content of the first and second substances,
the musical instrument debugging judgment module is used for acquiring debugging audio correspondingly generated by students in the process of debugging stringed musical instruments before the students play the stringed musical instruments; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged;
the playing video image acquisition and analysis module is used for shooting video images of hands of students playing stringed instruments in the process of playing stringed instruments after confirming that the stringed instruments are successfully debugged; analyzing the video image, and determining the action posture and action amplitude of fingers of the student in the process of playing the stringed instrument; the playing motion analysis module is used for determining that playing chapters corresponding to playing motion errors exist in the process of playing the stringed instrument by the students according to the motion postures and/or the motion amplitudes; the reminding module is used for feeding back corresponding reminding information to the students in the subsequent string instrument playing practice according to the wrong playing action of the students in playing the playing chapters.
7. The video surveillance-based stringed musical instrument playing teaching system of claim 6, wherein:
the musical instrument debugging judgment module is used for acquiring debugging audio correspondingly generated by students in the process of debugging stringed musical instruments before the students play the stringed musical instruments; analyzing the debugging audio to determine whether the stringed instrument is successfully debugged specifically comprises:
acquiring debugging audio correspondingly generated by students in the debugging process of stringed instruments from a plurality of different azimuth angles before the students play the stringed instruments by using a microphone array;
performing Kalman filtering processing on the debugging audio so as to eliminate background noise existing in the debugging audio;
extracting corresponding sound waveform information from the debugging audio; analyzing the sound waveform information so as to determine whether the duration time that the actual sound frequency of the stringed instrument is greater than a preset sound frequency threshold value exceeds a preset time threshold value in the debugging process, and if not, determining that the stringed instrument is successfully debugged; if yes, determining that the stringed instrument fails to be debugged, and feeding back a prompt message for continuously debugging the stringed instrument to the student until the stringed instrument is successfully debugged.
8. The video surveillance-based stringed musical instrument playing teaching system of claim 6, wherein:
the playing video image acquisition and analysis module is used for shooting video images of hands of students playing stringed instruments in the process of playing stringed instruments after confirming that the stringed instruments are successfully debugged; analyzing the video image, and determining the action posture and action amplitude of the fingers of the student in the process of playing the stringed instrument specifically comprises the following steps:
after the fact that the stringed instrument is successfully debugged is determined, binocular shooting is carried out on the action of hands of the students for playing the stringed instrument in the process of playing the stringed instrument, and therefore corresponding binocular video images are obtained;
determining the image parallax corresponding to the binocular video images so as to generate a three-dimensional video image corresponding to the action of playing the stringed instrument by the hands;
extracting a plurality of frame images of different time periods in the process of playing the stringed instrument by the student from the three-dimensional video image; and identifying the finger action state of the student in each frame of image, thereby determining the six-degree-of-freedom action posture of the finger and the action amplitude of the finger in the process of playing the stringed instrument by the student.
9. The video surveillance-based stringed musical instrument playing teaching system of claim 8, wherein:
the playing motion analysis module is configured to determine, according to the motion posture and/or the motion amplitude, that a playing chapter corresponding to a playing motion error exists in the process of playing the stringed instrument by the student, and specifically includes:
determining fingertip orientation change information of fingers in a three-dimensional space in the process of playing the stringed instrument by a student according to the six-degree-of-freedom action posture of the fingers; determining fingertip action amplitude change information of the fingers in a three-dimensional space in the process of playing the stringed instrument by the student according to the action amplitude of the fingers; determining whether the student has wrong playing actions in the process of playing the stringed musical instrument according to the fingertip orientation change information and the fingertip action amplitude change information;
determining the stringed instrument playing chapters corresponding to the error playing action at the moment according to the time points of the error playing action, and marking the stringed instrument playing chapters as key attention playing chapters;
and the number of the first and second groups,
the reminding module is used for feeding back corresponding reminding messages to the students in the subsequent string instrument playing exercises according to the wrong playing actions of the students in playing the playing chapters, and specifically comprises the following steps: and in the subsequent string instrument playing practice of the students, feeding back corresponding reminding messages to the students when the important attention is paid to the playing chapters.
CN202110673924.8A 2021-06-17 2021-06-17 String instrument playing teaching method and system based on video monitoring Pending CN113327482A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110673924.8A CN113327482A (en) 2021-06-17 2021-06-17 String instrument playing teaching method and system based on video monitoring

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110673924.8A CN113327482A (en) 2021-06-17 2021-06-17 String instrument playing teaching method and system based on video monitoring

Publications (1)

Publication Number Publication Date
CN113327482A true CN113327482A (en) 2021-08-31

Family

ID=77423721

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110673924.8A Pending CN113327482A (en) 2021-06-17 2021-06-17 String instrument playing teaching method and system based on video monitoring

Country Status (1)

Country Link
CN (1) CN113327482A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114783222A (en) * 2022-05-11 2022-07-22 张�浩 Musical instrument auxiliary teaching method and system based on AR technology

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105575220A (en) * 2016-03-22 2016-05-11 深圳视感文化科技有限公司 Interactive stringed instrument teaching system
JP2016194629A (en) * 2015-04-01 2016-11-17 ブラザー工業株式会社 Plucked string instrument performance evaluation device, music performance device, and plucked string instrument performance evaluation program
CN106531184A (en) * 2016-09-23 2017-03-22 南京凡豆信息科技有限公司 Intelligent hardware piano music finishing training guidance method based on melody characteristic matching
US9613541B1 (en) * 2016-01-12 2017-04-04 Dylan J. Badilla-Bradford Method and apparatus for learning to play a stringed instrument
CN206849295U (en) * 2017-04-27 2018-01-05 邵俊夫 Intelligent Piano Teaching assistor
CN109887375A (en) * 2019-04-17 2019-06-14 西安邮电大学 Piano practice error correction method based on image recognition processing
CN110796918A (en) * 2019-10-29 2020-02-14 维沃移动通信有限公司 Training method and device and mobile terminal
CN111563446A (en) * 2020-04-30 2020-08-21 郑州轻工业大学 Human-machine interaction safety early warning and control method based on digital twin
CN111932981A (en) * 2020-07-01 2020-11-13 江西昕光年智能科技有限公司 Violin for teaching and control method thereof
CN112883804A (en) * 2021-01-21 2021-06-01 小叶子（北京）科技有限公司 Error correction method and device for hand motion during musical instrument playing and electronic equipment

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016194629A (en) * 2015-04-01 2016-11-17 ブラザー工業株式会社 Plucked string instrument performance evaluation device, music performance device, and plucked string instrument performance evaluation program
US9613541B1 (en) * 2016-01-12 2017-04-04 Dylan J. Badilla-Bradford Method and apparatus for learning to play a stringed instrument
CN105575220A (en) * 2016-03-22 2016-05-11 深圳视感文化科技有限公司 Interactive stringed instrument teaching system
CN106531184A (en) * 2016-09-23 2017-03-22 南京凡豆信息科技有限公司 Intelligent hardware piano music finishing training guidance method based on melody characteristic matching
CN206849295U (en) * 2017-04-27 2018-01-05 邵俊夫 Intelligent Piano Teaching assistor
CN109887375A (en) * 2019-04-17 2019-06-14 西安邮电大学 Piano practice error correction method based on image recognition processing
CN110796918A (en) * 2019-10-29 2020-02-14 维沃移动通信有限公司 Training method and device and mobile terminal
CN111563446A (en) * 2020-04-30 2020-08-21 郑州轻工业大学 Human-machine interaction safety early warning and control method based on digital twin
CN111932981A (en) * 2020-07-01 2020-11-13 江西昕光年智能科技有限公司 Violin for teaching and control method thereof
CN112883804A (en) * 2021-01-21 2021-06-01 小叶子（北京）科技有限公司 Error correction method and device for hand motion during musical instrument playing and electronic equipment

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114783222A (en) * 2022-05-11 2022-07-22 张�浩 Musical instrument auxiliary teaching method and system based on AR technology
CN114783222B (en) * 2022-05-11 2022-12-20 张�浩 Musical instrument auxiliary teaching method and system based on AR technology

Similar Documents

Publication Publication Date Title
Yin et al. Digital violin tutor: an integrated system for beginning violin learners
CN112908355B (en) System and method for quantitatively evaluating teaching skills of teacher and teacher
Johnson et al. Detecting hand posture in piano playing using depth data
US20150242797A1 (en) Methods and systems for evaluating performance
Dalmazzo et al. Air violin: a machine learning approach to fingering gesture recognition
CN113674565B (en) Teaching system and method for piano teaching
CN101950495A (en) Novel video educational technology
CN115082266B (en) Comprehensive development analysis and evaluation system for student education subjects
CN105513580A (en) Keyboard musical instrument playing note identification system based on auxiliary camera
CN113327482A (en) String instrument playing teaching method and system based on video monitoring
CN107945257A (en) A kind of method for testing practice scene reduction
US11823496B2 (en) Professional dance evaluation method for implementing human pose estimation based on deep transfer learning
US20220143484A1 (en) Scoring system of automatically detecting body motion, scoring method of automatically detecting body motion, and non-transitory computer-readable storage medium
CN110991943B (en) Teaching quality evaluation system based on cloud computing
CN114170868A (en) Intelligent piano training method and system
CN112836945A (en) Teaching state quantitative evaluation system for teaching and teaching of professor
CN109344794B (en) Piano playing scoring method and device and computer storage medium
Wang et al. Educational violin transcription by fusing multimedia streams
CN113255470B (en) Multi-mode piano accompany training system and method based on hand gesture estimation
CN114783222B (en) Musical instrument auxiliary teaching method and system based on AR technology
Kikukawa et al. Development of A Learning Environment for Playing Erhu by Diagnosis and Advice regarding Finger Position on Strings.
Kashiwagi et al. A Study of Left Fingering Detection Using CNN for Guitar Learning
Ng Interactive multimedia for technology-enhanced learning with multimodal feedback
CN112016431A (en) Intelligent detection and analysis method and system for teaching quality
CN111311981A (en) Intelligent classroom that multidata supported

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination