CN113253837A - Air writing method and device, online live broadcast system and computer equipment - Google Patents
Air writing method and device, online live broadcast system and computer equipment Download PDFInfo
- Publication number
- CN113253837A CN113253837A CN202110354090.4A CN202110354090A CN113253837A CN 113253837 A CN113253837 A CN 113253837A CN 202110354090 A CN202110354090 A CN 202110354090A CN 113253837 A CN113253837 A CN 113253837A
- Authority
- CN
- China
- Prior art keywords
- writing
- user
- hand shape
- real
- distance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 86
- 238000012544 monitoring process Methods 0.000 claims abstract description 34
- 230000003993 interaction Effects 0.000 claims abstract description 28
- 238000004590 computer program Methods 0.000 claims description 16
- 230000033001 locomotion Effects 0.000 claims description 14
- 238000001514 detection method Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 11
- 230000037237 body shape Effects 0.000 claims description 7
- 238000010801 machine learning Methods 0.000 claims description 6
- 230000008878 coupling Effects 0.000 claims 1
- 238000010168 coupling process Methods 0.000 claims 1
- 238000005859 coupling reaction Methods 0.000 claims 1
- 238000012545 processing Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 7
- 238000012549 training Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 3
- 230000000644 propagated effect Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000009365 direct transmission Effects 0.000 description 1
- 230000004424 eye movement Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000003238 somatosensory effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/20—Education
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Tourism & Hospitality (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Educational Administration (AREA)
- Educational Technology (AREA)
- Evolutionary Computation (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention belongs to the field of education, and provides an air writing method, an online live broadcast system, an online live broadcast device and computer equipment, wherein the method comprises the following steps: monitoring the distance from a user to the camera device; when the monitored distance between the user and the camera device is within a preset distance range, detecting a real-time image of the user to monitor the hand shape of the user; when the hand shape of the user is monitored to be a writing hand shape, detecting the writing hand shape to identify a writing key point, and recording the position of the writing key point; and generating a writing track according to the positions of the writing key points of the plurality of time points. The method can improve the speed and the accuracy of writing identification, improve the use experience of the user and further optimize the air writing method in the online live course interaction.
Description
Technical Field
The invention belongs to the field of education, is particularly suitable for the field of online education, and more particularly relates to an aerial writing method, an aerial writing device, an online live broadcast system and computer equipment.
Background
In recent years, networks are increasingly popularized, live network course broadcasting becomes one of the latest development trends and research hotspots in modern remote education research and application, and the combination of network education with real-time interaction enables the network education to step into a new stage. The current online education mode is mostly online video playing courses and online video interaction courses and the like.
In fact, online video interactive courses involve a variety of interactive modes such as voice interaction, somatosensory interaction, eye movement interaction, and gesture interaction. For a gesture interaction mode, gesture recognition is required in the process of finishing gesture interaction, and the existing methods include a method for recognizing based on an image and a method for recognizing by using a gesture sensor. However, there is still much room for improvement in recognition accuracy, data processing speed, complexity of the calculation process, and user experience.
Therefore, there is a need to provide a method of over the air writing to solve the above problems.
Disclosure of Invention
Technical problem to be solved
The invention aims to solve the technical problems of low gesture interaction identification precision, low data processing speed, poor user experience and the like in online video interaction courses.
(II) technical scheme
In order to solve the above technical problem, an aspect of the present invention provides an over-the-air writing method for on-line live broadcasting, including the steps of: monitoring the distance from a user to the camera device; when the monitored distance between the user and the camera device is within a preset distance range, detecting a real-time image of the user to monitor the hand shape of the user; when the hand shape of the user is monitored to be a writing hand shape, detecting the writing hand shape to identify a writing key point, and recording the position of the writing key point; and generating a writing track according to the positions of the writing key points of the plurality of time points.
Optionally, the monitoring the distance from the user to the camera device includes: and detecting the real-time image of the user in real time to monitor the distance from the user to the camera device. Further optionally, the detecting the real-time image of the user in real time to monitor the distance from the user to the camera device includes: and detecting the area ratio of the body shape of the user in the real-time image, and calculating the distance from the user to the camera device according to the area ratio.
According to a preferred embodiment of the present invention, the detecting the real-time image of the user to monitor the hand shape of the user comprises: detecting whether the real-time image of the user contains the predefined writing hand shape by using a first image recognition model, wherein the image recognition model is a machine learning model which is trained and can recognize the writing hand shape of the human hand, and starting a detection process of writing key points when the real-time image is detected to contain the predefined writing hand shape.
According to a preferred embodiment of the present invention, the writing key point detection process includes: the writing keypoints are detected using a second image recognition model, which is a machine learning model trained to recognize writing keypoints in a writing hand shape.
According to a preferred embodiment of the present invention, the predefined writing hand shape is a posture in which the index finger is extended to hold the other fingers together, and the writing key point of the hand shape is the index finger tip.
According to a preferred embodiment of the present invention, the generating a writing trajectory according to the positions of the writing key points of the plurality of time points comprises: performing straight line or curve fitting on the positions of the writing key points of the multiple time points to generate the motion trail; and generating a writing track according to the motion track.
According to an alternative embodiment of the invention, a writing trace is generated as the user writes at the writing end and transmitted in real time to the opposite end interacting with the writing end.
According to an alternative embodiment of the invention, further comprising: the writing track generated at the writing end is coupled into the video stream and displayed or hidden in the user video stream according to user instructions at the writing end or the opposite end.
According to an alternative embodiment of the invention, further comprising: displaying the writing trajectory in a writing end and/or an opposite end.
A second aspect of the present invention provides an online live broadcasting system, including: a writing end and an opposite end; at least one server for interaction between a writing end and an opposite end during an online live class session, and the writing end or the at least one server generates a writing trajectory using the over-the-air writing method of any one of the above; the at least one server transmits the generated writing track to the opposite end in a video streaming mode, and displays the writing track in the writing end and/or the opposite end.
A third aspect of the present invention provides an aerial writing device for an online live class, the aerial writing device comprising: the first monitoring module is used for monitoring the distance from a user to the camera device; the second monitoring module is used for detecting the real-time image of the user to monitor the hand shape of the user when the monitored distance between the user and the camera device is within a preset distance range; the third monitoring module starts to detect the writing hand shape to identify a writing key point when the hand shape of the first user is monitored to be the writing hand shape; and the generating module is used for generating a writing track according to the positions of the writing key points of the plurality of time points.
A fourth aspect of the present invention provides a computer device, comprising a processor and a memory, wherein the memory is configured to store a computer executable program, and when the computer program is executed by the processor, the processor executes the air writing method of an online live course according to the present invention.
A fifth aspect of the present invention provides a computer program product comprising computer programs/instructions, characterized in that said computer programs/instructions, when executed by a processor, implement the method for over-the-air writing of an online live lesson according to the present invention.
(III) advantageous effects
Compared with the prior art, the method and the device start the hand shape recognition of the user when the distance between the user and the camera device is monitored to be within the preset distance range; when the hand shape of the user is monitored to be the writing hand shape, the process of recognizing the writing key points is started, the writing track is generated according to the position track of the writing key points, the speed and the accuracy of writing recognition can be improved, the use experience of the user can be improved, and the air writing method in online live broadcast class interaction can be further optimized.
Furthermore, the image monitoring model is used, so that the area ratio of the body shape of the first user in the real-time image can be detected more accurately, and more accurate position information of the first user can be obtained; whether the hand shape of the first user is a writing hand shape or not can be detected more accurately through the first image recognition model; through the second image recognition model, the positions of the writing key points can be determined more accurately, and therefore the writing track written in the air can be determined more quickly and more accurately; the speed and the accuracy of writing identification can be improved, various display modes of writing tracks in interaction can be realized, the user experience can be improved, and the air writing method in online live class interaction can be further optimized.
Drawings
FIG. 1 is a flowchart of an example of an aerial writing method of embodiment 1 of the present invention;
FIG. 2 is a flowchart of another example of an over-the-air writing method of embodiment 1 of the present invention;
FIG. 3 is a schematic view of an example of a writing hand shape in the air writing method of embodiment 1 of the present invention;
FIG. 4 is a schematic view of another example of a writing hand shape in the air writing method of embodiment 1 of the present invention;
FIG. 5 is a flowchart of still another example of the space writing method of embodiment 1 of the present invention;
fig. 6 is a schematic diagram of an example of a live online system of embodiment 2 of the present invention.
Fig. 7 is a schematic view of an example of an aerial writing apparatus of embodiment 3 of the present invention;
fig. 8 is a schematic view of another example of an aerial writing device of embodiment 3 of the present invention;
fig. 9 is a schematic view of still another example of an aerial writing device of embodiment 3 of the present invention;
fig. 10 is a schematic view of still another example of an aerial writing device of embodiment 3 of the present invention;
FIG. 11 is a schematic structural diagram of a computer device of one embodiment of the present invention;
FIG. 12 is a schematic diagram of a computer program product of an embodiment of the invention.
Detailed Description
In describing particular embodiments, specific details of structures, properties, effects, or other features are set forth in order to provide a thorough understanding of the embodiments by one skilled in the art. However, it is not excluded that a person skilled in the art may implement the invention in a specific case without the above-described structures, performances, effects or other features.
The flow chart in the drawings is only an exemplary flow demonstration, and does not represent that all the contents, operations and steps in the flow chart are necessarily included in the scheme of the invention, nor does it represent that the execution is necessarily performed in the order shown in the drawings. For example, some operations/steps in the flowcharts may be divided, some operations/steps may be combined or partially combined, and the like, and the execution order shown in the flowcharts may be changed according to actual situations without departing from the gist of the present invention.
The block diagrams in the figures generally represent functional entities and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and/or processing unit devices and/or microcontroller devices.
The same reference numerals denote the same or similar elements, components, or parts throughout the drawings, and thus, a repetitive description thereof may be omitted hereinafter. It will be further understood that, although the terms first, second, third, etc. may be used herein to describe various elements, components, or sections, these elements, components, or sections should not be limited by these terms. That is, these phrases are used only to distinguish one from another. For example, a first device may also be referred to as a second device without departing from the spirit of the present invention. Furthermore, the term "and/or", "and/or" is intended to include all combinations of any one or more of the listed items.
In order to further optimize the air writing method, the invention provides an air writing method which is used for on-line live broadcast lessons and comprises the steps of detecting the hand shape of a first user through a plurality of detection steps, detecting and identifying the writing key points of the writing hand shape when the hand shape is detected, and recording the writing key points and the corresponding writing tracks thereof so as to display the writing tracks in the writing end and/or the opposite end. The opposite end may interact or communicate with the writing end. Therefore, the method can improve the speed and the accuracy of writing identification, improve the use experience of the user and further optimize the air writing method in the online live broadcast course interaction.
In order that the objects, technical solutions and advantages of the present invention will become more apparent, the present invention will be further described in detail with reference to the accompanying drawings in conjunction with the following specific embodiments.
It should be noted that, for convenience of understanding, the embodiment of the present invention is described in the context of an online live session, but those skilled in the art will understand that the application of the present invention is not limited thereto. The invention can also be used for other live scenes, or can be used as a video processing method for processing the existing video, recognizing the writing action of the character in the video and generating a writing track according to the writing action, and the writing track is coupled to the existing video or transmitted/displayed in a mode of being independent of the existing video.
Fig. 1 is a flowchart of an example of an aerial writing method of embodiment 1 of the present invention.
As shown in fig. 1, the present invention provides a method of over-the-air writing, the method comprising:
step S101, monitoring a distance from the user to the image pickup device.
And S102, when the distance between the user and the camera device is monitored to be within a preset distance range, detecting a real-time image of the user to monitor the hand shape of the user.
And step S103, when the hand shape of the user is monitored to be the writing hand shape, detecting the writing hand shape to identify a writing key point, and recording the position of the writing key point.
And step S104, generating a writing track according to the positions of the writing key points of the plurality of time points.
In this example, in an application scenario of an online live class, the online live system operating on one or both of a writing end and an opposite end, the over-the-air writing method is used for user interaction between the writing end and the opposite end during the online live class, the writing end and the opposite end may be both connected with a camera device, the respective camera device continuously shoots real-time images to form respective video streams, and the video streams formed at the writing end can be transmitted to the opposite end through a server of the online live system. The specific process of the air writing method will be described in detail below.
First, in step S101, the distance of the user from the image pickup device is monitored.
In this example, the writing end is a first client corresponding to a first user, which may be a teacher, and the opposite end is a second client interacting with the writing end, which corresponds to a second user, which may be a student.
Specifically, during an online live class, a camera device (e.g., a camera device of a computer, a mobile phone, etc.) of the first client monitors a real-time image of the teacher in real time.
Alternatively, the real-time image of the user may be detected in real time to monitor the distance from the user to the camera device, for example, an area ratio of the shape of the user in the real-time image may be detected, and the distance from the user to the camera device may be calculated according to the area ratio.
Further, the distance of the first user from the camera of the first client may be monitored using an image monitoring model.
It should be noted that the above description is only given by way of example, and the present invention is not limited thereto. In other examples, the monitoring of the distance of the user from the camera device may also be implemented by hardware of the first client or the second client, such as an infrared remote distance meter, a distance sensor, and the like.
Fig. 2 is a flowchart showing another example of an air writing method of an online live class of embodiment 1 of the present invention. As shown in fig. 2, the method further includes a step S201 of establishing an image monitoring model.
In step S201, an image monitoring model is established, and training data is used to train the image monitoring model, where the training data set includes historical human body frame data, pictures including human body in a historical live broadcast video stream, and position information of human body in each picture that has been manually labeled.
Specifically, each picture monitored by the first client is input into the image monitoring model, and the position information of the first user in the picture is output, for example, the area ratio of the shape of the first user in each real-time image can be output.
Further, whether the position of the first user is within a specific distance range is calculated according to the area ratio so as to further judge whether to perform next detection processing.
Therefore, by using the image monitoring model, the area ratio of the body shape of the first user in the real-time image can be detected more accurately, and more accurate position information of the first user can be obtained.
It should be noted that the above description is only given as an alternative example, and the invention is not to be construed as being limited thereto.
Next, in step S102, when it is monitored that the distance from the user to the camera device is within a preset distance range, a real-time image of the user is detected to monitor the hand shape of the user.
Specifically, in the case that the calculated position of the first user is within a specific distance range, the first image recognition model is started to monitor. The first image recognition model is used to detect a user's written hand shape.
Further, using the first image recognition model, it is monitored in real time whether a predefined hand shape is included in each image, i.e. whether the hand shape of the first user is a predefined writing hand shape is detected.
Preferably, according to different application scenarios, a writing hand shape for writing, drawing or marking is predefined and used as a detection comparison to determine whether the detected hand shape is the predefined writing hand shape.
In this example, the predefined writing hand shape may be a gesture in which the index finger is stretched out and the remaining fingers are held together, as shown in FIG. 3. Alternatively, the predefined writing hand shape may further include a common pen-holding gesture as shown in FIG. 4.
Further, a first image recognition model may be established based on a convolutional neural network, and trained using a training data set, where the training data set includes pictures including hands in a historical live stream, and hand information in each picture, where the hand information includes hand positions and hand shape (hand shape) category information. The written hand shapes in these pictures have been marked by the necessary user.
Specifically, the monitored corresponding pictures (pictures within a specific distance range) are input into the first image recognition model, and the hand position and hand shape category information of the first user in each picture is output.
Preferably, the specific distance ranges from 0.1m to 1 m.
Thus, by using the first image recognition model, it is possible to more accurately detect whether the hand shape of the first user is a predefined writing hand shape.
It should be noted that the above description is only given by way of example, and the present invention is not limited thereto.
Next, in step S103, when the hand shape of the user is detected as the writing hand shape, the writing hand shape starts to be detected to identify the writing key point, and the position of the writing key point is recorded.
Specifically, when the first image recognition model monitors that the hand shape of the user is the predefined writing hand shape, a writing key point detection process is started, that is, the writing key point of the writing hand shape is detected, and the writing key point is identified.
Fig. 3 is a schematic diagram of an example of writing a hand shape in the air writing method of the live online class according to embodiment 1 of the present invention. As shown in fig. 3, the key point of the writing hand shape is the tip of the index finger (or the position of the tip of the index finger).
It should be noted that the above is only described as a preferred example, and is not to be construed as limiting the present invention, and in other examples, for example, in the case that the user uses a special pen to write, the writing hand shape may also include a posture that the user holds the pen with a common hand, and a key point of writing of the writing hand shape is a pen point, see fig. 4 in particular.
In this example, a second image recognition model is used to monitor writing keypoints of the writing hand shape.
Specifically, a second image recognition model may be pre-established, and the second image recognition model may be trained by using training data, where the training data includes pictures containing written hand shapes in the historical live broadcast stream, hand shape category information, and writing key points corresponding to each written hand shape that has been labeled, and the like.
And further, processing the picture recognized as the writing hand shape and the corresponding hand shape category information by using a trained second image recognition model so as to monitor and recognize writing key points and recognize the writing key points of the writing hand shape.
In this example, the live online system includes at least one server that can record and store information data related to the over-the-air writing for displaying the over-the-air writing content in live broadcasting or replaying.
Specifically, when a server of the online live broadcast system identifies a writing key point, the position of the writing key point is recorded, and the position information of the writing key point is recorded.
For example, when the image recognition model is used for detection, when it is detected that the index finger of a first user (e.g., a teacher) stretches out and the other fingers are held together, it is determined that the hand shape of the first user is a writing hand shape, and detection is performed on a writing key point of the first user, where the writing key point is the position of the tip of the index finger, and the position of the tip of the index finger is recorded in real time to a server of an online live broadcast system.
Thus, the position of the writing key point can be determined more accurately by a plurality of detection processes.
It should be noted that the above description is only given by way of example, and the present invention is not limited thereto. For example, the detection and recognition of the writing hand shape of the user and the recognition of the writing key point of the writing hand shape may be performed at the writing end, which is not limited in this embodiment.
Next, in step S104, a writing trajectory is generated from the positions of the writing key points at a plurality of time points.
In this example, the movement trajectory of the writing keypoints is calculated as the writing trajectory according to the positions of the writing keypoints identified at a plurality of time points.
Specifically, the positions of the writing key points are identified according to a plurality of time points, or the positions of the writing key points are identified from each frame image of a preset interval frame number, so that the positions of the writing key points identified by the time points are obtained.
More specifically, the predetermined interval frame number is 0, 1, 2 or other numbers, etc., and when 0, it is all frames. Therefore, since the time required for the recognition processing is relatively long for all the frame images, the frame images are recognized from each frame image of the predetermined number of frames at intervals. This can improve the speed of data recognition processing in the air.
Further, calculating the motion trail of the writing key point to be used as the writing trail.
Specifically, the positions of the writing key points detected at the plurality of time points are subjected to straight line or curve fitting to generate a motion trajectory, and the generated motion trajectory is taken as a writing trajectory, thereby generating the writing trajectory. Therefore, the writing track of the air writing can be determined more quickly and more accurately by the above method.
For example, when the position of the index finger tip of the teacher is detected, the movement trajectory of the position of the index finger tip of the teacher is started to be continuously recorded, but when the position of the index finger tip is not detected or the writing hand shape of the teacher is changed, the recording of the position of the index finger tip is stopped. And fitting the positions of the plurality of writing key points to generate a motion track as a writing track.
It should be noted that the above description is only given by way of example, and the present invention is not limited thereto.
Fig. 5 is a flowchart of still another example of an air writing method of an online live class according to embodiment 1 of the present invention.
As shown in fig. 5, a step S501 of displaying the writing trace in the writing end and/or the opposite end is further included.
In this example, the interaction between the writing end and the opposite end is through an online live system on which the written trace may be saved for re-display upon playback.
Preferably, the writing track is separately saved in the first client and/or the second client, so that the writing track is displayed or hidden in the video stream according to the operation of the first user or the second user.
Further, the written trace is coupled into the video stream to display the written trace in real time in a video picture of the video stream of the first client and/or the second client.
Optionally, the writing track may be displayed or hidden in a user video stream of a writing end according to a user instruction of the writing end; alternatively, the written trace may be displayed or hidden in the user video stream of the opposite end according to a user instruction of the opposite end.
It should be noted that the generated writing track may be superimposed on the video stream, and the writing track may be hidden/removed as needed. The foregoing is described by way of preferred examples only and is not to be construed as limiting the invention. For example, the writing trace generated at the writing end may not be superimposed on the video stream, and may be stored and transmitted independently.
Preferably, the writing trace may be displayed in a courseware screen corresponding to an online live lesson.
Therefore, multiple display modes of the writing track in interaction can be realized, and the use experience of the user can be improved.
It should be noted that the above description is only given by way of example, and the present invention is not limited thereto.
The above-described procedure of the air writing method is only for illustrating the present invention, and the order and number of steps are not particularly limited. In addition, the steps in the method can be split into two or three steps, or some steps can be combined into one step, and the steps are adjusted according to practical examples.
Compared with the prior art, the method and the device have the advantages that the image monitoring model is used, so that the area ratio of the body shape of the first user in the real-time image can be detected more accurately, and more accurate position information of the first user can be obtained; whether the hand shape of the first user is a writing hand shape or not can be detected more accurately through the first image recognition model; through the second image recognition model, the position of the writing key point can be determined more accurately, and therefore the writing track written in the air can be determined more quickly and more accurately; the speed and the accuracy of writing identification can be improved, various display modes of writing tracks in interaction can be realized, the user experience can be improved, and the air writing method in online live class interaction can be further optimized. The model can be arranged at a server end, and the writing end equipment does not need to be improved, and only needs to upload real-time video, so that the model can have an aerial writing function in online live broadcasting.
Example 2
Referring to fig. 6, a live online system of embodiment 2 of the present invention will be explained.
According to a second aspect of the present invention, the present invention also provides an online live broadcasting system, including: a writing end and an opposite end; at least one server for on-line live session in-class interaction between the writing end and the opposite end.
The writing track generated by one server in the at least one server by using the air writing method in embodiment 1 is transmitted to the opposite end and/or the writing end in a video streaming mode and is displayed in the writing end and/or the opposite end. Optionally, in some embodiments, the writing end uses the writing track generated by the air writing method described in embodiment 1 and uploads the writing track in a video stream.
Preferably, the server couples the generated writing track to the video stream by means of direct transmission, transmits the writing track to the opposite end (i.e. the second client) and displays the writing track in real time in the picture of the video stream of the opposite end.
Specifically, the writing end and the opposite end correspond to a first user and a second user, respectively.
In another example, the writing trajectory may be displayed in real time in the writing end and/or the opposite end, respectively.
Further, the motion trajectory may be displayed or hidden in the video stream according to an operation of the first user or the second user. Specifically, the writing track can be displayed or hidden in a user video stream of a writing end according to a user instruction of the writing end; alternatively, the written trace may be displayed or hidden in the user video stream of the opposite end according to a user instruction of the opposite end.
It should be noted that the foregoing is only described as a preferred example, and is not to be construed as limiting the present invention, and in other examples, video streaming may be performed by other server forwarding methods. In embodiment 2, the same portions as those in embodiment 1 are not described.
Compared with the prior art, the system can determine the writing track of the aerial writing more quickly and more accurately by using the aerial writing method of the embodiment; the speed and the accuracy of writing recognition can be improved, various display modes of writing tracks in interaction can be realized, and the use experience of a user can be improved.
Example 3
Embodiments of the apparatus of the present invention are described below, which may be used to perform method embodiments of the present invention. The details described in the device embodiments of the invention should be regarded as complementary to the above-described method embodiments; reference is made to the above-described method embodiments for details not disclosed in the apparatus embodiments of the invention.
Referring to fig. 7 to 10, an in-flight writing apparatus 700 for a live session in embodiment 3 of the present invention will be described.
According to a third aspect of the present invention, the present invention further provides an aerial writing device 700 for online live class, where the aerial writing device 700 includes a first monitoring module 701 for monitoring a distance from a user to a camera device; a second monitoring module 702, configured to detect a real-time image of the user to monitor a hand shape of the user when it is monitored that a distance from the user to the camera device is within a preset distance range; the third monitoring module 703, when monitoring that the hand shape of the first user is a writing hand shape, starts to detect the writing hand shape to identify a writing key point; the generating module 704 generates a writing track according to the positions of the writing key points of the plurality of time points.
Preferably, real-time images of the user are detected in real time to monitor the distance from the user to the camera device; optionally, the detecting the real-time image of the user in real time to monitor the distance from the user to the camera device includes: and detecting the area ratio of the body shape of the user in the real-time image, and calculating the distance from the user to the camera device according to the area ratio.
As shown in fig. 8, the aerial writing device 700 may further include a model building module 801, where the model building module 801 is configured to build a first image recognition model, so that the second monitoring module 702 uses the first image recognition model to detect whether the real-time image of the user includes the predefined writing hand shape, where the image recognition model is a machine learning model of a writing hand shape trained to recognize the shape of a human hand, and when the real-time image includes the predefined writing hand shape, a writing keypoint detection process is started.
Preferably, the third monitoring module 703 may detect the writing keypoints of the shape of the human hand using a second image recognition model, which is a machine learning model trained to recognize the writing keypoints of the shape of the human hand. The model building module 801 may also be used to build the second image recognition model.
Preferably, the predefined writing hand shape is a posture that the index finger stretches out and the other fingers are held together, and the writing key point of the hand shape is the index finger tip.
Preferably, the generating module 704 may generate the writing track according to the positions of the writing key points of the plurality of time points, and this step may include: performing straight line or curve fitting on the positions of the writing key points of the multiple time points to generate the motion trail; and generating a writing track according to the motion track.
Preferably, the writing trace is generated as the user writes at the writing end and transmitted in real time to the opposite end interacting with the writing end.
Preferably, the method further comprises the following steps: a written trace generated at a writing end is coupled into a video stream and the written trace may be displayed or hidden in the video stream.
As shown in fig. 9, the device further comprises a display module 901, and the display module 901 is used for displaying the writing track in the writing end and/or the opposite end.
In another example, as shown in fig. 10, the on-air writing device 700 of the on-line live class is used for interaction between a first client (i.e., a writing end) and a second client (i.e., opposite ends) during the on-line live class, the first client is connected with a camera device, and the camera device continuously shoots real-time images to form a video stream and transmits the video stream to the second client through the first client. The aerial writing device 700 includes: a first monitoring module 701, configured to monitor a distance from a first user to a camera device of the first client by performing real-time detection on the real-time image; a second monitoring module 702, configured to start detecting the real-time image to monitor a hand shape of the first user when the distance is monitored to be within a specific range; the third monitoring module 703, when the monitored hand shape of the first user is a writing hand shape, starts to detect the writing hand shape to identify a writing key point; the generation module includes a recording module 604 and a calculation module 605. A recording module 604, configured to record a position of the writing key point when the writing key point is identified; a calculating module 605, configured to calculate, according to the positions of the writing key points identified by the multiple time points, a motion trajectory of the writing key points as a writing trajectory; and the display module 606 displays the writing track in the first client and/or the second client.
In example 3, the same portions as those in examples 1 and 2 are not described.
Those skilled in the art will appreciate that the modules in the above-described embodiments of the apparatus may be distributed as described in the apparatus, and may be correspondingly modified and distributed in one or more apparatuses other than the above-described embodiments. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
Compared with the prior art, the method and the device have the advantages that the image monitoring model is used, so that the area ratio of the body shape of the first user in the real-time image can be detected more accurately, and more accurate position information of the first user can be obtained; whether the hand shape of the first user is a writing hand shape or not can be detected more accurately through the first image recognition model; through the second image recognition model, the position of the writing key point can be determined more accurately, and therefore the writing track written in the air can be determined more quickly and more accurately; the speed and the accuracy of writing identification can be improved, various display modes of writing tracks in interaction can be realized, the user experience can be improved, and the air writing method in online live class interaction can be further optimized.
Example 4
In the following, embodiments of the computer apparatus of the present invention are described, which may be seen as specific physical embodiments for the above-described embodiments of the method and apparatus of the present invention. The details described in the computer device embodiment of the invention should be considered as additions to the method or apparatus embodiment described above; for details which are not disclosed in the embodiments of the computer device of the invention, reference may be made to the above-described embodiments of the method or apparatus.
FIG. 11 is a schematic block diagram of a computer device according to an embodiment of the present invention, the computer device including a processor and a memory, the memory storing a computer executable program, the processor executing the method according to any one of the embodiments when the computer program is executed by the processor, including but not limited to the method of FIG. 1.
As shown in fig. 11, the computer device is in the form of a general purpose computing device. The processor can be one or more and can work together. The invention also does not exclude that distributed processing is performed, i.e. the processors may be distributed over different physical devices. The computer device of the present invention is not limited to a single entity, and may be a sum of a plurality of entity devices.
The memory stores a computer executable program, typically machine readable code. The computer readable program may be executed by the processor to enable a computer device to perform the method of the invention, or at least some of the steps of the method.
The memory may include volatile memory, such as Random Access Memory (RAM) and/or cache memory, and may also be non-volatile memory, such as read-only memory (ROM).
Optionally, in this embodiment, the computer device further includes an I/O interface, which is used for data exchange between the computer device and an external device. The I/O interface may be a local bus representing one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, and/or a memory storage device using any of a variety of bus architectures.
It should be understood that the computer device shown in fig. 11 is only one example of the present invention, and elements or components not shown in the above examples may also be included in the computer device of the present invention. For example, some computer devices also include display units such as display screens, and some computer devices also include human-computer interaction elements such as buttons, keyboards, and the like. The computer device can be considered to be covered by the present invention as long as the computer device can execute the computer readable program in the memory to implement the method of the present invention or at least part of the steps of the method.
FIG. 12 is a schematic diagram of a computer program product of an embodiment of the invention. As shown in fig. 12, a computer program product has a computer executable program stored therein, and when the computer executable program is executed, the above-mentioned space writing method of the present invention is realized. The computer program product may comprise a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. The computer program product may be transmitted, propagated, or transported by a computer to be used by or in connection with an instruction execution system, apparatus, or device. Program code embodied on the computer program product may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
From the above description of the embodiments, those skilled in the art will readily appreciate that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and electronic processing units, servers, clients, mobile phones, control units, processors, etc. included in the system. The invention may also be implemented by computer software for performing the method of the invention, e.g. control software executed by a microprocessor, an electronic control unit, a client, a server, etc. It should be noted that the computer software for executing the method of the present invention is not limited to be executed by one or a specific hardware entity, and can also be realized in a distributed manner by non-specific hardware. For computer software, the software product may be stored in a computer readable storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or may be distributed over a network, as long as it enables the computer device to perform the method according to the present invention.
While the foregoing detailed description has described the objects, aspects and advantages of the present invention in further detail, it should be appreciated that the present invention is not inherently related to any particular computer, virtual machine, or computer apparatus, as various general purpose devices may implement the present invention. The invention is not to be considered as limited to the specific embodiments thereof, but is to be understood as being modified in all respects, all changes and equivalents that come within the spirit and scope of the invention.
Claims (10)
1. An air writing method for on-line live broadcast lessons, which is characterized by comprising the following steps:
monitoring the distance from a user to the camera device;
when the monitored distance between the user and the camera device is within a preset distance range, detecting a real-time image of the user to monitor the hand shape of the user;
when the hand shape of the user is monitored to be a writing hand shape, detecting the writing hand shape to identify a writing key point, and recording the position of the writing key point;
and generating a writing track according to the positions of the writing key points of the plurality of time points.
2. The aerial writing method of claim 1, wherein the monitoring of the distance of the user from the camera device comprises: detecting the real-time image of the user in real time to monitor the distance from the user to the camera device;
optionally, the detecting the real-time image of the user in real time to monitor the distance from the user to the camera device includes: and detecting the area ratio of the body shape of the user in the real-time image, and calculating the distance from the user to the camera device according to the area ratio.
3. The aerial writing method of claim 1, wherein the detecting the real-time image of the user to monitor the hand shape of the user comprises:
detecting whether a real-time image of the user contains a predefined writing hand shape or not by using a first image recognition model, wherein the image recognition model is a trained machine learning model capable of recognizing the writing hand shape, and when the real-time image is detected to contain the predefined writing hand shape, starting a detection process of writing key points;
optionally, the predefined writing hand shape is a posture that the index finger is stretched out and the other fingers are held together, and the writing key point of the hand shape is the index finger tip.
4. The aerial writing method of claim 3, wherein the detection process of the writing keypoints comprises:
the writing keypoints are detected using a second image recognition model, which is a machine learning model trained to recognize writing keypoints in a writing hand shape.
5. The aerial writing method of claim 1, wherein generating a writing trajectory from the positions of the writing keypoints for a plurality of time points comprises:
performing straight line or curve fitting on the positions of the writing key points of the multiple time points to generate a motion trail of the writing key points;
and generating a writing track according to the motion track of the writing key point.
6. The aerial writing method of claim 1, further comprising: transmitting the writing track generated by the user at a writing end to an opposite end interacting with the writing end in real time; optionally, the step comprises:
coupling the writing track generated at a writing end into a user video stream, and displaying or hiding the writing track in the user video stream according to a user instruction at the writing end or an opposite end;
optionally, the air writing method further includes: displaying the writing trajectory in a writing end and/or an opposite end.
7. An online live broadcasting system, comprising:
a writing end and an opposite end;
at least one server for on-line live session in-class interaction between the writing end and the opposite end, and
the writing end or the at least one server generating a writing trajectory using the aerial writing method of claim 1; the at least one server transmits the generated writing track to the opposite end in a video streaming mode, and displays the writing track in the writing end and/or the opposite end.
8. An aerial writing device, comprising:
the first monitoring module is used for monitoring the distance from a user to the camera device;
the second monitoring module is used for detecting the real-time image of the user to monitor the hand shape of the user when the monitored distance between the user and the camera device is within a preset distance range;
the third monitoring module starts to detect the writing hand shape to identify a writing key point when the hand shape of the first user is monitored to be the writing hand shape;
and the generating module is used for generating a writing track according to the positions of the writing key points of the plurality of time points.
9. A computer device comprising a processor and a memory, the memory for storing a computer executable program, characterized in that:
the computer program, when executed by the processor, performs a method of over-the-air writing of a live online class as claimed in any one of claims 1-7.
10. A computer program product comprising computer programs/instructions, characterized in that the computer programs/instructions, when executed by a processor, implement the method of over-the-air writing of live online lessons of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110354090.4A CN113253837A (en) | 2021-04-01 | 2021-04-01 | Air writing method and device, online live broadcast system and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110354090.4A CN113253837A (en) | 2021-04-01 | 2021-04-01 | Air writing method and device, online live broadcast system and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113253837A true CN113253837A (en) | 2021-08-13 |
Family
ID=77181332
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110354090.4A Pending CN113253837A (en) | 2021-04-01 | 2021-04-01 | Air writing method and device, online live broadcast system and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113253837A (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110254765A1 (en) * | 2010-04-18 | 2011-10-20 | Primesense Ltd. | Remote text input using handwriting |
| CN103608761A (en) * | 2011-04-27 | 2014-02-26 | Nec软件系统科技有限公司 | Input device, input method and recording medium |
| CN105320248A (en) * | 2014-06-03 | 2016-02-10 | 深圳Tcl新技术有限公司 | Mid-air gesture input method and device |
| CN105718878A (en) * | 2016-01-19 | 2016-06-29 | 华南理工大学 | Egocentric vision in-the-air hand-writing and in-the-air interaction method based on cascade convolution nerve network |
| CN109344793A (en) * | 2018-10-19 | 2019-02-15 | 北京百度网讯科技有限公司 | Aerial hand-written method, apparatus, equipment and computer readable storage medium for identification |
| CN111062312A (en) * | 2019-12-13 | 2020-04-24 | RealMe重庆移动通信有限公司 | Gesture recognition method, gesture control method, device, medium and terminal device |
| CN211293894U (en) * | 2019-11-27 | 2020-08-18 | 华南理工大学 | Hand-written interaction device in air |
| CN111626297A (en) * | 2020-04-13 | 2020-09-04 | 小船出海教育科技（北京）有限公司 | Character writing quality evaluation method and device, electronic equipment and recording medium |
| CN111950514A (en) * | 2020-08-26 | 2020-11-17 | 重庆邮电大学 | Depth camera-based aerial handwriting recognition system and method |
| CN112199015A (en) * | 2020-09-15 | 2021-01-08 | 安徽鸿程光电有限公司 | Intelligent interaction all-in-one machine and writing method and device thereof |
| CN112286411A (en) * | 2020-09-30 | 2021-01-29 | 北京大米科技有限公司 | Display mode control method and device, storage medium and electronic equipment |
-
2021
- 2021-04-01 CN CN202110354090.4A patent/CN113253837A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110254765A1 (en) * | 2010-04-18 | 2011-10-20 | Primesense Ltd. | Remote text input using handwriting |
| CN103608761A (en) * | 2011-04-27 | 2014-02-26 | Nec软件系统科技有限公司 | Input device, input method and recording medium |
| CN105320248A (en) * | 2014-06-03 | 2016-02-10 | 深圳Tcl新技术有限公司 | Mid-air gesture input method and device |
| CN105718878A (en) * | 2016-01-19 | 2016-06-29 | 华南理工大学 | Egocentric vision in-the-air hand-writing and in-the-air interaction method based on cascade convolution nerve network |
| CN109344793A (en) * | 2018-10-19 | 2019-02-15 | 北京百度网讯科技有限公司 | Aerial hand-written method, apparatus, equipment and computer readable storage medium for identification |
| CN211293894U (en) * | 2019-11-27 | 2020-08-18 | 华南理工大学 | Hand-written interaction device in air |
| CN111062312A (en) * | 2019-12-13 | 2020-04-24 | RealMe重庆移动通信有限公司 | Gesture recognition method, gesture control method, device, medium and terminal device |
| CN111626297A (en) * | 2020-04-13 | 2020-09-04 | 小船出海教育科技（北京）有限公司 | Character writing quality evaluation method and device, electronic equipment and recording medium |
| CN111950514A (en) * | 2020-08-26 | 2020-11-17 | 重庆邮电大学 | Depth camera-based aerial handwriting recognition system and method |
| CN112199015A (en) * | 2020-09-15 | 2021-01-08 | 安徽鸿程光电有限公司 | Intelligent interaction all-in-one machine and writing method and device thereof |
| CN112286411A (en) * | 2020-09-30 | 2021-01-29 | 北京大米科技有限公司 | Display mode control method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11871109B2 (en) | Interactive application adapted for use by multiple users via a distributed computer-based system | |
| Chow et al. | Music education using augmented reality with a head mounted display | |
| KR102114207B1 (en) | Learning Support System And Method Using Augmented Reality And Virtual reality based on Artificial Intelligence | |
| Iqbal et al. | Acceptance of dance training system based on augmented reality and technology acceptance model (TAM) | |
| US20220415197A1 (en) | Virtual and augmented reality instruction system | |
| US20210104169A1 (en) | System and method for ai based skill learning | |
| Ng et al. | 3D augmented mirror: a multimodal interface for string instrument learning and teaching with gesture support | |
| CN108875785A (en) | The attention rate detection method and device of Behavior-based control Characteristic Contrast | |
| US10720187B2 (en) | Method and apparatus to extract key video frames from online education | |
| CN110544399A (en) | Graphical remote teaching system and graphical remote teaching method | |
| US20150301726A1 (en) | Systems and Methods for Displaying Free-Form Drawing on a Contact-Sensitive Display | |
| Faisal et al. | Towards a reference model for sensor-supported learning systems | |
| US20140310640A1 (en) | Interactive digital art apparatus | |
| CN113253838A (en) | AR-based video teaching method and electronic equipment | |
| JP2020086075A (en) | Learning support system and program | |
| CN113253837A (en) | Air writing method and device, online live broadcast system and computer equipment | |
| CN113780051A (en) | Method and device for evaluating concentration degree of student | |
| CN110766997A (en) | Copy display method, device and storage medium | |
| Ahmad et al. | Towards a Low‐Cost Teacher Orchestration Using Ubiquitous Computing Devices for Detecting Student’s Engagement | |
| JP2021064101A (en) | Information processing apparatus, control method, and program | |
| CN115052194B (en) | Learning report generation method, device, electronic equipment and storage medium | |
| CN210119873U (en) | Supervision device based on VR equipment | |
| US11990059B1 (en) | Systems and methods for extended reality educational assessment | |
| US20230137560A1 (en) | Assistance system and method for guiding exercise postures in live broadcast | |
| CN108091186B (en) | Teaching method and teaching system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |