CN107909022B - Video processing method and device, terminal equipment and storage medium - Google Patents
Video processing method and device, terminal equipment and storage medium Download PDFInfo
- Publication number
- CN107909022B CN107909022B CN201711103557.8A CN201711103557A CN107909022B CN 107909022 B CN107909022 B CN 107909022B CN 201711103557 A CN201711103557 A CN 201711103557A CN 107909022 B CN107909022 B CN 107909022B
- Authority
- CN
- China
- Prior art keywords
- writing
- image
- display screen
- touch display
- track
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/32—Digital ink
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/62—Control of parameters via user interfaces
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2621—Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a video processing method, a video processing device, terminal equipment and a storage medium. The method comprises the following steps: acquiring a writing image corresponding to an area where a touch display screen is located from video data shot by a camera facing the touch display screen; acquiring a writing track generated on a touch display screen in the shooting process of the camera, and mapping the acquisition time of the writing track to the time axis of the writing image; adding the writing trace to the written image based on the time axis and the relative position of the writing trace in the touch display screen. The invention solves the problem that the written content is shielded by the hands of the user in the video teaching process, and improves the video teaching efficiency and the attention of students in the learning process.
Description
Technical Field
Embodiments of the present invention relate to image processing technologies, and in particular, to a video processing method and apparatus, a terminal device, and a storage medium.
Background
With the continuous development of networks, teaching assistance means also progress. In the 21 st century, network education has been promoted along with the development of network technology as PCs and notebook devices have been popularized.
In the prior art, the lecture process of a teacher can be recorded into a multimedia file, students can be helped to learn, review and pre-learn conveniently, resource sharing can be formed between teachers and schools, and the teaching force of teachers and resources is improved. Meanwhile, the remote video teaching can be carried out on learners in different regions through the network.
In the actual teaching process, when a teacher explains the writing on the writing board, the student is attentive to the content written by the teacher, but a part of the content being written or the written content may be blocked by the hand of the teacher, as shown in fig. 1, the content written by the teacher in the first line is p (c) ═ 0.01 ═ 1%, but since the teacher blocks the part of the content in the first line from the hand when writing the content in the second line, the part of the written content can be received by the student only after the teacher moves the hand, thereby affecting the attention of the student in the learning process.
Disclosure of Invention
The invention provides a video processing method, a video processing device, terminal equipment and a storage medium, which are used for improving the efficiency of video teaching and the attention of students in the learning process.
In a first aspect, an embodiment of the present invention provides a video processing method, where the method includes:
acquiring a writing image corresponding to an area where a touch display screen is located from video data shot by a camera facing the touch display screen;
acquiring a writing track generated on a touch display screen in the shooting process of the camera, and mapping the acquisition time of the writing track to the time axis of the writing image;
adding the writing trace to the written image based on the time axis and the relative position of the writing trace in the touch display screen.
In a second aspect, an embodiment of the present invention further provides a video processing apparatus, where the apparatus includes:
the device comprises a writing image acquisition module, a writing image acquisition module and a writing image acquisition module, wherein the writing image acquisition module is used for acquiring a writing image corresponding to the area of the touch display screen from video data shot by a camera facing the touch display screen;
the acquisition writing track module is used for acquiring a writing track generated on a touch display screen in the shooting process of the camera and mapping the acquisition time of the writing track to the time axis of the writing image;
and the writing track adding module is used for adding the writing track to the writing image based on the time axis and the relative position of the writing track in the touch display screen.
In a third aspect, an embodiment of the present invention further provides a terminal device, where the terminal device includes:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a video processing method as in any one of the above.
In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the video processing method as described in any one of the above.
According to the technical scheme, the writing image corresponding to the area where the touch display screen is located is obtained from the video data shot by the camera facing the touch display screen; acquiring a writing track generated on a touch display screen in the shooting process of a camera, and mapping the acquisition time of the writing track to a time axis of a written image; the technical means that the writing track is added to the writing image based on the relative positions of the time axis and the writing track in the touch display screen solves the problem that the writing content is shielded by hands in the video teaching process, and improves the efficiency of the video teaching and the attention effect of students in the learning process.
Drawings
Fig. 1 is a diagram illustrating a state in which written contents are occluded by a hand in the related art.
Fig. 2 is a flowchart of a video processing method according to a first embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a video recording system according to a first embodiment of the present invention.
Fig. 4 is a flowchart of a video processing method according to a second embodiment of the present invention.
Fig. 5 is a diagram illustrating a state where written contents are displayed on a hand in the second embodiment of the present invention.
Fig. 6 is a schematic structural diagram of a video processing apparatus according to a third embodiment of the present invention.
Fig. 7 is a schematic structural diagram of a terminal device in the fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 2 is a flowchart of a video processing method according to an embodiment of the present invention, where the present embodiment is applicable to a situation where a written content is prevented from being blocked by a hand during a course recording process or a written content is prevented from being blocked by a hand during a remote teaching process. The video processing method specifically comprises the following steps:
and S110, acquiring a writing image corresponding to the area where the touch display screen is located from the video data shot by the camera facing the touch display screen.
In the embodiment of the present invention, a video recording is performed on a teaching process of a user through a camera, and as shown in fig. 3, the video recording system is a schematic structural diagram. In this embodiment, the tablet computer 14 with the touch display screen 10 and the camera 11 are connected by the USB connection line 13, and the camera 11 is installed right above the touch display screen 10, that is, the shooting direction of the camera 11 is toward the side of the touch display screen 10 of the tablet computer 14. The touch display screen is not limited to a tablet computer, and other electronic devices with the touch display screen may also be used, for example, a smart phone with the touch display screen, a notebook computer, and the like.
As an example and not by way of limitation, after a user connects a camera and a tablet computer through a USB connection line, a writing software in the tablet computer is turned on, and a writing mode is entered, that is, a writing pen writes on a screen of a touch display screen of the tablet computer, and when a handwriting display mode is similar to a pen mode in a drawing board, directions of the camera and the tablet computer are adjusted to enable the screen of the tablet computer to be located as far as possible under the camera, and the camera is turned on to start recording, at this time, the user writes content to be explained on the screen of the touch display screen, and simultaneously, an image written by the user in an area where the touch display screen is located is obtained through the camera.
And S120, acquiring a writing track generated on the touch display screen in the shooting process of the camera, and mapping the acquisition time of the writing track to the time axis of the writing image.
The time axis is started when the camera is turned on and starts to record the content written by the user on the touch display screen, and is generated along with the time until the recording of the course or the remote teaching is finished, and the time axis is used for recording the time state in the process of acquiring the writing image and the writing track and the time correlation between the time state and the time state. The writing track is composed of a plurality of writing point sets, when a user (a teacher or other people who record videos) writes on the touch display screen, the touch display screen collects the writing point sets in real time and combines all the writing point sets to form the writing track, the writing track is stored in a memory of the tablet personal computer, and meanwhile, the time of collecting the writing track corresponds to the time of obtaining the writing image one by one.
In the specific embodiment of the invention, when a user starts to write content on the touch display screen, a touch event of the writing software system is triggered, the writing software in the tablet computer acquires a writing track generated on the touch display screen through the touch event provided by the system, and the acquisition time of the writing track is mapped to the time axis of a writing image.
And S130, adding the writing track to the writing image based on the time axis and the relative position of the writing track in the touch display screen.
In the scheme, the writing images need to be acquired continuously, but the relative position of the writing track in the touch display screen is changed all the time when the user writes the content on the touch display screen by using the writing pen. And according to the time sequence of obtaining the writing images and the relative positions of the writing tracks in the touch display screen, carrying out one-to-one correspondence on the writing tracks and the writing images, and adding the writing tracks into the writing images.
Generally speaking, during the writing process of the user, the refreshing speed of the writing image is relatively faster than the refreshing speed of the writing track, that is, the writing track at least corresponds to one frame of the writing image, each frame of the writing image is a static image, and the relative position of the writing track on the touch display screen is also changed. According to the relative position of the device on the touch display screen and at least one corresponding frame of writing image, the collected writing track is covered in the writing image, namely, the writing point set is covered on the corresponding pixel point in the corresponding writing image, so that the writing point set and the corresponding pixel point are combined, and therefore the writing point set is displayed on the display screen as a frame of video picture, the writing content is prevented from being shielded by the hands of a user, a learner watching the video content is not influenced by the user, and the learning efficiency of the learner is reduced.
Specifically, when a user writes content on the touch display screen, the written content on the first line is "P (C) ═ 0.01 ═ 1%", and the written content on the second line is "P (Pos | C) ═ 0.9 ═ 90%", if the content written by the user is added to the written image, when the user explains the content on the first line again and places the hand at the position of the first line, the learner viewing the video content can also see the content on the second line, thereby improving the efficiency of the user explaining the content and the learning efficiency of the learner.
According to the technical scheme of the embodiment, the writing image corresponding to the area where the touch display screen is located is obtained from the video data shot by the camera facing the touch display screen; acquiring a writing track generated on a touch display screen in the shooting process of a camera, and mapping the acquisition time of the writing track to a time axis of a written image; the technical means that the writing track is added to the writing image based on the relative positions of the time axis and the writing track in the touch display screen solves the problem that the writing content is shielded by hands in the video teaching process, and improves the efficiency of the video teaching and the attention effect of students in the learning process.
Further, on the basis of the above technical solution, after the writing trace is added to the written image based on the time axis and the relative position of the writing trace in the touch display screen, the video processing method further includes: and synchronously acquiring audio based on a time axis to acquire audio data, and associating the audio data with the written image to generate video data.
Of course, the video data shot by the camera includes, on one hand, the content written by the user on the touch display screen during the video recording process, and on the other hand, the audio data explaining the written content by the user during the video recording process.
In the scheme, based on the time sequence of acquiring the writing images, the audio data corresponding to the acquisition time of the writing images are stored in a correlation mode to generate the video data of the network video course, and the related video data information is stored in the terminal equipment related to the camera.
Of course, the video picture viewed by the learner viewing the video through the display screen is actually a still video image with certain relation displayed according to a certain refresh frequency, and usually a "frame" is taken as the unit of the video image. Since human eyes have a function of "persistence of vision", a plurality of discrete video images are displayed at a certain frequency, and thus a continuous video picture can be seen. Correspondingly, in the process of recording videos by the camera, the video images are acquired from one frame to another.
According to the technical scheme, audio data are acquired through synchronous audio acquisition based on a time axis, the audio data are associated with the writing image to generate video data, and the video image combined with the audio data is played and displayed in the terminal device, so that a learner can watch and learn the teaching content.
Example two
Fig. 4 is a flowchart of a video processing method according to a second embodiment of the present invention, where this embodiment is applicable to a case where written content is occluded by a hand during recording of a micro-class or remote teaching, and this embodiment is further optimized by acquiring a written image corresponding to an area where a touch display screen is located from video data captured by a camera facing the touch display screen and adding a writing trajectory to the written image based on a time axis and a relative position of the writing trajectory in the touch display screen, and specifically includes the following steps:
s210, reading an initial image which is shot by a camera facing the touch display screen and contains the area where the touch display screen is located.
In the specific embodiment of the present invention, the camera is installed right above the touch display screen, but in the specific recording process, the video image that can be captured by the camera is not only the video image of the area where the touch display screen is located, but may also include video images other than the touch display screen.
Meanwhile, writing software in the tablet personal computer is used for writing contents, most writing software is provided with a specified frame, and a user can write the contents in the specified frame, so that the initial image acquired by the camera also contains related video images outside the specified frame.
S220, performing image recognition on the initial image to extract an initial writing image of the area where the touch display screen is located.
Specifically, in order to subsequently correspond the writing track written by the user to the video image acquired by the camera one by one, the initial image needs to be preprocessed to identify and extract the initial writing image of the area where the touch display screen is located.
Illustratively, a boundary of an area where the touch display screen is located in the initial image is detected through a contour detection algorithm in an Opencv open source library, and image recognition and extraction are performed on the initial image according to the boundary to obtain an initial writing image of the area where the touch display screen is located. Specifically, the contour detection algorithm can be divided into still image contour detection and dynamic video contour detection, wherein the first category of dynamic video contour detection utilizes an optical flow method to acquire motion cues. The optical flow method calculates the variation field of the brightness of an object with time and space based on a space-time gradient model. The principle of the method is that the instantaneous change rate of a pixel point is calculated by optical flow, and the principle is that the gray value of the pixel point is kept unchanged when the pixel point moves from one position to other positions, so that the instantaneous change rate of the motion of a certain pixel point can be calculated according to the optical flow estimation. The second category uses frame difference method to obtain motion cue, and the frame difference method is also one of the common methods in detecting moving objects. The basic principle is that the pixel difference of the subtraction of the two frames before and after is calculated, and the foreground moving target is obtained by using a fixed closed value to carry out binarization processing. If the target has no motion, the pixel difference of the subtraction of the two frames is zero; if the target has obvious motion, the pixel difference of the subtraction of the two frames is larger than a closed value, and the position of the moving target in the video frame can be obtained through binarization processing.
In the scheme, the initial image acquired by the camera can be identified and extracted through static image contour detection, so that only the writing image of the area where the touch display screen is located is reserved, and the writing track and the writing image can be combined in the follow-up process.
And S230, carrying out distortion correction on the initial writing image to obtain a writing image corresponding to the display shape of the touch display screen.
Specifically, the camera is installed right above the touch display screen, but in a specific video recording process, the writing pen is in a motion state in the area where the touch display screen is located all the time. Similarly, the relative position of the track written by the writing pen on the touch display screen in the touch display screen also changes, that is, for the camera, the relative position of the track written by the writing pen on the touch display screen is not directly below the camera, and accordingly, the written image acquired by the camera will have a certain angle with the position of the camera, which causes distortion of the acquired video image.
By way of example and not limitation, in the solution of this embodiment, the Opencv library function is used to perform distortion correction on the initial writing image, such as icvconeimage (this function is used to copy the image content and does not need to create a memory, when programming, the original image is copied to a new position by using this function to facilitate the subsequent distortion correction), 2 cvgetphotoperceptictive transform (projection transform by four-point calculation, when programming, the coordinates of the left side of the four vertices of the original image and the four vertices of the standard rectangle need to be obtained, and then the numerical calculation for conversion between the two is performed by this function), and 34 cvwaperspective (this function is used to perform perspective transform on the image, that is, the coordinates of the four vertices of the original image are converted to the new four vertices), and the above three functions can be used to perform distortion correction on the initial writing image to obtain the writing image corresponding to the display shape of the touch display screen.
Of course, in this embodiment, the Opencv library function is used to perform distortion correction on the initial writing image, so that the obtained writing image corresponds to the display shape of the touch display screen. Specifically, the screen of the touch display screen is rectangular, and the initial writing image is subjected to distortion correction, so that the corrected writing image is also rectangular, and the distortion phenomenon that a camera acquires a video image is avoided.
S240, a writing track generated on the touch display screen in the shooting process of the camera is collected, and the collection time of the writing track is mapped to the time axis of the writing image.
And S250, scaling the writing track in an equal proportion according to the proportional relation between the size of the writing image and the size of the touch display screen.
Specifically, if the screen of the touch display screen of the tablet personal computer is rectangular at a ratio of 3:4, but when the content is written by using the writing software, the writing image has a size smaller than the screen size of the touch display screen, for example, when the size of the writing frame specified by the writing software is 2.7:3.6, the initial writing image is subjected to distortion correction by the Opencv library function so that the display shape of the writing image also becomes rectangular at a ratio of 2.7:3.6, and the size of the writing trajectory is adjusted by the image scaling algorithm so that the size ratio of the writing trajectory also becomes 2.7: 3.6.
Illustratively, if the size of the touch display screen of the tablet computer is 18cm × 24cm, the size of the writing frame specified by the writing software in the tablet computer is 16.2cm × 21.6cm, and correspondingly, the size of the writing image is also 16.2cm × 21.6cm, and in order to achieve a better combination effect between the writing image and the writing track, the writing track is scaled by the image scaling algorithm so that the size of the writing track is also 16.2cm × 21.6 cm.
It is to be understood that, when a user starts writing content on the left side of a designated frame of writing software, for example, when the writing content on the first line is "P (C) ═ 0.01 ═ 1%", and the writing content on the second line is "P (Pos | C) ═ 0.9 ═ 90%", and only images having writing contents such as P (C) ═ 0.01 ═ 1% and P (Pos | C) ═ 0.9 ═ 90% are recognized and extracted from the initial image, if the initial writing image is an image, the size of the writing image obtained by distortion correction of the initial writing image is reduced accordingly, that is, the size of the writing image is smaller than the size of the touch display screen, and the writing trajectory is also scaled by an image scaling algorithm so that the size of the writing trajectory corresponds to the size of the touch display screen, so that the composite effect of the writing track and the writing image is better.
Wherein image scaling is the process of resizing a digital image. As the size of an image increases, the visibility of the pixels making up the image becomes higher, causing the image to appear "soft". Conversely, reducing an image will enhance its smoothness and sharpness. Generally, an interpolation method is adopted for image amplification, that is, a proper interpolation algorithm is adopted to insert new elements between pixel points on the basis of original image pixels. In the scheme, the writing track is written on the screen of the tablet computer, the size of the writing track is matched with the size of the screen of the tablet computer, but the size of a writing image obtained by distortion correction is not completely the same as the size of the screen of the tablet computer, and in order to ensure the composite effect of the writing image and the writing track, the writing track needs to be scaled in an equal proportion according to the proportional relation between the size of the writing image and the size of the touch display screen.
S260, acquiring a first pixel coordinate of the writing image and a writing point set of the zoomed writing track.
Specifically, each frame of writing image is composed of a plurality of pixel points. Meanwhile, the writing track is composed of a plurality of writing point sets. As the writing image and the writing track are zoomed, the first pixel coordinate of the writing image and the writing point set of the writing track are correspondingly changed.
S270, confirming at least one frame of writing image corresponding to the writing track based on the time axis, and covering the writing point set on corresponding pixel points in the corresponding writing image based on the first pixel coordinate.
It should be noted that the length of time for the camera to record one frame of video image is determined by the frame rate of the camera, for example, the frame rate of the camera is 30 frames/second, and then the length of time for the camera to record one frame of video image is 1/30 seconds; alternatively, the frame rate of the camera is 60 frames/second, and the length of time for which the camera records one frame of video image is 1/60 seconds.
In the actual recording process, according to the normal writing speed, the writing content on the touch display screen does not change much in the time period of writing an image in one frame, that is, at least one frame of writing image is included in the time period of writing one letter, number and Chinese character. Therefore, in the process of writing one letter, there may be multiple frames of written images, and similarly, one letter corresponds to multiple pixel coordinates, that is, one letter corresponds to different pixel points.
By the technical scheme, when a user writes contents on the touch display screen, the writing contents on the first line are sequentially 'P (C) < 0.01 ≧ 1%', and the writing contents on the second line are sequentially 'P (Pos | C) < 0.9 ≧ 90%', the acquired first pixel coordinate is acquired when the P letter of the first line starts to be written, similarly, the writing tracks of the first line and the second line are zoomed by the method for zooming the writing point set to acquire the writing point set of the zoomed writing track, and the writing point sets corresponding to the first line and the second line are overlaid on the corresponding pixel points in the corresponding writing image based on acquiring the time axis corresponding to the writing of the first line and the second line and based on writing the first pixel coordinate corresponding to the first line and the second line, so that the contents of the writing point set are correspondingly added to the corresponding writing image, that is, when a user writes content on the touch display screen, other content on the touch display screen is correspondingly presented on the user's hand, so that a learner watching video content can see the content on the entire touch display screen without having to obstruct the written content from the user's hand, affecting the effect of the user's explanation, and the learning efficiency of the learner.
By way of example and not limitation, as shown in fig. 5, when a user writes content on a touch display screen (a specific writing may be a finger or a writing pen corresponding to a touch detection scheme, and in fig. 5, the writing pen), a set of writing points of the written content is collected by the above method and is merged into a written image, that is, the content written by the user in the first row is: all the writing point sets of the contents such as "p (c) ═ 0.01 ═ 1%" are covered in the pixel points of the corresponding writing image, so that the user can perform the following steps on the contents of the second line: when the user writes "P (Pos | C) ═ 0.9 ═ 90%", the content "P (C) ═ 0.01 ═ 1%" in the first line is displayed in the writing image, that is, the written content is on the hand of the user, so that the learner watching the video can not see the content written on the touch display screen because of being blocked by the hand of the user, and the effect of explaining the content by the user and the learning efficiency of the learner watching the video are improved.
Similarly, due to the writing habits of different users, the written content of the user may be blocked by the hands of the user during the process of writing the content, which may also cause poor effect of the learner watching the video when the user explains the content.
According to the technical scheme of the embodiment, an initial image which is shot by a camera facing a touch display screen and contains an area where the touch display screen is located is read; performing image recognition on the initial image to extract an initial writing image of the area where the touch display screen is located; distortion correction is carried out on the initial writing image to obtain a writing image corresponding to the display shape of the touch display screen; acquiring a writing track generated on a touch display screen in the shooting process of a camera, and mapping the acquisition time of the writing track to a time axis of a written image; carrying out equal-scale zooming on the writing track according to the proportional relation between the size of the writing image and the size of the touch display screen; acquiring a first pixel coordinate of a writing image and a writing point set of a zoomed writing track; the technical means that at least one frame of writing image corresponding to the writing track is confirmed based on the time axis, and the writing point set covers corresponding pixel points in the corresponding writing image based on the first pixel coordinate solves the problem that writing content is shielded by hands of a user in a video teaching process, improves the effect of explaining the content by the user, and improves the learning efficiency of learners watching videos.
EXAMPLE III
Fig. 6 is a schematic structural diagram of a video processing apparatus according to a third embodiment of the present invention. The embodiment may be applicable to the case of recording or remote teaching a video, and the apparatus may be implemented in a software and/or hardware manner, as shown in fig. 6, the video processing apparatus specifically includes: a get writing image module 310, a collect writing trace module 320, and an add writing trace module 330.
The module 310 for obtaining a writing image is configured to obtain a writing image corresponding to an area where the touch display screen is located from video data captured by a camera facing the touch display screen;
the acquisition writing track module 320 is used for acquiring a writing track generated on the touch display screen in the shooting process of the camera and mapping the acquisition time of the writing track to the time axis of a writing image;
an add writing trace module 330 for adding a writing trace to the written image based on the time axis and the relative position of the writing trace in the touch display screen.
According to the technical scheme of the embodiment, the writing image corresponding to the area where the touch display screen is located is obtained from the video data shot by the camera facing the touch display screen; acquiring a writing track generated on a touch display screen in the shooting process of a camera, and mapping the acquisition time of the writing track to a time axis of a written image; the technical means that the writing track is added to the writing image based on the relative positions of the time axis and the writing track in the touch display screen solves the problem that the writing content is shielded by hands in the video teaching process, and improves the efficiency of the video teaching and the attention effect of students in the learning process.
Further, a capture writing image module 310 includes:
an initial image reading unit 311, configured to read an initial image that is captured by a camera facing the touch display screen and includes an area where the touch display screen is located;
an initial writing image extracting unit 312, configured to perform image recognition on the initial image to extract an initial writing image of the area where the touch display screen is located;
and a writing image acquiring unit 313 for performing distortion correction on the initial writing image to acquire a writing image corresponding to the display shape of the touch display screen.
Further, add writing trace module 330 includes:
a zooming writing track unit 331, configured to perform equal-scale zooming on the writing track according to a proportional relationship between a size of the writing image and a size of the touch display screen;
an obtaining unit 332, configured to obtain a first pixel coordinate of the writing image and a writing point set of the zoomed writing track;
the covering unit 333 is configured to determine at least one frame of writing image corresponding to the writing track based on the time axis, and cover the writing point set on a corresponding pixel point in the corresponding writing image based on the first pixel coordinate.
Further, the video processing apparatus further includes: and a video data generation module 340, configured to perform audio acquisition based on time axis synchronization to obtain audio data, and associate the audio data with a written image to generate video data.
The video processing device can execute the video processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 7 is a schematic structural diagram of a terminal device according to a fourth embodiment of the present invention. FIG. 7 illustrates a block diagram of an exemplary computer device 412 suitable for use in implementing embodiments of the present invention. The computer device 412 shown in FIG. 7 is only one example and should not impose any limitations on the functionality or scope of use of embodiments of the present invention.
As shown in FIG. 7, computer device 412 is in the form of a general purpose computing device. Components of computer device 412 may include, but are not limited to: one or more processors or processing units 416, a system memory 428, and a bus 418 that couples the various system components including the system memory 428 and the processing unit 416.
The system memory 428 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)430 and/or cache memory 432. The computer device 412 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 434 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 7, commonly referred to as a "hard drive"). Although not shown in FIG. 7, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 418 by one or more data media interfaces. System memory 428 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 440 having a set (at least one) of program modules 442 may be stored, for instance, in system memory 428, such program modules 442 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. The program modules 442 generally perform the functions and/or methodologies of the described embodiments of the invention.
The computer device 412 may also communicate with one or more external devices 414 (e.g., keyboard, pointing device, display 424, etc.), with one or more devices that enable a user to interact with the computer device 412, and/or with any devices (e.g., network card, modem, etc.) that enable the computer device 412 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 422. Also, computer device 412 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet) through network adapter 420. As shown, network adapter 420 communicates with the other modules of computer device 412 over bus 418. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the computer device 412, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The processing unit 416 executes various functional applications and data processing by executing programs stored in the system memory 428, for example, to implement the video processing method provided by the embodiment of the present invention:
acquiring a writing image corresponding to the area where the touch display screen is located from video data shot by a camera facing the touch display screen;
acquiring a writing track generated on a touch display screen in the shooting process of a camera, and mapping the acquisition time of the writing track to a time axis of a written image;
the writing trace is added to the written image based on the time axis and the relative position of the writing trace in the touch display screen.
Further, acquiring a writing image corresponding to the area where the touch display screen is located from video data shot by a camera facing the touch display screen includes:
reading an initial image which is shot by a camera facing the touch display screen and contains an area where the touch display screen is located;
performing image recognition on the initial image to extract an initial writing image of the area where the touch display screen is located;
and carrying out distortion correction on the initial writing image to obtain a writing image corresponding to the display shape of the touch display screen.
Further, adding the writing trace to the written image based on the time axis and the relative position of the writing trace in the touch display screen includes:
carrying out equal-scale zooming on the writing track according to the proportional relation between the size of the writing image and the size of the touch display screen;
acquiring a first pixel coordinate of a writing image and a writing point set of a zoomed writing track;
and confirming at least one frame of writing image corresponding to the writing track based on the time axis, and covering the writing point set on corresponding pixel points in the corresponding writing image based on the first pixel coordinate.
Further, the video processing method further comprises:
and synchronously acquiring audio based on a time axis to acquire audio data, and associating the audio data with the written image to generate video data.
EXAMPLE five
The fifth embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the video processing method provided in all the embodiments of the present invention of the present application:
acquiring a writing image corresponding to the area where the touch display screen is located from video data shot by a camera facing the touch display screen;
acquiring a writing track generated on a touch display screen in the shooting process of a camera, and mapping the acquisition time of the writing track to a time axis of a written image;
the writing trace is added to the written image based on the time axis and the relative position of the writing trace in the touch display screen.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (8)
1. A video processing method, comprising:
acquiring a writing image corresponding to an area where a touch display screen is located from video data shot by a camera facing the touch display screen;
acquiring a writing track generated on a touch display screen in the shooting process of the camera, and mapping the acquisition time of the writing track to the time axis of the writing image;
adding the writing trace to the writing image based on the time axis and the relative position of the writing trace in the touch display screen;
the adding the written trace to the written image based on the time axis and the relative position of the written trace in the touch display screen includes:
scaling the writing track in an equal proportion according to the proportional relation between the size of the writing image and the size of the touch display screen;
acquiring a first pixel coordinate of the writing image and a writing point set of the zoomed writing track;
and confirming at least one frame of writing image corresponding to the writing track based on the time axis, and covering the corresponding pixel points in the corresponding writing image with the writing point set based on the first pixel coordinate.
2. The method of claim 1, wherein the obtaining of the writing image corresponding to the area of the touch display screen from the video data captured by the camera facing the touch display screen comprises:
reading an initial image which is shot by a camera facing the touch display screen and contains an area where the touch display screen is located;
performing image recognition on the initial image to extract an initial writing image of the area where the touch display screen is located;
and carrying out distortion correction on the initial writing image to obtain a writing image corresponding to the display shape of the touch display screen.
3. The method of claim 1, further comprising:
and synchronously acquiring audio based on the time axis to acquire audio data, and associating the audio data with the written image to generate video data.
4. The method of claim 1, wherein the camera is mounted directly above the touch display screen.
5. A video processing apparatus, comprising:
the device comprises a writing image acquisition module, a writing image acquisition module and a writing image acquisition module, wherein the writing image acquisition module is used for acquiring a writing image corresponding to the area of the touch display screen from video data shot by a camera facing the touch display screen;
the acquisition writing track module is used for acquiring a writing track generated on a touch display screen in the shooting process of the camera and mapping the acquisition time of the writing track to the time axis of the writing image;
a writing trace adding module for adding the writing trace to the writing image based on the time axis and the relative position of the writing trace in the touch display screen;
the writing trace adding module comprises:
the zooming writing track unit is used for zooming the writing track in an equal proportion according to the proportional relation between the size of the writing image and the size of the touch display screen;
the acquisition unit is used for acquiring a first pixel coordinate of the writing image and a writing point set of the zoomed writing track;
and the covering unit is used for confirming at least one frame of writing image corresponding to the writing track based on the time axis and covering the corresponding pixel points in the corresponding writing image with the writing point set based on the first pixel coordinate.
6. The apparatus of claim 5, wherein the means for obtaining a written image comprises:
the initial image reading unit is used for reading an initial image which is shot by a camera facing the touch display screen and contains the area where the touch display screen is located;
an initial writing image extracting unit, configured to perform image recognition on the initial image to extract an initial writing image of an area where the touch display screen is located;
and the acquisition writing image unit is used for carrying out distortion correction on the initial writing image to acquire a writing image corresponding to the display shape of the touch display screen.
7. A terminal device, characterized in that the device comprises:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a video processing method as claimed in any one of claims 1 to 4.
8. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a video processing method according to any one of claims 1 to 4.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711103557.8A CN107909022B (en) | 2017-11-10 | 2017-11-10 | Video processing method and device, terminal equipment and storage medium |
| PCT/CN2017/116748 WO2019090903A1 (en) | 2017-11-10 | 2017-12-17 | Video processing method and apparatus, terminal device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711103557.8A CN107909022B (en) | 2017-11-10 | 2017-11-10 | Video processing method and device, terminal equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107909022A CN107909022A (en) | 2018-04-13 |
| CN107909022B true CN107909022B (en) | 2020-06-16 |
Family
ID=61844613
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711103557.8A Active CN107909022B (en) | 2017-11-10 | 2017-11-10 | Video processing method and device, terminal equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN107909022B (en) |
| WO (1) | WO2019090903A1 (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10873709B2 (en) * | 2018-07-10 | 2020-12-22 | Fuji Xerox Co., Ltd. | Systems and methods for automatic detection and insetting of digital streams into a 360-degree video |
| JP2020098420A (en) * | 2018-12-17 | 2020-06-25 | ソニー株式会社 | Image processing apparatus, image processing method and program |
| CN109905733A (en) * | 2019-01-21 | 2019-06-18 | 侯力宇 | A kind of novel video purchase system |
| CN113014846B (en) * | 2019-12-19 | 2022-07-22 | 华为技术有限公司 | Video acquisition control method, electronic equipment and computer readable storage medium |
| CN112770160A (en) * | 2020-12-24 | 2021-05-07 | 沈阳麟龙科技股份有限公司 | Stock analysis video creation system and method |
| CN112580574A (en) * | 2020-12-28 | 2021-03-30 | 北京翰立教育科技有限公司 | Intelligent learning method and device based on handwritten character recognition |
| CN113554904B (en) * | 2021-07-12 | 2023-04-07 | 江苏欧帝电子科技有限公司 | Intelligent processing method and system for multi-mode collaborative education |
| CN114120337B (en) * | 2021-10-15 | 2024-04-09 | 华南理工大学 | Method, system, device and storage medium for recovering paper writing in real time |
| CN114527922A (en) * | 2022-01-13 | 2022-05-24 | 珠海视熙科技有限公司 | Method for realizing touch control based on screen identification and screen control equipment |
| CN116844166B (en) * | 2023-08-24 | 2023-11-24 | 青岛罗博数码科技有限公司 | Video positioning device and method based on learning behavior |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102929387A (en) * | 2012-09-25 | 2013-02-13 | 长安大学 | Man-machine interaction method and man-machine interaction system based on common paper and pen |
| CN103945160A (en) * | 2014-02-27 | 2014-07-23 | 广州视睿电子科技有限公司 | Method and device for realizing remote video conversation in Android system |
| CN105451065A (en) * | 2014-08-26 | 2016-03-30 | 天脉聚源（北京）教育科技有限公司 | Simultaneous recording method for a variety of media |
| CN105635783A (en) * | 2015-12-31 | 2016-06-01 | 田雪松 | Manufacturing method for multimedia file |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100693343B1 (en) * | 2005-12-29 | 2007-03-12 | 이코리아시스템 (주) | A cyber eletronic-board |
| CN101105895A (en) * | 2007-08-10 | 2008-01-16 | 上海迈辉信息技术有限公司 | Audio and video frequency multi-stream combination teaching training system and realization method |
| CN101436353A (en) * | 2007-11-16 | 2009-05-20 | 北京新学堂网络科技有限公司 | Preparation method for remote schoolteaching video simulating true teaching room schoolteaching effect |
| CN103177611A (en) * | 2011-12-23 | 2013-06-26 | 李云峰 | Method for realizing multimedia courseware on E-ink book |
| CN102903265A (en) * | 2012-09-19 | 2013-01-30 | 河南智游网络技术有限公司 | Method for automatically recording classroom teaching |
| CN104575120B (en) * | 2015-01-09 | 2017-02-22 | 代四广 | Display system for aided teaching |
| CN104575137B (en) * | 2015-01-19 | 2018-01-12 | 广州贝远信息技术有限公司 | Split type scene interaction multi-medium intelligent terminal |
| CN105120195B (en) * | 2015-09-18 | 2019-04-26 | 谷鸿林 | Content recordal, playback system and method |
| CN105513438A (en) * | 2016-01-21 | 2016-04-20 | 郑尚勤 | Teaching demonstration instrument |
| CN106970681A (en) * | 2017-02-21 | 2017-07-21 | 广州视源电子科技股份有限公司 | Write display methods and its system |
-
2017
- 2017-11-10 CN CN201711103557.8A patent/CN107909022B/en active Active
- 2017-12-17 WO PCT/CN2017/116748 patent/WO2019090903A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102929387A (en) * | 2012-09-25 | 2013-02-13 | 长安大学 | Man-machine interaction method and man-machine interaction system based on common paper and pen |
| CN103945160A (en) * | 2014-02-27 | 2014-07-23 | 广州视睿电子科技有限公司 | Method and device for realizing remote video conversation in Android system |
| CN105451065A (en) * | 2014-08-26 | 2016-03-30 | 天脉聚源（北京）教育科技有限公司 | Simultaneous recording method for a variety of media |
| CN105635783A (en) * | 2015-12-31 | 2016-06-01 | 田雪松 | Manufacturing method for multimedia file |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019090903A1 (en) | 2019-05-16 |
| CN107909022A (en) | 2018-04-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107909022B (en) | Video processing method and device, terminal equipment and storage medium | |
| US11849196B2 (en) | Automatic data extraction and conversion of video/images/sound information from a slide presentation into an editable notetaking resource with optional overlay of the presenter | |
| US11335379B2 (en) | Video processing method, device and electronic equipment | |
| Karambakhsh et al. | Deep gesture interaction for augmented anatomy learning | |
| CN109313812B (en) | Shared experience with contextual enhancements | |
| US9049482B2 (en) | System and method for combining computer-based educational content recording and video-based educational content recording | |
| WO2021035223A1 (en) | Automatic data extraction and conversion of video/images/sound information from a board-presented lecture into an editable notetaking resource | |
| WO2021213067A1 (en) | Object display method and apparatus, device and storage medium | |
| CN112243583A (en) | Multi-endpoint mixed reality conference | |
| US20190130193A1 (en) | Virtual Reality Causal Summary Content | |
| CN113934297A (en) | Interaction method and device based on augmented reality, electronic equipment and medium | |
| US20230353702A1 (en) | Processing device, system and method for board writing display | |
| CN109461203B (en) | Gesture three-dimensional image generation method and device, computer equipment and storage medium | |
| CN109857244B (en) | Gesture recognition method and device, terminal equipment, storage medium and VR glasses | |
| US11557065B2 (en) | Automatic segmentation for screen-based tutorials using AR image anchors | |
| CN116434253A (en) | Image processing method, device, equipment, storage medium and product | |
| CN114222065A (en) | Image processing method, image processing apparatus, electronic device, storage medium, and program product | |
| CN112817441A (en) | Method and device for combining key and human eye identification | |
| CN108805951B (en) | Projection image processing method, device, terminal and storage medium | |
| US20240135501A1 (en) | Video generation method and apparatus, device and medium | |
| CN113840177B (en) | Live interaction method and device, storage medium and electronic equipment | |
| TWI762830B (en) | System for displaying hint in augmented reality to play continuing film and method thereof | |
| JP2023167630A (en) | Image processing apparatus and image processing method | |
| US20230326094A1 (en) | Integrating overlaid content into displayed data via graphics processing circuitry and processing circuitry using a computing memory and an operating system memory | |
| CN113784077B (en) | Information processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |