CN111918089A - Video stream processing method, video stream display method, device and equipment - Google Patents

Video stream processing method, video stream display method, device and equipment Download PDF

Info

Publication number
CN111918089A
CN111918089A CN202010797090.7A CN202010797090A CN111918089A CN 111918089 A CN111918089 A CN 111918089A CN 202010797090 A CN202010797090 A CN 202010797090A CN 111918089 A CN111918089 A CN 111918089A
Authority
CN
China
Prior art keywords
video frame
video
teeth
irregular
tooth
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010797090.7A
Other languages
Chinese (zh)
Inventor
吴文艺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Fanxing Huyu IT Co Ltd
Original Assignee
Guangzhou Fanxing Huyu IT Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Fanxing Huyu IT Co Ltd filed Critical Guangzhou Fanxing Huyu IT Co Ltd
Priority to CN202010797090.7A priority Critical patent/CN111918089A/en
Publication of CN111918089A publication Critical patent/CN111918089A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application

Abstract

The application discloses a video stream processing method, a video stream display device and video stream display equipment, and belongs to the technical field of audio and video processing. The method comprises the following steps: acquiring a first video frame in a first video stream of a main broadcast account; identifying an irregular tooth in the first video frame, the irregular tooth comprising at least one tooth of the plurality of teeth; and in response to the first video frame including the irregular teeth, correcting the irregular teeth in the first video frame to obtain a second video frame, wherein the second video frame includes target regular teeth obtained by correcting the irregular teeth. When the video picture is displayed according to the second video frame, a picture for further beautifying the image of the anchor can be displayed, and the beautifying effect of the displayed human face is improved.

Description

Video stream processing method, video stream display method, device and equipment
Technical Field
The present application relates to the field of audio and video processing technologies, and in particular, to a video stream processing method, a video stream display device, and an apparatus.
Background
In the live broadcast platform, a main broadcast client side sends the collected live broadcast stream of the main broadcast user to a plurality of audience client sides through a live broadcast server. And the audience client displays the live broadcast picture of the anchor user according to the live broadcast stream.
When the anchor user starts the beautifying function, the anchor client side can conduct beautifying processing on the live stream when the live stream of the anchor user is collected. Such as eye enlargement, skin whitening, face thinning, etc., of the anchor face in the video frames of the live stream. And then the anchor client sends the live stream after the beauty treatment to the audience client through the live server, and the audience client displays a live broadcast picture comprising the anchor face after the beauty treatment according to the live stream after the beauty treatment.
At present, the anchor client performs the beautifying processing on the live stream of the anchor user in general by adopting the above mode. The face-beautifying effect of the displayed anchor face is limited.
Disclosure of Invention
The application provides a video stream processing method, a video stream display device and video stream display equipment, which can improve the beauty effect of a displayed face. The technical scheme is as follows:
according to an aspect of the present application, there is provided a video stream processing method, the method including:
acquiring a first video frame in a first video stream of a main broadcast account;
identifying an irregular tooth in the first video frame, the irregular tooth comprising at least one tooth of a plurality of teeth;
and in response to the first video frame including the irregular teeth, correcting the irregular teeth in the first video frame to obtain a second video frame including target regular teeth obtained by correcting the irregular teeth.
According to another aspect of the present application, there is provided a video stream display method, the method including:
displaying a first video picture in a user interface according to a first video frame in a first video stream of a main broadcast account, wherein irregular teeth are displayed in the first video picture;
in response to receiving a dental beautification operation, obtaining a second video stream, a second video frame in the second video stream being obtained by correcting the irregular teeth in the first video frame;
and displaying a second video picture in a user interface according to the second video frame, wherein teeth displayed in the second video picture are tidy teeth.
According to another aspect of the present application, there is provided a video stream processing apparatus, the apparatus including:
the acquisition module is used for acquiring a first video frame in a first video stream of the anchor account;
an identification module to identify an irregular tooth in the first video frame, the irregular tooth comprising at least one of a plurality of teeth;
and the correcting module is used for responding to the first video frame including the irregular teeth, correcting the irregular teeth in the first video frame to obtain a second video frame, and the second video frame includes target regular teeth obtained by correcting the irregular teeth.
Optionally, the identification module is configured to:
and calling a machine learning model, and identifying irregular teeth in the first video frame, wherein the machine learning model is obtained by training an irregular tooth sample set.
Optionally, the machine learning model comprises an image segmentation model and an image classification model; the identification module is configured to:
according to the image segmentation model, segmenting a tooth image region corresponding to a single tooth in the first video frame, wherein the tooth comprises a regular tooth and an irregular tooth;
identifying the out-of-order teeth in the dental image region according to the image classification model.
Optionally, the corrective module is configured to:
determining first two-side vertexes of each irregular tooth and second two-side vertexes of each regular tooth;
determining a correction curve through a polynomial fitting device according to the first two-side vertex and the second two-side vertex;
and correcting the irregular teeth in the first video frame according to the correction curve.
Optionally, the correction module is configured to:
establishing a coordinate system according to the first two-side vertex and the second two-side vertex;
determining coordinates of the first two-side vertex and the second two-side vertex in the coordinate system;
and determining a correction function through a polynomial fitting device according to the coordinates of the first two-side vertex and the coordinates of the second two-side vertex, wherein a function image of the correction function is the correction curve.
Optionally, the corrective module is configured to:
determining target correction points corresponding to the first two-side vertex points in the first video frame, wherein the target correction points are on the correction curve;
and according to the target correction point, carrying out image deformation on the tooth image region corresponding to the irregular tooth by a moving least square method to obtain the target regular tooth, wherein the third two side vertexes of the target regular tooth are the target correction point.
Optionally, the apparatus further comprises:
a replacing module, configured to replace the first video frame in the first video stream with the second video frame to obtain a second video stream;
and the sending module is used for sending the second video stream to the audience client.
According to another aspect of the present application, there is provided a video stream display apparatus, the apparatus including:
the display module is used for displaying a first video picture in a user interface according to a first video frame in a first video stream of a main broadcast account, wherein irregular teeth are displayed in the first video picture;
an obtaining module, configured to obtain a second video stream in response to receiving a dental beautification operation, where a second video frame in the second video stream is obtained by correcting the irregular teeth in the first video frame;
and the display module is further used for displaying a second video picture in a user interface according to the second video frame, wherein the teeth displayed in the second video picture are tidy teeth.
According to another aspect of the present application, there is provided a computer device comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by the processor to implement a video stream processing method or a video stream display method as described above.
According to another aspect of the present application, there is provided a computer readable storage medium having stored therein at least one instruction, at least one program, code set, or set of instructions that is loaded and executed by a processor to implement a video stream processing method or a video stream display method as described above.
According to another aspect of the application, a computer program product or computer program is provided, comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to execute the video stream processing method or the video stream display method provided in various alternative implementations of the above aspects
The beneficial effect that technical scheme that this application provided brought includes at least:
the second video frame is obtained by identifying the irregular teeth in the first video frame and correcting the irregular teeth to obtain regular teeth. When the video picture is displayed based on the second video frame, a picture for further beautifying the character of the anchor can be displayed. When the method is used in scenes such as short videos or video on demand, the method can also realize the dental correction of irregular teeth in the video frames, thereby displaying pictures for further beautifying the character images in the video frames. The beauty effect of the displayed face is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram illustrating an effect of straightening teeth in a video frame according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of a video stream processing system according to an exemplary embodiment of the present application;
fig. 3 is a schematic flowchart of a video stream processing method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of a video stream display method according to an embodiment of the present application;
fig. 5 is a schematic flowchart of another video stream processing method provided in the embodiment of the present application;
FIG. 6 is a schematic diagram of an implementation process for identifying irregular teeth in a first video frame according to an embodiment of the present disclosure;
FIG. 7 is a schematic diagram of an implementation process for correcting irregular teeth in a first video frame according to an embodiment of the present disclosure;
FIG. 8 is a schematic diagram of an implementation process for determining a calibration curve provided by an embodiment of the present application;
FIG. 9 is a diagram illustrating an implementation process for correcting irregular teeth in a first video frame according to a correction curve according to an embodiment of the present disclosure;
fig. 10 is a block diagram of a video stream processing apparatus according to an embodiment of the present application;
fig. 11 is a block diagram of another video stream processing apparatus provided in an embodiment of the present application;
fig. 12 is a block diagram of a video stream display apparatus according to an embodiment of the present application;
fig. 13 is a schematic structural diagram of a server provided in an embodiment of the present application;
fig. 14 is a schematic structural diagram of a terminal according to an embodiment of the present application.
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
Detailed Description
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
Fig. 1 is a schematic diagram illustrating an effect of straightening teeth in a video frame according to an embodiment of the present application. As shown in fig. 1 (a), after receiving a first video stream of a anchor account sent by an anchor client, a server acquires a first video frame 101 in the first video stream. The first video frame is any video frame in the first video stream. As shown in fig. 1 (b), the server segments a tooth image region corresponding to each tooth in the first video frame 101, and identifies an irregular tooth in the tooth image region. First tooth 102a, second tooth 102b, third tooth 102c, and fourth tooth 102d are irregular teeth, and fifth tooth 102e and sixth tooth 102f are regular teeth. As shown in fig. 1 (c), the server determines a first two-sided vertex of each irregular tooth and a second two-sided vertex of each regular tooth. The first tooth 102a has vertices at both sides a1 and a2, respectively, the second tooth 102b has vertices at both sides a3 and a4, respectively, the third tooth 102c has vertices at both sides a5 and a6, respectively, the fourth tooth 102d has vertices at both sides a7 and a8, respectively, the fifth tooth 102e has vertices at both sides a9 and a10, respectively, and the sixth tooth 102f has vertices at both sides a11 and a12, respectively. As shown in fig. 1 (d), the server establishes a rectangular coordinate system according to the first two-side vertexes of each irregular tooth and the second two-side vertexes of each regular tooth, and determines the coordinates of the first two-side vertexes and the second two-side vertexes. And a correction curve 103 is determined by a polynomial fitting method. The server then determines the target correction points for the first bilateral vertices of each irregular tooth on the correction curve 103. The target correction point corresponding to a1 is b1, the target correction point corresponding to a2 is b2, the target correction point corresponding to a3 is b3, the target correction point corresponding to a4 is b4, the target correction point corresponding to a5 is b5, the target correction point corresponding to a6 is b6, the target correction point corresponding to a7 is b7, and the target correction point corresponding to a8 is b 8. As shown in fig. 1 (e), the server performs image deformation on a tooth image region corresponding to the irregular tooth by a moving least square method according to the target correction point b1-b8, and obtains a second video frame. The second video frame includes target aligned teeth obtained by correcting the misaligned teeth. Among them, the first target aligned tooth 104a is obtained by correcting the first tooth 102a, the second target aligned tooth 104b is obtained by correcting the second tooth 102b, the third target aligned tooth 104c is obtained by correcting the third tooth 102c, and the fourth target aligned tooth 104d is obtained by correcting the fourth tooth 102 d. Third both-side vertexes of the first target aligned tooth are b1 and b2, third both-side vertexes of the second target aligned tooth are b3 and b4, third both-side vertexes of the third target aligned tooth are b5 and b6, and third both-side vertexes of the fourth target aligned tooth are b7 and b 8. Wherein b1, b2, b3, b4, b5, b6, b7 and b8 are on the correction curve. And the server replaces the first video frame in the first video stream with the second video frame to obtain a second video stream. Optionally, the server corrects all video frames in the first video stream by using the method described above, so as to obtain the second video stream. The server then sends the second video stream to the viewer client for playback.
The second video stream is obtained by correcting irregular teeth in the first video frame of the first video stream of the anchor account number to regular teeth. When the audience client plays the second video stream, the video picture which further beautifies the image of the anchor can be displayed, and the beautifying effect of the displayed face is improved.
Fig. 2 is a schematic structural diagram of a video stream processing system according to an exemplary embodiment of the present application, as shown in fig. 2, the system includes: a server 210 and at least one viewer terminal 220. Optionally, when the system is used in a short video scene or a live scene, the system further comprises an anchor terminal 230. The viewer terminal 220 is used to display video frames in the video stream transmitted by the server 210. Anchor terminal 230 is used to record a live stream or short video stream of the anchor account.
The server 210 may be a server, a server cluster composed of several servers, or a virtual server in a cloud computing service center, and the like, which is not limited herein. The anchor terminal 230 may be a terminal device including a camera, such as a smart phone, a tablet computer, a desktop computer, a notebook computer, and the like. The viewer terminal 220 may be a smart phone, a computer, a television, a multimedia player, an e-reader, etc. The connection between the server 210 and the anchor terminal 230 may be established through a wired network or a wireless network, and the connection between the server 210 and the viewer terminal 220 may be established through a wired network or a wireless network. As shown in fig. 2, in the embodiment of the present application, the anchor terminal 230 is a desktop computer, and the viewer terminal 220 is a smart phone.
The viewer terminal 220 is provided with a viewer client, and the viewer terminal 220 is connected to the server 210 through the viewer client, and the server 210 corresponds to the viewer client. The anchor terminal 230 has an anchor client installed thereon, and the anchor terminal 230 is connected to the server 210 through the anchor client. The spectator client on the spectator terminal and the anchor client on the anchor terminal may be the same or different.
Optionally, the video stream processing method provided in the embodiment of the present application may be executed by a server, an anchor client, or a viewer client. The video stream in the embodiment of the present application refers to a live stream in a live scene or a short video stream in a short video scene, and also refers to a multimedia stream of a movie or a multimedia stream of a tv show in a video-on-demand scene. When the video stream in the embodiment of the application is a live stream, the server is a live server. When the video stream in the embodiment of the present application is a short video stream, the server is a short video server. The embodiment of the application is mainly explained by being applied to a live scene.
Fig. 3 is a schematic flowchart of a video stream processing method according to an embodiment of the present application. The method may be used for a server in a system as shown in fig. 1. As shown in fig. 3, the method includes:
step 301, a first video frame in a first video stream of a primary account is acquired.
The anchor account is any anchor account in the server. The first video stream of the anchor account is any video stream of the anchor account. The first video frame in the first video stream is any video frame in the first video stream.
Optionally, when the first video stream is a live stream, the first video stream is a live stream of any anchor account in the server. When the first video stream is a short video stream, the first video stream is a short video stream of any anchor account in the server.
Step 302, identify irregular teeth in a first video frame.
Irregular teeth means that the tooth is skewed relative to other teeth. The irregular teeth include at least one of a plurality of teeth in the first video frame.
Optionally, the server processes the first video frame through a Watershed Algorithm (Watershed Algorithm), a threshold segmentation method, and the like, and can obtain a binarized image corresponding to the first video frame, that is, an image with a gray value of a pixel point of 0 or 255. Therefore, the tooth image area corresponding to each tooth in the first video frame can be divided, and the outline of each tooth in the first video frame can be determined. Comparing the contour of each tooth to the known contour of a clean tooth allows the determination of a clean tooth in the first video frame.
Optionally, the server identifies irregular teeth in the first video frame by a template matching method. One or more tooth aligning templates are established in the server, and the server performs template matching with the first video frame through the one or more tooth aligning templates. When a region with a correlation coefficient higher than a first threshold value with one or more templates of the clean teeth appears in the first video frame, the region is a clean teeth image region corresponding to the clean teeth; when a region with the correlation coefficient of the template of one or more regular teeth higher than a second threshold and lower than a first threshold appears in the first video frame, the region is a regular tooth image region corresponding to irregular teeth. Wherein the correlation coefficient is used to indicate how similar a region in the first video frame is to a template of one or more clean teeth. The first threshold and the second threshold are set by the server.
Optionally, the server is also capable of identifying irregular teeth in the first video frame by invoking a machine learning model. The machine learning model is obtained by training an irregular tooth sample set. Optionally, the sample set of irregular teeth includes an image of the irregular teeth and information reflecting that the image belongs to the irregular teeth. Optionally, the machine learning model is a Convolutional Neural Networks (CNN) model. The server can identify whether irregular teeth are included in the first video frame according to the machine learning model.
And 303, responding to the first video frame including the irregular teeth, correcting the irregular teeth in the first video frame to obtain a second video frame, wherein the second video frame includes target regular teeth obtained by correcting the irregular teeth.
Optionally, the server determines a target correction position corresponding to the irregular tooth, and performs image deformation on a tooth image region of the irregular tooth in the first video frame based on the target correction position by a moving least square method. That is, irregular teeth are corrected to regular teeth according to the target orthodontic position in the first video frame, thereby obtaining a second video frame.
In summary, the video stream processing method provided in the embodiment of the present application obtains the second video frame by identifying the irregular teeth in the first video frame and correcting the irregular teeth to obtain the regular teeth. When the video picture is displayed based on the second video frame, a picture for further beautifying the character of the anchor can be displayed. When the method is used in scenes such as short videos or video on demand, the method can also realize the dental correction of irregular teeth in the video frames, thereby displaying pictures for further beautifying the character images in the video frames. The beauty effect of the displayed face is improved.
Fig. 4 is a schematic flowchart of a video stream display method according to an embodiment of the present application. The method may be used in a viewer terminal or a viewer client on a viewer terminal in a system such as that shown in fig. 1. As shown in fig. 4, the method includes:
step 401, displaying a first video image in a user interface according to a first video frame in a first video stream of a main broadcast account, wherein irregular teeth are displayed in the first video image.
The first video stream is transmitted by the server to the viewer client. The first video frame is any video frame in the first video stream. Optionally, when the first video stream is a live stream, the user interface is a live room interface corresponding to the anchor account. When the first video stream is a short video stream, the user interface is a short video playing interface.
Step 402, in response to receiving the dental beautification operation, obtaining a second video stream, wherein a second video frame in the second video stream is obtained by correcting irregular teeth in the first video frame.
Optionally, the server corrects the irregular teeth in the first video frame by a machine learning model. The machine learning model is obtained by training an irregular tooth sample set. The set of irregular tooth samples includes an image of the irregular teeth and information reflecting that the image belongs to the irregular teeth.
Optionally, the second video stream is obtained by the server correcting irregular teeth in the first video frame, obtaining a second video frame corresponding to the first video frame, and replacing the first video frame with the second video frame. Optionally, the server rectifies all video frames in the first video stream, resulting in the second video stream.
Optionally, the viewer client receives the dental beautification operation when the viewer client detects a touch operation of the viewer user against a "beautify teeth" button in the user interface.
And step 403, displaying a second video image in the user interface according to the second video frame, wherein the teeth displayed in the second video image are tidy teeth.
The second video frame includes aligned teeth that are obtained by correcting for irregular teeth in the first video frame in the first video stream.
In summary, the video stream display method provided in the embodiment of the present application displays the second video frame according to the second video frame, where the second video frame is obtained by correcting irregular teeth in the first video frame. I.e. the second video stream shows video pictures comprising aligned teeth. When the second video picture is displayed according to the second video frame, a picture for further beautifying the character of the anchor can be displayed. When the method is used in scenes such as short videos or video on demand, pictures for further beautifying the character images in the video frames can be displayed. The beauty effect of the displayed face is improved.
Fig. 5 is a schematic flowchart of another video stream processing method according to an embodiment of the present application. The method may be used in a system as shown in fig. 1. As shown in fig. 5, the method includes:
step 501, a server acquires a first video frame in a first video stream of a main broadcast account.
Optionally, when the anchor client sends the first video stream to the server and sends an orthodontic request for the first video stream, the server acquires a first video frame in the first video stream and performs the step of orthodontic. Or when the server receives the video stream of any anchor account, acquiring the first video frame in the stream and executing the dental correction. Illustratively, when the "american teeth" function is turned on in the anchor client, the request for orthodontics is simultaneously sent when the first video stream is sent to the server.
Step 502, the server identifies irregular teeth in the first video frame.
The irregular tooth includes at least one tooth of the plurality of teeth. Optionally, the server identifies irregular teeth in the first video frame by a watershed algorithm, a threshold segmentation method and the like. And the server realizes the identification of irregular teeth in the first video frame by a template matching method. Alternatively, the server can also identify irregular teeth in the first video by invoking a machine learning model. The machine learning model is obtained by training an irregular tooth sample set. Optionally, the irregular tooth sample set includes at least one of an image of irregular teeth and information for reflecting that the image belongs to irregular teeth, and an image of regular teeth and information for reflecting that the image belongs to regular teeth. The server, when identifying the irregular teeth in the first video frame, can also identify the regular teeth in the first video frame.
Optionally, the machine learning model includes an image segmentation model and an image classification model. The image segmentation model is a model based on a Single Shot multi box Detector (SSD). The image classification model is a Convolutional Neural Networks (CNN) model. As shown in fig. 6, the implementation process of step 502 includes the following steps 5021 and 5022:
in step 5021, according to the image segmentation model, a tooth image region corresponding to a single tooth in the first video frame is segmented, wherein the tooth comprises a regular tooth and an irregular tooth.
Optionally, the tooth image region corresponding to a single tooth includes a tooth image region corresponding to a clean tooth and a tooth image region corresponding to an irregular tooth. Illustratively, with continued reference to fig. 1, the tooth image region corresponding to the irregular tooth 102a is a display region of the irregular tooth 102a in the first video frame 101.
Optionally, when the server detects that the number of the tooth image areas in the first video frame is less than a preset value, the server determines that irregular teeth are not included in the first video frame. For example, when the server detects that the number of dental image regions in the first video frame is less than 4, the server determines that irregular teeth are not included in the first video frame.
In step 5022, irregular teeth in the tooth image region are identified according to the image classification model.
And the server identifies the tooth image region corresponding to each tooth in the first video frame according to the image classification model, so that the irregular teeth in the first video frame and the tooth image region corresponding to the irregular teeth are determined.
Step 503, in response to the first video frame including the irregular teeth, the server corrects the irregular teeth in the first video frame to obtain a second video frame, and the second video frame includes target regular teeth obtained by correcting the irregular teeth.
Optionally, as shown in fig. 7, the implementation process of step 503 includes the following steps 5031 to 5033:
in step 5031, a first two-sided vertex for each out-of-order tooth and a second two-sided vertex for each in-order tooth are determined.
Optionally, the server identifies a tooth image region corresponding to each tooth in the first video frame through a target detection model based on a yolo (young Only Look once) algorithm, so as to determine a first two-side vertex of each irregular tooth and a second two-side vertex corresponding to each regular tooth. The first two side vertexes of the irregular teeth comprise two left and right vertexes of the lower end of the irregular maxillary teeth and/or two left and right vertexes of the upper end of the irregular mandibular teeth. The second lateral vertexes of the aligned teeth include left and right vertexes of a lower end of the aligned maxillary tooth and/or left and right vertexes of an upper end of the aligned mandibular tooth.
Optionally, the server can identify the tooth image region corresponding to each tooth in the first video frame through the image classification model in the step 5022, so as to determine the clean teeth in the first video frame. Alternatively, the server determines the teeth other than the irregular teeth in the first video frame as the regular teeth.
In step 5032, a correction curve is determined by a polynomial fitting method according to the first two-sided vertex and the second two-sided vertex.
The calibration curve is used to calibrate out-of-alignment teeth.
Optionally, as shown in fig. 8, the implementation of step 5032 includes the following steps 5032a-5032 c:
in step 5032a, a coordinate system is established according to the first two-sided vertex and the second two-sided vertex.
Optionally, the server establishes a coordinate system according to the first two-side vertex and the second two-side vertex, which means that the server establishes the coordinate system in the first video frame where the first two-side vertex and the second two-side vertex are located. For example, the server uses the center point of the first video frame as the origin of the coordinate system, and establishes the coordinate system according to the horizontal direction and the vertical direction in the first video frame.
In step 5032b, the coordinates of the first two-sided vertex and the second two-sided vertex in the coordinate system are determined.
Optionally, the origin of the coordinate system is in the first video frame, and the server determines the coordinates of the first two-side vertex and the second two-side vertex in the coordinate system according to the display positions of the first two-side vertex and the second two-side vertex in the first video frame.
In step 5032c, a correction function is determined by a polynomial fitting method according to the coordinates of the first two side vertices and the coordinates of the second two side vertices, and a function image of the correction function is a correction curve.
Alternatively, when the correction function is determined using a polynomial fitting method, the coefficient of the fitting function is determined by a least square method, thereby obtaining the fitting function. The server fits a preselected function using polynomials of a plurality of orders, and determines the curvature of the function image at the first both-side vertex and the second both-side vertex regions to conform to the preselected function of the set value as the fitted function. The curvature set value is used for ensuring that the correction curve is a smooth curve with a certain radian, thereby ensuring the tidying effect after the irregular teeth are corrected.
In step 5033, the irregular teeth in the first video frame are corrected according to the correction curve.
Optionally, as shown in fig. 9, the implementation process of step 5033 includes the following steps 5033a and 5033 b:
in step 5033a, target correction points corresponding to the vertices of the first two sides are determined in the first video frame, and the target correction points are on the correction curve.
Optionally, the server determines the projection point of the first two side vertices on the correction curve as the target correction point. Illustratively, with continued reference to fig. 1, the projected point of a1 on the correction curve 103 is b1, and the projected point of a2 on the correction curve 103 is b 2. Then b1 is the target correction point corresponding to a1 on the correction curve, and b2 is the target correction point corresponding to a2 on the correction curve
In step 5033b, the tooth image region corresponding to the irregular tooth is subjected to image deformation by a moving least square method according to the target correction point to obtain a target regular tooth, and the third both side vertexes of the target regular tooth are the target correction points.
Optionally, the dental image region is obtained by the image segmentation model in step 5021. The dental image region is determined according to the template matching method in step 502. Alternatively, the dental image region is obtained from the binarized image corresponding to the first video frame obtained by the watershed algorithm and the threshold segmentation method in step 502.
And the server carries out image deformation on the tooth image region corresponding to the irregular tooth by a mobile least square method, so that the top points of the first two sides of the irregular tooth after the image deformation are superposed with the target correction point corresponding to the irregular tooth, and the target regular tooth is obtained. The image deformation is carried out by the moving least square method, so that smooth and real deformation can be generated in the tooth image area corresponding to the irregular teeth, and the tooth straightening presents a natural effect.
Optionally, the server performs image deformation on an image area corresponding to at least one irregular tooth in the first video frame.
Step 504, the server replaces the first video frame in the first video stream with the second video frame to obtain a second video stream.
Optionally, the server performs the step of orthodontics on all video frames in the first video stream, thereby obtaining the second video stream.
Step 505, the audience client displays a first video image in the user interface according to a first video frame in a first video stream of the anchor account, wherein irregular teeth are displayed in the first video image.
The first video stream is transmitted by the server to the viewer client. Optionally, when the first video stream is a live stream, the first video frame is a live frame of the anchor account. When the first video stream is a short video stream, the first video frame is a frame of a short video of the main broadcast account.
In response to receiving the dental beautification operation, the viewer client obtains a second video stream, where a second video frame of the second video stream is obtained by straightening irregular teeth of the first video frame, step 506.
Optionally, when the viewer client receives the dental beautification operation, a dental beautification request is sent to the server, and the server performs the step of orthodontically straightening the first video stream and sends the second video stream to the viewer client.
Optionally, the audience client further sends the dental beautification request to a anchor client through a server, the anchor client performing the step of orthodontics on the first video stream and sending the second video stream to the audience client through the server. Alternatively, the viewer client performs the dental correction step on the received first video stream transmitted by the server, thereby obtaining the second video stream.
And step 507, the audience client displays a second video image in the user interface according to the second video frame, wherein the teeth displayed in the second video image are tidy teeth.
The second video frame includes aligned teeth that are obtained by correcting for irregular teeth in the first video frame in the first video stream.
The steps executed by the server can be independently realized to be a video stream processing method at the server side; the above steps performed by the viewer client may be implemented separately as a video stream display method on the viewer client side.
In summary, the video stream processing method provided in the embodiment of the present application obtains the second video frame by identifying the irregular teeth in the first video frame and correcting the irregular teeth to obtain the regular teeth. When the video picture is displayed based on the second video frame, a picture for further beautifying the character of the anchor can be displayed. When the method is used in scenes such as short videos or video on demand, the method can also realize the dental correction of irregular teeth in the video frames, thereby displaying pictures for further beautifying the character images in the video frames. The beauty effect of the displayed face is improved.
In addition, by calling the machine learning model, the tooth image region in the first video frame is firstly identified, and then the irregular teeth are identified in the tooth image region, so that the accuracy of identifying the irregular teeth is improved. The correction curve is determined by a polynomial fitting method, so that the correction curve is a smooth curve with a certain radian, and the tidying effect of the untidy teeth after correction is ensured. The image deformation is carried out by the moving least square method, so that smooth and real deformation can be generated in the tooth image area corresponding to the irregular teeth, and the tooth straightening presents a natural effect.
It should be noted that, the order of the steps of the method provided in the embodiments of the present application may be appropriately adjusted, and the steps may also be increased or decreased according to the circumstances, and any method that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application shall be covered by the protection scope of the present application, and therefore, the detailed description thereof is omitted.
In one specific example, when the above method is used in a live scene: when the anchor client acquires the first live stream of the anchor account, the anchor client sends the first live stream to the audience client through the live server. And the audience client displays a first direct-broadcasting picture according to a first video frame in the first direct-broadcasting stream, wherein irregular teeth are displayed in the first direct-broadcasting picture. And then the audience client receives the tooth beautification operation and sends a tooth beautification request to the live broadcast server. And the live broadcast server executes the step of tooth rectification on all video frames in the first live broadcast stream according to the tooth beautifying request to obtain a second live broadcast stream and sends the second live broadcast stream to the audience client. And the audience client displays a second live broadcast picture according to a second video frame in the second live broadcast stream, wherein the second live broadcast picture is a picture obtained by correcting irregular teeth in the first live broadcast picture.
When the above method is used in short video scenes: and the anchor client sends the recorded first short video stream and the tooth beautifying request to a short video server. And the short video server performs tooth straightening on all video frames in the first video stream according to the tooth beautifying request, so that irregular teeth in all video frames in the first video stream are corrected to be regular teeth. And then when the audience client needs to play the first video stream, the server sends the second short video stream to the audience client, so that the teeth in the short video image displayed by the audience client are ensured to be regular teeth.
The above method can also be used in video on demand scenarios: the server performs a dental correction step on the multimedia stream of the movie in the database and sends it to the viewer client for playing. The teeth of the characters in the movie picture displayed in the audience client are corrected to be regular teeth, and the user experience of the audience user is improved.
Fig. 10 is a block diagram of a video stream processing apparatus according to an embodiment of the present application. The apparatus may be used in a server in a system as shown in figure 1. The apparatus 100 comprises:
the obtaining module 1001 is configured to obtain a first video frame in a first video stream of a main broadcast account.
An identifying module 1002 is configured to identify an irregular tooth in the first video frame, the irregular tooth comprising at least one tooth of the plurality of teeth.
The correcting module 1003 is configured to correct the irregular teeth in the first video frame to obtain a second video frame in response to the first video frame including the irregular teeth, where the second video frame includes target regular teeth obtained by correcting the irregular teeth.
Optionally, the identifying module 1002 is configured to:
and calling a machine learning model to identify irregular teeth in the first video frame. The machine learning model is obtained by training an irregular tooth sample set.
Optionally, the machine learning model comprises an image segmentation model and an image classification model. An identifying module 1002 for:
and according to the image segmentation model, segmenting a tooth image region corresponding to a single tooth in the first video frame, wherein the tooth comprises regular teeth and irregular teeth.
And identifying irregular teeth in the tooth image area according to the image classification model.
Optionally, the rectification module 1003 is configured to:
a first two-sided vertex for each out-of-alignment tooth and a second two-sided vertex for each in-alignment tooth are determined.
And determining a correction curve through a polynomial fitting device according to the first two-side vertex and the second two-side vertex.
And correcting irregular teeth in the first video frame according to the correction curve.
Optionally, the correcting module 1003 is configured to:
and establishing a coordinate system according to the first two side vertexes and the second two side vertexes.
And determining the coordinates of the first two-side vertex and the second two-side vertex in a coordinate system.
And determining a correction function through a polynomial fitting device according to the coordinates of the first two side vertexes and the coordinates of the second two side vertexes, wherein a function image of the correction function is a correction curve.
Optionally, the rectification module 1003 is configured to:
and determining target correction points corresponding to the top points of the first two sides in the first video frame, wherein the target correction points are on the correction curve.
And according to the target correction point, carrying out image deformation on a tooth image region corresponding to the irregular tooth by a moving least square method to obtain a target regular tooth, wherein the vertexes of the third two sides of the target regular tooth are the target correction points.
Optionally, as shown in fig. 11, the apparatus 100 further includes:
a replacing module 1004, configured to replace the first video frame in the first video stream with the second video frame, so as to obtain a second video stream.
A sending module 1005 for sending the second video stream to the viewer client.
Fig. 12 is a block diagram of a video stream display apparatus according to an embodiment of the present application. The apparatus may be used in a viewer terminal or a viewer client on a viewer terminal in a system such as that shown in fig. 1. The apparatus 120 comprises:
the display module 1201 is configured to display a first video image in the user interface according to a first video frame in a first video stream of the anchor account, where the first video image has irregular teeth.
An obtaining module 1202 for obtaining a second video stream in response to receiving the dental beautification operation, wherein a second video frame in the second video stream is obtained by correcting irregular teeth in the first video frame.
The display module 1201 is further configured to display a second video image in the user interface according to the second video frame, where teeth displayed in the second video image are regular teeth.
It should be noted that: the video stream processing apparatus provided in the foregoing embodiment is only illustrated by dividing the functional modules, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the video stream processing apparatus and the video stream processing method provided in the foregoing embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments and are not described herein again.
Similarly, the video stream display apparatus provided in the foregoing embodiment is only illustrated by the division of the functional modules, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the video stream display apparatus and the video stream display method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are detailed in the method embodiments and are not described herein again.
Embodiments of the present application further provide a computer device, including: the system comprises a processor and a memory, wherein at least one instruction, at least one program, code set or instruction set is stored in the memory, and the at least one instruction, the at least one program, code set or instruction set is loaded by the processor and executed to realize the video stream processing method or the video stream display method provided by the method embodiments.
Optionally, the computer device is a server. Illustratively, fig. 13 is a schematic structural diagram of a server provided in an embodiment of the present application.
The server 1300 includes a Central Processing Unit (CPU) 1301, a system Memory 1304 including a Random Access Memory (RAM) 1302 and a Read-Only Memory (ROM) 1303, and a system bus 1305 connecting the system Memory 1304 and the CPU 1301. The computer device 1300 also includes a basic Input/Output system (I/O system) 1306, a mass storage device 1307 for storing an operating system 1313, application programs 1314, and other program modules 1315, which facilitate the transfer of information between the various components within the computer device.
The basic input/output system 1306 includes a display 1308 for displaying information and an input device 1309, such as a mouse, keyboard, etc., for a user to input information. Wherein the display 1308 and input device 1309 are connected to the central processing unit 1301 through an input-output controller 1310 connected to the system bus 1305. The basic input/output system 1306 may also include an input/output controller 1310 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, input-output controller 1310 also provides output to a display screen, a printer, or other type of output device.
The mass storage device 1307 is connected to the central processing unit 1301 through a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer-readable storage media provide non-volatile storage for the server 1300. That is, the mass storage device 1307 may include a computer-readable storage medium (not shown) such as a hard disk or Compact Disc-Only Memory (CD-ROM) drive.
Without loss of generality, the computer-readable storage media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable storage instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash Memory or other solid state Memory devices, CD-ROM, Digital Versatile Disks (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the foregoing. The system memory 1304 and mass storage device 1307 described above may be collectively referred to as memory.
The memory stores one or more programs configured to be executed by the one or more central processing units 1301, the one or more programs containing instructions for implementing the method embodiments described above, and the central processing unit 1301 executes the one or more programs to implement the methods provided by the various method embodiments described above.
The server 1300 may also operate as a remote server connected to a network via a network, such as the internet, according to various embodiments of the present application. That is, the server 1300 may be connected to the network 1312 through a network interface unit 1311 coupled to the system bus 1305, or alternatively, the network interface unit 1311 may be used to connect to other types of networks or remote server systems (not shown).
The memory also includes one or more programs, which are stored in the memory, and the one or more programs include instructions for performing the steps performed by the server in the methods provided by the embodiments of the present application.
Optionally, the computer device is installed with a main broadcasting client or a spectator client, and the computer device is a terminal. Illustratively, fig. 14 is a schematic structural diagram of a terminal provided in an embodiment of the present application.
In general, terminal 1400 includes: a processor 1401, and a memory 1402.
Processor 1401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so forth. The processor 1401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1401 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also referred to as a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 1401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content that the display screen needs to display. In some embodiments, processor 1401 may further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.
Memory 1402 may include one or more computer-readable storage media, which may be non-transitory. Memory 1402 may also include high speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 1402 is used to store at least one instruction for execution by processor 1401 to implement a video stream processing method or a video stream display method provided by method embodiments herein.
In some embodiments, terminal 1400 may further optionally include: a peripheral device interface 1403 and at least one peripheral device. The processor 1401, the memory 1402, and the peripheral device interface 1403 may be connected by buses or signal lines. Each peripheral device may be connected to the peripheral device interface 1403 via a bus, signal line, or circuit board. Specifically, the peripheral device includes: at least one of radio frequency circuitry 1404, a display 1405, a camera assembly 1406, audio circuitry 1407, a positioning assembly 1408, and a power supply 1409.
The peripheral device interface 1403 can be used to connect at least one peripheral device related to I/O (Input/Output) to the processor 1401 and the memory 1402. In some embodiments, the processor 1401, memory 1402, and peripheral interface 1403 are integrated on the same chip or circuit board; in some other embodiments, any one or both of the processor 1401, the memory 1402, and the peripheral device interface 1403 may be implemented on a separate chip or circuit board, which is not limited by the embodiments of the present application.
The Radio Frequency circuit 1404 is used for receiving and transmitting RF (Radio Frequency) signals, also called electromagnetic signals. The radio frequency circuitry 1404 communicates with communication networks and other communication devices via electromagnetic signals. The rf circuit 1404 converts an electrical signal into an electromagnetic signal to transmit, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1404 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so forth. The radio frequency circuit 1404 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocols include, but are not limited to: the world wide web, metropolitan area networks, intranets, generations of mobile communication networks (2G, 3G, 4G, and 5G), Wireless local area networks, and/or WiFi (Wireless Fidelity) networks. In some embodiments, the radio frequency circuit 1404 may further include NFC (Near Field Communication) related circuits, which are not limited in this application.
The display screen 1405 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 1405 is a touch display screen, the display screen 1405 also has the ability to capture touch signals at or above the surface of the display screen 1405. The touch signal may be input to the processor 1401 for processing as a control signal. At this point, the display 1405 may also be used to provide virtual buttons and/or virtual keyboards, also referred to as soft buttons and/or soft keyboards. In some embodiments, the display 1405 may be one, providing the front panel of the terminal 1400; in other embodiments, display 1405 may be at least two, respectively disposed on different surfaces of terminal 1400 or in a folded design; in still other embodiments, display 1405 may be a flexible display disposed on a curved surface or on a folded surface of terminal 1400. Even further, the display 1405 may be arranged in a non-rectangular irregular figure, i.e., a shaped screen. The Display 1405 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), and the like.
The camera assembly 1406 is used to capture images or video. Optionally, camera assembly 1406 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal 1400 and the rear camera is disposed on the rear side of the terminal. In some embodiments, the number of the rear cameras is at least two, and each rear camera is any one of a main camera, a depth-of-field camera, a wide-angle camera and a telephoto camera, so that the main camera and the depth-of-field camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize panoramic shooting and VR (Virtual Reality) shooting functions or other fusion shooting functions. In some embodiments, camera assembly 1406 may also include a flash. The flash lamp can be a monochrome temperature flash lamp or a bicolor temperature flash lamp. The double-color-temperature flash lamp is a combination of a warm-light flash lamp and a cold-light flash lamp, and can be used for light compensation at different color temperatures.
The audio circuit 1407 may include a microphone and a speaker. The microphone is used for collecting sound waves of a user and the environment, converting the sound waves into electric signals, and inputting the electric signals to the processor 1401 for processing or inputting the electric signals to the radio frequency circuit 1404 to realize voice communication. For stereo capture or noise reduction purposes, multiple microphones may be provided, each at a different location of terminal 1400. The microphone may also be an array microphone or an omni-directional pick-up microphone. The speaker is then used to convert electrical signals from the processor 1401 or the radio frequency circuit 1404 into sound waves. The loudspeaker can be a traditional film loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, the speaker can be used for purposes such as converting an electric signal into a sound wave audible to a human being, or converting an electric signal into a sound wave inaudible to a human being to measure a distance. In some embodiments, the audio circuit 1407 may also include a headphone jack.
The positioning component 1408 serves to locate the current geographic position of the terminal 1400 for navigation or LBS (Location Based Service). The Positioning component 1408 may be based on the Positioning component of the GPS (Global Positioning System) in the united states, the beidou System in china, or the galileo System in russia.
Power supply 1409 is used to power the various components of terminal 1400. The power source 1409 may be alternating current, direct current, disposable or rechargeable. When the power source 1409 comprises a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery charged through a wired line, and the wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery may also be used to support fast charge technology.
In some embodiments, terminal 1400 also includes one or more sensors 1410. The one or more sensors 1410 include, but are not limited to: acceleration sensor 1411, gyroscope sensor 1412, pressure sensor 1413, fingerprint sensor 1414, optical sensor 1415, and proximity sensor 1416.
The acceleration sensor 1411 may detect the magnitude of acceleration on three coordinate axes of a coordinate system established with the terminal 1400. For example, the acceleration sensor 1411 may be used to detect components of the gravitational acceleration in three coordinate axes. The processor 1401 can control the touch display 1405 to display a user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1411. The acceleration sensor 1411 may also be used for the acquisition of motion data of a game or a user.
The gyro sensor 1412 may detect a body direction and a rotation angle of the terminal 1400, and the gyro sensor 1412 and the acceleration sensor 1411 may cooperate to collect a 3D motion of the user on the terminal 1400. The processor 1401 can realize the following functions according to the data collected by the gyro sensor 1412: motion sensing (such as changing the UI according to a user's tilting operation), image stabilization at the time of photographing, game control, and inertial navigation.
Pressure sensors 1413 may be disposed on the side bezel of terminal 1400 and/or underlying touch display 1405. When the pressure sensor 1413 is disposed on the side frame of the terminal 1400, the user's holding signal of the terminal 1400 can be detected, and the processor 1401 performs left-right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1413. When the pressure sensor 1413 is disposed at the lower layer of the touch display 1405, the processor 1401 controls the operability control on the UI interface according to the pressure operation of the user on the touch display 1405. The operability control comprises at least one of a button control, a scroll bar control, an icon control and a menu control.
The fingerprint sensor 1414 is used for collecting a fingerprint of a user, and the processor 1401 identifies the user according to the fingerprint collected by the fingerprint sensor 1414, or the fingerprint sensor 1414 identifies the user according to the collected fingerprint. Upon recognizing that the user's identity is a trusted identity, processor 1401 authorizes the user to perform relevant sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying for, and changing settings, etc. Fingerprint sensor 1414 may be disposed on the front, back, or side of terminal 1400. When a physical button or vendor Logo is provided on terminal 1400, fingerprint sensor 1414 may be integrated with the physical button or vendor Logo.
The optical sensor 1415 is used to collect ambient light intensity. In one embodiment, processor 1401 can control the display brightness of touch display 1405 based on the ambient light intensity collected by optical sensor 1415. Specifically, when the ambient light intensity is high, the display luminance of the touch display 1405 is increased; when the ambient light intensity is low, the display brightness of the touch display 1405 is turned down. In another embodiment, the processor 1401 can also dynamically adjust the shooting parameters of the camera assembly 1406 according to the intensity of the ambient light collected by the optical sensor 1415.
Proximity sensor 1416, also known as a distance sensor, is typically disposed on the front panel of terminal 1400. The proximity sensor 1416 is used to collect the distance between the user and the front surface of the terminal 1400. In one embodiment, when proximity sensor 1416 detects that the distance between the user and the front face of terminal 1400 is gradually decreased, processor 1401 controls touch display 1405 to switch from a bright screen state to a dark screen state; when proximity sensor 1416 detects that the distance between the user and the front face of terminal 1400 is gradually increasing, processor 1401 controls touch display 1405 to switch from a breath-screen state to a bright-screen state.
Those skilled in the art will appreciate that the configuration shown in fig. 14 is not intended to be limiting with respect to terminal 1400 and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components may be employed.
The embodiment of the present application further provides a computer-readable storage medium, where at least one instruction, at least one program, a code set, or a set of instructions is stored in the computer-readable storage medium, and when the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by a processor of a computer device, the video stream processing method or the video stream display method provided by the above method embodiments is implemented.
The present application also provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the video stream processing method or the video stream display method provided by the above method embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer readable storage medium, and the above readable storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only an example of the present application and should not be taken as limiting, and any modifications, equivalent switches, improvements, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims (12)

1. A method for processing a video stream, the method comprising:
acquiring a first video frame in a first video stream of a main broadcast account;
identifying an irregular tooth in the first video frame, the irregular tooth comprising at least one tooth of a plurality of teeth;
and in response to the first video frame including the irregular teeth, correcting the irregular teeth in the first video frame to obtain a second video frame including target regular teeth obtained by correcting the irregular teeth.
2. The method of claim 1, wherein the identifying irregular teeth in the first video frame comprises:
and calling a machine learning model, and identifying irregular teeth in the first video frame, wherein the machine learning model is obtained by training an irregular tooth sample set.
3. The method of claim 2, wherein the machine learning model comprises an image segmentation model and an image classification model;
the invoking a machine learning model to identify irregular teeth in the first video frame, comprising:
according to the image segmentation model, segmenting a tooth image region corresponding to a single tooth in the first video frame, wherein the tooth comprises a regular tooth and an irregular tooth;
identifying the out-of-order teeth in the dental image region according to the image classification model.
4. The method of claim 1, wherein said correcting said irregular teeth in said first video frame to obtain a second video frame in response to said irregular teeth being included in said first video frame comprises:
determining first two-side vertexes of each irregular tooth and second two-side vertexes of each regular tooth;
determining a correction curve by a polynomial fitting method according to the first two-side vertex and the second two-side vertex;
and correcting the irregular teeth in the first video frame according to the correction curve.
5. The method of claim 4, wherein determining a correction curve from the first two-sided vertex and the second two-sided vertex by a polynomial fitting method comprises:
establishing a coordinate system according to the first two-side vertex and the second two-side vertex;
determining coordinates of the first two-side vertex and the second two-side vertex in the coordinate system;
and determining a correction function by a polynomial fitting method according to the coordinates of the first two side vertexes and the coordinates of the second two side vertexes, wherein a function image of the correction function is the correction curve.
6. The method of claim 4, wherein said correcting said irregular teeth in said first video frame according to said correction curve comprises:
determining target correction points corresponding to the first two-side vertex points in the first video frame, wherein the target correction points are on the correction curve;
and according to the target correction point, carrying out image deformation on a tooth image region corresponding to the irregular tooth by a moving least square method to obtain the target regular tooth, wherein the third two side vertexes of the target regular tooth are the target correction point.
7. The method of any of claims 1 to 6, wherein after said correcting said irregular teeth in said first video frame for a second video frame in response to said first video frame including said irregular teeth, said method further comprises:
replacing the first video frame in the first video stream with the second video frame to obtain a second video stream;
and transmitting the second video stream to the viewer client.
8. A method for displaying a video stream, the method comprising:
displaying a first video picture in a user interface according to a first video frame in a first video stream of a main broadcast account, wherein irregular teeth are displayed in the first video picture;
in response to receiving a dental beautification operation, obtaining a second video stream, a second video frame in the second video stream being obtained by correcting the irregular teeth in the first video frame;
and displaying a second video picture in a user interface according to the second video frame, wherein teeth displayed in the second video picture are tidy teeth.
9. A video stream processing apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring a first video frame in a first video stream of the anchor account;
an identification module to identify an irregular tooth in the first video frame, the irregular tooth comprising at least one of a plurality of teeth;
and the correcting module is used for responding to the first video frame including the irregular teeth, correcting the irregular teeth in the first video frame to obtain a second video frame, and the second video frame includes target regular teeth obtained by correcting the irregular teeth.
10. A video stream display apparatus, characterized in that the apparatus comprises:
the display module is used for displaying a first video picture in a user interface according to a first video frame in a first video stream of a main broadcast account, wherein irregular teeth are displayed in the first video picture;
an obtaining module, configured to obtain a second video stream in response to receiving a dental beautification operation, where a second video frame in the second video stream is obtained by correcting the irregular teeth in the first video frame;
and the display module is further used for displaying a second video picture in the user interface according to the second video frame, wherein the teeth displayed in the second video picture are tidy teeth.
11. A computer device comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, the at least one instruction, the at least one program, the set of codes, or the set of instructions being loaded and executed by the processor to implement the video stream processing method of any of claims 1 to 7, or the video stream display method of claim 8.
12. A computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by a processor to implement the video stream processing method of any of claims 1 to 7 or the video stream display method of claim 8.
CN202010797090.7A 2020-08-10 2020-08-10 Video stream processing method, video stream display method, device and equipment Pending CN111918089A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010797090.7A CN111918089A (en) 2020-08-10 2020-08-10 Video stream processing method, video stream display method, device and equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010797090.7A CN111918089A (en) 2020-08-10 2020-08-10 Video stream processing method, video stream display method, device and equipment

Publications (1)

Publication Number Publication Date
CN111918089A true CN111918089A (en) 2020-11-10

Family

ID=73283662

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010797090.7A Pending CN111918089A (en) 2020-08-10 2020-08-10 Video stream processing method, video stream display method, device and equipment

Country Status (1)

Country Link
CN (1) CN111918089A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112511850A (en) * 2020-11-20 2021-03-16 广州繁星互娱信息科技有限公司 Wheat connecting method, live broadcast display method, device, equipment and storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106228549A (en) * 2016-07-14 2016-12-14 嘉兴学院 A kind of triangle gridding tooth dividing method based on path planning
CN108470365A (en) * 2018-03-07 2018-08-31 天津工业大学 A kind of arch wire method for drafting based on upper lower tooth jaw
CN108510471A (en) * 2018-03-05 2018-09-07 广东欧珀移动通信有限公司 Image orthodontic method, device and terminal device
CN109903217A (en) * 2019-01-25 2019-06-18 北京百度网讯科技有限公司 Image distortion method and device
CN109903218A (en) * 2019-02-28 2019-06-18 维沃移动通信有限公司 A kind of image processing method and terminal
CN110136229A (en) * 2019-05-27 2019-08-16 广州亮风台信息科技有限公司 A kind of method and apparatus changed face for real-time virtual
US20200000552A1 (en) * 2018-06-29 2020-01-02 Align Technology, Inc. Photo of a patient with new simulated smile in an orthodontic treatment review software
CN110929651A (en) * 2019-11-25 2020-03-27 北京达佳互联信息技术有限公司 Image processing method, image processing device, electronic equipment and storage medium

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106228549A (en) * 2016-07-14 2016-12-14 嘉兴学院 A kind of triangle gridding tooth dividing method based on path planning
CN108510471A (en) * 2018-03-05 2018-09-07 广东欧珀移动通信有限公司 Image orthodontic method, device and terminal device
CN108470365A (en) * 2018-03-07 2018-08-31 天津工业大学 A kind of arch wire method for drafting based on upper lower tooth jaw
US20200000552A1 (en) * 2018-06-29 2020-01-02 Align Technology, Inc. Photo of a patient with new simulated smile in an orthodontic treatment review software
CN109903217A (en) * 2019-01-25 2019-06-18 北京百度网讯科技有限公司 Image distortion method and device
CN109903218A (en) * 2019-02-28 2019-06-18 维沃移动通信有限公司 A kind of image processing method and terminal
CN110136229A (en) * 2019-05-27 2019-08-16 广州亮风台信息科技有限公司 A kind of method and apparatus changed face for real-time virtual
CN110929651A (en) * 2019-11-25 2020-03-27 北京达佳互联信息技术有限公司 Image processing method, image processing device, electronic equipment and storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
周浩敏，钱政: "《智能传感技术与系统》", 30 September 2008 *
马国武，朱卫东，王福，王东红: "《2011国家执业医师资格考试口腔助理医师考试指南2011版》", 31 January 2011 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112511850A (en) * 2020-11-20 2021-03-16 广州繁星互娱信息科技有限公司 Wheat connecting method, live broadcast display method, device, equipment and storage medium
CN112511850B (en) * 2020-11-20 2023-08-08 广州繁星互娱信息科技有限公司 Wheat connecting method, live broadcast display device, equipment and storage medium

Similar Documents

Publication Publication Date Title
US11436779B2 (en) Image processing method, electronic device, and storage medium
CN110427110B (en) Live broadcast method and device and live broadcast server
CN110992493B (en) Image processing method, device, electronic equipment and storage medium
JP2022527708A (en) Image processing methods and head-mounted display devices
CN109947338B (en) Image switching display method and device, electronic equipment and storage medium
CN109302632B (en) Method, device, terminal and storage medium for acquiring live video picture
CN110933452B (en) Method and device for displaying lovely face gift and storage medium
CN111028144B (en) Video face changing method and device and storage medium
CN110533585B (en) Image face changing method, device, system, equipment and storage medium
CN111753784A (en) Video special effect processing method and device, terminal and storage medium
CN111083513B (en) Live broadcast picture processing method and device, terminal and computer readable storage medium
CN110839174A (en) Image processing method and device, computer equipment and storage medium
CN111586444B (en) Video processing method and device, electronic equipment and storage medium
CN111723803A (en) Image processing method, device, equipment and storage medium
CN112565806A (en) Virtual gift presenting method, device, computer equipment and medium
CN113384880A (en) Virtual scene display method and device, computer equipment and storage medium
CN108965769B (en) Video display method and device
CN111565309A (en) Display equipment and distortion parameter determination method, device and system thereof, and storage medium
CN112419143A (en) Image processing method, special effect parameter setting method, device, equipment and medium
CN111918089A (en) Video stream processing method, video stream display method, device and equipment
CN112235650A (en) Video processing method, device, terminal and storage medium
CN112967261B (en) Image fusion method, device, equipment and storage medium
CN111757146B (en) Method, system and storage medium for video splicing
CN114093020A (en) Motion capture method, motion capture device, electronic device and storage medium
CN112950535A (en) Video processing method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20201110