CN110730375A - Data processing method and device and computer readable storage medium - Google Patents
Data processing method and device and computer readable storage medium Download PDFInfo
- Publication number
- CN110730375A CN110730375A CN201810785516.XA CN201810785516A CN110730375A CN 110730375 A CN110730375 A CN 110730375A CN 201810785516 A CN201810785516 A CN 201810785516A CN 110730375 A CN110730375 A CN 110730375A
- Authority
- CN
- China
- Prior art keywords
- content
- synthesized
- video data
- interface
- displaying
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N21/4331—Caching operations, e.g. of an advertisement for later insertion during playback
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44004—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving video buffer management, e.g. video decoder buffer or video display buffer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
Abstract
The invention discloses a data processing method, which comprises the steps of collecting video data through a camera and sending the video data to a system buffer area; determining contents to be synthesized and sending the contents to be synthesized to the system buffer area; overlapping the video data and the content to be synthesized in the system buffer area; the invention also discloses a data processing device and a computer readable storage medium. By adopting the scheme of the invention, displayable content supported by the terminal can be added in the video data according to the requirements of the user.
Description
Technical Field
The embodiment of the invention relates to the technical field of communication, in particular to a data processing method, a data processing device and a computer readable storage medium.
Background
With the development of mobile networks, the bandwidth that the network can provide is wider and faster. Voice calls are gradually replaced by video calls. Not only wechat, Skype, provides video chat functionality, but more and more telecom operators are beginning to promote vilte (video over lte). The software for video chat has many features, such as dynamic expressions, providing virtual Reality chat scenarios in combination with Augmented Reality (AR) technology, and the reduction of network charges drives more consumers to be willing to use videos for chat, and the network speeds of 4G and 5G can ensure the definition of videos.
Although simple and easy to use, the complicated information cannot be clearly interpreted by several sentences. For example, many business people cannot go into the study without drawing or text explanation when they use video chat to communicate information. In order to solve the problem, the video chat software is required to provide an additional window for performing functions of text chat, picture sharing, desktop sharing and the like. When the WeChat and the Skype video chat, information such as screenshot can be sent through a text chat interface, and the information which cannot be clearly explained by using the speech can be explained. Sometimes limited by the limitation of chat software or protocols, and cannot provide functions such as text windows and document sharing. Camera's App is supported, and the choices offered to the user are also very few. During the video chat, any modification to the video is dependent on the support of the App. For example, functions such as beauty and expression are all implemented in the App and are completely limited by the App. The ViLTE call is taken as a video call technology mainly pushed by the current telecom operator, the support is provided by an IMS related protocol, and text communication is not provided in the video call process, so that files cannot be directly transmitted. When a user wants to transmit text information, the text information can be transmitted only by means of shooting a file by a Camera. However, this photographing process is very difficult, and a slight shake may cause unclear characters.
Disclosure of Invention
In order to solve the existing technical problem, embodiments of the present invention desirably provide a data processing method, an apparatus, and a computer-readable storage medium, which can add displayable content supported by a terminal to video data according to a user's requirement.
The technical scheme of the embodiment of the invention is realized as follows:
the invention provides a data processing method, which comprises the following steps:
acquiring video data through a camera, and sending the video data to a system buffer area;
determining contents to be synthesized and sending the contents to be synthesized to the system buffer area;
and overlapping the video data and the content to be synthesized in the system buffer area.
The present invention also provides a data processing apparatus, the apparatus comprising: a first processing module (1401), a second processing module (1402) and a synthesis module (1403); wherein the content of the first and second substances,
the first processing module (1401) is used for acquiring video data through a camera and sending the video data to a system buffer area;
the second processing module (1402) is configured to determine content to be synthesized and send the content to be synthesized to the system buffer;
the compositing module (1403) is configured to overlay the video data and the content to be composited in the system buffer.
The invention also provides a data processing apparatus comprising a processor and a memory for storing a computer program capable of running on the processor; wherein the processor is configured to execute the steps of the data processing method according to any one of the above aspects when the computer program is executed.
The invention also provides a computer-readable storage medium, on which a computer program is stored, which is characterized in that the computer program, when executed by a processor, implements the steps of the data processing method according to any of the above aspects.
According to the data processing method, the data processing device and the computer readable storage medium, video data are collected through the camera and sent to the system buffer area; determining contents to be synthesized and sending the contents to be synthesized to the system buffer area; overlapping the video data and the content to be synthesized in the system buffer area; therefore, displayable content supported by the terminal can be added into the video data according to the requirements of the user, and the user experience is improved.
Drawings
FIG. 1 is a schematic diagram of a data processing method according to an embodiment of the present invention;
fig. 2 is a schematic diagram illustrating that an Android system encapsulates video data according to a second embodiment of the present invention;
fig. 3 is a schematic diagram illustrating that an Android system acquires video data according to a second embodiment of the present invention;
FIG. 4 is a schematic diagram of a blending process for adding visual content between Camera and APP in the second embodiment of the present invention;
FIG. 5 is a schematic diagram of a mixed flow process of inserting PPT into Surface according to a second embodiment of the present invention;
FIG. 6 is a diagram illustrating a display effect of a control interface according to a second embodiment of the present invention;
fig. 7 is a diagram illustrating the display effect of displaying the text/picture contents to the video data according to the second embodiment of the present invention;
fig. 8 is a diagram illustrating the display effect of displaying PPT content to video data according to a second embodiment of the present invention;
FIG. 9 is a diagram illustrating the display effect of the handwritten pattern displayed to video data according to the second embodiment of the present invention;
FIG. 10 is a diagram illustrating the display effect of the floating button according to the second embodiment of the present invention;
fig. 11 is a diagram illustrating a display effect of a floating PPT play window according to a second embodiment of the present invention;
FIG. 12 is a diagram illustrating the display effect of a floating text entry window according to a second embodiment of the present invention;
FIG. 13 is a diagram illustrating an effect of setting a setting interface according to a second embodiment of the present invention;
FIG. 14 is a first schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention;
FIG. 15 is a second schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention;
fig. 16 is a schematic structural diagram of a data processing apparatus according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Example one
Fig. 1 is a schematic flow chart of an implementation of a data processing method according to a first embodiment of the present invention, as shown in fig. 1, the method includes the following steps:
step 101: acquiring video data through a camera, and sending the video data to a system buffer area;
generally, video data are collected through a camera, the camera sends the collected video data to a system buffer area, and when an application program is used for displaying the video data, the video data are obtained from the system buffer area through an interface between the application program and a system and are displayed.
The system buffer area is used for caching video data so as to preview or display the video data.
Step 102: determining contents to be synthesized and sending the contents to be synthesized to the system buffer area;
wherein, the content to be synthesized may include: and the displayable contents supported by the terminals such as characters, pictures, dynamic expressions, AR effect data, shared documents, handwritten patterns and the like.
In an embodiment, the determining the content to be synthesized comprises: displaying an adding interface, and receiving adding operation through the adding interface; the adding operation is used for adding the content to be synthesized; and determining the content to be synthesized according to the adding operation.
The adding interface can be an adding interface for displaying the type of the content to be synthesized.
And displaying an adding interface on the terminal, performing adding operation on the adding interface, determining the content to be synthesized according to the received adding operation by the adding interface, and sending the content to be synthesized to the system buffer area.
In an embodiment, the determining content to be synthesized according to the adding operation includes: determining an operation control corresponding to the adding operation, and determining a content path corresponding to the operation control; different operation controls correspond to different types of contents to be synthesized; displaying a content display interface corresponding to the content path; and receiving selection operation aiming at the content display interface, and determining the content to be synthesized according to the selection operation.
And providing a plurality of operation controls in the adding interface, and adding different types of contents to be synthesized through the operation of different operation controls. Here, the operation for the addition interface is referred to as an addition operation. The content path corresponding to different operation controls is different from the type of the content to be synthesized, adding operation is carried out on an adding interface, the corresponding content display interface is displayed according to the content path corresponding to different operation controls and the type of the content to be synthesized, and the content to be synthesized is determined according to the received selection operation in the content display interface.
Such as: the adding interface displays that: when the adding operation is to click a picture, a picture content path corresponding to the operation control is determined according to the operation control corresponding to the picture, a picture content display interface corresponding to the picture content path is displayed, the content to be synthesized is selected in the picture content display interface, and after the selecting operation of the picture content display interface is received, the content to be synthesized is determined according to the selected picture.
In an embodiment, the determining content to be synthesized according to the adding operation includes: displaying the input window of the content to be synthesized according to the adding operation; receiving an input operation aiming at the input window, and determining the content to be synthesized according to the input operation.
When adding operation is carried out on an adding interface, an input window of the content to be synthesized is displayed according to the trigger of the adding operation, the input operation is carried out on the input window of the content to be synthesized, and after the input operation of the input window is received, the content to be synthesized is determined according to the input operation.
Here, the input operation includes two cases: one input operation is an input operation trajectory, and the other is an input operation received through a keyboard.
Such as: displaying on the adding interface: adding controls such as characters and handwritten patterns. And when the adding operation is to click the handwritten pattern, displaying an input window according to the handwritten pattern adding operation, performing input operation in the input window by a user through handwriting, wherein the handwritten operation track is the character 'you', and determining that the content to be synthesized is 'you' after the terminal receives the handwritten operation track of the input window. When the adding operation is the character clicking, an input window is displayed according to the character adding operation, a user performs input operation in the input window through a keyboard to input the character 'I', and the terminal determines that the content to be synthesized is 'I' after receiving the input operation of the input window.
In an embodiment, before the displaying the add interface, the method further includes: displaying a control, wherein the control is a system control; receiving a trigger operation aiming at the control, and acquiring display parameters according to the trigger operation; accordingly, the displaying an add interface includes: and displaying the adding interface according to the display parameters.
Before the adding interface is displayed, the control is triggered, the system receives the triggering operation aiming at the control, the display parameters of the adding interface are obtained, and the adding interface is displayed according to the adding parameters. The triggering operation can be single-click operation, double-click operation and the like; the display parameters may include: font, content, display position and other parameters related to the display effect of the added interface.
Such as: the system display interface display control is a floating button, a user clicks the floating button, the system receives clicking operation aiming at the floating button, and the obtained display parameters are as follows: the font is Song style, and the content is: and displaying the adding interface according to the acquired display parameters of the adding interface by the characters and the pictures.
It should be noted that, when video data is collected by the camera, the control is displayed on the terminal system interface at the same time.
Step 103: and overlapping the video data and the content to be synthesized in the system buffer area.
The system buffer area receives the video data collected by the camera and the content to be synthesized, and the content to be synthesized is superposed on the video data.
In an embodiment, the method further comprises: displaying a superposition setting interface; the superposition setting interface is an interface for setting the content to be synthesized. Receiving a setting operation aiming at the superposition setting interface, and determining superposition parameters according to the setting operation; adjusting the content to be synthesized according to the superposition parameters to obtain the adjusted content to be synthesized; correspondingly, the overlaying the video data and the content to be synthesized includes: and overlapping the video data and the adjusted content to be synthesized.
When the video data and the content to be synthesized are superimposed, the display effect of the content to be synthesized can be set. The terminal displays a superposition setting interface, sets the content to be synthesized on the superposition setting interface, determines superposition parameters according to the setting operation after receiving the setting operation aiming at the superposition setting interface, adjusts the content to be synthesized according to the superposition parameters to obtain the adjusted content to be synthesized, and superposes the adjusted content to be synthesized on the video data.
Such as: the method comprises the steps that a content to be synthesized is PPT, a terminal displays a superposition setting interface, the PPT is set on the superposition setting interface, a full-screen playing mode is clicked on the superposition setting interface, after the situation that the setting operation aiming at the superposition setting interface is the full-screen playing mode is received, superposition parameters are determined to be full-screen playing, the playing mode of the PPT is adjusted according to the superposition parameters, the adjusted PPT is obtained and is the PPT of full-screen playing, and the PPT of full-screen playing is superposed on video data.
Such as: the method comprises the steps that a content to be synthesized is a picture, a terminal displays a superposition setting interface, the picture is set on the superposition setting interface, the transparency of the picture is set to be 50% on the superposition setting interface, after the setting operation aiming at the superposition setting interface is received, the transparency is 50%, the superposition parameter is determined to be 50%, the transparency of the picture is adjusted according to the superposition parameter, the adjusted picture is the picture with the transparency of 50%, and the picture with the transparency of 50% is superposed on video data.
In an embodiment, the method further comprises: and adjusting the content to be synthesized according to a set de-synthesis strategy, wherein the de-synthesis strategy is used for removing the content to be synthesized from the synthesized video data.
And displaying the synthesized video data on a terminal display interface, and if the content to be synthesized in the synthesized video data is no longer needed, adjusting the content to be synthesized in the synthesized video data according to a de-synthesis strategy to remove the content to be synthesized which is not needed. Here, the decombination policy may be a user active cleanup or timeout policy.
Such as: the content to be synthesized in the synthesized video data displayed on the terminal display interface is 10 characters, the user actively clears the last character, and according to the received de-synthesis strategy for clearing the last character, the last character of the content to be synthesized is cleared, so that the adjusted content to be synthesized is 9 characters.
Such as: and when the animation expression is displayed for 50s, the animation expression is removed according to the animation expression 50s overtime strategy.
In an embodiment, the method further comprises: when receiving display operation aiming at an application program, generating a display instruction according to the display operation; acquiring the synthesized video data from the system buffer area according to the display instruction, wherein the synthesized video data is obtained by overlapping the video data and the content to be synthesized; and displaying the synthesized video data on a display interface of the application program.
The method comprises the steps of obtaining composite video data after the video data and the content to be synthesized are overlapped, when the composite video data need to be displayed, receiving display operation aiming at an application program by a system, generating a display instruction according to the display operation, obtaining the composite video data from a system cache region according to the display instruction, and displaying the obtained composite video data on a display interface of the application program.
It should be noted that the video data and the content to be synthesized may be superimposed to generate synthesized video data, and then the display operation for the application program is received, or the display operation for the application program may be received first, and then the video data and the content to be synthesized are superimposed to generate synthesized video data.
In the embodiment of the invention, video data are collected through a camera and sent to a system buffer area; determining contents to be synthesized and sending the contents to be synthesized to the system buffer area; overlapping the video data and the content to be synthesized in the system buffer area; therefore, displayable content supported by the terminal can be added into the video data according to the requirements of the user, and the user experience is improved.
Example two
In this embodiment, a data processing method provided by the embodiment of the present invention is described by taking an Android system as an example.
The embodiment of the invention provides a method which can realize the function of transmitting pictures, characters, PPT and the like at the same time. According to the embodiment of the invention, the content is directly mixed on the video, so that the information sharing can be realized: graphics, handwriting, expression and the like. During the call, the supported displayable content may be sent at will. And the network protocol does not need to be adjusted, and even WeChat, Skype and the like can be compatible.
The embodiment of the invention is directly modified by the data of the Camera (Camera), and is not limited to WeChat, ViLTE and other applications. Through the fusion of the Camera data and information such as pictures and texts in the system layer, the related content can be sent.
The video call function requires support of a Session Initiation Protocol (SIP), a Transmission Control Protocol (TCP)/a User Datagram Protocol (UDP)/an interconnection Protocol (Internet Protocol, IP) between networks, and a Real-time Transport Protocol (RTP) related network Protocol, and further requires a terminal to provide Camera. Wechat and Skype have both made improvements to network protocols in order to provide video calls. The mainstream implementation is: one path of data is created in the video, and another path of data is created again by other functions of text chatting. This data refers to a network connection with a server or an opposite end through which the data is transmitted and received.
Whereas ViLTE fully complies with 3GPP protocol specifications, mainly provides voice and video functions, and cannot simultaneously communicate instant messages via IMS, which causes tariff problems if it is delivered via Short Message Service (SMS). After the ViLTE call is established, only video services can be provided according to the IMS protocol.
In the video call process, the image frames of the video are continuously transmitted to the other party. The content of the video is typically taken directly from the Camera of the terminal.
The embodiment of the invention provides a scheme which can realize the function of sharing characters and pictures in the video call process indiscriminately, and not only can support ViLTE, but also can support WeChat, Skype and the like.
In the related art, a video is collected from a Camera of a terminal, a video chat program App running on the terminal sets the Camera through an API provided by a system and obtains video frame data. The App encodes the frame data and then transmits the encoded frame data to the receiving party through an RTP (real-time transport protocol) mode. As shown in fig. 2, the Android calls a system interface MediaCodec to re-encode and encapsulate the obtained Camera data, which includes: the Audio Coding is encapsulated into RTP through Advanced Audio Coding (AAC), the video Coding is encapsulated into RTP through a Coding format H264, and the RTP is transmitted through an RTP protocol.
In the related art, Camera APIs provided by different operating systems are slightly different, but methods for acquiring Camera data are basically consistent. Now, taking an Android terminal as an example, a method for acquiring Camera data is shown in fig. 3:
1. cameradevice: a mobile terminal generally has two cameras (Camera devices) which are front-end and rear-end.
2. CameraRequest: some parameters of the photographic effect are defined in the request, and a target surface must be added to the request using addTarget () function, and then the data returned by the CameraDevice is sent to the target surface. In the API document of the android camera2, the target Surface can be MediaCodec, Surface View and Surface Texture, and the returned data is transferred to the preview interface; the data can also be a MediaRecorder or an ImageReader, and the returned data is transmitted to the two classes for further processing to form a video file or a picture.
3. Surface: corresponding to the screen buffer, may be used to display the video preview. All displayable areas of the Android system are controlled by Surface and Surface flickers. The video preview (local real-time video) obtained from Camera is put to Surface.
For streaming media, a codec encoder needs to be provided to set video related parameters. The encoder then provides a Surface to Camera for storing the video frames. The general flow is as follows:
1. creating an encoder
mMediaCodec＝MediaCodec.createEncoderByType("video/avc")；
mMediaCodec is a created encoder, which obtains the data of the video through Camera, and re-encodes the data by using MediaCodec, that is, the buffer area (Surface) created by MediaCodec is transferred to Camera, and after Camera puts the data in the buffer area, MediaCodec re-encodes the video.
2. Setting some parameters, such as frame rate and the like, through MediaFormat;
the MediaFormat is an API of an Android system, and parameters of audio and video can be set by using the MediaFormat.
3. One Surface is available from mMediaCodec: createinputsurface ();
having obtained the created encoder, the encoder mmediatecodec creates a buffer region.
4. Taking Surface as target;
and taking the created buffer area Surface as the target.
5. Sending a request mcamera device. createcapturerequest to Camera;
and sending a capture request to Camera and returning the captured data to Surface.
6. getOutputBuffer is obtained by mmedia codec.
And acquiring the data cached in the Surface through an encoder.
The data processing method provided by the embodiment of the invention can be as shown in fig. 4:
a layer of control (middle layer) is inserted between App and Camera for composing other visual content. Video frames of Camera, delivered to Surface through pipeline (pipeline). As long as the mixture of the visual contents of the user can be added between the pipeline and the Surface, the control right can be transferred to the user, so that some extended functions are realized, and the method is not limited to the limited functions provided by the App. After detecting that Camera is started, the user can add required data to Camera video and images after opening the middle layer.
When the video collected by the Camera and the data needing to be added are overlapped, the data needing to be added generate an image, the Camera places each frame of data of the collected video into a buffer area, the image generated by the data needing to be added is overlapped on each frame of data of the video in real time, and when the image needs to be displayed, the overlapped data in the buffer area are displayed.
In the process of video chat, the function of playing PPT to the opposite side can be realized by inserting the visual content of PPT into the Surface. This mixing process, as shown in fig. 5: and superposing the video data acquired by the camera with the PPT to obtain composite data, and displaying the composite data on a display interface of the mobile terminal, wherein the content displayed on the mobile display interface comprises the data acquired by the camera (part shown by smile expression) and the inserted PPT.
The embodiment of the invention can support delivery of any displayable content. That is, blending to Surface may be performed as long as the displayable content is supported by the system. In addition, the user can also set whether the PPT is displayed in a full screen mode or not and adjust the display position.
In order to realize the control of the video, some control interfaces are required to be added outside the App. As shown in fig. 6:
the method includes that a floating window S (control) is displayed on a display interface of the mobile terminal, after a user clicks the floating window S, an adding interface is displayed, and different types of controls for adding the contents to be synthesized are arranged in the adding interface, and the method includes the following steps: and adding a control 1, inputting characters, adding a control 2, selecting pictures, adding a control 3, dynamically expressing, adding a control 4, realizing AR effect, adding a control 5, sharing documents and adding the controls, and transmitting the characters and the pictures through displayed adding buttons.
When the user selects the text information, an input box is popped up for the user to input. After the input is finished, the character information is synthesized to the Surface, and the picture can be drawn to the Surface. This Surface corresponds to a screen buffer and any View is drawn on the Surface's Canvas.
Drawing API Canvas provided by Android relies on Surface to output to a user interface. But not all surfaces will be displayed, only the visible Surface will be displayed.
Canvas. drawtext ("text information");
drawing a picture to Surface by canvas.
The display effect of the embodiment of the invention for displaying the text/picture content on the video frame is as shown in the following figure 7:
the text and pictures sent out will remain displayed for a period of time until the user chooses to clear, or some timeout strategy.
1. It may be a user active purge;
2. alternatively, only a few, the earliest one may be displayed, with diffuse blanking.
The embodiment of the invention can send any displayable content, and the sending method of the PPT is described.
The process of playing PPT is shown in FIG. 8. In the chat interface, after the user clicks the float button S, the user can select to open a certain PPT. In the video process, the sending end can perform operations of page turning, play position adjustment, full-screen play and the like on the PPT.
PPT display content is rendered and inserted into the Surface in real time, and App cannot detect the change, such as ViLTE and WeChat video App.
In the video call process, after the user starts the 'handwritten pattern', the user can draw by hand in real time. The hand-drawn content overlay mode is displayed on the video chat, and the current normal video call is not influenced. And the handwritten pattern will be rendered and inserted into the Surface and then passed to the opposite end through App as shown in fig. 9.
The functions related to the embodiments of the present invention are realized by adding corresponding UI components to the add interface, and are described in detail as follows.
1. Floating button S
The float button S can be moved to a position where clicking the button S reveals additional functionality, as shown in fig. 10.
2. Floating PPT (Power Point) playing window
When the PPT is played, operations such as page turning, position moving, full screen, etc. can be performed, as shown in fig. 11.
3. Floating character input window
As much as normal chat, clicking on the input text can send the text into the video, as shown in fig. 12.
4. Interface setting
The user can configure the characters, handwriting effect, and the like during use, as shown in fig. 13. Wherein the content of the first and second substances,
(1) plug-in of support
The user may support more displayable content, requiring installation of a corresponding plug-in, such as a WORD plug-in.
(2) Signature setup
The method is used for setting the personalized signature of the user, and the personalized signature is displayed as a watermark or a certain position during video chat.
(3) Tattoo effect
Watermark settings, whether the watermark is displayed, and the color, shade, etc. of the watermark.
The embodiment of the invention can be applied to various scenes, for example, in the process of video chat, some knowledge needs to be explained to the opposite side, and a certain PPT document or picture is involved. At this time, PPT or pictures can be selected to be played to the other side; in the process of video chat, especially the deaf-mute can transmit the input characters to the opposite side in the form of converting the input characters into subtitles. There may also be occasions where muting is required; in the process of video call, corresponding accords can be directly sent or zoomed into characters to be sent in a direct handwriting mode.
In the embodiment of the invention, the data processing method of the embodiment can enhance the video call function, does not increase network load, and has a plurality of beneficial effects superposed on the current App. As long as the mobile terminal supports the embodiment of the invention, the characteristic function of the embodiment of the invention can be used in the video chat without being limited by App. That is, not only WeChat but also ViLTE and Skype can be used.
EXAMPLE III
The present embodiment provides a data processing apparatus, as shown in fig. 14, a data processing apparatus 140 includes: a first processing module 1401, a second processing module 1402, a synthesis module 1403; wherein the content of the first and second substances,
the first processing module 1401 is configured to collect video data through a camera, and send the video data to a system buffer area;
a second processing module 1402, configured to determine content to be synthesized; sending the content to be synthesized to the system buffer area;
a synthesizing module 1403, configured to superimpose the video data and the content to be synthesized in the system buffer.
In one embodiment, the second processing module 1402 determines the content to be synthesized includes: displaying an adding interface, and receiving adding operation through the adding interface; the adding operation is used for adding the content to be synthesized; and determining the content to be synthesized according to the adding operation.
In one embodiment, as shown in fig. 15, the data processing apparatus 140 further includes: a display module 1404, configured to generate a display instruction according to a display operation when the display operation for an application program is received; acquiring the synthesized video data from the system buffer area according to the display instruction, wherein the synthesized video data is obtained by overlapping the video data and the content to be synthesized; and displaying the synthesized video data on a display interface of the application program.
In an embodiment, the second processing module 1402 is further configured to determine an operation control corresponding to the adding operation, and determine a content path corresponding to the operation control; different operation controls correspond to different types of contents to be synthesized; displaying a content display interface corresponding to the content path; and receiving selection operation aiming at the content display interface, and determining the content to be synthesized according to the selection operation.
In an embodiment, the second processing module 1402 is further configured to display an input window of the content to be synthesized according to the adding operation; receiving an input operation aiming at the input window, and determining the content to be synthesized according to the input operation.
In one embodiment, as shown in fig. 15, the data processing apparatus 140 further includes: a control module 1405, configured to display a control, where the control is a system control; receiving a trigger operation aiming at the control, and acquiring display parameters according to the trigger operation; accordingly, the second processing module 1402, when displaying the add interface, is configured to: and displaying the adding interface according to the display parameters.
In one embodiment, as shown in fig. 15, the data processing apparatus 140 further includes: a setting module 1406 for displaying a superposition setting interface; the superposition setting interface is an interface for setting the content to be synthesized. Receiving a setting operation aiming at the superposition setting interface, and determining superposition parameters according to the setting operation; adjusting the content to be synthesized according to the superposition parameters to obtain the adjusted content to be synthesized; accordingly, the second processing module 1402 is configured to: and overlapping the video data and the adjusted content to be synthesized to obtain synthesized video data.
In an embodiment, the setting module 1406 is further configured to adjust the content to be composited according to a set de-compositing policy, where the de-compositing policy is used to remove the content to be composited from the composite video data.
It should be noted that, when the data processing device provided in the above embodiments performs data processing, the division of each program module is merely exemplified, and in practical applications, the processing may be distributed to different program modules according to needs, that is, the internal structure of the device may be divided into different program modules to complete all or part of the processing described above.
Example four
Based on the foregoing embodiments, an embodiment of the present invention provides a data processing apparatus, as shown in fig. 16, including a processor 1602 and a memory 1601 for storing a computer program capable of running on the processor 1602; wherein the processor 1602 is configured to implement, when running the computer program,:
acquiring video data through a camera, and sending the video data to a system buffer area;
determining contents to be synthesized and sending the contents to be synthesized to the system buffer area;
and overlapping the video data and the content to be synthesized in the system buffer area.
The method disclosed in the above embodiments of the present invention can be applied to the processor 1602, or implemented by the processor 1602. The processor 1602 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method can be implemented by hardware integrated logic circuits or instructions in software form in the processor 1602. The processor 1602 described above may be a general purpose processor, a DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The processor 1602 may implement or perform the methods, steps, and logic blocks disclosed in the embodiments of the present invention. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed by the embodiment of the invention can be directly implemented by a hardware decoding processor, or can be implemented by combining hardware and software modules in the decoding processor. The software modules may be located in a storage medium located in the memory 1601, and the processor 1602 reads the information in the memory 1601 and performs the steps of the method in conjunction with its hardware.
It is to be understood that the memory (memory 1601) of embodiments of the present invention can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a magnetic random access Memory (FRAM), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical Disc, or a Compact Disc Read Only Memory (CD-ROM); the magnetic surface storage may be disk storage or tape storage. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM, Double Data Synchronous Random Access Memory), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM, Enhanced Synchronous Dynamic Random Access Memory), Synchronous link Dynamic Random Access Memory (SLDRAM, Synchronous Dynamic Random Access Memory), Direct Memory bus (DRmb Access Memory, Random Access Memory). The described memory for embodiments of the present invention is intended to comprise, without being limited to, these and any other suitable types of memory.
Here, it should be noted that: the description of the terminal embodiment is similar to the description of the method, and has the same beneficial effects as the method embodiment, and therefore, the description is omitted. For technical details that are not disclosed in the terminal embodiment of the present invention, those skilled in the art should refer to the description of the method embodiment of the present invention to understand that, for brevity, detailed description is omitted here.
EXAMPLE five
In an exemplary embodiment, the present invention further provides a computer storage medium, specifically a computer readable storage medium, for example, a memory 1601 storing a computer program, which can be executed by a processor 1602 to perform the steps of the foregoing method. The computer readable storage medium may be Memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface Memory, optical disk, or CD-ROM.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements:
acquiring video data through a camera, and sending the video data to a system buffer area;
determining contents to be synthesized and sending the contents to be synthesized to the system buffer area;
and overlapping the video data and the content to be synthesized in the system buffer area.
Here, it should be noted that: the description of the terminal embodiment is similar to the description of the method, and has the same beneficial effects as the method embodiment, and therefore, the description is omitted. For technical details that are not disclosed in the terminal embodiment of the present invention, those skilled in the art should refer to the description of the method embodiment of the present invention to understand that, for brevity, detailed description is omitted here.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.
Claims (11)
1. A method of data processing, the method comprising:
acquiring video data through a camera, and sending the video data to a system buffer area;
determining contents to be synthesized and sending the contents to be synthesized to the system buffer area;
and overlapping the video data and the content to be synthesized in the system buffer area.
2. The method of claim 1, wherein the determining the content to be synthesized comprises:
displaying an adding interface, and receiving adding operation through the adding interface; the adding operation is used for adding the content to be synthesized;
and determining the content to be synthesized according to the adding operation.
3. The method of claim 1, further comprising:
when receiving display operation aiming at an application program, generating a display instruction according to the display operation;
acquiring the synthesized video data from the system buffer area according to the display instruction, wherein the synthesized video data is obtained by overlapping the video data and the content to be synthesized;
and displaying the synthesized video data on a display interface of the application program.
4. The method of claim 1, wherein the determining content to be synthesized according to the adding operation comprises:
determining an operation control corresponding to the adding operation, and determining a content path corresponding to the operation control; different operation controls correspond to different types of contents to be synthesized;
displaying a content display interface corresponding to the content path;
and receiving selection operation aiming at the content display interface, and determining the content to be synthesized according to the selection operation.
5. The method of claim 1, wherein the determining content to be synthesized according to the adding operation comprises:
displaying the input window of the content to be synthesized according to the adding operation;
receiving an input operation aiming at the input window, and determining the content to be synthesized according to the input operation.
6. The method of claim 2, wherein prior to displaying the add interface, the method further comprises:
displaying a control, wherein the control is a system control;
receiving a trigger operation aiming at the control, and acquiring display parameters according to the trigger operation;
accordingly, the displaying an add interface includes:
and displaying the adding interface according to the display parameters.
7. The method according to any one of claims 1, 4 and 5, further comprising:
displaying a superposition setting interface; the superposition setting interface is an interface for setting the content to be synthesized;
receiving a setting operation aiming at the superposition setting interface, and determining superposition parameters according to the setting operation;
adjusting the content to be synthesized according to the superposition parameters to obtain the adjusted content to be synthesized;
correspondingly, the overlaying the video data and the content to be synthesized includes:
and overlapping the video data and the adjusted content to be synthesized.
8. The method of claim 3, further comprising:
and adjusting the content to be synthesized according to a set de-synthesis strategy, wherein the de-synthesis strategy is used for removing the content to be synthesized from the synthesized video data.
9. A data processing apparatus, characterized in that the apparatus comprises: a first processing module (1401), a second processing module (1402) and a synthesis module (1403); wherein the content of the first and second substances,
the first processing module (1401) is used for acquiring video data through a camera and sending the video data to a system buffer area;
the second processing module (1402) is configured to determine content to be synthesized and send the content to be synthesized to the system buffer;
the compositing module (1403) is configured to overlay the video data and the content to be composited in the system buffer.
10. A data processing apparatus comprising a processor and a memory for storing a computer program operable on the processor; wherein the processor is adapted to perform the steps of the method of any one of claims 1 to 8 when running the computer program.
11. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810785516.XA CN110730375A (en) | 2018-07-17 | 2018-07-17 | Data processing method and device and computer readable storage medium |
| PCT/CN2019/094736 WO2020015539A1 (en) | 2018-07-17 | 2019-07-04 | Data processing method and device, and computer-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810785516.XA CN110730375A (en) | 2018-07-17 | 2018-07-17 | Data processing method and device and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110730375A true CN110730375A (en) | 2020-01-24 |
Family
ID=69163940
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810785516.XA Pending CN110730375A (en) | 2018-07-17 | 2018-07-17 | Data processing method and device and computer readable storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110730375A (en) |
| WO (1) | WO2020015539A1 (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020078176A1 (en) * | 2000-12-15 | 2002-06-20 | Tetsuya Nomura | Video-on-demand system |
| CN103259919A (en) * | 2013-02-01 | 2013-08-21 | 广州网视通信息科技有限公司 | Method for overlapping scrolling subtitles on mobile phone real-time monitoring videos |
| CN103702040A (en) * | 2013-12-31 | 2014-04-02 | 广州华多网络科技有限公司 | Real-time video graphic decoration superposing processing method and system |
| WO2016159613A1 (en) * | 2015-03-31 | 2016-10-06 | 스타십벤딩머신 주식회사 | Object tracking method and system for image synthesis |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101764957B (en) * | 2009-12-28 | 2012-07-04 | 深圳华为通信技术有限公司 | Method and device for inserting picture in conference caption |
| CN104780439B (en) * | 2014-01-15 | 2019-08-13 | 腾讯科技（深圳）有限公司 | The processing method and processing device of video |
| CN108347653B (en) * | 2018-01-29 | 2020-03-06 | 广州虎牙信息科技有限公司 | Interaction method, device, equipment and storage medium |
-
2018
- 2018-07-17 CN CN201810785516.XA patent/CN110730375A/en active Pending
-
2019
- 2019-07-04 WO PCT/CN2019/094736 patent/WO2020015539A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020078176A1 (en) * | 2000-12-15 | 2002-06-20 | Tetsuya Nomura | Video-on-demand system |
| CN103259919A (en) * | 2013-02-01 | 2013-08-21 | 广州网视通信息科技有限公司 | Method for overlapping scrolling subtitles on mobile phone real-time monitoring videos |
| CN103702040A (en) * | 2013-12-31 | 2014-04-02 | 广州华多网络科技有限公司 | Real-time video graphic decoration superposing processing method and system |
| WO2016159613A1 (en) * | 2015-03-31 | 2016-10-06 | 스타십벤딩머신 주식회사 | Object tracking method and system for image synthesis |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020015539A1 (en) | 2020-01-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111386708B (en) | System and method for broadcasting live media streams | |
| EP3562163B1 (en) | Audio-video synthesis method and system | |
| US20190238919A1 (en) | Video broadcast system and a method of disseminating video content | |
| WO2019086037A1 (en) | Video material processing method, video synthesis method, terminal device and storage medium | |
| EP2569937B1 (en) | Systems and methods for real-time multimedia communication across multiple standards and proprietary devices | |
| EP2348671B1 (en) | Conference terminal, conference server, conference system and method for data processing | |
| EP3151548A1 (en) | Video recording method and device | |
| US20100217884A2 (en) | Method and system of providing multimedia content | |
| TW201143348A (en) | Switching cameras during a video conference of a multi-camera mobile device | |
| CN112804459A (en) | Image display method and device based on virtual camera, storage medium and electronic equipment | |
| JP2009518103A (en) | Method and system for enabling a user to play a large screen game with a mobile device | |
| KR20130140904A (en) | System and method for video caption re-overlaying for video adaptation and retargeting | |
| US11128894B2 (en) | Method and mobile terminal for processing data | |
| CN111246270B (en) | Method, device, equipment and storage medium for displaying bullet screen | |
| JP2014049865A (en) | Monitor camera system | |
| CN112866619A (en) | Teleconference control method and device, electronic equipment and storage medium | |
| CN114222149A (en) | Plug flow method, device, medium and computer equipment | |
| KR101915792B1 (en) | System and Method for Inserting an Advertisement Using Face Recognition | |
| CN113596571B (en) | Screen sharing method, device, system, storage medium and computer equipment | |
| KR102029604B1 (en) | Editing system and editing method for real-time broadcasting | |
| CN110730375A (en) | Data processing method and device and computer readable storage medium | |
| US10165365B2 (en) | Sound sharing apparatus and method | |
| CN113784084B (en) | Processing method and device | |
| CN114915852B (en) | Video call interaction method, device, computer equipment and storage medium | |
| CN114285836A (en) | Video playing method, device and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200124 |