JP2022511828A - Video data processing methods, video data processing equipment, computer equipment, and computer programs - Google Patents

Abstract

The embodiments of the present application disclose video data processing methods, devices, and storage media. This method determines the target object from the key video frame of the target video in response to a trigger operation on the target video and obtains the multimedia information associated with the target object, and the target pixel point in the target object. The step of determining the locus acquisition request corresponding to the target pixel point based on the position information in the key video frame, and the target locus information associated with the position information in the key video frame of the target pixel point based on the locus acquisition request. Display multimedia information based on the step to be acquired and the position information of the target pixel point in the target locus information in the next video frame of the key video frame when playing the next video frame of the key video frame. Including steps.

Description

This application claims priority based on a Chinese patent application filed on April 30, 2019, with an application number of 2019103558569.8 and the title of the invention being "Video Data Processing Methods and Related Devices". , All of which are incorporated herein by reference.

The present application relates to the technical field of the Internet, and particularly to video data processing methods and related devices.

While the user is watching the network video through the user terminal, the user can see the user characters and user comments posted by the user or another user on the video playback interface. In the conventional user character display method, the user character output to the video reproduction interface is usually output and displayed via a fixed character display track in the video reproduction interface.

The embodiments of the present application provide video data processing methods and related devices.

In one aspect of the embodiments of the present application, a video data processing method applied to a computer device is provided, and the method is described.
A step of determining a target pixel point from a key video frame of the target video in response to a trigger operation on the target video and acquiring multimedia information associated with the target pixel point, wherein the key video frame is a step. A step and a step, which is a video frame in which the trigger operation is located, and the target pixel point is a pixel point corresponding to the trigger operation in the key video frame.
A step of determining a locus acquisition request corresponding to the target pixel point based on the position information of the target pixel point in the key video frame, and
The step of acquiring the target locus information associated with the position information of the target pixel point in the key video frame based on the locus acquisition request, wherein the target locus information is the key video of the target pixel point. The position information of the target pixel point in the next video frame of the key video frame, including the position information in the video frame next to the frame, is obtained by tracking the target pixel point. ,
When playing back the video frame next to the key video frame, the multimedia information is displayed based on the position information of the target pixel point in the video frame next to the key video frame in the target locus information. Including steps.

In one aspect of the embodiments of the present application, a video data processing method applied to a service server is provided, wherein the method is described.
A step of acquiring trajectory information associated with a target video in response to a trajectory acquisition request for a target pixel point in a key video frame, wherein the key video frame is a video frame in the target video and the target pixel. The points are the pixel points in the key video frame, and the locus information is determined by the position information of the pixel points in each video frame of the target video.
It is a step of selecting the target locus information associated with the position information of the target pixel point in the key video frame from the locus information associated with the target video, and returning the target locus information. The locus information includes the target position information, and the target position information is for triggering to display the multimedia information associated with the target pixel point in the next video frame of the key video frame. , Steps and, including.

In one aspect of the embodiments of the present application, a video data processing method is provided, wherein the method is described.
The step of acquiring the adjacent first video frame and second video frame from the target video, and
A step of determining an average displacement matrix corresponding to the first video frame based on the optical flow tracking rule corresponding to the target video, the pixel points in the first video frame, and the pixel points in the second video frame. When,
A step of tracking the position information of the pixel points in the first video frame based on the average displacement matrix and determining the position information of the traced pixel points in the second video frame.
A step of generating trajectory information associated with the target video based on the position information of the pixel points in the first video frame and the position information of the tracked pixel points in the second video frame. The locus information includes a step that includes target locus information for tracking and displaying multimedia information associated with a target pixel point in a target video.

In one aspect of the embodiments of the present application, a video data processing device applied to a computer device is provided, and the device is used.
An object determination module that determines a target pixel point from a key video frame of the target video in response to a trigger operation on the target video and acquires multimedia information associated with the target pixel point, and is the key video frame. Is a video frame in which the trigger operation is located, and the target pixel point is an object determination module, which is a pixel point corresponding to the trigger operation in the key video frame.
A request determination module that determines a locus acquisition request corresponding to the target pixel point based on the position information of the target pixel point in the key video frame.
A locus acquisition module that acquires target locus information associated with position information of the target pixel point in the key video frame based on the locus acquisition request, wherein the target locus information is the target pixel point of the target pixel point. The position information of the target pixel point in the next video frame of the key video frame, including the position information in the video frame next to the key video frame, is obtained by tracking the target pixel point. Trajectory acquisition module and
When playing back the video frame next to the key video frame, the multimedia information is displayed based on the position information of the target pixel point in the video frame next to the key video frame in the target locus information. Includes a text display module.

In one aspect of the embodiments of the present application, a video data processing device applied to a service server is provided, and the device is used.
A request response module that acquires trajectory information associated with a target video in response to a trajectory acquisition request for a target pixel point in a key video frame, wherein the key video frame is a video frame in the target video, and the said. The target pixel point is a pixel point in the key video frame, and the locus information is determined by the position information of the pixel point in each video frame of the target video.
A locus selection module that selects target locus information associated with the position information of the target pixel point in the key video frame from the locus information associated with the target video, and returns the target locus information. The target locus information includes target position information, and the target position information is for triggering display of multimedia information associated with the target pixel point in the next video frame of the key video frame. Includes a locus sorting module, which is.

In one aspect of the embodiments of the present application, a video data processing apparatus is provided, wherein the apparatus is
The first acquisition module that acquires the adjacent first video frame and second video frame from the target video,
A matrix that determines the average displacement matrix corresponding to the first video frame based on the optical flow tracking rule corresponding to the target video, the pixel points in the first video frame, and the pixel points in the second video frame. Get module and
A position tracking module that tracks the position information of the pixel points in the first video frame based on the average displacement matrix and determines the position information of the tracked pixel points in the second video frame.
There is a tracking generation module that generates trajectory information associated with the target video based on the position information of the pixel points in the first video frame and the position information of the tracked pixel points in the second video frame. , The trajectory information includes a tracking generation module, including target trajectory information for tracking and displaying multimedia information associated with a target pixel point in a target video.

In one aspect of the embodiments of the present application, a computer device is provided, wherein the computer device comprises a processor, a memory, a network interface, and the like.
The processor is connected to the memory and the network interface, the network interface provides a data communication function, the memory stores a computer program, and the processor calls the computer program. The method described in one embodiment of the embodiment is carried out.

In one embodiment of the present embodiment, a computer storage medium is provided, the computer storage medium stores a computer program, the computer program includes program instructions, and the program instructions are delivered by a processor. When executed, the method described in one embodiment of the present application is executed.

In order to more clearly explain the configuration in the embodiment or the prior art of the present application, the drawings necessary for the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings in the following description show only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained from these drawings without any creative effort.

It is a schematic diagram of the configuration of the network architecture provided in the embodiment of this application. FIG. 3 is a schematic diagram of a plurality of video frames in the target video provided in the embodiments of the present application. FIG. 3 is a schematic diagram of a scenario for acquiring a target video provided in an embodiment of the present application. It is a schematic diagram of the flow of the video data processing method provided in the Example of this application. It is a schematic diagram of acquisition of multimedia information provided in the Example of this application. It is a schematic diagram of all pixel tracking provided in the Example of this application. It is a schematic diagram of the tracking of barrage data in a plurality of consecutive video frames provided in the embodiment of the present application. It is a schematic diagram of another video data processing method provided in the Example of this application. It is a method of determining the effective pixel point provided in the embodiment of this application. It is a schematic diagram of the barrage data display based on the locus information provided in the embodiment of the present application. It is a schematic diagram of the structure of the video data processing apparatus provided in the Example of this application. It is a schematic diagram of the structure of the computer equipment provided in the Example of this application. It is a schematic diagram of the configuration of another video data processing apparatus provided in the embodiment of this application. It is a schematic diagram of the configuration of another computer device provided in the embodiment of this application. It is a schematic diagram of the configuration of another video data processing apparatus provided in the embodiment of this application. It is a schematic diagram of the configuration of another computer device provided in the embodiment of this application.

Hereinafter, the configuration of the embodiment of the present application will be clearly and sufficiently described with reference to the drawings of the embodiment of the present application, but as is clear, the embodiment described is only a part of the embodiment of the present application. Not all examples. All other embodiments that one of ordinary skill in the art obtains from the embodiments of the present application without creative labor fall within the scope of protection of the present application.

In the conventional network video playback process, the user characters displayed on the video playback interface are independent of the video content played on the video playback interface, so there is a constant distance between the displayed user characters and the video content. There is no correlation. Further, when the user intends to transmit the user character, the user terminal outputs the acquired user character in a predetermined character display track. Therefore, since all the user characters transmitted from each user are output on the same character display track, it is impossible to make a unique comment on the video content.

See FIG. 1, which is a schematic diagram of the configuration of the network architecture provided in the examples of the present application. As shown in FIG. 1, the network architecture may include a service server 2000 (or application server 2000) and a user terminal cluster. The service server 2000 may be a server cluster composed of a large number of servers, for example, a cloud server (or abbreviated as cloud). The user terminal cluster may include a plurality of user terminals, and specifically, as shown in FIG. 1, may include a user terminal 3000a, a user terminal 3000b, a user terminal 3000c, ..., And a user terminal 3000n. As shown in FIG. 1, the user terminal 3000a, the user terminal 3000b, the user terminal 3000c, ..., And the user terminal 3000n may each be network-connected to the service server 2000. As a result, each user terminal can exchange data with the service server 2000 via the network connection.

As shown in FIG. 1, the target application may be integrated and installed in any of the user terminals of the user terminal cluster. When the target application is executed on each user terminal, data can be exchanged with the service server 2000 shown in FIG. 1, respectively. Here, the target application may include an application having a video playback function, such as a multimedia application, a social application, and an entertainment application. In order to facilitate understanding, in the embodiment of the present application, the service data display is performed by the target user terminal integrated with the target application, taking as an example the case where one of the plurality of user terminals is used as the target user terminal. A specific process for realizing data exchange with the service server 2000 by the platform will be described. Here, the target user terminal in the embodiment of the present application may include a mobile terminal in which the target application is integrated, such as a personal computer, a tablet computer, a notebook computer, and a smartphone. Here, the service server 2000 may be a background server of the target application, and the service database corresponding to the background server stores each service data information displayed on the service data display platform. The service data information may include internet information such as video data. It should be understood that the service data display platform may display a plurality of videos, and the target user may display one of the plurality of videos on the target user terminal via the service data display platform. When triggered, the video data corresponding to the video may be acquired, the video data may be played back on the target user terminal, and the video data currently being played on the target user terminal may be targeted. It can be called a video. The target video is video data returned by the service server 2000 based on a data load instruction transmitted from the target user terminal.

Here, the target video may include a plurality of video frames. Each video frame can be referred to as one image data. Also, each video frame corresponds to one playback time stamp (ie, one time) within the playback period of the target video. As a result, when the target user terminal subsequently loads and plays the target video, each video frame is displayed on the playback display interface based on the playback time stamp corresponding to each video frame in the target video. Can be done.

Here, the service server 2000 may perform framing processing on each video of the video set stored in the service database in the video preprocessing stage. As a result, a plurality of video frames included in each video can be divided into one picture. Further, for ease of understanding, see FIG. 2, which is a schematic diagram of a plurality of video frames in the target video provided in the embodiments of the present application. Here, the target video may be video A in the service database described above, and as shown in FIG. 2, the video A is n video frames (n is a positive integer greater than 0). May include. The service server 2000 may predivide the n video frames in the video A into n pictures. Of these n pictures, two pictures adjacent to each other in the front and back can be called one image pair. For example, as shown in FIG. 2, in the embodiment of the present application, the video frame corresponding to the first time shown in FIG. 2 and the video frame corresponding to the second time can be referred to as one image pair. The video frame corresponding to the second time and the video frame corresponding to the third time can be called one image pair, and the video frame of the n-1th time and the video frame corresponding to the nth time can be referred to as one image pair. It can be called a first image pair. In other words, for one target video, multiple image pairs can be determined from multiple video frames of the target video, and each image pair has a video corresponding to two adjacent times before and after. Frames may be included, i.e., each image pair may contain two adjacent video frames.

For ease of understanding, in the embodiments of the present application, the first image pair among the plurality of image pairs will be taken as an example. In the video preprocessing stage, in the embodiment of the present application, one video frame of the first image pair (for example, the video frame corresponding to the first time shown in FIG. 2) can be referred to as a first video frame. , The other video frame of the image pair (ie, the video frame corresponding to the second time) can be referred to as the second video frame, and further, based on the optical flow tracking rules, the first video frame of the image pair. By tracking the position information of all the pixel points in the above, it is possible to acquire the appearance position information of each pixel point in the second video frame in the first video frame. Since each of the image pairs contains two adjacent video frames, it is possible to calculate the appearance position information of the pixel points in the first video frame of each image pair in the next video frame, and finally. In the service server 2000, the movement loci of all the pixel points in the video A in all the video frames can be determined, and the movement loci of these pixel points can be collectively referred to as the locus information of the pixel points. ..

Since the service server 2000 can calculate the locus information of all the pixel points in the video A in advance, the video A currently being played when the target user plays the video A on the target user terminal. Can be called the target video. During playback of the target video, if the target user needs to track an object, the target user may perform a trigger operation on the object to be tracked (that is, the target object) on the target user terminal. .. The pixel point corresponding to the trigger operation is called the target pixel point, that is, the target pixel point is determined by the trigger operation performed by the target user on the target object in the currently playing video frame, and the target pixel point is determined. Trigger operations can be used to select the target object that needs to be tracked from the currently playing video frame. Here, in the embodiment of the present application, the video frame corresponding to the trigger operation can be referred to as a key video frame. In other words, in the embodiment of the present application, the video frame currently including the target pixel point can be referred to as a key video frame in the target video. As should be understood, the key video frame may be a video frame corresponding to the first time in the embodiment corresponding to FIG. 2 above. Optionally, the key video frame may be a video frame corresponding to the second time in the embodiment corresponding to FIG. 2, but is not exhaustively mentioned here.

As to be understood, in the embodiment of the present application, the key video frame, the target pixel point in the key video frame, and the position information of the target pixel point are given to the service server 2000 in the embodiment corresponding to FIG. Can be done. As a result, the service server 2000 selects the target pixel point from the locus information of all the pixel points in the target video calculated in advance based on the position information of the target pixel point in the key video frame. The locus information that matches the position information can be selected as the target locus information. Here, the target locus information may include the position coordinates of the target pixel point in the next video frame of the key video frame. Further, the service server 2000 may return the target locus information to the target user terminal. As a result, when the target user terminal plays the video frame next to the key video frame, the position information of the target pixel point in the video frame next to the key video frame is further based on the target locus information. That is, the target position information of the target pixel point can be acquired, and further, the multimedia information corresponding to the target object can be displayed based on the target position information.

Further, see FIG. 3, which is a schematic diagram of a scenario for acquiring a target video provided in an embodiment of the present application. The target user terminal shown in FIG. 3 may be the user terminal 3000a in the embodiment corresponding to FIG. 1 above. As shown in FIG. 3, after entering the target application, the target user may display the service data display platform of the target application on the target user terminal (for example, a smartphone). The service data display platform may display video 10a, video 20a, video 30a, and video 40a shown in FIG. When the target user needs to play the video 30a shown in FIG. 3 (the video 30a may be the video A in the embodiment corresponding to the above FIG. 3) on the target user terminal, the video 30a A playback operation may be performed on the located area (for example, the target user may perform a click operation on the video 30a), and the target identification information corresponding to the video 30a may be obtained. In addition to the data load instruction shown in FIG. 3, the data load instruction may be further given to the application server having a network connection relationship with the target user terminal. The application server may be the service server 2000 in the embodiment corresponding to FIG. 1 above. As can be understood, when the application server obtains the data load instruction, the video data corresponding to the target identification information can be searched from the service database, and the detected video data may be collectively referred to as the target data. can. Thereby, the target data can be given to the target user terminal shown in FIG. 3, and the target user terminal can reproduce the video data by the video reproduction interface shown in FIG. At this time, the target user terminal can refer to the video 30a selected and played by the target user as the target video. That is, at this time, the target user terminal can reproduce each video frame in the video A according to the reproduction time stamp shown in FIG.

Here, for a specific process of acquiring the target object and acquiring the target locus information by the target user terminal, the realization method provided in the examples corresponding to FIGS. 4 to 7 below may be referred to. Further, the specific processes of acquiring the position information of the pixel points in the second video frame by the service server 2000 and selecting the target locus information corresponding to the target pixel points are described in FIGS. 8 to 10 below. The implementation method provided in the embodiment corresponding to the above may be referred to.

See FIG. 4, which is a schematic diagram of the flow of the video data processing method provided in the examples of the present application. As shown in FIG. 4, this method is applicable to the target user terminal in the embodiment corresponding to FIG. 1 above. This method may include the following steps.

In step S101, in response to the trigger operation for the target video, the target pixel point is determined from the key video frame of the target video, and the multimedia information associated with the target pixel point is acquired. It is a video frame in which the trigger operation is located, and the target pixel point is a pixel point corresponding to the trigger operation in the key video frame.

Specifically, the target user terminal may display a service data display platform for mounting a plurality of service data information on the display interface of the target application when accessing the target application. For example, each service data information on the service data exhibition platform may be one video. The service data information displayed on the service data display platform is selected by the application server having a network connection relationship with the target user terminal based on the user profile data of the target user (for example, the behavior history data of the target user). It may be determined by doing. When the target user executes a playback operation on one service data information (for example, one video) on the service data exhibition platform, the video data corresponding to the video is obtained from the service database corresponding to the application server. It may be loaded, and the loaded video data may be played back on the video playback interface of the target user terminal. Further, the target user terminal may acquire the trigger operation performed by the target user on the target object (that is, the object that needs to be tracked) in the video playback interface while playing the video data. .. The trigger operation is, for example, clicking or touching a certain point of the target object in the video frame displayed on the display of the target user terminal with the mouse. The video frame corresponding to the trigger operation can be called a key video frame, and the pixel point corresponding to the trigger operation in the key video frame can be called a target pixel point. Pixel points are individual points in an image (eg, a video frame). When the image is a picture having a resolution of 640 × 480, 640 × 480 pixel points are distributed in the image. Usually, pixel points in an image have spatial position and color (or gradation) attributes. At the same time, the target user terminal may create a text box in a subwindow independent of the video playback interface. This allows the target user to enter multimedia information related to the target object in the text box. When the target user inputs the multimedia information in the text box, the target user terminal can acquire the multimedia information associated with the target object. That is, the multimedia information associated with the target object can be collectively referred to as user characters and user comments input by the target user.

Here, the target user terminal may be a terminal device having a video data reproduction function. The target user terminal may be the user terminal 3000a in the embodiment corresponding to FIG. 1. The target user terminal may be understood as a mobile terminal. Here, the application server may be the service server 2000 in the embodiment corresponding to FIG. 1.

Further, for ease of understanding, see FIG. 5, which is a schematic diagram of the acquisition of multimedia information provided in the embodiments of the present application. As shown in FIG. 5, while the target user terminal is playing the video 30a in the embodiment corresponding to FIG. 3, the target user terminal may use the video 30a currently being played as the target video. As can be seen, the target user may perform a trigger operation on any one of the plurality of video frames included in the video 30a at any time during which the video 30a is being played. .. The target user terminal may use the video frame corresponding to the trigger operation as the key video frame. For example, as shown in FIG. 5, the target user may select the object A as the target object in the video reproduction interface 100a shown in FIG. In this case, the target user terminal can refer to the video frame currently being reproduced by the video reproduction interface 100a as a key video frame. In other words, the target user terminal may use the video frame corresponding to the selection operation (that is, the trigger operation) as the key video frame, and the pixel point corresponding to the selection operation in the key video frame as the target pixel point. May be good. At this time, the target pixel point is the pixel point of the key video frame in the target video acquired by the target user terminal.

As shown in FIG. 5, when the target user performs a trigger operation on the object A in the video playback interface 100a shown in FIG. 5, the text box shown in FIG. 5 pops up in the video playback interface 200a shown in FIG. The text box may also be referred to as a dialog box. The text box shown in FIG. 5 can be understood as a floating window independent of the video playback interface 200a. Further, the text box shown in FIG. 5 may have a relational relationship with the object A shown in FIG. 5 (for example, there may be a relative relationship between the target pixel point in the object A and the display position. The relationship between the target pixel point of the target object in the video 30a and the multimedia information associated with the target object is constructed). The realization of the floating window may be similar to or the same as the realization of the video playback interface. It should be understood that the multimedia information entered in the dialog box of the embodiments of the present application may include data such as user characters, user pictures, and user facial expressions, and the user characters entered by the target user in the dialog box. (That is, text information), user pictures (that is, picture information), user facial expressions (that is, facial expression information), and the like can be collectively referred to as dialog data. The display of the barrage data may be similar to that of subtitles.

Therefore, when the target user inputs the text information A in the text box of the video reproduction interface 200a shown in FIG. 5, the input text information A may be displayed in the video reproduction interface 300a shown in FIG. The input text information A may be text information having a certain position interval distance from the target pixel point of the object A as shown in FIG. The text information A displayed on the video reproduction interface 300a can be referred to as barrage data associated with the target object.

In step S102, the locus acquisition request corresponding to the target pixel point is determined based on the position information of the target pixel point in the key video frame.

Specifically, the target user terminal may determine the position information of the target pixel point in the key video frame, and the frame number in the target video of the key video frame and the key video of the target pixel point. A locus acquisition request corresponding to the target pixel point may be generated based on the position information in the frame. As a result, step S103 can be further executed.

Here, the locus acquisition request selects the locus information matching the target pixel point from the locus information corresponding to each of all the pixel points in the target video calculated in advance to the application server. Can be used to instruct.

In step S103, the target locus information associated with the position information of the target pixel point in the key video frame is acquired based on the locus acquisition request.

Here, the target locus information includes the position information of the target pixel point in the video frame next to the key video frame, and the position information of the target pixel point in the video frame next to the key video frame is. It is obtained by tracking the target pixel point.

In the embodiment of the present application, the target user terminal has a movement locus in all video frames of all pixel points in the target video calculated in advance by the application server (the movement locus of each pixel point is one locus information). Based on (which can be generically referred to), the movement locus of the pixel point matching the target pixel point is associated with the position information of the target pixel point in the key video frame from the movement loci corresponding to these pixel points. It may be selected as the target locus information obtained. In other words, when the target user terminal acquires the target locus information, the above figure is based on the position information of the target pixel point included in the target locus information in the next video frame of the key video frame. To quickly determine the position spacing distance between the target pixel point in the target object and the multimedia information associated with the target object, or the position spacing between the target pixel point and the multimedia information in the embodiment corresponding to 5. That is, the appearance position information of the multimedia information in the next video frame of the key video frame can be quickly acquired.

As can be understood, the position spacing distance may be understood as the relative position spacing distance between the target pixel point in the key video frame and the corresponding barrage data. That is, the position separation distance may include a relative position separation distance in the horizontal direction (that is, the lateral direction), or may include a relative position separation distance in the vertical direction (that is, the vertical direction). As a result, when the target user terminal acquires the position information of the target pixel point in the video frame next to the key video frame, the text information A next to the key video frame is based on the relative position spacing distance. It is possible to ensure that the position information in the video frame can be calculated quickly. That is, at this time, the position information of the text information A displayed on the video reproduction interface 300a in the embodiment corresponding to FIG. 5 is based on the position information of the target pixel point in another video frame after the key video frame. It will be decided. When the calculation performance of the target user terminal is normal, the target user terminal targets the locus information matching the position information of the target pixel point in the key video frame from the application server having a network connection relationship with the target user terminal. It may be acquired as trajectory information. As a result, when the target user terminal acquires the target locus information of the target pixel point calculated in advance by the application server, the target pixel point in the target locus information is the next video frame of the key video frame. Based on the appearance position information in, high-speed tracking of the barrage data can be realized quickly and accurately within a valid period. In this way, the amount of calculation of the target user terminal can be effectively reduced, and even when the calculation performance of the target user terminal is normal, it is possible to ensure that the barrage data can be tracked at high speed. ..

Here, the valid period may be a display period corresponding to the barrage data. That is, the target user terminal can track the barrage data associated with the target object within the display period.

It should be understood that the movement locus of each pixel point in the target video (that is, the locus information of each pixel point) is determined by the position information of each pixel point in each video frame of the target video. Here, in the case of a target video including a plurality of video frames, in the embodiment of the present application, any two adjacent video frames among the plurality of video frames may be determined as one image pair. It should be understood that, of the two video frames contained in each image pair determined from the plurality of video frames, one video frame can be referred to as the first video frame, and the other video frame can be referred to as the first video frame. It can be called a second video frame. Regarding the image pair 1 including the video frame corresponding to the first time and the video frame corresponding to the second time in the embodiment corresponding to FIG. 2, the video frame corresponding to the first time in the image pair 1 Can be referred to as a first video frame, and a video frame corresponding to the second time can be referred to as a second video frame. Further, based on the pre-calculated average displacement matrix between the two video frames in the image pair 1, all the pixel points in the first video frame are tracked and all the pixels in the first video frame are traced. The appearance position information of the point in the second video frame can be determined. Similarly, for the image pair 2 including the video frame corresponding to the second time and the video frame corresponding to the third time in the embodiment corresponding to FIG. 2, the video frame corresponding to the second time is the second. It can be called one video frame, and the video frame corresponding to the third time can be called the second video frame. Thereby, based on the pre-calculated average displacement matrix between the two video frames in the image pair 2, all the pixel points in the first video frame are tracked, and all the pixel points in the first video frame are traced. The appearance position information of the pixel point in the second video frame can be determined. In this way, in the embodiment of the present application, the average displacement matrix corresponding to each image pair can be acquired, and the average displacement matrix corresponding to each image pair is set in the first video frame in each image pair. It can be called the corresponding average displacement matrix, and the average displacement matrix corresponding to each first video frame maps all the pixel points in the first video frame to the second video frame, and in the second video frame, It can be used to accurately acquire the position information of these mapped pixel points. It should be understood that the average displacement matrix in the embodiments of the present application may include a longitudinal average displacement matrix and a lateral average displacement matrix. The vertical average displacement matrix performs vertical coordinate conversion on the first vertical coordinate value (for example, y value) of each pixel point in the first video frame, and maps it to the second video frame of each pixel point. The second vertical coordinate obtained can be obtained. Similarly, the lateral average displacement matrix is used to perform lateral coordinate conversion on the first lateral coordinate value (for example, x value) of each pixel point in the first video frame, and the second video of each pixel point. The second lateral coordinates mapped to the frame can be obtained. As should be understood, in the embodiment of the present application, the first horizontal coordinate and the first vertical coordinate value of each pixel point in the first video frame are referred to as the first position information of each pixel point in the first video frame. The second horizontal coordinate and the second vertical coordinate value of each pixel point mapped to the first video frame are referred to as the second position information of each mapped pixel point in the second video frame. be able to. Since each image pair corresponds to one average displacement matrix, the corresponding second position information can be calculated based on the first position information of the pixel points in the first video frame, and the calculated second position information can be calculated. The second position information of each mapped pixel point in the two video frames can be stored, and the position information in each video frame of the same pixel point is integrated to move the locus of all the pixel points in the video frame. Can be obtained. Thereby, it is possible to realize the tracking of all the pixel points in all the video frames of the target video.

It should be understood that the plurality of videos in the target video shown in the embodiment corresponding to FIG. 2 above may be a plurality of consecutive image frames. Therefore, after the target video shown in FIG. 2 above is divided, a video frame number corresponding to the playback order can be set for each divided image frame (that is, a video frame). For example, the video frame number of the video frame acquired at the first time may be 1, and the video frame number 1 is such that the video frame acquired at the first time is the first in the target video. It can be used to indicate that it is one frame. Similarly, the video frame number of the video frame acquired at the second time may be 2, and the video frame number 2 is such that the video frame acquired at the second time is the target video. It can be used to indicate that it is the second frame. In this way, the video frame number of the video frame acquired at the above-mentioned n-1 time may be n-1, and the video frame number n-1 is acquired at the n-1 time. The resulting video frame can be used to indicate that it is the n-1th frame in the target video, and the video frame number of the video frame acquired at the above nth time may be n. The video frame number n can be used to indicate that the video frame acquired at the nth time is the nth frame in the target video, that is, the last frame in the target video.

In order to facilitate understanding, in the embodiment of the present application, in order to explain a specific process for realizing pixel tracking by translating a pixel point in the first frame to a second frame by an average displacement matrix. Of the plurality of video frames shown in FIG. 2, the image pair composed of the first frame and the second frame can be referred to as the first image pair. Here, the first frame in the first image pair corresponds to the video frame corresponding to the first time in the embodiment corresponding to FIG. 2, and the second frame in the first image pair corresponds to FIG. 2 above. It is a video frame corresponding to the second time in the embodiment. Further, see FIG. 6, which is a schematic diagram of all pixel tracking provided in the examples of the present application. The image pair (1, 2) shown in FIG. 6 may be the first image pair described above. The first video frame in the first image pair may be the video frame corresponding to the first time described above (that is, the first frame), and the second video frame in the first image pair may be the first video frame described above. It may be a video frame (that is, a second frame) corresponding to two times. Here, it should be understood that the numerical value 1 of the image pair (1, 2) is the video frame number of the first frame, and the numerical value 2 is the video frame number of the second frame. Therefore, any two video frames adjacent to each other before and after in the target video can be represented by the video frame number of each video frame in the target video. The pixel point display area 600a shown in FIG. 6 may include all the pixel points extracted from the first video frame of the image pair. For example, each pixel point in the pixel point display area 600a is 1. It can correspond to one area identifier. The pixel point display area 600a in FIG. 6 is merely for illustration purposes, and the pixel point display area 600a can also be referred to as a pixel point area or the like. As a matter of understanding, in the embodiment of the present application, only the case where the pixel points acquired from the first video frame are 20 pixel points is taken as an example, but in reality, the acquisition is performed from the first video frame. The number of pixel points created is much higher than the 20 listed in the examples of the present application. It should be understood that since a plurality of video frames in the same video are acquired by image collection by the same terminal, the number of pixel points in each video frame included in the same video is the same.

As shown in FIG. 6, after each pixel point in the first video frame is acquired, all the acquired pixel points can be collectively referred to as a pixel point, and further, an average displacement matrix shown in FIG. Therefore, all the pixel points in the pixel point display area 600a can be tracked, and further, the position information of the mapped pixel points can be determined in the pixel point display area 700a corresponding to the second video frame. .. For example, taking the pixel point A shown in FIG. 6 as an example, the position information of the pixel point A in the pixel point display area 600a shown in FIG. 6 may be the coordinate position information of the area identifier 5, and the average thereof. The pixel point A can be mapped to the pixel point display area 700a shown in FIG. 6 by the displacement matrix, and the position information of the pixel point A in the pixel point display area 700a shown in FIG. 6 is the area identifier 10. It may be coordinate position information. In the embodiment of the present application, after calculating the position information of the pixel point A in the second video frame, the position information can be stored. Since each image pair in the target video can correspond to one average displacement matrix, the position information of the pixel point in each first video frame, which is mapped to the second video frame, is calculated. Can be done. By integrating the appearance position information of the same pixel point in each image pair in consecutive video frames, it is possible to acquire the appearance position information of the pixel point A in each video frame of the target video, and further, the said The movement locus of the pixel point A can be acquired based on the position information of the pixel point A in each video frame of the target video.

Similarly, for the other pixel points of all the pixel points in the video frame, the average displacement matrix corresponding to each image pair (that is, the average displacement matrix corresponding to the first video frame in each image pair). It is possible to determine the position information of each of the other pixel points in each video frame of the target video, and further, the movement locus of the other pixel points is acquired to obtain the first video frame of each image pair. It is possible to realize all pixel tracking of all the pixel points in the above, and further, it is possible to acquire the position information in each video frame of all the pixel points of the target video. As should be understood, in the embodiment of the present application, the locus information corresponding to each of all the pixel points in the target video can be collectively referred to as the locus information corresponding to the pixel points.

Here, when it is difficult to satisfy the calculation request for tracking a large number of pixel points, the calculation performance of the target user terminal has a network connection relationship with the target user terminal in order to reduce the calculation amount of the target user terminal. A certain application server may pre-calculate the movement locus of all pixel points in the target video. As a result, when the target user terminal actually plays the target video, the application server receives the position information in the key video frame of the target pixel point transmitted from the target user terminal, and is calculated in advance. From the locus information corresponding to the pixel points, the locus information matching the target pixel points can be selected as the target locus information, and the target locus information can be returned to the target user terminal. As a result, the target user terminal can further execute step S104 based on the acquired target locus information. Here, the target pixel point is a pixel point in the key video frame selected by the target user.

Optionally, if the target user terminal has good calculation performance, the movement loci of all the pixel points in the target video may be calculated in advance in the target user terminal. As a result, when the target user terminal actually plays the target video, further, based on the target pixel points in the target object selected by the target user, from the locus information corresponding to these pixel points, The locus information matching the target pixel point can be selected as the target locus information, and step S104 can be further executed.

In step S104, when the video frame next to the key video frame is played back, the multimedia is based on the position information of the target pixel point in the video frame next to the key video frame in the target locus information. Display information.

Further, for ease of understanding, see FIG. 7, which is a schematic diagram of tracking barrage data in a plurality of consecutive video frames provided in the embodiments of the present application. It should be understood that in the embodiments of the present application, a plurality of consecutive video frames for barrage tracking are located after the key video frame currently being played and the key video frame in the target video that has not yet been played. It may include a video frame to be used. For example, when the video frame 10 shown in FIG. 7 is used as a key video frame, it appears in the video frame 10 in each video frame after the key video frame (for example, a video frame such as a video frame 20 and a video frame 30). Barrage tracking may be performed on the barrage data. Here, the video frame 10 shown in FIG. 7 may be the video frame displayed on the video playback interface 300a in the embodiment corresponding to FIG. 5, that is, the video shown in FIG. 7. The frame 10 may be a key video frame associated with the target user in the target video currently being played on the target user terminal. In other words, the key video frame in the embodiment of the present application can be understood as a video frame corresponding to the trigger operation performed when the target user selects the target object. As should be understood, the target object in the embodiment of the present application may include an object such as a person, an animal, or a plant selected by the target user by a click operation in the video frame being played. In other words, the target user terminal can call the object selected by the target user the target object, and the pixel point corresponding to the trigger operation in the target object of the key video frame is set as the target pixel point. Further, from the locus information of all the pixel points in the target video calculated in advance by the application server, the locus information associated with the target pixel point in the target object can be acquired and acquired. The locus information can be the target locus information corresponding to the target pixel point. The target locus information may include position information of the target pixel point in the key video frame, and each video frame of the target pixel point after the key video frame (for example, the next video of the key video frame). The position information in the frame) may also be included. It should be understood that the multimedia information (ie, also associated with the target pixel point) associated with the target object (ie, also associated with the target pixel point) based on the position information of the target pixel point in each video frame after the key video frame. The position information in each video frame after the key video frame of the (bullet data in the embodiment corresponding to FIG. 5) is quickly calculated to realize high-speed tracking of the bullet data associated with the target object. Can be done. Thereby, when the target user terminal plays the next video frame of the key video frame, the barrage data is calculated in the next video frame based on the position information of the barrage data in the key video frame. Allows barrage data to be displayed in real time. The display of the barrage data may be similar to the display of subtitles.

It should be understood that in the embodiments of the present application, the barrage data can be associated with the target object in the key video frame to realize the barrage data and the target object that follow the shape of the shadow, that is, the user. The barrage entered by can move relative to this tracked target object throughout the valid tracking period. For example, in the target video, if the target object is present in any of the plurality of consecutive video frames after the key video frame, the target pixel points in the target object are in some of these consecutive video frames. Based on the position information, the barrage data (that is, the above-mentioned text information A) associated with the target object can be displayed.

The target user terminal may transmit the barrage data (multimedia information) input by the user and the calculated position information of the barrage data in each video frame of the target video to the server. Alternatively, the server receives the frame number of the key video frame in the target video clicked by the user, the target pixel point coordinates, and the input bullet data (multimedia information) transmitted from the target user terminal, and receives the input bullet data (multimedia information) of the target pixel point. , The target locus information in each video frame of the target video is calculated, the position information of the bullet data in each video frame of the target video is calculated based on the target locus information, and the position information of the bullet data is saved. May be good. When receiving the information transmitted from the target user terminal, the server may also receive information such as an identifier of the target user terminal and / or a user identifier in which the user logs in to the target application at the target user terminal. Then, when playing the target video on another user terminal, the server transmits the bullet data, the position information of the bullet data in each video frame of the target video, and the user identifier to the other user terminal. However, another user terminal may display the bullet data in each video frame of the target video based on the position information of the bullet data.

It should be understood that for any video played on the target user terminal, the target user needs to be tracked by the target user from the currently playing video frame when the current time is time T1. You can select the object you think you have. The selected object can be called a target object. Further, the target user terminal selects the locus information associated with the target pixel point in the target object from the previously calculated locus information corresponding to each of all the pixel points in the video. The target locus information corresponding to the target pixel point in the target object can be quickly acquired. Here, it should be understood that the trajectory information corresponding to any of the pixel points in the video, which is calculated in advance, is used to describe the position information of the pixel points in each video frame of the video. Can be done. Therefore, when the video frame reproduced at time T1 is used as the key video frame, the target user terminal can acquire the target pixel point in the target object in the key video frame, and further, the target pixel point is set to the target pixel point. From the corresponding target locus information, the position information of the target pixel point in each video frame after the key video frame can be quickly acquired, and the position information is associated with the target object based on the target locus information. It is possible to display the provided multimedia information. Here, as can be understood, when the locus information formed by the target pixel points in each video frame is a circle, the multimedia information associated with the target object may rotate synchronously according to the locus information. can. In this way, by tracking all the pixel points in the target video in advance, the locus information corresponding to each pixel point can be acquired in advance. As a result, when the target video is being played back on the target user terminal, the pixel point in the target object corresponding to the trigger operation is set as the target pixel point based on the trigger operation executed by the target user, and the target pixel point is set as the target pixel point. The associated locus information can be acquired as the target locus information, and further, based on the acquired target locus information, accurate tracking of the multimedia information associated with the target object can be quickly realized. ..

Therefore, for different objects in the key video frame, it is possible to acquire a movement locus corresponding to each of the target pixel points in the different objects. This allows the barrage data associated with different target objects to move in different trajectories, making the barrage data more relevant to the target object of the barrage data, and the visuals of the barrage data. The exhibition effect can be enriched, and the flexibility of the barrage data display mode can be increased.

In the embodiment of the present application, when the target user's trigger operation for the target video is acquired, the video frame corresponding to the trigger operation in the target video can be used as a key video frame. Thereby, the target pixel point is determined from the key video frame, and the multimedia information associated with the target pixel point and the target object in which the target pixel point is located (for example, the multimedia information is the user in the target video). It may be multimedia data such as characters, pictures, and facial expressions). Further, the locus acquisition request corresponding to the target pixel point is determined based on the position information of the target pixel point in the key video frame. Further, based on the locus acquisition request, the target locus information associated with the position information of the target pixel point in the key video frame can be acquired. Thereby, when playing the next video frame of the key video frame, the barrage data associated with the target pixel point and the target object in which the target pixel point is located is displayed based on the target locus information. Can be done. As can be seen from this, in the embodiment of the present application, when the key video frame is determined, the locus information of the target pixel point is further selected and selected from the locus information of all the pixel points in the key video frame. The locus information of the target pixel point can be used as the target locus information. As a result, the barrage data display effect can be enriched based on the acquired target trajectory information. For example, the acquired target trajectory information may be different for the target pixel points in different target objects, and as a result, the barrage data display effect is different. Further, based on the relationship between the target object and the barrage data, the position information of the barrage data in each video frame after the key video frame can be quickly determined. In other words, the barrage data can fluctuate according to the target object throughout the target video, and can further enhance the visual display effect of the user characters in the video, with the barrage data and the target object, or comment. It can be more relevant to the object in the target video.

Further, for ease of understanding, see FIG. 8, which is a schematic diagram of another video data processing method according to an embodiment of the present application. This method is mainly used to describe the process of data exchange between the target user terminal and the application server. The method may include the following steps.

In step S201, the adjacent first video frame and second video frame are acquired from the target video.

Specifically, the application server can determine a plurality of image pairs from a plurality of video frames included in the target video, and each of the plurality of image pairs is two adjacent videos in the target video. It consists of a frame.

In other words, the application server first frames the target video in the video preprocessing stage for the target video, and converts the plurality of video frames in the target video into one picture at a time according to the playback time series. It may be divided. As a result, it is possible to acquire a plurality of video frames arranged in the reproduction time series as shown in FIG. By dividing each video frame of the target video, a picture corresponding to each video frame can be acquired, that is, one image can be regarded as one image frame. Further, the application server may perform pixel tracking for pixel points in two video frames of each image pair by the front-back optical flow method. For example, in the case of a target video containing n video frames, the application server may use two video frames with adjacent frame numbers as one image pair based on the video frame number of each video frame of the target video. You may decide. In other words, the application server may determine a video frame having a video frame number of 1 and a video frame having a video frame number of 2 as one image pair. Similarly, the application server may determine a video frame having a video frame number of 2 and a video frame having a video frame number of 3 as one image pair. In this way, the application server may determine the video frame having the video frame number n-1 and the video frame having the video frame number n as one image pair.

For each image pair in the target video, two video frames in each image pair can be described by the video frame number of each video frame. Therefore, in the case of the target video including the above-mentioned n video frames, n-1 image pairs can be acquired, and these n-1 image pairs are (1, 2), (2, 2). 3), (3,4), ... (N-1, n) can be expressed. Here, in the image pair, the video frame having the video frame number 1 can be referred to as the first frame of the target video, and the video frame having the video frame number 2 is the second frame of the target video. In this way, the video frame having the video frame number n-1 in the image pair can be called the n-1th frame of the target video, and the video having the video frame number n. The frame can be referred to as the nth frame of the target video. Further, the application server may track the pixel points in each image pair of the target video by the cloud front-back optical flow method. Here, the cloud anterior-posterior optical flow method can be collectively referred to as an optical flow method, and this optical flow method can be used to calculate pixel point displacement between two video frames in each image pair. can.

As you can see, each image pair consists of two adjacent video frames in the front and back, so one video frame in each image pair can be called the first video frame, and in each image pair. The other video frame can be called a second video frame. This makes it possible to further execute step S202.

Here, for ease of understanding, in the embodiment of the present application, the two video frames in each image pair acquired from the target video can be collectively referred to as a first video frame and a second video frame. That is, the application server can acquire the adjacent first video frame and second video frame from the target video.

In step S202, the average displacement matrix corresponding to the first video frame is based on the optical flow tracking rule corresponding to the target video, the pixel points in the first video frame, and the pixel points in the second video frame. To decide.

Specifically, the application server may extract all the pixel points of the first video frame. All the extracted pixel points can be collectively referred to as pixel points. Here, the optical flow tracking rule corresponding to the target video may include the cloud forward / backward optical flow method described above, or may include a cloud displacement integration method and a cloud displacement difference method. It should be understood that the optical flow tracking rule performs an optical flow operation on the pixel points in the first video frame and the pixel points in the second video frame of each image pair to correspond to each image pair. The optical flow tracking result can be obtained, and further, the target state matrix and the target displacement matrix corresponding to each image pair can be determined based on the optical flow tracking result. It is possible to use the target state matrix and the target displacement matrix directly to track the pixel points in the video frame, but the tracking results may not be very accurate. In order to improve the accuracy of tracking, information on the displacement of the pixel points around the pixel points may be taken into consideration when calculating the displacement of the pixel points in the first video frame and the second video frame. For example, the application server includes, for each pixel point in the first video frame, an image block around the pixel point (the pixel point and the pixel points around the pixel point) around the pixel point. ), And the average displacement of all the pixel points in the image block may be calculated and used as the displacement of the pixel points. With such a processing method, the amount of calculation may be large. According to the embodiment of the present application, the optical flow tracking rule further performs a displacement integral calculation on the target state matrix and the target displacement matrix corresponding to each image pair, and the state corresponding to each image pair. It is also possible to obtain an integral matrix and a displacement integral matrix. Further, according to the optical flow tracking rule, it is also possible to perform a displacement difference calculation on the state integral matrix and the displacement integral matrix corresponding to each image pair to obtain the average displacement matrix corresponding to each image pair. .. In other words, the optical flow tracking rule can accurately obtain an average displacement matrix that can be used to accurately track the position information of pixel points in the first video frame of each image pair. According to the displacement integration calculation and the displacement difference calculation described above, the application server can calculate the average displacement between the pixel points in the first video frame and the pixel points in the second video frame in a batch, so that the calculation speed can be determined. It can be improved and the processing efficiency of the pixel point and the video frame can be improved.

Here, in the cloud forward / backward optical flow method, the calculation by the forward / reverse optical flow method is performed synchronously for the first video frame and the second video frame in each image pair, and each image pair is subjected to the calculation. It can be used to obtain the corresponding optical flow tracking results. In other words, the optical flow tracking results obtained by the application server may include a forward displacement matrix corresponding to the first video frame in each image pair, or the inverse corresponding to the second video frame in each image pair. It may include a directional displacement matrix. In the embodiments of the present application, each matrix element of the forward displacement matrix and the reverse displacement matrix may contain displacements of two dimensions (eg, (Δx, Δy)). Here, these two dimensional displacements can be understood as horizontal displacements (ie, Δx) and vertical displacements (ie, Δy) of the same pixel point. It should be understood that for any of the image pairs in the target video, when calculated by the optical flow method, a forward horizontal displacement matrix, a forward vertical displacement matrix, an inverse horizontal displacement matrix, The inverse vertical displacement matrix can be acquired, and the acquired four matrices can be called the optical flow result. Further, the application server may set an initial state matrix for the first video frame in each image pair, and further, based on the previously acquired forward displacement matrix and reverse displacement matrix, each image. It may be determined whether or not the pixel points in the first video frame of each pair satisfy the target selection condition. When the first video frame has pixel points satisfying the target selection condition, the application server may determine the pixel points satisfying the target selection condition as the effective pixel points, and further, based on the determined effective pixel points. Then, the initial state matrix corresponding to the first video frame and the forward displacement matrix may be modified to obtain the target state matrix and the target displacement matrix corresponding to the first video frame in each image pair. Further, the application server determines the average displacement matrix corresponding to the first video frame in each image pair by the above-mentioned cloud displacement integration method and cloud displacement difference method, and the acquired target state matrix and target displacement matrix. You may.

Here, in the embodiment of the present application, the forward horizontal displacement matrix and the forward vertical displacement matrix can be collectively referred to as a forward displacement matrix, and the reverse horizontal displacement matrix and the reverse vertical displacement matrix can be reversed. It can be collectively referred to as a directional displacement matrix. For ease of understanding, in the embodiments of the present application, one of a plurality of image pairs is taken as an example, and the average corresponding to the image pair is taken from the first video frame and the second video frame in the image pair. The process of acquiring the displacement matrix will be described. Here, the first video frame in the image pair may be a video frame having the above-mentioned video frame number of 1, and the second video frame is a video frame having the above-mentioned video frame number of 2. May be good. Therefore, an image pair consisting of a video frame having a video frame number of 1 and a video frame having a video frame number of 2 is called an image pair 1, and the image pair 1 can be represented by (1, 2). ..

Here, the forward displacement matrix corresponding to the image pair 1 calculated by the optical flow method is a forward horizontal displacement matrix (for example, even if the forward horizontal displacement matrix is the matrices Q1 _{, 2, x} ). May be included) and a forward vertical displacement matrix (eg, the forward vertical displacement matrix may be matrices Q1 _{, 2, y} ). Here, it should be understood that each matrix element of the matrix Q1 _{, 2, x} can be understood as the horizontal displacement of the pixel point in the first video frame in the second video frame. That is, each matrix element of the forward horizontal displacement matrix can be referred to as a first lateral displacement corresponding to a pixel point in the first video frame. Similarly, each matrix element of the matrix Q1 _{, 2, y} can be understood as the vertical displacement of the pixel points in the first video frame in the second video frame. That is, each matrix element of the forward horizontal displacement matrix can be referred to as a first longitudinal displacement corresponding to a pixel point in the first video frame. In other words, the matrix size of these two matrices (ie, matrix Q _{1,2, x} and matrix Q _{1, 2, x} ) obtained by the optical flow calculation method is the same as the size of the first video frame. That is, one matrix element can correspond to one pixel point in the first video frame.

Similarly, the reverse displacement matrix corresponding to the image pair 1 calculated by the optical flow method may be a reverse horizontal displacement matrix (that is, the reverse horizontal displacement matrix may be matrices Q _{2, 1, x} ). May be included) and a reverse vertical displacement matrix (ie, the reverse vertical displacement matrix may be matrices Q _{2, 1, y} ). Here, it should be understood that each matrix element of the matrix Q _{2, 1, x} can be understood as the horizontal displacement of the pixel points in the second video frame in the first video frame. That is, each matrix element of the reverse horizontal displacement matrix can be referred to as a second lateral displacement corresponding to a pixel point in the second video frame. Similarly, each matrix element of the matrix Q _{2, 1, y} can be understood as the vertical displacement of the pixel points in the second video frame in the first video frame. That is, each matrix element of the reverse vertical displacement matrix can be referred to as a second vertical displacement corresponding to a pixel point in the second video frame. In other words, the matrix size of these two matrices (ie, matrix Q _{2,1, x} and matrix Q _{2,1, y} ) obtained by the optical flow calculation method is the same as the size of the second video frame. That is, one matrix element can correspond to one pixel point in the second video frame.

It should be understood that for each video frame in the target video, the number of pixel points in each video frame is the same, so these four matrices (ie,) corresponding to the image pair 1 obtained by the optical flow calculation method. , Matrix Q _{1, 2, x} , Matrix Q _{1, 2, y} , Matrix Q _{2, 1, x} , Matrix Q _{2, 1, y} ) have the same matrix size. For example, when the number of pixel points in each video frame is m × n, the matrix size of the acquired four matrices may be m × n. As can be seen, each matrix element of the forward horizontal displacement matrix and the forward vertical displacement matrix can correspond to the pixel points in the first video frame. Therefore, each matrix element of the forward displacement matrix corresponding to the image pair 1 can represent a two-dimensional displacement of the pixel points in the first video frame in the second video frame. The forward displacement matrix corresponding to the image pair 1 can be collectively referred to as a forward displacement matrix corresponding to the first video frame. Similarly, each matrix element of the reverse displacement matrix corresponding to the image pair 1 can represent a two-dimensional displacement of the pixel points in the second video frame in the first video frame. The reverse displacement matrix corresponding to the image pair 1 can be collectively referred to as a reverse displacement matrix corresponding to the second video frame.

As can be seen from this, the application server transfers the pixel points in the first video frame to the second video frame based on the first position information of the pixel points in the first video frame and the optical flow tracking rule. By mapping in the forward direction, it is possible to determine the second position information of the mapped first mapping point in the second video frame, and further, the first position information of the pixel point and the first mapping point. Based on the second position information, the forward displacement matrix corresponding to the first video frame can be determined. Further, the application server maps the pixel points in the second video frame to the first video frame in the reverse direction based on the second position information of the pixel points in the second video frame and the optical flow tracking rule. Then, in the first video frame, the third position information of the mapped second mapping point can be determined, and further, the second position information of the first mapping point and the third position information of the second mapping point. Based on the position information, the reverse displacement matrix corresponding to the second video frame can be determined. Here, the first mapping point and the second mapping point are both pixel points in which the pixel points in one video frame of the image pair are mapped to the other video frame by the optical flow method.

Further, the application server uses the pixel points that satisfy the target selection condition among the pixel points based on the first position information of the pixel points in the first video frame, the forward displacement matrix, and the reverse displacement matrix. A point can be determined as an effective pixel point. Here, the specific process of determining the effective pixel point by the application server can be described as follows.

The application server acquires the first pixel point from the pixel points in the first video frame, determines the first position information of the first pixel point in the first video frame, and determines the first position information of the first pixel point, and the forward displacement matrix. Therefore, the first lateral displacement and the first longitudinal displacement corresponding to the first pixel point may be determined. Further, the application server determines the first pixel point as the first pixel point based on the first position information of the first pixel point and the first lateral displacement and the first vertical displacement corresponding to the first pixel point. It may be mapped in the forward direction to the two video frames, and the second position information of the mapped second pixel point may be determined in the second video frame. Further, the application server determines the second lateral displacement and the second longitudinal displacement corresponding to the second pixel point from the reverse displacement matrix, and obtains the second position information of the second pixel point and the second pixel point. Based on the second lateral displacement and the second longitudinal displacement corresponding to the two pixel points, the second pixel point was mapped in the opposite direction to the first video frame and mapped in the first video frame. The third position information of the third pixel point may be determined. Further, the application server determines the error distance between the first pixel point and the third pixel point based on the first position information of the first pixel point and the third position information of the third pixel point. Then, based on the first position information of the first pixel point and the second position information of the second pixel point, the image block including the first pixel point and the image block including the second pixel point The correlation coefficient may be determined. Further, the application server may determine, among the pixel points, the pixel points whose error distance is less than the error distance threshold and whose correlation coefficient is larger than the correlation coefficient threshold as effective pixel points.

In order to verify the accuracy of the matrix elements of the forward displacement matrix and the reverse displacement matrix in the optical flow tracking result obtained in this way, in the embodiment of the present application, the matrix of the above-mentioned four displacement matrices is obtained by matrix transformation. The elements may be selected, that is, depending on the change status of the matrix elements at the positions corresponding to the respective pixel points in the constructed initial state matrix, the four matrices may be changed to the positions corresponding to the respective pixel points. Certain matrix elements with large displacement errors may be removed. Thereby, the effective pixel point can be determined from the pixel points of the first video frame.

Further, for ease of understanding, see FIG. 9, which is a method of determining effective pixel points provided in the embodiments of the present application. As shown in FIG. 9, the application server may _first initialize the state matrix S1 having the same size as the first video before selecting the matrix elements of the four matrices. In this case, the application server can call the state matrix S ₁ the initial state matrix. Here, in the initial state matrix, the value of the matrix element corresponding to each pixel point can be called a first numerical value. At this time, the first numerical value in the initial state matrix is zero. The change state of the value of the matrix element of the initial state matrix can be used to indicate whether or not the pixel point in the first video frame satisfies the target selection condition. Thereby, the pixel points satisfying the target selection condition can be set as the effective tracking pixel points (that is, the effective pixel points).

Here, the first image frame shown in FIG. 9 may be a video frame having a video frame number of 1 in the above-mentioned image pair 1. The pixel point in the first video frame may include the first pixel point p1 as shown in FIG. 9, that is, the first pixel point p1 is one of all the pixel points of the first video frame. The position information in the first video frame of the first pixel point p1 can be referred to as the first position information. Further, the application server finds the first lateral displacement corresponding to the first pixel point p1 from the forward horizontal displacement matrix of the forward displacement matrix, and finds the forward vertical displacement matrix of the forward displacement matrix. From, the first longitudinal displacement corresponding to the first pixel point p1 can be found, and further, the first position information of the first pixel point p1 and the first horizontal direction corresponding to the first pixel point p1. Based on the displacement and the first longitudinal displacement, the first pixel point p1 is forward-mapped to the second video frame shown in FIG. 9, and in the second video frame, the mapped second pixel point p2. The second position information can be determined. As can be understood, at this time, the second pixel point p2 is a pixel point obtained by matrix-converting the first pixel point p1. Further, the application server can determine the second lateral displacement and the second longitudinal displacement corresponding to the second pixel point p2 from the above-mentioned reverse displacement matrix, and the second pixel point p2 can be determined. Based on the two position information and the second lateral displacement and the second longitudinal displacement corresponding to the second pixel point p2, the second pixel point p2 is moved in the opposite direction to the first video frame shown in FIG. It can be mapped, and in the first video frame, the third position information of the mapped third pixel point p1'can be determined. As can be understood, at this time, the third pixel point p1'is a pixel point obtained by matrix-converting the second pixel point p2 to which the first pixel point p1 is mapped.

Further, the application server has a position error t between two positions of the first position information of the first pixel point p1 and the third position information of the matrix-converted third pixel point p1'in the first video frame. _11'can be determined. Further, the application server displays an image block 10 having a size of k * k pixels (for example, 8 * 8 pixels) centered on the first position information of the first pixel point p1 in the first video frame shown in FIG. You may choose. Further, as shown in FIG. 9, the application server similarly has an image block having a size of k * k pixels centered on the second position information of the second pixel point p2 in the second video frame shown in FIG. 20 may be selected, and a correlation coefficient between the two image blocks (the correlation coefficient may be N ₁ , 2) may be calculated.

Here, the calculation formulas of the correlation coefficients N ₁ and 2 are as follows.

Patch ₁ (a, b) in Equation 1 can represent the pixel value of the pixel point at the position of row a and column b of the image block 10 shown in FIG. The pixel value may be a pixel gradation value between 0 and 255. E (patch ₁ ) represents the average pixel value of the image block 10 shown in FIG. Patch ₂ (a, b) represents the pixel value of the pixel point at the position of row a and column b of the image block 20 shown in FIG. E (patch ₂ ) represents the average pixel value of the image block 20 shown in FIG.

Here, as can be understood, in the embodiment of the present application, after the error distance between the first pixel point p1 and the third pixel point p1'shown in FIG. 9 is calculated, the calculated error distance is set in advance. It may be compared with the error distance. If t _11' < _TB and N 1, ₂ ≧ _TA (where _TB is the set error distance threshold and _TA is the set correlation coefficient threshold), the first video frame. Since the first pixel point p1 in 1 satisfies the above-mentioned target selection condition, it can be determined that the first pixel point p1 is an effective pixel point.

Further, the application server may set the value of the matrix element at the position corresponding to the _first pixel point p1 in the initial state matrix S1 to the second numerical value. For example, in order to indicate that the first pixel point p1 in the first video frame is an effective pixel point, the value of the element corresponding to the _first pixel point p1 in the initial state matrix S1 is switched from 0 to 1. May be good. On the contrary, when t _11' ≧ _TB and / or N 1, ₂ < _TA , it means that the first pixel point p1 in the first video frame does not satisfy the above-mentioned target selection condition. In this case, the application server can determine that the first pixel point p1 shown in FIG. 9 is an invalid tracking pixel point. That is, in the initial state matrix S1, the value of the element corresponding to the _first pixel point p1 remains 0. At the same time, the application server is further located at a position corresponding to the first pixel point p1 in the forward displacement matrix (that is, the matrix Q1 _{, 2, x} and the matrix Q _{1, 2, y} ). The value of a matrix element may be set to 0. Thereby, the forward displacement matrix including the first numerical value can be determined as the target displacement matrix (for example, the forward horizontal displacement matrix Q _x1 and the forward horizontal displacement matrix Q _y1 ). That is, the matrix elements at these positions in the target displacement matrix are used to represent the matrix determined after the error tracking displacements with large errors are excluded from the forward displacement matrix described above. Can be done.

As can be understood, for the other pixel points shown in FIG. 9, the above steps of sequentially selecting the pixel points from the first video frame shown in FIG. 9 and using them as the first pixel points to determine the effective pixel points are repeated. You may. If any of all the pixel points in the first video frame is set as the first pixel point, all the effective pixel points in the first video frame can be determined. Thereby, the matrix element in the initial state matrix can be updated based on the position information of the effective pixel point in the initial state matrix, and the initial state matrix including the second numerical value can be obtained by the first video frame. Can be determined as the target state matrix _S1 corresponding to. Then, the target displacement matrix corresponding to the first video frame (that is, the target horizontal displacement matrix Q _{x, 1} and the target horizontal displacement matrix Q _{y, 1} ) can be acquired. Similarly, for other image pairs among the plurality of image pairs, the target corresponding to the first video frame in each of the remaining image pairs can be obtained by repeating the above step of determining the effective pixel points from the image pair 1. The state matrix and target displacement matrix can be obtained. For example, taking a plurality of consecutive video frames having video frame numbers 1, 2, 3, 4, ..., N in the target video as an example, the plurality of configured image pairs are each (1, 1,). It can be expressed as 2), (2,3), (3,4), ..., (N-1, n). Then, from the optical flow tracking result corresponding to each image pair, the target state matrix S1 corresponding to the image pair ( ₁ , 2) and the image pair (1, 2) are obtained by the above-mentioned effective pixel point determination method. The corresponding target displacement matrix Q ₁ (that is, the target horizontal displacement matrix Q _{x, 1} and the target vertical displacement matrix Q _{y, 1} described above) can be finally obtained. In this way, the target state matrix S _n-1 corresponding to the image pair (n-1, n) and the target displacement matrix Q _n-1 corresponding to the image pair (1, 2) (that is, the above-mentioned target horizontal). The displacement matrix Q _{x, n-1} and the target horizontal displacement matrix Q _{y, n-1} ) can be acquired.

Further, the application server performs an integral calculation on the target state matrix S1 corresponding to the image pair ₁ and the target displacement matrix _Q1 by the cloud displacement integration method, and corresponds to the pixel points in the first video frame. The state integral matrix S _in (x, y) and the displacement integral matrix Q _in (x, y) can be obtained. Here, the displacement integral matrix Q _in (x, y) may include the lateral displacement integral matrix Q _{x, in} (x, y) and the longitudinal displacement integral matrix Q _{y, in} (x, y). good. Further, the state integral matrix S _in (x, y), the lateral displacement integral matrix Q _{x, in} (x, y), and the longitudinal displacement integral matrix Q _{y, in} (x, y) are the following matrix integrals. It can be obtained by an expression.

The x and y in Equations 2, 3, and 4 can be used to represent the coordinates of all the matrix elements _in the state-integrated matrix and the displacement-integrated matrix corresponding to the first video frame, for example, Sin (x). , Y) can represent the values of the matrix elements of the x-by-y column of the state integration matrix. Further, x'and y'in Equation 2, Equation 3, and Equation 4 can represent the coordinates of the matrix elements in the target state matrix and the target displacement matrix. For example, S (x', y') is the target state. It can represent the value of a matrix element in the x'row y'column of a matrix.

Further, the application server may select a target box having a height of M and a width of N in the first video frame by the cloud displacement difference method, and further, within the target box, Equation 2 may be selected. , The displacement difference calculation is performed on these three integral matrices acquired by the equations 3 and 4, and the state difference matrix S _dif (x, y) and the displacement difference matrix Q _dif (x, y) are acquired, respectively. be able to. The target box is for selecting all the pixel points in a certain area around the pixel points and calculating the average displacement, for example, having a size of 80 × 80 pixels.

Here, the displacement difference matrix Q _dif (x, y) may include the lateral displacement difference matrix Q _{x, dif} (x, y) and the vertical displacement integration matrix Q _{y, dif} (x, y). good. The state integral matrix S _dif (x, y), the lateral displacement difference matrix Q _{x, dif} (x, y), and the longitudinal displacement integral matrix Q _{y, dif} (x, y) are the following matrix difference equations (x, y). It can be obtained by the formula 5).

Here, as can be understood, in the embodiment of the present application, the region where the target block is located in the first video frame can be referred to as a difference region. Thereby, the average displacement matrix corresponding to the first video frame can be determined based on the size information of the difference region, the state integral matrix, the lateral displacement integral matrix, and the longitudinal displacement integral matrix. Here, M and N in the displacement difference calculation formula are the length value and the width value of the difference region. Here, x and y in the displacement difference calculation formula are the position information of each pixel point in the first video frame, respectively. By the displacement difference calculation formula, the average value of all the pixel points in the difference region can be quickly obtained. For example, in the case of a state integration matrix, the state difference matrix S _dif (x, y) corresponding to the state integration matrix S _in (x, y) can be acquired. In the case of the lateral displacement integral matrix Q _{x, in} (x, y) and the longitudinal displacement integral matrix Q _{y, in} (x, y), the lateral displacement difference matrix Q _{x, def} (x, y) and the longitudinal direction. The displacement difference matrix Q _{y, dif} (x, y) can be acquired.

Further, the application server sets the ratio of the lateral displacement difference matrix Q _{x, def} (x, y) and the state difference matrix S _div (x, y) to the lateral average displacement matrix Q _{x, F} (x, y). ), And the ratio of the vertical displacement difference matrix Q _{y, in} (x, y) to the state difference matrix S _dif (x, y) is the vertical average displacement matrix Q _{y, F} (x, y). May be determined as.

E in Equations 6 and 7 represents an artificially set relatively small number, such as 0.001. That is, e in the equations 6 and 7 is for avoiding direct division by 0 when all the values of the matrix elements in the state difference matrix S _dif (x, y) are 0. As a result, step S203 can be further executed, and the appearance position information of the pixel points in the first video frame in the second video frame can be calculated in advance on the target user terminal.

In step S203, the position information of the pixel points in the first video frame is tracked based on the average displacement matrix, and the position information of the tracked pixel points in the second video frame is determined.

Specifically, the application server uses the average displacement matrix acquired in step S203 described above (the average displacement matrix is the horizontal average displacement matrix Q _{x, F} (x, y) and the vertical average displacement matrix Q _y ). _{, F} (x, y), and may also be included), and further, the appearance of the pixel points in the first video frame in the next video frame (ie, the second video frame in the image pair 1 above). The position information can be traced quickly and accurately, that is, the position information of the pixel points obtained by tracking the pixel points in the first video frame in the second video frame by performing displacement conversion can be obtained. Can be decided.

In Equation 8, x is the lateral position coordinate of the pixel point in the first video frame, and Q _{x, F} (x, y) is the lateral average displacement matrix corresponding to the first video frame. According to the equation 8, the lateral position coordinates of the pixel points in the first video frame can be coordinate-converted, and the lateral position coordinates of the pixel points in the first video frame can be obtained in the next video frame. .. Similarly, y in Equation 9 is the vertical position coordinate of the pixel point in the first video frame, and Q _{x, y} (x, y) is the vertical average displacement matrix corresponding to the first video frame. In addition, according to the equation 9, the vertical position coordinates of the pixel points in the first video frame are coordinate-converted, and the vertical position coordinates of the pixel points in the first video frame are obtained in the next video frame. Can be done.

As you can see, for any of the image pairs, the average displacement matrix corresponding to the first video frame in the image pair allows the pixel points in the first video frame of the image pair to be quickly tracked. In the second video frame of the image pair, the position coordinates of the tracked pixel points can be determined, i.e., in the second video frame of each image pair, the position information of the tracked pixel points can be determined. can. The application server may further store the location information of the pixel points tracked in each image pair to allow further execution of step S204.

In step S204, the locus information associated with the target video is generated based on the position information of the pixel points in the first video frame and the position information of the tracked pixel points in the second video frame.

Here, the locus information includes target locus information for tracking and displaying multimedia information associated with the target object in the target video.

In step S205, in response to the trigger operation for the target video, the target pixel point is determined from the key video frame of the target video, and the multimedia information associated with the target pixel point is acquired.

In step S206, a locus acquisition request corresponding to the target pixel point is determined based on the position information of the target pixel point in the key video frame.

Here, for the specific implementation method of step S205 and step S206, the description of the target user terminal in the embodiment corresponding to FIG. 4 may be referred to, but further description thereof will be omitted here.

In step S207, the locus information associated with the target video is acquired in response to the locus acquisition request for the target pixel point in the key video frame.

Specifically, the application server may receive the locus acquisition request transmitted by the target user terminal based on the target pixel point in the key video frame, and further, the target calculated in advance by the application server described above. The locus information associated with each of all the pixel points in the video may be acquired. This makes it possible to further execute step S208.

In step S208, the target locus information associated with the position information of the target pixel point in the key video frame is selected from the locus information associated with the target video, and the target locus information is returned.

Specifically, the application server may acquire the video frame number in the target video of the key video frame and the position information of the target pixel point in the key video frame from the trajectory acquisition request, and further, the said. The trajectory information associated with the target pixel point may be selected from the trajectory information associated with the target video acquired in advance by the application server, and the selected trajectory information can be referred to as target trajectory information. Further, the target locus information may be returned to the target user terminal. As a result, the target user terminal receives the target based on the frame number of the key video frame until the appearance position information of the target pixel point in each video frame after the key video frame is acquired. From the locus information, the appearance position information of the target pixel point in the next video of the key video frame can be quickly found. In this case, the target user terminal obtains new locus information consisting of appearance position information of the target pixel point in each video frame after the key video frame. Optionally, it should be understood that when the application server acquires the frame number of the key video frame, the appearance position information of the target pixel point in each video frame after the key video frame is acquired. From the selected locus information, the appearance position information of the target pixel point in the next video of the key video frame may be quickly found. In this case, the application server can call the new locus information consisting of the appearance position information of the target pixel point in each video frame after the key video frame as the target locus information.

Here, as can be understood, when the computational performance of the target user terminal cannot meet the demand for object tracking, the target user terminal generates a trajectory acquisition request corresponding to the target pixel point, and the trajectory acquisition request is applied to the application. It may be sent to the server. As a result, the application server can acquire the target locus information associated with the position information of the target pixel point in the key video frame based on the locus acquisition request, and the acquired target locus information can be obtained by the target user. You can reply to the terminal.

Optionally, if the computational performance of the target user terminal can meet the demand for object tracking, the target user terminal may execute the above-mentioned steps S201 to S204 on the target user terminal. As a result, in the target user terminal, all pixel tracking of all pixel points in the target video is performed in advance, position information of all pixel points in the target video in each video frame is acquired in advance, and further, the target It is possible to integrate the positions of the position information of each pixel point in the video in each video frame and acquire the locus information corresponding to each pixel point in the target video. In this case, the target user terminal sets the target locus information associated with the position information of the target pixel point in the key video frame based on the position information of the target pixel point in the target object in the key video frame. It may be acquired directly on the user terminal. This makes it possible to further execute step S209.

Here, the target locus information includes the position information of the target pixel point in the video frame next to the key video frame, and the position information in the video frame next to the key video frame tracks the target object. It was acquired by.

For example, take the case where a plurality of consecutive video frames included in the currently played video (that is, the target video) are the following six video frames. The six video frames may be a video frame a, a video frame b, a video frame c, a video frame d, a video frame e, and a video frame f. Therefore, the application server may preprocess each video frame in the target video during the video preprocessing stage, i.e., two in the target video based on the optical flow tracking rules described above. The average displacement matrix corresponding to each of the image pairs consisting of adjacent video frames may be determined, and further, the average displacement matrix corresponding to each image pair (corresponding to the first video frame in each image pair). All pixel points in the first video frame may be tracked based on (which can also be referred to as an average displacement matrix). As a result, the appearance position information of all the pixel points in the first video frame in the second video frame can be acquired, and further, the position information of all the pixel points of the target video in each video frame can be acquired. (That is, the position information of all the pixel points of the target video in the above-mentioned video frame a, video frame b, video frame c, video frame d, video frame e, and video frame f can be acquired). Based on the position information of all the pixel points of the target video in each video frame, the locus information corresponding to each of all the pixel points of the target video can be acquired. The locus information corresponding to each of all the pixel points of the target video is referred to as locus information associated with the target video.

Further, for example, when the application server performs preprocessing on the target video, the pixel point A in the target video (for example, the pixel point A is one of all the pixel points in the target video). The locus information corresponding to the pixel point A may be calculated in advance, and the locus information corresponding to the pixel point A includes each video frame of the target video of the pixel point A (that is, the video frame). When the position information in a, the video frame b, the video frame c, the video frame d, the video frame e, and the video frame f) is included, the key video frame corresponding to the target pixel point in the target user terminal is the video of the target video. If it is the frame c, the pixel point A in the target object in the video frame c may be further set as the target pixel point, and further, when the target pixel point is the pixel point A, the locus of the pixel point A in the application server. Information may be selected, and further, based on the selected locus information of the pixel point A, each video frame of the pixel point A after the key video frame (that is, video frame d, video frame e, The position information in the video frame f) can be acquired.

In the embodiment of the present application, when the key video frame is the first video frame in the target video, the target locus information acquired by the target user terminal may be the locus information calculated in advance. For example, in the case where the key video frame described above is the video frame a (first video frame) in the target video, the target locus information acquired by the target user terminal is the video frame a at the target pixel point. , The video frame b, the video frame c, the video frame d, the video frame e, and the position information in the video frame f may be included. Optionally, when the key video frame is the non-first video frame in the target video, the target locus information acquired by the target user terminal consists of partial position information determined from the previously calculated locus information. It may be trajectory information. For example, in the case where the key video frame described above is the video frame c (that is, the non-first video frame) in the target video, the target locus information acquired by the target user terminal is the video of the target pixel point. The position information in the frame d, the video frame e, and the video frame f may be included, and the position information in the video frame d, the video frame e, and the video frame f of the target pixel point can be referred to as partial position information.

As can be understood, optionally, the target user terminal has a locus information including position information in a key video frame of the target pixel point searched by the application server (that is, a locus corresponding to the pixel point A). Information) can also be collectively referred to as target trajectory information. In this case, the target locus information can be regarded as locus information corresponding to the pixel point A matching the target pixel point, which is searched from all the pixel points of the target video. Since the locus information can include the position information of the pixel point A in each video frame of the target video, naturally, from the locus information, each of the target pixel points after the key video. It is also possible to quickly acquire the position information in the video frame.

In step S209, when the video frame next to the key video frame is played back, the multimedia is based on the position information of the target pixel point in the video frame next to the key video frame in the target locus information. Display information.

Here, as can be understood, in the embodiment of the present application, when the target pixel points in the target object selected by the target user are acquired, the locus information corresponding to each of all the pixel points calculated in advance is included. Therefore, the locus information associated with the position information of the target pixel point in the key video frame can be selected, and the selected locus information can be called the target locus information. In the embodiment of the present application, pixel tracking of pixel points in each video frame of the video can be performed in advance. Therefore, when the average displacement matrix corresponding to the first video frame in each image pair is acquired, each pixel in the video is obtained. The position information of the point in the corresponding video frame can be quickly acquired. It should be understood that the pre-calculated position information of each pixel point in the corresponding video frame represents the position information of each pixel point in the video currently being played on the video playback interface in the corresponding video frame. Can be used for Therefore, when the target user terminal acquires the target pixel point in the target object and the multimedia information associated with the target object, the target pixel point is selected from the locus information corresponding to each of the target pixel points. The locus information corresponding to the above can be quickly called the target locus information, and the target locus information can be returned to the target user terminal. As a result, the target user terminal is attached to the target locus information, and the multimedia associated with the target object is based on the position information of the target pixel point in each video frame after the key video frame. Information (eg, barrage data) can be tracked and displayed.

As can be understood, when the locus formed by the position information of the target pixel point in a plurality of consecutive video frames after the key video frame is a circle, the barrage data is obtained in the target user terminal. It can be displayed while rotating according to the trajectory.

For ease of understanding, see FIG. 10, which is a schematic diagram of the barrage data display based on the trajectory information provided in the embodiments of the present application. The video frame 100 as shown in FIG. 10 may include a plurality of objects, and may include, for example, the object 1, the object 2, and the object 3 shown in FIG. When the target user uses the object 1 shown in FIG. 10 as the target object on the target user terminal, the video frame 100 can be called a key video frame, and the pixel points corresponding to the trigger operation by the target user in the target object. Can be called a target pixel point. When the target user terminal has strong calculation performance, the target user terminal may pre-calculate the position information in each video frame of each pixel point in the target video. As a result, the locus information associated with the target video can be acquired in the target user terminal. For example, the locus information 1 shown in FIG. 10 can be acquired as calculated in advance, that is, the position information in the locus information 1 is each of the pixel points in the target video of the target video. It is determined by the position information in the video frame. Therefore, the target user terminal can quickly consider the locus information 1 shown in FIG. 10 as the target locus information based on the position information of the target pixel point in the object 1, and the object 1 in the locus information 1 can be regarded as the target locus information 1. , Multimedia information associated with the target object (ie, object 1) based on position information in each video frame after the key video frame (ie, video frame 200 and video frame 300 shown in FIG. 10). That is, the bullet data 1 “BBBBBB”) shown in FIG. 10 can be quickly tracked and displayed. That is, the barrage data displayed on the video frame 200 and the video frame 300 shown in FIG. 10 are both determined by the position information in the locus information 1 shown in FIG.

As can be understood, the locus information associated with the target video shown in FIG. 10 may be pre-calculated by the application server described above. As a result, even when the application server receives the locus acquisition request for the target pixel point in the object 1 described above, the target pixel point in the key video frame is selected from the locus information associated with the target video shown in FIG. It is possible to quickly acquire the trajectory information associated with the position information of. That is, by executing all pixel tracking of all pixel points in the target video on the application server, the amount of calculation of the target user terminal can be effectively reduced, so that the target user terminal has the locus information shown in FIG. When 1 is acquired, it is possible to ensure that the barrage data 1 shown in FIG. 10 can be quickly tracked and displayed based on the position information in the trajectory information 1, and the flexibility of displaying the barrage data can be improved. can. It should be understood that the video playback interface may have multiple barrages associated with the target object, but the target user terminal may overlap the multiple barrages present in the video playback interface. If detected, these barrages with overlap may be merged to retain the barrage most recently acquired by the target user terminal among these barrages with overlap.

In the embodiment of the present application, when the target user's trigger operation for the target video is acquired, the video frame corresponding to the trigger operation in the target video can be called a key video frame. Thereby, the target pixel point is determined from the key video frame, and the multimedia information associated with the target pixel point and the target object in which the target pixel point is located (for example, the multimedia information is the user in the target video). It may be multimedia data such as characters, pictures, and facial expressions). Further, the locus acquisition request corresponding to the target pixel point is determined based on the position information of the target pixel point in the key video frame. Further, based on the locus acquisition request, the target locus information associated with the position information of the target pixel point in the key video frame can be acquired. Thereby, when playing the next video frame of the key video frame, the barrage data associated with the target pixel point and the target object in which the target pixel point is located is displayed based on the target locus information. Can be done. As can be seen from this, in the embodiment of the present application, when the key video frame is determined, the locus information of the target pixel point is further selected and selected from the locus information of all the pixel points in the key video frame. The locus information of the target pixel point can be called the target locus information. As a result, the barrage data display effect can be enriched based on the acquired target trajectory information. For example, the acquired target trajectory information may be different for the target pixel points in different target objects, and as a result, the barrage data display effect is different. Further, based on the relationship between the target object and the barrage data, the position information of the barrage data in each video frame after the key video frame can be quickly determined. In other words, the barrage data can fluctuate according to the target object throughout the target video, and can further enhance the visual display effect of the user characters in the video, with the barrage data and the target object, or comment. It can be more relevant to the object in the target video.

Further, refer to FIG. 11, which is a schematic diagram of the configuration of the video data processing apparatus provided in the embodiment of the present application. As shown in FIG. 11, the video data processing device 1 can be applied to the target user terminal in the embodiment corresponding to the above FIG. The video data processing device 1 may include an object determination module 1101, a request determination module 1102, a locus acquisition module 1103, and a text display module 1104.

The object determination module 1101 determines the target pixel point from the key video frame of the target video in response to the trigger operation for the target video, acquires the multimedia information associated with the target pixel point, and obtains the multimedia information associated with the target pixel point. Is a video frame in which the trigger operation is located, and the target pixel point is a pixel point corresponding to the trigger operation in the key video frame.

The request determination module 1102 determines a locus acquisition request corresponding to the target pixel point based on the position information of the target pixel point in the key video frame.

The locus acquisition module 1103 acquires the target locus information associated with the position information of the target pixel point in the key video frame based on the locus acquisition request, and the target locus information is the target pixel point of the target pixel point. The position information of the target pixel point in the video frame next to the key video frame, including the position information in the video frame next to the key video frame, is obtained by tracking the target pixel point.

When the text display module 1104 reproduces the video frame next to the key video frame, the text display module 1104 is based on the position information of the target pixel point in the video frame next to the key video frame in the target locus information. Display multimedia information.

Here, for the specific execution method of the object determination module 1101, the request determination module 1102, the locus acquisition module 1103, and the text display module 1104, refer to the description of steps S101 to S104 in the embodiment corresponding to FIG. However, further description will be omitted here.

In the embodiment of the present application, when the target user's trigger operation for the target video is acquired, the video frame corresponding to the trigger operation in the target video can be used as a key video frame. Thereby, the target pixel point is determined from the key video frame, and the multimedia information associated with the target pixel point and the target object in which the target pixel point is located (for example, the multimedia information is the user in the target video). It may be multimedia data such as characters, pictures, and facial expressions). Further, the locus acquisition request corresponding to the target pixel point is determined based on the position information of the target pixel point in the key video frame. Further, based on the locus acquisition request, the target locus information associated with the position information of the target pixel point in the key video frame can be acquired. Thereby, when playing the next video frame of the key video frame, the barrage data associated with the target pixel point and the target object in which the target pixel point is located is displayed based on the target locus information. Can be done. As can be seen from this, in the embodiment of the present application, when the key video frame is determined, the locus information of the target pixel point is further selected and selected from the locus information of all the pixel points in the key video frame. The locus information of the target pixel point can be used as the target locus information. As a result, the barrage data display effect can be enriched based on the acquired target trajectory information. For example, the acquired target trajectory information may be different for the target pixel points in different target objects, and as a result, the barrage data display effect is different. Further, based on the relationship between the target object and the barrage data, the position information of the barrage data in each video frame after the key video frame can be quickly determined. In other words, the barrage data can fluctuate according to the target object throughout the target video, and can further enhance the visual display effect of the user characters in the video, with the barrage data and the target object, or comment. It can be more relevant to the object in the target video.

Further, see FIG. 12, which is a schematic diagram of the configuration of the computer equipment provided in the embodiments of the present application. As shown in FIG. 12, the computer device 1000 may be the target user terminal in the embodiment corresponding to FIG. 1 above. The computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Further, the computer device 1000 may include a user interface 1003 and at least one communication bus 1002. Here, the communication bus 1002 is for realizing connection communication between these components. Here, the user interface 1003 may include a display (Display) and a keyboard (Keyboard). The optional user interface 1003 may include standard wired and wireless interfaces. The network interface 1004 may optionally include standard wired and wireless interfaces (eg, Wi-Fi interfaces). The memory 1004 may be a high-speed RAM or a non-volatile memory, for example, at least one magnetic disk memory. The memory 1005 may optionally be at least one storage device located away from the processor 1001 described above. As shown in FIG. 12, the memory 1005, which is a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

In the computer device 1000 shown in FIG. 12, the network interface 1004 can provide a network communication function, the user interface 1003 mainly provides an interface for input to the user, and the processor 1001 is stored in the memory 1005. By calling the device control application
A step of determining a target pixel point from a key video frame of the target video in response to a trigger operation on the target video and acquiring multimedia information associated with the target pixel point, wherein the key video frame is a step. A step and a step, which is a video frame in which the trigger operation is located, and the target pixel point is a pixel point corresponding to the trigger operation in the key video frame.
A step of determining a locus acquisition request corresponding to the target pixel point based on the position information of the target pixel point in the key video frame, and
The step of acquiring the target locus information associated with the position information of the target pixel point in the key video frame based on the locus acquisition request, wherein the target locus information is the key video of the target pixel point. The position information of the target pixel point in the next video frame of the key video frame, including the position information in the video frame next to the frame, is obtained by tracking the target pixel point. ,
When playing back the video frame next to the key video frame, the multimedia information is displayed based on the position information of the target pixel point in the video frame next to the key video frame in the target locus information. Steps and
Can be used to realize.

As should be understood, the computer device 1000 described in the embodiment of the present application may be executed according to the description of the video data processing method in the embodiment corresponding to FIG. 4, and in the embodiment corresponding to FIG. 11 It may be executed according to the above description of the video data processing device 1, but further description thereof will be omitted here. Further, further description of the beneficial effect when the same method is used will be omitted.

Further, it should be pointed out here that, in the embodiment of the present application, a computer storage medium is further provided, and the computer storage medium stores a computer program executed by the video data processing device 1 mentioned above. The computer program includes a program instruction, and the processor can execute the program instruction according to the description of the video data processing method in the embodiment corresponding to FIG. 4, and further the description thereof will be described here. Omit. Further, further description of the beneficial effect when the same method is used will be omitted. For technical details not shown in the examples of computer storage media according to the present application, refer to the description of the examples of the method of the present application.

Further, see FIG. 13, which is a schematic diagram of the configuration of another video data processing apparatus provided in the embodiments of the present application. As shown in FIG. 13, the video data processing device 2 can be applied to the application server in the embodiment corresponding to FIG. 8, and the application server is the service server 2000 in the embodiment corresponding to FIG. May be good. The video data processing device 2 may include a request response module 1301 and a locus selection module 1302.

The request response module 1301 acquires the locus information associated with the target video in response to the locus acquisition request for the target pixel point in the key video frame, and the key video frame is a video frame in the target video. The target pixel point is a pixel point in the key video frame, and the locus information is determined by the position information of the pixel point in each video frame of the target video.

The locus selection module 1302 selects the target locus information associated with the position information of the target pixel point in the key video frame from the locus information associated with the target video, and returns the target locus information. The target locus information includes target position information, and the target position information is for triggering display of multimedia information associated with the target pixel point in the next video frame of the key video frame. Is.

Here, for the specific implementation method of the request response module 1301 and the locus selection module 1302, the description of steps S207 and S208 in the embodiment corresponding to FIG. 8 may be referred to, but further description is given here. Is omitted.

Further, see FIG. 14, which is a schematic diagram of the configuration of another computer device provided in the embodiments of the present application. As shown in FIG. 14, the computer device 2000 may be the target service server 2000 in the embodiment corresponding to FIG. 1 above. The computer device 2000 may include a processor 2001, a network interface 2004, and a memory 2005. Further, the computer equipment 2000 may include a user interface 2003 and at least one communication bus 2002. Here, the communication bus 2002 is for realizing connection communication between these components. Here, the user interface 2003 may include a display and a keyboard, and the optional user interface 2003 may include a standard wired interface and a wireless interface. The network interface 2004 may optionally include standard wired and wireless interfaces (eg, Wi-Fi interfaces). The memory 2004 may be a high-speed RAM or a non-volatile memory, for example, at least one magnetic disk memory. The memory 2005 may optionally be at least one storage device located away from the aforementioned processor 2001. As shown in FIG. 14, memory 2005, which is a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

In the computer equipment 2000 shown in FIG. 14, the network interface 2004 can provide a network communication function, the user interface 2003 mainly provides an input interface to the user, and the processor 2001 is stored in the memory 2005. By calling the device control application
A step of acquiring trajectory information associated with a target video in response to a trajectory acquisition request for a target pixel point in a key video frame, wherein the key video frame is a video frame in the target video and the target pixel. The points are the pixel points in the key video frame, and the locus information is determined by the position information of the pixel points in each video frame of the target video.
It is a step of selecting the target locus information associated with the position information of the target pixel point in the key video frame from the locus information associated with the target video, and returning the target locus information. The locus information includes the target position information, and the target position information is for triggering to display the multimedia information associated with the target pixel point in the next video frame of the key video frame. , Steps and
Can be used to realize.

As a matter of understanding, the computer equipment 2000 described in the embodiment of the present application may be executed according to the description of the video data processing method in the embodiment corresponding to FIG. 8, and in the embodiment corresponding to FIG. It may be executed according to the above description of the video data processing device 2, but further description thereof will be omitted here. Further, further description of the beneficial effect when the same method is used will be omitted.

Further, it should be pointed out here that, in the embodiment of the present application, a computer storage medium is further provided, and the computer storage medium stores a computer program executed by the video data processing device 2 mentioned above. The computer program includes a program instruction, and the processor can execute the program instruction according to the description of the video data processing method in the embodiment corresponding to FIG. 8, and further the description thereof will be described here. Omit. Further, further description of the beneficial effect when the same method is used will be omitted. For technical details not shown in the examples of computer storage media according to the present application, refer to the description of the examples of the method of the present application.

Further, see FIG. 15, which is a schematic diagram of the configuration of another video data processing apparatus provided in the embodiments of the present application. As shown in FIG. 15, the video data processing device 3 may be applied to the service server 2000 in the embodiment corresponding to FIG. 1, or may be applied to the target user terminal in the embodiment corresponding to FIG. You may. The video data processing device 3 may include a first acquisition module 310, a matrix acquisition module 410, a position tracking module 510, and a locus generation module 610.

The first acquisition module 310 acquires the adjacent first video frame and second video frame from the target video.

The matrix acquisition module 410 is based on the optical flow tracking rule corresponding to the target video, the pixel points in the first video frame, and the pixel points in the second video frame, and the average corresponding to the first video frame. Determine the displacement matrix.

Here, the matrix acquisition module 410 may include a first determination unit 4001, a matrix determination unit 4002, a pixel point selection unit 4003, a matrix correction unit 4004, and a second determination unit 4005.

The first determination unit 4001 acquires the optical flow tracking rule corresponding to the target video, determines the position information of the pixel points in the first video frame as the first position information, and determines the position information of the pixel points in the second video frame. The position information is determined as the second position information.

The matrix determination unit 4002 is based on the optical flow tracking rule, the first position information of the pixel points in the first video frame, and the second position information of the pixel points in the second video frame. The forward displacement matrix corresponding to the frame is acquired, and the reverse displacement matrix corresponding to the second video frame is acquired.

Here, the matrix determination unit 4002 includes a first tracking subunit 4021 and a second tracking subunit 4022.

The first tracking subsystem 4021 forwards the pixel points in the first video frame to the second video frame based on the first position information of the pixel points in the first video frame and the optical flow tracking rule. In the second video frame, the second position information of the mapped first mapping point is determined, and based on the first position information of the pixel point and the second position information of the first mapping point. The forward displacement matrix corresponding to the first video frame is determined.

The second tracking subsystem 4022 uses the first mapping point in the second video frame as the first video frame based on the second position information of the pixel point in the second video frame and the optical flow tracking rule. Mapping in the reverse direction, in the first video frame, the third position information of the mapped second mapping point is determined, the second position information of the first mapping point and the third position of the second mapping point. Based on the information, the reverse displacement matrix corresponding to the second video frame is determined.

Here, for the specific implementation method of the first tracking subunit 4021 and the second tracking subunit 4022, the description of the cloud forward / backward optical flow method in the embodiment corresponding to FIG. 8 may be referred to. Then, further explanation will be omitted.

The pixel point selection unit 4003 is a pixel that satisfies the target selection condition among the pixel points based on the first position information of the pixel points in the first video frame, the forward displacement matrix, and the reverse displacement matrix. The point is determined as an effective pixel point.

Here, the pixel point sorting unit 4003 includes a first position-fixing subunit 4031, a second position-fixing subunit 4032, a third position-fixing subunit 4033, an error determination subunit 4034, and an effective sorting subunit 4035. And, including.

The first position determination subunit 4031 acquires the first pixel point from the pixel points in the first video frame, determines the first position information of the first pixel point in the first video frame, and the said. From the forward displacement matrix, the first lateral displacement and the first longitudinal displacement corresponding to the first pixel point are determined.

The second position-determining subunit 4032 is based on the first position information of the first pixel point and the first lateral displacement and the first vertical displacement corresponding to the first pixel point, and the first pixel point. Is forward-mapped to the second video frame, and in the second video frame, the second position information of the mapped second pixel point is determined.

The third position determination subunit 4033 determines the second lateral displacement and the second longitudinal displacement corresponding to the second pixel point from the reverse displacement matrix, and together with the second position information of the second pixel point. , The second pixel point is mapped in the reverse direction to the first video frame based on the second lateral displacement and the second longitudinal displacement corresponding to the second pixel point, and in the first video frame. The third position information of the mapped third pixel point is determined.

The error determination subsystem 4034 determines the error distance between the first pixel point and the third pixel point based on the first position information of the first pixel point and the third position information of the third pixel point. An image block including the first pixel point and an image block including the second pixel point are determined and based on the first position information of the first pixel point and the second position information of the second pixel point. Determine the correlation coefficient of.

The effective selection subunit 4035 determines, among the pixel points, the pixel points whose error distance is less than the error distance threshold value and whose correlation coefficient is equal to or larger than the correlation coefficient threshold value are defined as effective pixel points.

Here, the specific implementation method of the first position-fixing subunit 4031, the second position-fixing subunit 4032, the third position-fixing subunit 4033, the error determination subunit 4034, and the effective selection subunit 4035 is shown in the above figure. Although the description of the specific process for determining the effective pixel point in the embodiment corresponding to No. 8 may be referred to, further description thereof will be omitted here.

The matrix correction unit 4004 corrects the initial state matrix corresponding to the first video frame and the forward displacement matrix based on the effective pixel points, thereby modifying the target state matrix and the target corresponding to the first video frame. Get the displacement matrix.

Here, the matrix correction unit 4004 includes an initial acquisition subunit 4041, a numerical switching subunit 4042, and a displacement setting subunit 4043.

The initial acquisition subunit 4041 acquires an initial state matrix corresponding to the first video frame, the state values of each matrix element of the initial state matrix are all first numerical values, and one matrix element is the pixel. Corresponds to one of the points.

The numerical switching subsystem 4042 switches the state value of the matrix element corresponding to the effective pixel point from the first numerical value to the second numerical value in the initial state matrix, and the initial state matrix including the second numerical value is the first video. Determined as the target state matrix corresponding to the frame.

In the forward displacement matrix, the displacement setting subsystem 4043 sets the displacement of the matrix element corresponding to the remaining pixel points to the first numerical value, and sets the forward displacement matrix including the first numerical value as the target displacement matrix. The remaining pixel points are determined and are pixel points other than the effective pixel points among the pixel points.

Here, the displacement setting subsystem 4043 specifically, when the forward displacement matrix includes the initial lateral displacement matrix and the initial longitudinal displacement matrix, the remaining pixel points in the initial lateral displacement matrix. The first lateral displacement of the matrix element corresponding to is set to the first numerical value, and the initial lateral displacement including the first numerical value is determined as the lateral displacement matrix corresponding to the first video frame.

Further, the displacement setting subsystem 4043 specifically sets the first vertical displacement of the matrix element corresponding to the remaining pixel points to the first numerical value in the initial vertical displacement matrix, and sets the first numerical value. The initial vertical displacement including the numerical value is determined as the vertical displacement matrix corresponding to the first video frame.

Further, the displacement setting subunit 4043 specifically determines the lateral displacement matrix corresponding to the first video frame and the longitudinal displacement matrix corresponding to the first video frame as the target displacement matrix.

Regarding the specific implementation method of the initial acquisition subunit 4041, the numerical switching subunit 4042, and the displacement setting subunit 4043, the description of modifying the initial state matrix and the forward displacement matrix in the embodiment corresponding to FIG. 8 above. However, further description thereof will be omitted here.

The second determination unit 4005 determines the average displacement matrix corresponding to the first video frame based on the target state matrix and the target displacement matrix.

Here, the second determination unit 4005 includes a first integral subunit 4051, a second integral subunit 4052, a third integral subunit 4053, and a difference calculation subunit 4054.

The first integration subunit 4051 acquires a state integration matrix corresponding to a pixel point in the first video frame by performing a displacement integration operation on the target state matrix in the first video frame.

In the first video frame, the second integration subsystem 4052 performs a displacement integration calculation on the lateral displacement matrix in the target state matrix, thereby performing a lateral displacement integral corresponding to the pixel points in the first video frame. Get the matrix.

The third integration subsystem 4053 performs a displacement integration calculation on the vertical displacement matrix in the target state matrix in the first video frame, thereby performing a vertical displacement integration corresponding to the pixel points in the first video frame. Get the matrix.

The difference calculation subsystem 4054 determines a difference region corresponding to the displacement difference calculation from the first video frame, and determines the size information of the difference region, the state integral matrix, the lateral displacement integral matrix, and the vertical displacement integral. Based on the matrix, the average displacement matrix corresponding to the first video frame is determined.

Here, the difference calculation subunit 4054 includes a first difference subunit 4055, a second difference subunit 4056, a third difference subunit 4057, and an average determination subunit 4058.

The first difference subsystem 4055 performs a displacement difference operation on the state integral matrix based on the length information and the width information corresponding to the difference region, so that the state difference matrix corresponding to the first image frame is performed. To get.

The second difference subsystem 4056 performs a displacement difference calculation on each of the lateral displacement integral matrix and the longitudinal displacement integral matrix based on the length information and the width information corresponding to the difference region. The horizontal displacement difference matrix and the vertical displacement difference matrix corresponding to the first image frame are acquired.

The third difference subsystem 4057 determines the ratio between the lateral displacement difference matrix and the state difference matrix as the horizontal average displacement matrix, and the ratio between the vertical displacement difference matrix and the state difference matrix is the vertical average. Determined as a displacement matrix.

The average determination subunit 4058 determines the longitudinal displacement difference matrix and the longitudinal average displacement matrix as the average displacement matrix corresponding to the first video frame.

Here, regarding the specific implementation method of the first integral subunit 4051, the second integral subunit 4052, the third integral subunit 4053, and the difference calculation subunit 4054, the cloud displacement integral in the embodiment corresponding to FIG. 8 above. The description of the method and the cloud displacement difference method may be referred to, but further description thereof will be omitted here.

Here, the specific implementation method of the first determination unit 4001, the matrix determination unit 4002, the pixel point selection unit 4003, the matrix correction unit 4004, and the second determination unit 4005 is described in step S202 in the embodiment corresponding to FIG. However, further description will be omitted here.

The position tracking module 510 tracks the position information of the pixel points in the first video frame based on the average displacement matrix, and determines the position information of the tracked pixel points in the second video frame.

The locus generation module 610 generates locus information associated with the target video based on the position information of the pixel points in the first video frame and the position information of the tracked pixel points in the second video frame. However, the locus information includes target locus information for tracking and displaying multimedia information associated with a target pixel point in a target video.

Here, for specific implementation methods of the first acquisition module 310, the matrix acquisition module 410, the position tracking module 510, and the locus generation module 610, refer to the description of steps S201 to S204 in the embodiment corresponding to FIG. However, further description will be omitted here.

Further, see FIG. 16, which is a schematic diagram of the configuration of another computer device provided in the embodiments of the present application. As shown in FIG. 16, the computer device 3000 can be applied to the service server 2000 in the embodiment corresponding to FIG. 1. The computer device 3000 may include a processor 3001, a network interface 3004, and a memory 3005. Further, the video data processing device 3000 may include a user interface 3003 and at least one communication bus 3002. Here, the communication bus 3002 is for realizing connection communication between these components. Here, the user interface 3003 may include a display and a keyboard, and the optional user interface 3003 may include a standard wired interface and a wireless interface. The network interface 3004 may optionally include standard wired and wireless interfaces (eg, Wi-Fi interfaces). The memory 3004 may be a high-speed RAM or a non-volatile memory, for example, at least one magnetic disk memory. The memory 3005 may optionally be at least one storage device located away from the processor 3001 described above. As shown in FIG. 16, the memory 3005, which is a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

In the computer equipment 3000 shown in FIG. 16, the network interface 3004 can provide a network communication function, the user interface 3003 mainly provides an input interface to the user, and the processor 3001 is stored in the memory 3005. By calling the device control application
The step of acquiring the adjacent first video frame and second video frame from the target video, and
A step of determining an average displacement matrix corresponding to the first video frame based on the optical flow tracking rule corresponding to the target video, the pixel points in the first video frame, and the pixel points in the second video frame. When,
A step of tracking the position information of the pixel points in the first video frame based on the average displacement matrix and determining the position information of the traced pixel points in the second video frame.
A step of generating trajectory information associated with the target video based on the position information of the pixel points in the first video frame and the position information of the tracked pixel points in the second video frame. The locus information includes a step and a target locus information for tracking and displaying multimedia information associated with a target pixel point in a target video.
Can be used to realize.

As should be understood, the computer device 3000 described in the embodiment of the present application may be executed according to the description of the video data processing method in the embodiment corresponding to FIG. 8, and in the embodiment corresponding to FIG. It may be executed according to the above description of the video data processing device 3, but further description thereof will be omitted here. Further, further description of the beneficial effect when the same method is used will be omitted.

Further, it should be pointed out here that, in the embodiment of the present application, a computer storage medium is further provided, and the computer storage medium stores a computer program executed by the video data processing device 3 mentioned above. The computer program includes a program instruction, and the processor can execute the program instruction according to the description of the video data processing method in the embodiment corresponding to FIG. 8, and further the description thereof will be described here. Omit. Further, further description of the beneficial effect when the same method is used will be omitted. For technical details not shown in the examples of computer storage media according to the present application, refer to the description of the examples of the method of the present application.

As will be appreciated by those skilled in the art, all or part of the procedure of the method in the above embodiment may be performed by instructing the relevant hardware via a computer program. The above program may be stored in a computer-readable storage medium. When this program is executed, the procedure of the embodiment of each method as described above is executed. Here, the storage medium described above may be a magnetic disk, an optical disk, a read-only memory (ROM: Read-Only Memory), a random access memory (RAM: Random Access Memory), or the like.

Since the above notices are merely preferred embodiments of the present application and, of course, are not intended to limit the scope of rights of the present application, equivalent changes based on the claims of the present application are still within the scope of protection of the present application. Belongs.

2000 Service server 3000a, 3000b, 3000c, 3000n User terminal 1,2,3 Video data processing device 310 First acquisition module 410 Matrix acquisition module 510 Position tracking module 610 Trajectory generation module 1000, 2000, 3000 Computer equipment 1001, 2001, 3001 Processor 1002,2002,3002 Communication bus 1003,2003,3003 User interface 1004,2004,3004 Network interface 1005,2005,3005 Memory 1101 Object determination module 1102 Request determination module 1103 Trajectory acquisition module 1104 Text display module 1301 Request response module 1302 Trajectory Sorting module 4001 1st decision unit 4002 Matrix decision unit 4003 Pixel point sort unit 4004 Matrix correction unit 4005 2nd decision unit 4021 1st tracking sub unit 4022 2nd tracking sub unit 4031 1st positioning sub unit 4032 2nd positioning sub Unit 4033 3rd Position Determination Subunit 4034 Error Determination Subunit 4035 Effective Sort Subunit 4041 Initial Acquisition Subunit 4042 Numerical Switching Subunit 4043 Displacement Setting Subunit 4051 1st Integral Subunit 4052 2nd Integral Subunit 4053 3rd Integral Subunit Unit 4054 Difference calculation sub-unit 4055 First difference sub-unit 4056 Second difference sub-unit 4057 Third difference sub-unit 4058 Average determination sub-unit

It is a schematic diagram of the configuration of the network architecture provided in the embodiment of this application. FIG. 3 is a schematic diagram of a plurality of video frames in the target video provided in the embodiments of the present application. FIG. 3 is a schematic diagram of a scenario for acquiring a target video provided in an embodiment of the present application. It is a schematic diagram of the flow of the video data processing method provided in the Example of this application. It is a schematic diagram of acquisition of multimedia information provided in the Example of this application. It is a schematic diagram of all pixel tracking provided in the Example of this application. It is a schematic diagram of the tracking of barrage data in a plurality of consecutive video frames provided in the embodiment of the present application. It is a schematic diagram of another video data processing method provided in the Example of this application. It is a schematic diagram of the method of determining the effective pixel point provided in the Example of this application. It is a schematic diagram of the barrage data display based on the locus information provided in the embodiment of the present application. It is a schematic diagram of the structure of the video data processing apparatus provided in the Example of this application. It is a schematic diagram of the structure of the computer equipment provided in the Example of this application. It is a schematic diagram of the configuration of another video data processing apparatus provided in the embodiment of this application. It is a schematic diagram of the configuration of another computer device provided in the embodiment of this application. It is a schematic diagram of the configuration of another video data processing apparatus provided in the embodiment of this application. It is a schematic diagram of the configuration of another computer device provided in the embodiment of this application.

Here, the service server 2000 may perform framing processing on each video of the video set stored in the service database in the video preprocessing stage. As a result, a plurality of video frames included in each video can be divided into one picture. Further, for ease of understanding, see FIG. 2, which is a schematic diagram of a plurality of video frames in the target video provided in the embodiments of the present application. Here, the target video may be video A in the service database described above, and as shown in FIG. 2, the video A is n video frames (n is a positive integer greater than 0). May include. The service server 2000 may predivide the n video frames in the video A into n pictures. Of these n pictures, two pictures adjacent to each other in the front and back can be called one image pair. For example, as shown in FIG. 2, in the embodiment of the present application, the video frame corresponding to the first time shown in FIG. 2 and the video frame corresponding to the second time can be referred to as one image pair. The video frame corresponding to the second time and the video frame corresponding to the third time can be called one image pair, and the video frame corresponding to the n-1th time and the video frame corresponding to the nth time can be called one. Can be called one image pair. In other words, for one target video, multiple image pairs can be determined from multiple video frames of the target video, and each image pair has a video corresponding to two adjacent times before and after. Frames may be included, i.e., each image pair may contain two adjacent video frames.

Further, see FIG. 3, which is a schematic diagram of a scenario for acquiring a target video provided in an embodiment of the present application. The target user terminal shown in FIG. 3 may be the user terminal 3000a in the embodiment corresponding to FIG. 1 above. As shown in FIG. 3, after entering the target application, the target user may display the service data display platform of the target application on the target user terminal (for example, a smartphone). The service data display platform may display video 10a, video 20a, video 30a, and video 40a shown in FIG. When the target user needs to play the video 30a shown in FIG. 3 (the video 30a may be the video A in the embodiment corresponding to the above FIG. 2 ) on the target user terminal, the video 30a A playback operation may be performed on the located area (for example, the target user may perform a click operation on the video 30a), and the target identification information corresponding to the video 30a may be obtained. In addition to the data load instruction shown in FIG. 3, the data load instruction may be further given to the application server having a network connection relationship with the target user terminal. The application server may be the service server 2000 in the embodiment corresponding to FIG. 1 above. As can be understood, when the application server obtains the data load instruction, the video data corresponding to the target identification information can be searched from the service database, and the detected video data may be collectively referred to as the target data. can. Thereby, the target data can be given to the target user terminal shown in FIG. 3, and the target user terminal can reproduce the video data by the video reproduction interface shown in FIG. At this time, the target user terminal can refer to the video 30a selected and played by the target user as the target video. That is, at this time, the target user terminal can reproduce each video frame in the video 30a according to the reproduction time stamp shown in FIG.

It should be understood that the movement locus of each pixel point in the target video (that is, the locus information of each pixel point) is determined by the position information of each pixel point in each video frame of the target video. Here, in the case of a target video including a plurality of video frames, in the embodiment of the present application, any two adjacent video frames among the plurality of video frames may be determined as one image pair. It should be understood that, of the two video frames contained in each image pair determined from the plurality of video frames, one video frame can be referred to as the first video frame, and the other video frame can be referred to as the first video frame. It can be called a second video frame. Regarding the image pair 1 including the video frame corresponding to the first time and the video frame corresponding to the second time in the embodiment corresponding to FIG. 2, the video frame corresponding to the first time in the image pair 1 Can be referred to as a first video frame, and a video frame corresponding to the second time can be referred to as a second video frame. Further, based on the pre-calculated average displacement matrix between the two video frames in the image pair 1, all the pixel points in the first video frame are tracked and all the pixels in the first video frame are traced. The appearance position information of the point in the second video frame can be determined. Similarly, for the image pair 2 including the video frame corresponding to the second time and the video frame corresponding to the third time in the embodiment corresponding to FIG. 2, the video frame corresponding to the second time is the second. It can be called one video frame, and the video frame corresponding to the third time can be called the second video frame. Thereby, based on the pre-calculated average displacement matrix between the two video frames in the image pair 2, all the pixel points in the first video frame are tracked, and all the pixel points in the first video frame are traced. The appearance position information of the pixel points in the second video frame can be determined. In this way, in the embodiment of the present application, the average displacement matrix corresponding to each image pair can be acquired, and the average displacement matrix corresponding to each image pair is set in the first video frame in each image pair. It can be called the corresponding average displacement matrix, and the average displacement matrix corresponding to each first video frame maps all the pixel points in the first video frame to the second video frame, and in the second video frame, It can be used to accurately acquire the position information of these mapped pixel points. It should be understood that the average displacement matrix in the embodiments of the present application may include a longitudinal average displacement matrix and a lateral average displacement matrix. The vertical average displacement matrix performs vertical coordinate conversion on the first vertical coordinate value (for example, y value) of each pixel point in the first video frame, and maps it to the second video frame of each pixel point. The second vertical coordinate obtained can be obtained. Similarly, the lateral average displacement matrix is used to perform lateral coordinate conversion on the first lateral coordinate value (for example, x value) of each pixel point in the first video frame, and the second video of each pixel point. The second lateral coordinates mapped to the frame can be obtained. As a matter of understanding, in the embodiment of the present application, the first horizontal coordinate value and the first vertical coordinate value of each pixel point in the first video frame are referred to as the first position information of each pixel point in the first video frame. The second horizontal coordinate value and the second vertical coordinate value of each pixel point mapped to the second video frame can be referred to as the second position information of each mapped pixel point in the second video frame. Can be called. Since each image pair corresponds to one average displacement matrix, the corresponding second position information can be calculated based on the first position information of the pixel points in the first video frame, and the calculated second position information can be calculated. The second position information of each mapped pixel point in the two video frames can be stored, and the position information in each video frame of the same pixel point is integrated to move the locus of all the pixel points in the video frame. Can be obtained. Thereby, it is possible to realize the tracking of all the pixel points in all the video frames of the target video.

It should be understood that the plurality of video frames in the target video shown in the embodiment corresponding to FIG. 2 above may be a plurality of consecutive image frames. Therefore, after the target video shown in FIG. 2 above is divided, a video frame number corresponding to the playback order can be set for each divided image frame (that is, a video frame). For example, the video frame number of the video frame acquired at the first time may be 1, and the video frame number 1 is such that the video frame acquired at the first time is the first in the target video. It can be used to indicate that it is one frame. Similarly, the video frame number of the video frame acquired at the second time may be 2, and the video frame number 2 is such that the video frame acquired at the second time is the target video. It can be used to indicate that it is the second frame. In this way, the video frame number of the video frame acquired at the above-mentioned n-1 time may be n-1, and the video frame number n-1 is acquired at the n-1 time. The resulting video frame can be used to indicate that it is the n-1th frame in the target video, and the video frame number of the video frame acquired at the above nth time may be n. The video frame number n can be used to indicate that the video frame acquired at the nth time is the nth frame in the target video, that is, the last frame in the target video.

As shown in FIG. 6, after each pixel point in the first video frame is acquired, all the acquired pixel points can be collectively referred to as a pixel point, and further, an average displacement matrix shown in FIG. Therefore, all the pixel points in the pixel point display area 600a can be tracked, and further, the position information of the mapped pixel points can be determined in the pixel point display area 700a corresponding to the second video frame. .. For example, taking the pixel point A shown in FIG. 6 as an example, the position information of the pixel point A in the pixel point display area 600a shown in FIG. 6 may be the coordinate position information of the area identifier 5, and the average thereof. The pixel point A can be mapped to the pixel point display area 700a shown in FIG. 6 by the displacement matrix, and the position information of the pixel point A in the pixel point display area 700a shown in FIG. 6 is the area identifier 10. It may be coordinate position information. In the embodiment of the present application, after calculating the position information of the pixel point A in the second video frame, the position information can be stored. Since each image pair in the target video can correspond to one average displacement matrix, the position information of the pixel point in each first video frame, which is mapped to the second video frame, is calculated. Can be done. By integrating the appearance position information of the same pixel point in each image pair in consecutive video frames, it is possible to acquire the appearance position information of the pixel point A in each video frame of the target video, and further, the said The movement locus of the pixel point A can be acquired based on the appearance position information of the pixel point A in each video frame of the target video.

The target user terminal may transmit the barrage data (multimedia information) input by the user and the calculated position information of the barrage data in each video frame of the target video to the server. Alternatively, the server receives the frame number of the key video frame in the target video clicked by the user, the target pixel point coordinates, and the input bullet data (multimedia information) transmitted from the target user terminal, and receives the input bullet data (multimedia information) of the target pixel point. , The target locus information in each video frame of the target video is calculated, the position information of the bullet data in each video frame of the target video is calculated based on the target locus information, and the position information of the bullet data is saved. May be good. When the server receives the information transmitted from the target user terminal, it also receives information such as the identifier of the target user terminal and / or the user identifier used when the user logs in to the target application on the target user terminal. You may. Then, when playing the target video on another user terminal, the server transmits the bullet data, the position information of the bullet data in each video frame of the target video, and the user identifier to the other user terminal. However, another user terminal may display the bullet data in each video frame of the target video based on the position information of the bullet data.

Therefore, for different objects in the key video frame, it is possible to acquire a movement locus corresponding to each of the target pixel points in the different objects. This allows the barrage data associated with different target objects to move in different trajectories, making the barrage data more relevant to the target object of the barrage data, and further visualizing the barrage data. It is possible to enrich the display effect and increase the flexibility of the barrage data display mode.

Here, in the cloud forward / backward optical flow method, the calculation by the forward / reverse optical flow method is performed synchronously for the first video frame and the second video frame in each image pair, and each image pair is subjected to the calculation. It can be used to obtain the corresponding optical flow tracking results. In other words, the optical flow tracking results obtained by the application server may include a forward displacement matrix corresponding to the first video frame in each image pair, or the inverse corresponding to the second video frame in each image pair. It may include a directional displacement matrix. In the embodiments of the present application, each matrix element of the forward displacement matrix and the reverse displacement matrix may contain displacements of two dimensions (eg, (Δx, Δy)). Here, these two dimensional displacements can be understood as horizontal displacements (ie, Δx) and vertical displacements (ie, Δy) of the same pixel point. It should be understood that for any of the image pairs in the target video, when calculated by the optical flow method, the forward horizontal displacement matrix, the forward vertical displacement matrix, and the inverse horizontal variation are performed. The position matrix and the inverse vertical displacement matrix can be acquired, and the acquired four matrices can be called the optical flow result. Further, the application server may set an initial state matrix for the first video frame in each image pair, and further, based on the previously acquired forward displacement matrix and reverse displacement matrix, each image. It may be determined whether or not the pixel points in the first video frame of each pair satisfy the target selection condition. When the first video frame has pixel points satisfying the target selection condition, the application server may determine the pixel points satisfying the target selection condition as the effective pixel points, and further, based on the determined effective pixel points. Then, the initial state matrix corresponding to the first video frame and the forward displacement matrix may be modified to obtain the target state matrix and the target displacement matrix corresponding to the first video frame in each image pair. Further, the application server determines the average displacement matrix corresponding to the first video frame in each image pair by the above-mentioned cloud displacement integration method and cloud displacement difference method, and the acquired target state matrix and target displacement matrix. You may.

Here, in the embodiment of the present application, the forward horizontal displacement matrix and the forward vertical displacement matrix can be collectively referred to as a forward horizontal displacement matrix, and the reverse horizontal displacement matrix and the reverse vertical displacement matrix can be collectively referred to. The position matrix can be collectively referred to as a reverse displacement matrix. For ease of understanding, in the embodiments of the present application, one of a plurality of image pairs is taken as an example, and the average corresponding to the image pair is taken from the first video frame and the second video frame in the image pair. The process of acquiring the displacement matrix will be described. Here, the first video frame in the image pair may be a video frame having the above-mentioned video frame number of 1, and the second video frame is a video frame having the above-mentioned video frame number of 2. May be good. Therefore, an image pair consisting of a video frame having a video frame number of 1 and a video frame having a video frame number of 2 is called an image pair 1, and the image pair 1 can be represented by (1, 2). ..

Here, the forward displacement matrix corresponding to the image pair 1 calculated by the optical flow method is a forward horizontal displacement matrix (for example, even if the forward horizontal displacement matrix is the matrices Q1 _{, 2, x} ). May be included) and a forward vertical displacement matrix (eg, the forward vertical displacement matrix may be matrices Q1 _{, 2, y} ). Here, it should be understood that each matrix element of the matrix Q1 _{, 2, x} can be understood as the horizontal displacement of the pixel point in the first video frame in the second video frame. That is, each matrix element of the forward horizontal displacement matrix can be referred to as a first lateral displacement corresponding to a pixel point in the first video frame. Similarly, each matrix element of the matrix Q1 _{, 2, y} can be understood as the vertical displacement of the pixel points in the first video frame in the second video frame. That is, each matrix element of the forward vertical displacement matrix can be referred to as a first longitudinal displacement corresponding to a pixel point in the first video frame. In other words, the matrix size of these two matrices calculated by the optical flow method (that is, the matrix Q _{1, 2, x} and the matrix Q _{1, 2, y} ) is the same as the size of the first video frame. That is, one matrix element can correspond to one pixel point in the first video frame.

Similarly, the reverse displacement matrix corresponding to the image pair 1 calculated by the optical flow method may be a reverse horizontal displacement matrix (that is, the reverse horizontal displacement matrix may be matrices Q _{2, 1, x} ). May be included) and a reverse vertical displacement matrix (ie, the reverse vertical displacement matrix may be matrices Q _{2, 1, y} ). Here, it should be understood that each matrix element of the matrix Q _{2, 1, x} can be understood as the horizontal displacement of the pixel points in the second video frame in the first video frame. That is, each matrix element of the reverse horizontal displacement matrix can be referred to as a second lateral displacement corresponding to a pixel point in the second video frame. Similarly, each matrix element of the matrix Q _{2, 1, y} can be understood as the vertical displacement of the pixel points in the second video frame in the first video frame. That is, each matrix element of the reverse vertical displacement matrix can be referred to as a second vertical displacement corresponding to a pixel point in the second video frame. In other words, the matrix size of these two matrices (ie, matrix Q _{2,1, x} and matrix Q _{2,1, y} ) calculated by the optical flow method is the same as the size of the second video frame. That is, one matrix element can correspond to one pixel point in the second video frame.

It should be understood that for each video frame in the target video, the number of pixel points in each video frame is the same, so these four matrices (ie,) corresponding to the image pair 1 calculated by the optical flow method . , Matrix Q _{1, 2, x} , Matrix Q _{1, 2, y} , Matrix Q _{2, 1, x} , Matrix Q _{2, 1, y} ) have the same matrix size. For example, when the number of pixel points in each video frame is m × n, the matrix size of the acquired four matrices may be m × n. As can be seen, each matrix element of the forward horizontal displacement matrix and the forward vertical displacement matrix can correspond to the pixel points in the first video frame. Therefore, each matrix element of the forward displacement matrix corresponding to the image pair 1 can represent a two-dimensional displacement of the pixel points in the first video frame in the second video frame. The forward displacement matrix corresponding to the image pair 1 can be collectively referred to as a forward displacement matrix corresponding to the first video frame. Similarly, each matrix element of the reverse displacement matrix corresponding to the image pair 1 can represent a two-dimensional displacement of the pixel points in the second video frame in the first video frame. The reverse displacement matrix corresponding to the image pair 1 can be collectively referred to as a reverse displacement matrix corresponding to the second video frame.

The application server acquires the first pixel point from the pixel points in the first video frame, determines the first position information of the first pixel point in the first video frame, and determines the first position information of the first pixel point, and the forward displacement matrix. Therefore, the first lateral displacement and the first longitudinal displacement corresponding to the first pixel point may be determined. Further, the application server determines the first pixel point as the first pixel point based on the first position information of the first pixel point and the first lateral displacement and the first vertical displacement corresponding to the first pixel point. It may be mapped in the forward direction to the two video frames, and the second position information of the mapped second pixel point may be determined in the second video frame. Further, the application server determines the second lateral displacement and the second longitudinal displacement corresponding to the second pixel point from the reverse displacement matrix, and obtains the second position information of the second pixel point and the second pixel point. Based on the second lateral displacement and the second longitudinal displacement corresponding to the two pixel points, the second pixel point was mapped in the opposite direction to the first video frame and mapped in the first video frame. The third position information of the third pixel point may be determined. Further, the application server determines the error distance between the first pixel point and the third pixel point based on the first position information of the first pixel point and the third position information of the third pixel point. Then, based on the first position information of the first pixel point and the second position information of the second pixel point, the image block including the first pixel point and the image block including the second pixel point The correlation coefficient may be determined. Further, the application server may determine, among the pixel points, the pixel points whose error distance is less than the error distance threshold and whose correlation coefficient is equal to or larger than the correlation coefficient threshold as effective pixel points.

Further, for ease of understanding, see FIG. 9, which is a schematic diagram of the method of determining effective pixel points provided in the embodiments of the present application. As shown in FIG. 9, the application server may _first initialize the state matrix S1 having the same size as the first video frame before selecting the matrix elements of the four matrices. In this case, the application server can call the state matrix S ₁ the initial state matrix. Here, in the initial state matrix, the value of the matrix element corresponding to each pixel point can be called a first numerical value. At this time, the first numerical value in the initial state matrix is zero. The change state of the value of the matrix element of the initial state matrix can be used to indicate whether or not the pixel point in the first video frame satisfies the target selection condition. As a result, the pixel points satisfying the target selection condition can be set as effective tracking pixel points (that is, effective pixel points).

Further, the application server has an error distance between two positions of the first position information of the first pixel point p1 and the third position information of the matrix-converted third pixel point p1'in the first video frame. t _11'can be determined. Further, the application server displays an image block 10 having a size of k * k pixels (for example, 8 * 8 pixels) centered on the first position information of the first pixel point p1 in the first video frame shown in FIG. You may choose. Further, as shown in FIG. 9, the application server similarly has an image block having a size of k * k pixels centered on the second position information of the second pixel point p2 in the second video frame shown in FIG. 20 may be selected, and a correlation coefficient between the two image blocks (the correlation coefficient may be N ₁ , 2) may be calculated.

Here, as can be understood, in the embodiment of the present application, after the error distance between the first pixel point p1 and the third pixel point p1'shown in FIG. 9 is calculated, the calculated error distance is set in advance. It may be compared with the error distance threshold . If t _11' < _TB and N 1, ₂ ≧ _TA (where _TB is the set error distance threshold and _TA is the set correlation coefficient threshold), the first video frame. Since the first pixel point p1 in 1 satisfies the above-mentioned target selection condition, it can be determined that the first pixel point p1 is an effective pixel point.

Further, the application server may set the value of the matrix element at the position corresponding to the _first pixel point p1 in the initial state matrix S1 to the second numerical value. For example, in order to indicate that the first pixel point p1 in the first video frame is an effective pixel point, the value of the element corresponding to the _first pixel point p1 in the initial state matrix S1 is switched from 0 to 1. May be good. On the contrary, when t _11' ≧ _TB and / or N 1, ₂ < _TA , it means that the first pixel point p1 in the first video frame does not satisfy the above-mentioned target selection condition. In this case, the application server can determine that the first pixel point p1 shown in FIG. 9 is an invalid tracking pixel point. That is, in the initial state matrix S1, the value of the element corresponding to the _first pixel point p1 remains 0. At the same time, the application server is further located at a position corresponding to the first pixel point p1 in the forward displacement matrix (that is, the matrix Q1 _{, 2, x} and the matrix Q _{1, 2, y} ). The value of a matrix element may be set to 0. Thereby, the forward displacement matrix including the first numerical value can be determined as the target displacement matrix (for example, the forward horizontal displacement matrix Q _x1 and the forward vertical displacement matrix Q _y1 ). That is, the matrix elements at these positions in the target displacement matrix are used to represent the matrix determined after the error tracking displacements with large errors are excluded from the forward displacement matrix described above. Can be done.

As can be understood, for the other pixel points shown in FIG. 9, the above steps of sequentially selecting the pixel points from the first video frame shown in FIG. 9 and using them as the first pixel points to determine the effective pixel points are repeated. You may. If any of all the pixel points in the first video frame is set as the first pixel point, all the effective pixel points in the first video frame can be determined. Thereby, the matrix element in the initial state matrix can be updated based on the position information of the effective pixel point in the initial state matrix, and the initial state matrix including the second numerical value can be obtained by the first video frame. Can be determined as the target state matrix _S1 corresponding to. Then, the target displacement matrix corresponding to the first video frame (that is, the target horizontal displacement matrix Q _{x, 1} and the target vertical displacement matrix Q _{y, 1} ) can be acquired. Similarly, for other image pairs among the plurality of image pairs, the target corresponding to the first video frame in each of the remaining image pairs can be obtained by repeating the above step of determining the effective pixel points from the image pair 1. The state matrix and target displacement matrix can be obtained. For example, taking a plurality of consecutive video frames having video frame numbers 1, 2, 3, 4, ..., N in the target video as an example, the plurality of configured image pairs are each (1, 1,). It can be expressed as 2), (2,3), (3,4), ..., (N-1, n). Then, from the optical flow tracking result corresponding to each image pair, the target state matrix S1 corresponding to the image pair ( ₁ , 2) and the image pair (1, 2) are obtained by the above-mentioned effective pixel point determination method. The corresponding target displacement matrix Q ₁ (that is, the target horizontal displacement matrix Q _{x, 1} and the target vertical displacement matrix Q _{y, 1} described above) can be finally obtained. In this way, the target state matrix S _n-1 corresponding to the image pair ( n-1 , n ) and the target displacement matrix Q _n-1 corresponding to the image pair (n-1, n) (that is, described above). The target horizontal displacement matrix Q _{x, n-1} and the target horizontal position matrix Q _{y, n-1} ) can be acquired.

Here, the displacement difference matrix Q _dif (x, y) includes the lateral displacement difference matrix Q _{x, dif} (x, y) and the vertical displacement difference matrix Q _{y, dif} (x, y). But it may be. The state difference matrix S _dif (x, y), the lateral displacement difference matrix Q _{x, dif} (x, y), and the vertical displacement difference matrix Q _{y, dif} (x, y) are the following displacement differences. It can be obtained by the calculation formula (formula 5).

Further, the application server sets the ratio of the lateral displacement difference matrix Q _{x, def} (x, y) and the state difference matrix S _div (x, y) to the lateral average displacement matrix Q _{x, F} (x, y). ), And the ratio of the vertical displacement difference matrix Q _{y, dif} (x, y) to the state difference matrix S _dif (x, y) is the vertical average displacement matrix Q _{y, F} (x, y). May be determined as.

Specifically, the application server uses the average displacement matrix acquired in step S202 described above (the average displacement matrix is the horizontal average displacement matrix Q _{x, F} (x, y) and the vertical average displacement matrix Q. Based on _{y, F} (x, y) and may be included), further in the next video frame (ie, the second video frame in the image pair 1 above) of the pixel points in the first video frame. The appearance position information can be quickly and accurately tracked, that is, the position information of the pixel points obtained by tracking the pixel points in the first video frame in the second video frame by performing displacement conversion. Can be determined.

In Equation 8, x is the lateral position coordinate of the pixel point in the first video frame, and Q _{x, F} (x, y) is the lateral average displacement matrix corresponding to the first video frame. According to the equation 8, the lateral position coordinates of the pixel points in the first video frame can be coordinate-converted, and the lateral position coordinates of the pixel points in the first video frame can be obtained in the next video frame. .. Similarly, y in Equation 9 is the vertical position coordinate of the pixel point in the first video frame, and Q _{y , F} (x, y) is the vertical average displacement matrix corresponding to the first video frame. In addition, according to the equation 9, the vertical position coordinates of the pixel points in the first video frame are coordinate-converted, and the vertical position coordinates of the pixel points in the first video frame are obtained in the next video frame. Can be done.

Specifically, the application server may acquire the video frame number in the target video of the key video frame and the position information of the target pixel point in the key video frame from the trajectory acquisition request, and further, the said. The trajectory information associated with the target pixel point may be selected from the trajectory information associated with the target video acquired in advance by the application server, and the selected trajectory information can be referred to as target trajectory information. Further, the target locus information may be returned to the target user terminal. As a result, the target user terminal receives the target based on the frame number of the key video frame until the appearance position information of the target pixel point in each video frame after the key video frame is acquired. From the locus information, it is possible to quickly find the appearance position information of the target pixel point in the video frame next to the key video frame. In this case, the target user terminal can refer to new locus information consisting of appearance position information of the target pixel point in each video frame after the key video frame as target locus information . Optionally, it should be understood that when the application server acquires the frame number of the key video frame, the appearance position information of the target pixel point in each video frame after the key video frame is acquired. From the selected locus information, the appearance position information of the target pixel point in the video frame next to the key video frame may be quickly found. In this case, the application server can call the new locus information consisting of the appearance position information of the target pixel point in each video frame after the key video frame as the target locus information.

As can be understood, optionally, the target user terminal has a locus information including position information in a key video frame of the target pixel point searched by the application server (that is, a locus corresponding to the pixel point A). Information) can also be collectively referred to as target trajectory information. In this case, the target locus information can be regarded as locus information corresponding to the pixel point A matching the target pixel point, which is searched from all the pixel points of the target video. Since the locus information can include the position information of the pixel point A in each video frame of the target video, naturally, from the locus information, after the key video frame of the target pixel point. It is also possible to quickly acquire the position information in each video frame.

Here, as can be understood, in the embodiment of the present application, when the target pixel points in the target object selected by the target user are acquired, the locus information corresponding to each of all the pixel points calculated in advance is included. Therefore, the locus information associated with the position information of the target pixel point in the key video frame can be selected, and the selected locus information can be called the target locus information. In the embodiment of the present application, pixel tracking of pixel points in each video frame of the video can be performed in advance. Therefore, when the average displacement matrix corresponding to the first video frame in each image pair is acquired, each pixel in the video is obtained. The position information of the point in the corresponding video frame can be quickly acquired. It should be understood that the pre-calculated position information of each pixel point in the corresponding video frame represents the position information of each pixel point in the video currently being played on the video playback interface in the corresponding video frame. Can be used for Therefore, when the target user terminal acquires the target pixel point in the target object and the multimedia information associated with the target object, the target pixel point is selected from the locus information corresponding to each of the target pixel points. The locus information corresponding to can be called the target locus information, and the target locus information can be returned to the target user terminal. As a result, the target user terminal is attached to the target locus information, and the multimedia associated with the target object is based on the position information of the target pixel point in each video frame after the key video frame. Information (eg, barrage data) can be tracked and displayed.

Further, see FIG. 12, which is a schematic diagram of the configuration of the computer equipment provided in the embodiments of the present application. As shown in FIG. 12, the computer device 1000 may be the target user terminal in the embodiment corresponding to FIG. 1 above. The computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Further, the computer device 1000 may include a user interface 1003 and at least one communication bus 1002. Here, the communication bus 1002 is for realizing connection communication between these components. Here, the user interface 1003 may include a display (Display) and a keyboard (Keyboard). The optional user interface 1003 may include standard wired and wireless interfaces. The network interface 1004 may optionally include standard wired and wireless interfaces (eg, Wi-Fi interfaces). The memory 100 5 may be a high-speed RAM or a non-volatile memory, for example, at least one magnetic disk memory. The memory 1005 may optionally be at least one storage device located away from the processor 1001 described above. As shown in FIG. 12, the memory 1005, which is a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

Further, see FIG. 14, which is a schematic diagram of the configuration of another computer device provided in the embodiments of the present application. As shown in FIG. 14, the computer device 2000 may be the service server 2000 in the embodiment corresponding to FIG. 1 above. The computer device 2000 may include a processor 2001, a network interface 2004, and a memory 2005. Further, the computer equipment 2000 may include a user interface 2003 and at least one communication bus 2002. Here, the communication bus 2002 is for realizing connection communication between these components. Here, the user interface 2003 may include a display and a keyboard, and the optional user interface 2003 may include a standard wired interface and a wireless interface. The network interface 2004 may optionally include standard wired and wireless interfaces (eg, Wi-Fi interfaces). The memory 2005 may be a high - speed RAM or a non-volatile memory, for example, at least one magnetic disk memory. The memory 2005 may optionally be at least one storage device located away from the aforementioned processor 2001. As shown in FIG. 14, memory 2005, which is a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

In the forward displacement matrix, the displacement setting subsystem 4043 sets the displacement of the matrix element corresponding to the remaining pixel points to the first numerical value, and sets the forward displacement matrix including the first numerical value as the target displacement matrix. The remaining pixel points are determined and are pixel points other than the effective pixel points among the pixel points.

Here, the displacement setting subsystem 4043 specifically, when the forward displacement matrix includes the initial lateral displacement matrix and the initial longitudinal displacement matrix, the remaining pixel points in the initial lateral displacement matrix. The first lateral displacement of the matrix element corresponding to is set to the first numerical value, and the initial lateral displacement matrix including the first numerical value is determined as the lateral displacement matrix corresponding to the first video frame.

Further, the displacement setting subsystem 4043 specifically sets the first vertical displacement of the matrix element corresponding to the remaining pixel points to the first numerical value in the initial vertical displacement matrix, and sets the first numerical value. The initial vertical displacement matrix including the numerical values is determined as the vertical displacement matrix corresponding to the first video frame.

In the first video frame, the second integration subsystem 4052 performs a displacement integration calculation on the lateral displacement matrix in the target displacement matrix, thereby performing the lateral displacement integration corresponding to the pixel points in the first video frame. Get the matrix.

The third integration subsystem 4053 performs a displacement integration calculation on the vertical displacement matrix in the target displacement matrix in the first video frame, thereby performing a vertical displacement integration corresponding to the pixel points in the first video frame. Get the matrix.

The first difference subsystem 4055 performs a displacement difference operation on the state integral matrix based on the length information and the width information corresponding to the difference region, so that the state difference matrix corresponding to the first video frame is performed. To get.

The second difference subsystem 4056 performs a displacement difference calculation for each of the lateral displacement integral matrix and the vertical displacement integral matrix based on the length information and the width information corresponding to the difference region. The horizontal displacement difference matrix and the vertical displacement difference matrix corresponding to the first video frame are acquired.

The average determination subunit 4058 determines the horizontal average displacement matrix and the vertical average displacement matrix as the average displacement matrix corresponding to the first video frame.

Further, see FIG. 16, which is a schematic diagram of the configuration of another computer device provided in the embodiments of the present application. As shown in FIG. 16, the computer device 3000 can be applied to the service server 2000 in the embodiment corresponding to FIG. 1. The computer device 3000 may include a processor 3001, a network interface 3004, and a memory 3005. Further, the computer device 3000 may include a user interface 3003 and at least one communication bus 3002. Here, the communication bus 3002 is for realizing connection communication between these components. Here, the user interface 3003 may include a display and a keyboard, and the optional user interface 3003 may include a standard wired interface and a wireless interface. The network interface 3004 may optionally include standard wired and wireless interfaces (eg, Wi-Fi interfaces). The memory 300 5 may be a high-speed RAM or a non-volatile memory, for example, at least one magnetic disk memory. The memory 3005 may optionally be at least one storage device located away from the processor 3001 described above. As shown in FIG. 16, the memory 3005, which is a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

Claims (15)

A video data processing method applied to computer equipment.
A step of determining a target pixel point from a key video frame of the target video in response to a trigger operation on the target video and acquiring multimedia information associated with the target pixel point, wherein the key video frame is a step. A step and a step, which is a video frame in which the trigger operation is located, and the target pixel point is a pixel point corresponding to the trigger operation in the key video frame.
A step of determining a locus acquisition request corresponding to the target pixel point based on the position information of the target pixel point in the key video frame, and
The step of acquiring the target locus information associated with the position information of the target pixel point in the key video frame based on the locus acquisition request, wherein the target locus information is the key video of the target pixel point. The position information of the target pixel point in the next video frame of the key video frame, including the position information in the video frame next to the frame, is obtained by tracking the target pixel point. ,
When playing back the video frame next to the key video frame, the multimedia information is displayed based on the position information of the target pixel point in the video frame next to the key video frame in the target locus information. Steps and
A method characterized by including. Video data processing method applied to service servers
A step of acquiring trajectory information associated with a target video in response to a trajectory acquisition request for a target pixel point in a key video frame, wherein the key video frame is a video frame in the target video and the target pixel. The points are the pixel points in the key video frame, and the locus information is determined by the position information of the pixel points in each video frame of the target video.
It is a step of selecting the target locus information associated with the position information of the target pixel point in the key video frame from the locus information associated with the target video, and returning the target locus information. The locus information includes the target position information, and the target position information is for triggering to display the multimedia information associated with the target pixel point in the next video frame of the key video frame. , Steps and
A method characterized by including. It ’s a video data processing method.
The step of acquiring the adjacent first video frame and second video frame from the target video, and
A step of determining an average displacement matrix corresponding to the first video frame based on the optical flow tracking rule corresponding to the target video, the pixel points in the first video frame, and the pixel points in the second video frame. When,
A step of tracking the position information of the pixel points in the first video frame based on the average displacement matrix and determining the position information of the traced pixel points in the second video frame.
A step of generating trajectory information associated with the target video based on the position information of the pixel points in the first video frame and the position information of the tracked pixel points in the second video frame. The locus information includes a step and a target locus information for tracking and displaying multimedia information associated with a target pixel point in a target video.
A method characterized by including. The said, which determines the average displacement matrix corresponding to the first video frame based on the optical flow tracking rule corresponding to the target video, the pixel points in the first video frame, and the pixel points in the second video frame. The step is
The optical flow tracking rule corresponding to the target video is acquired, the position information of the pixel points in the first video frame is determined as the first position information, and the position information of the pixel points in the second video frame is the second position information. And the steps to decide as
Based on the optical flow tracking rule, the first position information of the pixel point in the first video frame, and the second position information of the pixel point in the second video frame, the forward direction corresponding to the first video frame. The step of acquiring the displacement matrix and acquiring the reverse displacement matrix corresponding to the second video frame,
Based on the first position information of the pixel points in the first video frame, the forward displacement matrix, and the reverse displacement matrix, the pixel points satisfying the target selection condition among the pixel points are determined as effective pixel points. Steps to do and
A step of acquiring a target state matrix and a target displacement matrix corresponding to the first video frame by modifying the initial state matrix and the forward displacement matrix corresponding to the first video frame based on the effective pixel points. When,
A step of determining an average displacement matrix corresponding to the first video frame based on the target state matrix and the target displacement matrix.
The method according to claim 3, wherein the method comprises. Based on the optical flow tracking rule, the first position information of the pixel point in the first video frame, and the second position information of the pixel point in the second video frame, the forward direction corresponding to the first video frame. The step of acquiring the displacement matrix and acquiring the reverse displacement matrix corresponding to the second video frame is
Based on the first position information of the pixel points in the first video frame and the optical flow tracking rule, the pixel points in the first video frame are forward-mapped to the second video frame, and the second video is used. In the frame, the second position information of the mapped first mapping point is determined, and the first video frame is set based on the first position information of the pixel point and the second position information of the first mapping point. Steps to determine the corresponding forward displacement matrix, and
Based on the second position information of the pixel points in the second video frame and the optical flow tracking rule, the pixel points in the second video frame are mapped in the opposite direction to the first video frame, and the first video is used. In the frame, the third position information of the mapped second mapping point is determined, and the second video is based on the second position information of the first mapping point and the third position information of the second mapping point. Steps to determine the reverse displacement matrix corresponding to the frame,
4. The method according to claim 4, wherein the method comprises. Based on the first position information of the pixel points in the first video frame, the forward displacement matrix, and the reverse displacement matrix, the pixel points satisfying the target selection condition among the pixel points are determined as effective pixel points. The steps to be performed are
The first pixel point is acquired from the pixel points in the first video frame, the first position information of the first pixel point is determined in the first video frame, and the first position is obtained from the forward displacement matrix. The step of determining the first lateral displacement and the first longitudinal displacement corresponding to the pixel points, and
Based on the first position information of the first pixel point and the first lateral displacement and the first vertical displacement corresponding to the first pixel point, the first pixel point is forward to the second video frame. And the step of determining the second position information of the mapped second pixel point in the second video frame.
From the reverse displacement matrix, the second lateral displacement and the second longitudinal displacement corresponding to the second pixel point are determined, and the second position information of the second pixel point and the second pixel point correspond to the second pixel point. Based on the second lateral displacement and the second longitudinal displacement, the second pixel point is mapped in the opposite direction to the first video frame, and in the first video frame, the mapped third pixel point is the third. 3 Steps to determine position information and
Based on the first position information of the first pixel point and the third position information of the third pixel point, the error distance between the first pixel point and the third pixel point is determined, and the first pixel. Based on the first position information of the point and the second position information of the second pixel point, the correlation coefficient between the image block including the first pixel point and the image block including the second pixel point is determined. Steps and
Among the pixel points, a step of determining a pixel point whose error distance is less than the error distance threshold value and whose correlation coefficient is equal to or larger than the correlation coefficient threshold value is determined as an effective pixel point.
4. The method according to claim 4, wherein the method comprises. The target state matrix and the target displacement matrix corresponding to the first video frame are acquired by modifying the initial state matrix and the forward displacement matrix corresponding to the first video frame based on the effective pixel points. The step is
In the step of acquiring the initial state matrix corresponding to the first video frame, the state values of each matrix element of the initial state matrix are all first numerical values, and one matrix element is the pixel point. Steps and steps that correspond to one of them,
In the initial state matrix, the state value of the matrix element corresponding to the effective pixel point is switched from the first numerical value to the second numerical value, and the initial state matrix including the second numerical value is the target state matrix corresponding to the first video frame. And the steps to decide as
In the forward displacement matrix, the displacement of the matrix element corresponding to the remaining pixel points is set to the first numerical value, and the forward displacement matrix including the first numerical value is determined as the target displacement matrix. The remaining pixel points are pixel points other than the effective pixel points among the pixel points.
4. The method according to claim 4, wherein the method comprises. In the forward displacement matrix, the step of setting the displacement of the matrix element corresponding to the remaining pixel points to the first numerical value and determining the forward displacement matrix including the first numerical value as the target displacement matrix is
When the forward displacement matrix includes an initial lateral displacement matrix and an initial longitudinal displacement matrix, the first lateral displacement of the matrix elements corresponding to the remaining pixel points in the initial lateral displacement matrix is the first. A step of setting a numerical value and determining the initial lateral displacement including the first numerical value as a lateral displacement matrix corresponding to the first video frame.
In the initial vertical displacement matrix, the first vertical displacement of the matrix element corresponding to the remaining pixel points is set to the first numerical value, and the initial vertical displacement including the first numerical value is set to the first video frame. And the steps to determine as the vertical displacement matrix corresponding to
A step of determining a lateral displacement matrix corresponding to the first video frame and a longitudinal displacement matrix corresponding to the first video frame as a target displacement matrix.
7. The method according to claim 7, wherein the method comprises. The step of determining the average displacement matrix corresponding to the first video frame based on the target state matrix and the target displacement matrix is
In the first video frame, a step of acquiring a state integral matrix corresponding to a pixel point in the first video frame by performing a displacement integral operation on the target state matrix, and
In the first video frame, a step of acquiring a lateral displacement integral matrix corresponding to a pixel point in the first video frame by performing a displacement integral operation on the lateral displacement matrix in the target state matrix.
In the first video frame, a step of acquiring a vertical displacement integral matrix corresponding to a pixel point in the first video frame by performing a displacement integral operation on the vertical displacement matrix in the target state matrix.
From the first video frame, a difference region corresponding to the displacement difference calculation is determined, and based on the size information of the difference region, the state integral matrix, the lateral displacement integral matrix, and the vertical displacement integral matrix, the said Steps to determine the average displacement matrix corresponding to the first video frame,
4. The method according to claim 4, wherein the method comprises. The step of determining the average displacement matrix corresponding to the first video frame based on the size information of the difference region, the state integral matrix, the lateral displacement integral matrix, and the longitudinal displacement integral matrix is the step.
A step of acquiring a state difference matrix corresponding to the first image frame by performing a displacement difference calculation on the state integral matrix based on the length information and the width information corresponding to the difference region.
The lateral displacement corresponding to the first image frame is performed by performing the displacement difference calculation for each of the lateral displacement integration matrix and the longitudinal displacement integration matrix based on the length information and the width information corresponding to the difference region. Steps to acquire the directional displacement difference matrix and the longitudinal displacement difference matrix,
A step of determining the ratio of the lateral displacement difference matrix to the state difference matrix as a lateral average displacement matrix, and determining the ratio of the longitudinal displacement difference matrix to the state difference matrix as a longitudinal average displacement matrix.
A step of determining the vertical displacement difference matrix and the vertical average displacement matrix as the average displacement matrix corresponding to the first video frame, and
9. The method of claim 9. A video data processing device applied to computer equipment.
An object determination module that determines a target pixel point from a key video frame of the target video in response to a trigger operation on the target video and acquires multimedia information associated with the target pixel point, and is the key video frame. Is a video frame in which the trigger operation is located, and the target pixel point is an object determination module, which is a pixel point corresponding to the trigger operation in the key video frame.
A request determination module that determines a locus acquisition request corresponding to the target pixel point based on the position information of the target pixel point in the key video frame.
A locus acquisition module that acquires target locus information associated with position information of the target pixel point in the key video frame based on the locus acquisition request, wherein the target locus information is the target pixel point of the target pixel point. The position information of the target pixel point in the next video frame of the key video frame, including the position information in the video frame next to the key video frame, is obtained by tracking the target pixel point. Trajectory acquisition module and
When playing back the video frame next to the key video frame, the multimedia information is displayed based on the position information of the target pixel point in the video frame next to the key video frame in the target locus information. Text display module and
A video data processing device characterized by including. A video data processing device applied to a service server.
A request response module that acquires trajectory information associated with a target video in response to a trajectory acquisition request for a target pixel point in a key video frame, wherein the key video frame is a video frame in the target video, and the said. The target pixel point is a pixel point in the key video frame, and the locus information is determined by the position information of the pixel point in each video frame of the target video.
A locus selection module that selects target locus information associated with the position information of the target pixel point in the key video frame from the locus information associated with the target video, and returns the target locus information. The target locus information includes target position information, and the target position information is for triggering display of multimedia information associated with the target pixel point in the next video frame of the key video frame. The trajectory selection module and
A video data processing device characterized by including. It ’s a video data processing device.
The first acquisition module that acquires the adjacent first video frame and second video frame from the target video,
A matrix that determines the average displacement matrix corresponding to the first video frame based on the optical flow tracking rule corresponding to the target video, the pixel points in the first video frame, and the pixel points in the second video frame. Get module and
A position tracking module that tracks the position information of the pixel points in the first video frame based on the average displacement matrix and determines the position information of the tracked pixel points in the second video frame.
There is a tracking generation module that generates trajectory information associated with the target video based on the position information of the pixel points in the first video frame and the position information of the tracked pixel points in the second video frame. The trajectory information includes a tracking generation module for tracking and displaying multimedia information associated with a target pixel point in a target video.
A video data processing device characterized by including. It ’s a computer device,
It has a processor, memory, and a network interface.
The processor is connected to the memory and the network interface, the network interface provides a data communication function, the memory stores a computer program, and the processor calls the computer program, thereby claiming. 1, 2, and the method according to any one of 3 to 10.
A computer device that features that. A computer-readable storage medium in which a computer program is stored, the computer program includes program instructions, and the program instructions are executed by a processor according to claims 1, 2, and 3 to 10. A computer-readable storage medium, characterized in that the method according to any one of the above is performed.

Images