JP7228608B2 - Video frame processing method and processing device, electronic device, storage medium and computer program - Google Patents

Description

TECHNICAL FIELD Embodiments of the present application relate to the field of computer technology, in particular to the field of computer vision technology, particularly to methods and devices for processing video frames, electronic devices, storage media and computer programs.

With the development of Internet technology, various Internet platforms such as video websites and live broadcasting platforms are emerging one after another. In order to diversify the display format of the video screen, various processing such as special effects, style conversion, etc. can be performed on the screen.

Consecutive video frames in a video are usually an image processing difficulty. There may be errors in the processed results of each video frame, and if the errors are large, there may be a blur problem in the processed results of consecutive video frames.

A video frame processing method, processing apparatus, electronic device, storage medium and computer program are provided.

In a first aspect, transforming a feature map of a preceding frame using an optical flow generated based on adjacent preceding and succeeding frames in a video to obtain a transformed feature map; determining weights of the transformed feature map based on the error, obtaining a fused feature map based on the weighted result of the features between the transformed feature map and the feature map of the subsequent frame; updating the feature map, wherein the updated feature map is a fused feature map.

In a second aspect, a transform configured to transform a feature map of a previous frame using an optical flow generated based on adjacent previous and subsequent frames in the video to obtain a transformed feature map. Determine the weights of the transformed feature map based on units and optical flow errors, and obtain a fused feature map based on the weighted result of the features between the transformed feature map and the feature map of the subsequent frame. and an update unit configured to update a feature map of a subsequent frame, wherein the updated feature map is a fused feature map. is provided.

In a third aspect, an electronic device comprising one or more processors and a storage device for storing one or more programs, wherein the one or more programs are executed by the one or more processors Provide an electronic device by which a method according to any embodiment of a method for processing video frames is implemented on one or more processors by being executed.

In a fourth aspect, a computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method of any one of the embodiments of the method of processing video frames. Provide a readable medium.

In a fifth aspect, embodiments of the present application provide a computer program product which, when executed by a processor, implements the method according to any of the embodiments of methods for processing video frames.

The solution of the present application utilizes the optical flow transformation results of previous frames to eliminate object position offsets between adjacent video frames and eliminate screen blur between adjacent video frames after image processing. can be effectively avoided. In addition, determining the weights of the transformed feature map based on the optical flow error helps to prevent the problem of deterioration of the accuracy of the fused features due to the optical flow error.

Other features, objects and advantages of the present application will become more apparent through reading the detailed description of the following non-limiting embodiments in conjunction with the drawings.

FIG. 1 illustrates an exemplary system architecture to which some embodiments of the present application apply. FIG. 2 is a flowchart of one embodiment of a method for processing video frames of the present application. FIG. 3 is a flowchart of one application scene according to the video frame processing method of the present application. FIG. 4 is a flow chart of one embodiment of determining the weights of a transformed feature map according to the video frame processing method of the present application. FIG. 5 is a structural schematic diagram of one embodiment according to the video frame processing device of the present application. FIG. 6 is a block diagram of an electronic device for implementing the video frame processing method of the embodiments of the present application.

Exemplary embodiments of the present application will now be described in conjunction with the drawings, and various details of the embodiments of the present application are included for ease of understanding and are exemplary only. As such, those skilled in the art should appreciate that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, descriptions of well-known functions and structures are omitted below for clarity and concise description.

It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other unless inconsistent. The present application will now be described in detail in combination with embodiments with reference to the drawings.

FIG. 1 shows an exemplary system architecture 100 of an embodiment to which the video frame processing method or video frame processing apparatus of the present application can be applied.

As shown in FIG. 1, system architecture 100 may include terminals 101 , 102 , 103 , network 104 and server 105 . Network 104 is used as a medium for providing communication links between terminals 101 , 102 , 103 and server 105 . Network 104 may include various connection types such as wired, wireless communication links, or fiber optic cables.

Users can interact with server 105 over network 104 using terminals 101 , 102 , 103 to receive or send messages and the like. Terminals 101, 102, 103 can be installed with various communication client applications such as video applications, live broadcast applications, instant messaging tools, email clients, social platform software, and so on.

Here, the terminal devices 101, 102, and 103 may be hardware or software. If the terminal devices 101, 102, 103 are hardware, they may be various electronic devices with displays, and may include smartphones, tablets, e-readers, laptop computers and desktop computers, etc., but they is not limited to If the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices. It may be implemented as multiple pieces of software or software modules (eg, multiple pieces of software or software modules for providing a distributed service), or it may be implemented as a single piece of software or software module. It is not specifically limited here.

Server 105 may be a server that provides various services, such as a back-end server that provides support for terminals 101 , 102 , 103 . The back-end server can perform processing such as analysis on received data, such as adjacent video frames, and feed back processing results (eg, updated feature maps) to the terminal device.

It should be noted that the video frame processing method provided by the embodiments of the present application can be executed by the server 105 or the terminal device 101, 102, 103, and correspondingly, the video frame processing device can be the server 105 or the terminal device. 101, 102, 103 may be installed.

It should be understood that the numbers of terminals, networks and servers in FIG. 1 are exemplary. Any number of terminals, networks and servers may be provided depending on the needs of the implementation.

As shown in FIG. 2, a flow 200 of one embodiment according to the video frame processing method of the present application is shown. The video frame processing method includes steps 201-203.

Step 201, transform the feature map of the previous frame using the optical flow generated based on the adjacent previous and subsequent frames in the video to obtain a transformed feature map.

In this embodiment, the execution entity (eg, the server or terminal device shown in FIG. 1) in which the video frame processing method is implemented recognizes a frame of previous time in the video, eg, the fifth frame ) and the optical flow generated in the subsequent frame (a frame of next time, for example, the sixth frame, which is the current frame), transform the acquired feature map of the preceding frame to acquire the transformed feature map. can. The transformed feature map and the feature map of the subsequent frame are similar (similarity greater than a predetermined threshold). In fact, the process of transformation may be called the process of warping, ie, it causes each pixel point to generate a position offset, the offset of which is the determined optical flow.

In fact, the above execution entity can obtain the feature map of the previous frame and the feature map of the subsequent frame of two adjacent video frames of the video. Specifically, the execution entity can directly obtain the feature map of the subsequent frame and the feature map of the previous frame from a local or other electronic device. In addition, the execution entity can obtain the subsequent frame and the preceding frame, detect the subsequent frame and the preceding frame, and obtain the feature map of the subsequent frame and the feature map of the preceding frame. In fact, the optical flow may be dense optical flow or sparse optical flow.

Specifically, the optical flow can be obtained in various ways. For example, the executor can obtain the optical flow directly from a local or other electronic device. In addition, the execution entity can further obtain the preceding frame and the subsequent frame to generate the optical flow. For example, the execution entity can generate an initial optical flow between a preceding frame and a succeeding frame, and perform preset processing on the initial optical flow to obtain the optical flow.

Step 202, determine the weight of the transformed feature map according to the optical flow error, and obtain a fused feature map according to the feature weighting result of the transformed feature map and the feature map of the subsequent frame.

In the present embodiment, the execution entity determines the weight of the transformed feature map, obtains the weighted result of weighting the transformed feature map and the feature map of the subsequent frame, and fuses according to the weighted result. It is possible to obtain a feature map that has been processed.

Optical flow error refers to the deviation between the generated optical flow and the actual optical flow. The execution entity uses optical flow to transform the reduced preceding frame, compares the transformed result with the reduced subsequent frame, and calculates, for example, the difference or absolute value between the transformed result and the reduced subsequent frame. to obtain the optical flow error.

In fact, the performing entity can determine the weights of the transformed feature map based on the optical flow error in various ways. For example, the agent may reduce the weight of the transformed feature map to a predetermined weight value, e.g. may be the maximum candidate weight of . In addition, the above execution entity acquires the correspondence between the optical flow error and the weight of the transformed feature map, such as a model or a correspondence table, to obtain the transformed feature map corresponding to the determined optical flow error. Weights may be obtained.

In fact, the performing entity can obtain the fused feature map based on the weighted results in various ways. For example, the executing entity may determine a weighted average value of the features of each feature map according to the weighted results, and the weighted average value may be the fused feature map. In addition, the above-mentioned executing entity further performs a preset process, such as directly converting the weighted result into a merged feature map, or multiplying the weighted result by a predetermined coefficient, where each of the weighted The sum of the feature map weights may be one.

Step 203, update the feature map of the subsequent frame, where the updated feature map is the fused feature map.

In this embodiment, the execution entity can update the feature map of subsequent frames to the fused feature map. In fact, the execution entity can further perform subsequent image processing, such as inputting the fused feature map as the feature map of the subsequent frame into a deep neural network. For example, it may be input to a fully connected layer of a convolutional neural network, or to a generator of a generative adversarial network.

The method provided by the above embodiment of the present application utilizes the optical flow transformation result of the previous frame to avoid the screen blurring between the adjacent frames after image processing. Eliminate position offsets. In addition, determining the weights of the transformed feature map based on the optical flow error helps prevent the problem of low precision of the fused features due to the optical flow error.

In some optional embodiments of this embodiment, the method includes adding features to the transformed feature map and the subsequent frame feature map based on the transformed feature map weight and the subsequent frame feature map weight. weighting to obtain a weighted result of the features of the transformed feature map and the feature map of the subsequent frame, wherein the greater the weight of the transformed feature map, the greater the weight of the feature map of the subsequent frame. small weight.

In these optional embodiments, the method may further include the performing entity weighting the transformed feature map and the feature map of subsequent frames. Specifically, the transformed feature map weights and the subsequent frame feature map weights may constrain each other.

Optionally, the sum of the transformed feature map weight and the feature map weight of the subsequent frame may be a predetermined number. Specifically, the sum of the weights of each feature map related to weighting may be a predetermined numerical value, such as one. For example, the weight of the feature map of the subsequent frame may be a predetermined number minus the weight of the transformed feature map, e.g., if the predetermined number is 1 and the weight of the transformed feature map is 1, For example, the weight of the feature map of the subsequent frame is zero. In practice, the weights may only contain 1's and 0's.

These embodiments can obtain a more accurate fused feature map by limiting the relationship between the weights.

In some optional embodiments of this embodiment, step 201 may comprise generating a feature map for the subsequent frame and a feature map for the previous frame of the video using an adversarial generation network; and The method further includes processing the updated feature map using a generative adversarial network to generate images of the target domain corresponding to subsequent frames.

In these optional embodiments, the actor can utilize an adversarial generation network to generate feature maps for subsequent frames and feature maps for previous frames. In addition, the execution entity further updates the feature map of the subsequent frame to the fused feature map, and then processes the updated feature map using the adversarial generation network to generate an image of the target domain. can do. Here, a generative adversarial network is used to generate an image of the target domain.

These embodiments can avoid blurring of objects in successive video frames after processing by a generative adversarial network, and between processing results of adjacent frames due to the inability of a generative adversarial network to process multiple video frames in batches. The problem of screen blurring caused by the difference in video resolution is eliminated, and the stability of the video screen is improved.

Next, as shown in FIG. 3, FIG. 3 is a schematic diagram of one application scene according to the video frame processing method of the present embodiment. In the application scene of FIG. 3, the execution entity 301 uses the optical flow 302 generated based on the adjacent previous frame (eg, the 7th frame) and the subsequent frame (eg, the 8th frame) in the video to Transform the feature map to obtain a transformed feature map 303, where the size of the feature map is the target size 32*32. The execution entity 301 determines the weights of the transformed feature map 303 based on the error of the optical flow 302, and generates the fused feature map 304 based on the feature weighting result of the transformed feature map 303 and the feature map of the subsequent frame. get. Actor 301 updates the feature map 305 of the subsequent frame, where the updated feature map is the fused feature map 304 .

In some optional embodiments of this embodiment, the optical flow may be dense optical flow. A video frame processing method of the present application reduces a preceding frame and a following frame to the size of a feature map, establishes a high density optical flow between the reduced preceding frame and the reduced succeeding frame, and The method further includes making the density optical flow an optical flow generated based on adjacent preceding and following frames in the video.

In these optional embodiments, the entity executing the video frame processing method (eg, the server or terminal shown in FIG. 1) reduces the size of the preceding and following frames of the video so that the following and The size of the previous frame can be reduced to the size of the feature map. Specifically, the size of the acquired feature map of the subsequent frame and the feature map of the preceding frame match. The execution entity can then determine the optical flow between the reduced previous frame and the reduced subsequent frame. Here, both the size of the feature map of the previous frame and the size of the transformed feature map are the size of the video frame after the reduction. To give an example, the preceding and following frames are the 9th and 10th frames of the video respectively, and the above execution entity reduces the 9th and 10th frames to 32*32, i.e. the size of the feature map. You may

These embodiments can increase the accuracy of the fused feature map by performing feature fusion on a pixel-by-pixel basis with dense optical flow.

Further, as shown in FIG. 4, an embodiment flow 400 for determining transformed feature map weights in a video frame processing method is shown. The flow 400 includes steps 401-404.

Step 401, using dense optical flow to transform the reduced predecessor frame to obtain a transformed predecessor frame.

In this embodiment, the optical flow is high density optical flow. An execution entity (e.g., the server or terminal device shown in FIG. 1) in which the video frame processing method is implemented transforms the reduced preceding frame using Dense Optical Flow, and transforms the result into the transformed preceding frame. can be The transformed predecessor frame is similar to the downscaled subsequent frame. Dense optical flow is ie compact optical flow. Dense optical flow can determine the positional offset between the reduced subsequent frame and the reduced previous frame for each pixel point.

Step 402, determine the error of the pixel point at the coordinate of the high-density optical flow according to the pixel value difference between the transformed previous frame and the reduced subsequent frame of the pixel point at each coordinate.

In this embodiment, the execution subject can determine the high-density optical flow for each pixel point. For each pixel point, determine the dense optical flow error of the pixel point at that coordinate based on the difference between the pixel values of the pixel point at that coordinate in the transformed previous frame and the reduced subsequent frame.

In practice, the executing entity can determine the error of the dense optical flow based on the difference value in various ways. For example, the execution entity may determine the absolute value of the difference value as the error of the dense optical flow, or directly determine the difference value as the error of the dense optical flow. In addition, the execution subject may further perform a preset process on the difference value, for example, multiply it by a predetermined coefficient or input it to a predetermined function, and use the result as the error of the high-density optical flow.

Step 403, it is determined whether a pixel point including the same object exists in the specific image for the pixel point of the coordinate in the image obtained by transforming the specific image with the high-density optical flow. to obtain the result of determination, where the size of the specific image is the target size.

In this embodiment, the execution entity can determine whether the image after transformation using high-density optical flow contains the same object as compared to the pixel point of the coordinates before transformation. That is, in adjacent frames before and after the video, the object may have changed position, and some content in the subsequent frame may be new content that did not appear in the previous frame. This step allows searching for the same content in adjacent frames. In fact, the particular image here may be various images, such as the previous frame after the reduction mentioned above.

In some optional embodiments of this embodiment, step 403 uses obtaining a specific image in which the pixel values of each pixel point are all predetermined pixel values, and using dense optical flow, transforming a specific image to obtain a transformed specific image; and if the pixel value of the pixel point at the coordinates in the transformed specific image is equal to or greater than a predetermined pixel value, the result of the determination is that the pixel point containing the same object exists. and determining that if the pixel value of the pixel point at the coordinates in the transformed specific image is less than a predetermined pixel value, the result of the determination is that there is no pixel point that includes the same object. may contain.

In these optional embodiments, the executor obtains a particular image, and the pixel values of all pixel points in the particular image may be predetermined pixel values, eg, 1 or 2. The execution entity can use high-density optical flow to transform the specific image to obtain a transformed specific image. Thus, the transformed particular image may have had positional offsets to objects in the image relative to the particular image. For a pixel point of one coordinate, if the pixel value in the converted image is equal to or greater than the predetermined pixel value, the object as the content of the pixel point not only exists in the converted specific image, but also in the specific image. exist. If the pixel value of the pixel point at the coordinates is less than the predetermined pixel value in the image after conversion, the object as the content of the pixel point exists only in the transformed specific image and does not exist in the specific image.

These embodiments can obtain the determination results based on a specific image in which the pixel values are all numerical values, which simplifies the calculation process and enhances the processing efficiency of the solution.

Step 404, determine the weight of the pixel point at that coordinate in the transformed feature map according to the error of the dense optical flow and the judgment result.

In this embodiment, the execution entity can determine the weight of the pixel point at that coordinate in the transformed feature map based on the error of the dense optical flow and the judgment result. In fact, the executing entity can determine its weight according to the error and the judgment result in various ways. For example, the execution entity acquires the correspondence between the error, the judgment result, and the weight (for example, a correspondence table or model), thereby obtaining the weight corresponding to the error and the judgment result. .

The present embodiment uses high-density optical flow to fuse features for each pixel point, thereby improving the effect of preventing screen blur. Moreover, the present embodiment determines the weight of the transformed feature map of the previous frame based on both the error of the dense optical flow and the result of the determination, so that the accuracy of the fused feature map can be enhanced.

In some optional embodiments of this embodiment, step 404 determines that the error of the dense optical flow is less than a predetermined error threshold and that there are pixel points that contain the same object. , determining that the weight of the pixel point at the coordinate in the transformed feature map is the first candidate weight, wherein the pixel at the pixel point at the coordinate in the subsequent frame after reduction is The larger the value, the larger the predetermined error threshold, and the error of the high-density optical flow is equal to or greater than the predetermined error threshold, and/or there is no pixel point that includes the same object as the determination result. responsive to determining that there is, determining that the weight of the pixel point at the coordinate in the transformed feature map is the second candidate weight, wherein the first candidate weight is greater than the second candidate weight; good.

In these optional embodiments, if both the error of the dense optical flow is less than a predetermined error threshold and the result of the determination is that there are pixel points containing the same object, then the transform It is determined that the weight of the pixel point at the coordinates in the finished feature map is the first candidate weight. Here, the numerical value of the first candidate weight may be large, such as 1, and the numerical value of the second candidate weight may be small, such as 0.

In practice, the predetermined error threshold may be associated with the pixel value of the coordinate point in the subsequent frame after downscaling, the larger the pixel value, the greater the predetermined error threshold. For example, if the pixel value of the pixel of the coordinate point in the subsequent frame after reduction is set to A, then the predetermined error threshold may be a*A+b, where a is a predetermined coefficient of A; It may be 0 to 1, and b may be a predetermined constant greater than 0.

These embodiments continue to use the post-transformation processing result of the preceding frame only if both the error in Dense Optical Flow is small and the content of the pixel point is present in both the preceding and succeeding frames; In this way, positional offsets of features in the fused feature map due to too large optical flow errors can be avoided, while at the same time the new content in subsequent frames will be replaced by the content of the previous frame when fused. Errors on the screen that occur can be avoided and the correctness of the content on the screen is ensured.

Further, as shown in FIG. 5, as an implementation of the method shown in the above figures, the present application provides one embodiment of a video frame processing apparatus, the embodiment of the apparatus is the method shown in FIG. In addition to the features corresponding to the embodiments and described below, the apparatus embodiments may also include features or effects that are the same as or correspond to the method embodiments shown in FIG. The device can be specifically applied in various electronic devices.

As shown in FIG. 5, the video frame processing apparatus 500 of this embodiment comprises a transform unit 501 , a fusion unit 502 and an update unit 503 . Here, the transformation unit 501 transforms the feature map of the preceding frame using the optical flow generated based on the adjacent preceding and following frames in the video to obtain a transformed feature map. A fusion unit 502 determines the weight of the transformed feature map based on the optical flow error, and the fused feature map based on the feature weighting result of the transformed feature map and the feature map of the subsequent frame. and the updating unit 503 is configured to update the feature map of the subsequent frame, where the updated feature map is the fused feature map.

In this embodiment, the specific processing and technical effects of the conversion unit 501, fusion unit 502 and update unit 503 of the video frame processing device 500 are respectively described in steps 201, 202 and 202 in the embodiment corresponding to FIG. The related description of step 203 can be referred to, so the description thereof is omitted here.

In some optional embodiments of this embodiment, the optical flow is dense optical flow, the apparatus reduces the preceding and following frames to the size of the feature map, and reduces the preceding and following frames to the size of the feature map. optical flow generation configured to determine a dense optical flow between a subsequent frame and the dense optical flow as an optical flow generated based on adjacent preceding and subsequent frames in the video Have more units.

In some optional embodiments of this embodiment, the fusion unit is further configured to perform determining the weights of the transformed feature map based on the optical flow error according to the following scheme. Using high-density optical flow, transform the reduced preceding frame, obtain the transformed preceding frame, and calculate the pixel value difference between the transformed preceding frame and the reduced succeeding frame for pixel points at each coordinate. Determine the error of the pixel point at the coordinates of the high-density optical flow based on, and for the pixel point at the coordinates in the image acquired after performing the transformation using the high-density optical flow on the specific image to determine whether there is a pixel point containing the same object in the specific image, and obtain the determination result, where the size of the specific image is the target size, and the dense optical flow Determine the weight of the pixel point at that coordinate in the transformed feature map, based on the error of and the result of determination.

In some optional embodiments of this embodiment, the fusion unit further converts a pixel point of its coordinates in an image obtained after transforming a particular image with Dense Optical Flow to a particular It is configured to determine whether or not a pixel point including the same object exists in the image, and obtain the result of the determination. Here, the size of a particular image is the target size. Obtaining a specific image in which the pixel values of each pixel point are all predetermined pixel values, and transforming the specific image using high-density optical flow to obtain a transformed specific image. For a pixel point at each coordinate in the transformed specific image, it is determined whether the pixel value of the pixel point at the coordinate in the transformed specific image is equal to or greater than a predetermined pixel value. When the pixel value of the pixel point at the coordinates in the transformed specific image is equal to or greater than the predetermined pixel value, it is determined that the pixel point including the same object exists as a determination result. When the pixel value of the pixel point at the coordinates in the transformed specific image is less than the predetermined pixel value, it is determined that the pixel point including the same object does not exist.

In some optional embodiments of this embodiment, the fusion unit further determines the weight of the pixel point at the coordinate in the transformed feature map based on the dense optical flow error and the determination result by the following method: Configured to perform a commit. In response to satisfying both that the error of the dense optical flow is less than a predetermined error threshold and that the result of the determination is that there is a pixel point containing the same object, the transformed feature map: The weight of the coordinate pixel point is determined to be the first candidate weight, where the greater the pixel value of the coordinate pixel point in the subsequent frame after reduction, the greater the predetermined error threshold. the coordinates in the transformed feature map in response to satisfying that the error of the dense optical flow is greater than or equal to a predetermined error threshold and/or that the result of the determination is that there are no pixel points containing the same object is the second candidate weight, where the first candidate weight is greater than the second candidate weight.

In some optional embodiments of this embodiment, the apparatus weights features for the transformed feature map and the subsequent frame feature map based on the transformed feature map weight and the subsequent frame feature map weight. to obtain the feature weighting result of the transformed feature map and the feature map of the subsequent frame, where the greater the weight of the transformed feature map, the smaller the weight of the feature map of the subsequent frame. may contain.

In some optional embodiments of this embodiment, the apparatus is further configured to utilize a generative adversarial network to generate feature maps for subsequent frames and feature maps for preceding frames of the video. may include units. The apparatus may also further include a target generation unit configured to process the updated feature map utilizing the adversarial generation network to generate an image of the target domain corresponding to subsequent frames.

According to embodiments of the present application, the application further provides an electronic device and a readable storage medium.
As shown in FIG. 6, it is a block diagram of an electronic device according to the video frame processing method of an embodiment of the present application. Electronic equipment refers to various forms of digital computers such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers and other suitable computers. Electronics can also refer to various forms of mobile devices such as personal digital assistants, cell phones, smart phones, wearable devices and other similar computing devices. It should be noted that the components, their connections and relationships, and their functionality shown herein are exemplary only and are not intended to limit the implementation of the application as described and/or claimed herein.

As shown in FIG. 6, the electronic device includes one or more processors 601, storage device 602, and interfaces (including high speed and low speed interfaces) for connecting components. Each component is connected to each other by a different bus, and may be mounted on a common motherboard, or may be mounted by other methods as required. The processor is capable of processing instructions executed within the electronic device and instructions for displaying graphical information of a graphical user interface (GUI) on an external input/output device such as a display device coupled to the interface. in or on a storage device. In other embodiments, multiple processors and/or multiple buses and multiple storage devices may be used, along with multiple storage devices, where appropriate. Also, multiple electronic devices may be connected, each device providing some required operation, for example, as a server array, blade server cluster, or multi-processor system. In FIG. 6, one processor 601 is taken as an example.

Storage device 602 is a non-transitory computer-readable storage medium provided by embodiments of the present application. Here, the storage device stores instructions executable by at least one processor to cause the at least one processor to perform the video frame processing method provided by the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to cause a computer to perform the video frame processing methods provided by the present application.

Storage device 602 is a non-transitory computer-readable storage medium containing non-transitory software programs, non-transitory computer-executable programs, and program commands/ It is used to store modules such as modules (eg, transform unit 501, fusion unit 502 and update unit 503 shown in FIG. 5). The processor 601 executes the non-transitory software programs, commands and modules stored in the storage device 602 to perform the various functional applications and data processing of the server, i.e. the processing of video frames in the above method embodiments. Implement a processing method.

The storage device 602 has a program storage area that can store an operating system and applications required for at least one function, and a data storage area that can store data created according to the use of the electronic device for processing video frames. and may include Storage 602 may also include high-speed random access memory and may also include non-transitory storage (eg, at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device). can be done. In some embodiments, storage device 602 can optionally include storage devices remotely located relative to processor 601, and these remotely located storage devices are connected to the present embodiment via a network. Can be connected to electronics for processing of video frames. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic device implementing the video frame processing method may further comprise an input device 603 and an output device 604 . The processor 601, storage device 602, input device 603 and output device 604 can be connected by a bus or other methods, and FIG. 6 is an example of connecting by a bus.

The input device 603 can receive input numeric and character information, and can generate input of key signals related to user settings and functional control of the electronic device for implementing the video frame processing method of the present embodiment. Examples of input device 603 include a touch panel, keypad, mouse, trackpad, touchpad, pointing device, one or more mouse buttons, trackball, joystick, and the like. The output device 604 can include a display device, an auxiliary lighting device, a haptic feedback device, etc., wherein the auxiliary lighting device is, for example, an LED, and the haptic feedback device is, for example, a vibration motor. The display device can include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch panel.

Various embodiments of the systems and techniques described herein may be digital electronic circuit systems, integrated circuit systems, Application Specific Integrated Circuits (ASICs), computer hardware, firmware, software, and/or combinations thereof. can be implemented in These various embodiments are implemented in one or more computer programs, which can be executed and/or interpreted in a programmable system that includes at least one programmable processor; The processor may be a dedicated or general purpose programmable processor capable of receiving data and instructions from a storage system, at least one input device and at least one output device, and transmitting data and instructions to the storage system, the at least one transmitting to one input device and the at least one output device.

These computer programs, also referred to as programs, software, software applications or code, contain programmable processor machine instructions and utilize high process and/or object oriented programming languages and/or assembly/machine language. can be implemented. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device for providing machine instructions and/or data to a programmable processor (e.g., magnetic disk, means an optical disc, storage device, programmable logic device (PLD)) and includes a machine-readable medium for receiving machine instructions as machine-readable signals. The term "machine-readable signal" means any signal for providing machine instructions and/or data to a programmable processor.

To provide for user interaction, the systems and techniques described herein combine a display device (e.g., a Cathode Ray Tube (CRT) or LCD (Liquid Crystal Display) monitor) to display information to the user. , a keyboard and a pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other types of devices can be used to provide further interaction with the user. For example, the feedback provided to the user may be any form of sensing feedback, such as, for example, visual, auditory, or tactile feedback, and may include any sound, audio, or tactile input. Input from the user may be received in the form of

The systems and techniques described herein may be implemented in computing systems that include background components (e.g., data servers) or may be implemented in computing systems that include middleware components (e.g., application servers). , or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or web browser), through which a user can interact with the systems and Embodiments of the technology may interact or be implemented in a computing system that includes any combination of such background, middleware or front-end components. Further, each component of the system may be connected by digital data communication via any form or medium such as a communication network. Communication networks include local area networks (LAN), wide area networks (WAN), the Internet, and the like.

The computer system can include clients and servers. A client and server are typically remote from each other and interact through a communication network. The relationship of client and server is created by running computer programs on the respective computers which have a client-server relationship to each other.

The flowcharts and block diagrams in the drawings illustrate possible system architectures, functionality, and operation of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in a flowchart or block diagram can represent a module, program segment, or portion of code, which implements a defined logic function. may include one or more executable instructions for It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutively-appearing blocks may be executed substantially in parallel or in the reverse order, depending on the functionality involved. It is noted that each block in the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, may be implemented in a dedicated hardware-based system to perform the specified functions or operations; It may also be implemented using a combination of dedicated hardware and computer instructions.

The units described in the embodiments of the present application may be implemented in software manner or may be implemented in hardware manner. The described units may be located in a processor and may for example be described as a processor comprising a conversion unit, a fusion unit and an update unit. Here, the names of these units do not, under certain circumstances, constitute a limitation of the units themselves, e.g. an update unit may also be described as "the unit that updates the feature map of subsequent frames". good.

As another aspect, the present application further provides a computer-readable medium, which may be included in the apparatus described in the above embodiments, or may exist independently without being incorporated in the apparatus. good. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device displays a Using the generated optical flow, transform the feature map of the preceding frame, obtain the transformed feature map, determine the weight of the transformed feature map based on the error of the optical flow, and combine the transformed feature map with the subsequent Obtain a fused feature map based on the feature weighting result with the feature map of the frame, and update the feature map of the subsequent frame, where the updated feature map is the fused feature map.

The above description is only a description of the preferred embodiments of the present application and the technical principles utilized. A person skilled in the art will appreciate that the scope of the invention referred to in the present application is not limited to the specific combination of the above technical features and that the above technical features or their equivalents can be used without departing from the above inventive concept. It should be understood that other technical solutions formed by any combination of features are also included at the same time. For example, technical solutions (but not limited to) formed by mutual replacement of the above features and technical features with similar functions disclosed in this application are also included.

Claims (13)

transforming a feature map of a preceding frame using an optical flow generated based on adjacent preceding and following frames in a video to obtain a transformed feature map;
Determining weights of the transformed feature map based on the optical flow error, and obtaining a fused feature map based on the feature weighting results of the transformed feature map and the feature map of the subsequent frame. ,
updating the feature map of the subsequent frame to obtain an updated feature map that is the fused feature map ;
the optical flow is a high density optical flow;
The processing method is
reducing the preceding frame and the succeeding frame to the size of a feature map; determining a dense optical flow between the reduced preceding frame and the succeeding frame after reduction; and applying the dense optical flow to the video. further comprising generating an optical flow based on adjacent preceding and following frames;
Determining weights of the transformed feature map based on the optical flow error includes:
transforming the reduced previous frame using the dense optical flow to obtain a transformed previous frame;
determining an error of a pixel point at that coordinate in the Dense Optical Flow based on a pixel value difference between the transformed previous frame and the reduced subsequent frame for the pixel point at each coordinate;
Whether or not a pixel point including the same object exists in the specific image with respect to the pixel point of the coordinates in the image obtained by transforming the specific image of the target size based on the high-density optical flow determining and obtaining a determination result;
determining weights of pixel points at the coordinates in the transformed feature map based on the dense optical flow error and the result of the determination;
How to process video frames. A pixel point containing the same object exists in the specific image with respect to the pixel point of the coordinates in the image obtained by transforming the specific image of the target size based on the high-density optical flow. Determining whether or not and obtaining the result of determination is
Acquiring a specific image in which the pixel value of each pixel point is a predetermined pixel value;
transforming the specific image using the high-density optical flow to obtain a transformed specific image;
Determining whether a pixel value of a pixel point at each coordinate in the transformed specific image is equal to or greater than the predetermined pixel value;
Determining that there is a pixel point including the same object when the pixel value of the pixel point at the coordinates in the transformed specific image is equal to or greater than the predetermined pixel value;
2. The processing method according to claim 1 , further comprising determining that there is no pixel point including the same object if the pixel value of the pixel point at the coordinates in the transformed specific image is less than the predetermined pixel value. . Determining the weight of the pixel point at the coordinate in the transformed feature map based on the error of the dense optical flow and the result of the determination,
the transformed feature map in response to satisfying both that the error of the dense optical flow is less than a predetermined error threshold and that there is a pixel point containing the same object as a result of the determination; is determined to be the first candidate weight, and the larger the pixel value of the pixel point at the coordinates in the subsequent frame after reduction, the greater the predetermined error threshold is big things and
of the coordinates in the transformed feature map in response to the fact that the error of the dense optical flow is greater than or equal to a predetermined error threshold and/or that there is no pixel point that includes the same object as the result of the determination. 2. The method of claim 1 , comprising determining that a pixel point weight is a second candidate weight that is less than said first candidate weight. performing feature weighting on the transformed feature map and the feature map of the subsequent frame based on the weight of the transformed feature map and the weight of the feature map of the subsequent frame; 2. The method of claim 1, further comprising obtaining a weighted result of features with a feature map, wherein the higher the weight of the transformed feature map, the lower the weight of the feature map of the subsequent frame. Processing method. utilizing a generative adversarial network to generate a feature map for a subsequent frame and a feature map for a preceding frame of a video;
2. The method of claim 1, further comprising processing the updated feature map using a generative adversarial network to generate an image of the target domain corresponding to the subsequent frame. a transformation unit configured to transform a feature map of a preceding frame using an optical flow generated based on adjacent preceding and following frames in a video to obtain a transformed feature map;
determining a weight of the transformed feature map based on the optical flow error, and obtaining a fused feature map based on the feature weighting result of the transformed feature map and the feature map of the subsequent frame. a fusion unit composed of
an update unit configured to update the feature map of the subsequent frame to obtain an updated feature map that is the fused feature map ;
the optical flow is a high density optical flow;
The processing device is
reducing the preceding frame and the succeeding frame to the size of a feature map; determining a dense optical flow between the reduced preceding frame and the succeeding frame after reduction; and applying the dense optical flow to the video. further comprising an optical flow generation unit configured to generate an optical flow based on adjacent preceding and following frames;
The fusion unit further comprises:
transforming the reduced preceding frame using the dense optical flow to obtain a transformed preceding frame;
determining an error of a pixel point at each coordinate of the pixel point at the coordinate of the high-density optical flow based on the pixel value difference between the transformed preceding frame and the reduced subsequent frame;
Whether or not a pixel point including the same object exists in the specific image with respect to the pixel point of the coordinates in the image obtained by transforming the specific image of the target size based on the high-density optical flow is determined, and the result of the determination is obtained,
configured to determine a weight of a pixel point at the coordinate in the transformed feature map based on the dense optical flow error and the result of the determination;
Video frame processor. The fusion unit further comprises:
obtaining a specific image in which the pixel values of each pixel point are all predetermined pixel values;
Using the high-density optical flow to transform the specific image to obtain a transformed specific image;
determining whether a pixel value of a pixel point at each coordinate in the transformed specific image is equal to or greater than the predetermined pixel value;
determining that a pixel point including the same object exists when the pixel value of the pixel point at the coordinates in the transformed specific image is equal to or greater than the predetermined pixel value;
7. The processing device according to claim 6, configured to determine that there is no pixel point including the same object when the pixel value of the pixel point at the coordinates in the transformed specific image is less than the predetermined pixel value. . The fusion unit further comprises:
the transformed feature map in response to satisfying both that the error of the dense optical flow is less than a predetermined error threshold and that there is a pixel point containing the same object as a result of the determination; is determined to be the first candidate weight, and the larger the pixel value of the pixel point at the coordinates in the subsequent frame after reduction, the greater the predetermined error threshold is big things and
of the coordinates in the transformed feature map in response to the fact that the error of the dense optical flow is greater than or equal to a predetermined error threshold and/or that there is no pixel point that includes the same object as the result of the determination. 7. The processor of claim 6 , configured to: determine that a pixel point weight is a second candidate weight that is less than the first candidate weight. performing feature weighting on the transformed feature map and the feature map of the subsequent frame based on the weight of the transformed feature map and the weight of the feature map of the subsequent frame; further comprising a weighted result obtaining unit configured to obtain a weighted result of the feature with the feature map;
7. The processing device of claim 6 , wherein the higher the weight of the transformed feature map, the lower the weight of the feature map of the subsequent frame. a feature generation unit configured to generate a feature map for a subsequent frame and a feature map for a preceding frame of a video using a generative adversarial network;
7. The process of claim 6 , further comprising a target generation unit configured to process the updated feature map utilizing an adversarial generation network to generate an image of the target domain corresponding to the subsequent frame. Device. An electronic device comprising one or more processors and a storage device for storing one or more programs,
An electronic device in which the processing method according to any one of claims 1 to 5 is implemented in the one or more processors by executing the one or more programs by the one or more processors. device. A computer-readable storage medium storing a computer program,
A computer readable storage medium on which the processing method according to any one of claims 1 to 5 is implemented when said computer program is executed by a processor. A computer program, which, when executed by a processor, implements the processing method according to any one of claims 1 to 5 .

Images