CN111862250B - Video color conversion method and device, electronic equipment and storage medium - Google Patents
Video color conversion method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111862250B CN111862250B CN202010537912.8A CN202010537912A CN111862250B CN 111862250 B CN111862250 B CN 111862250B CN 202010537912 A CN202010537912 A CN 202010537912A CN 111862250 B CN111862250 B CN 111862250B
- Authority
- CN
- China
- Prior art keywords
- coloring
- image
- video
- model
- color
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 74
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000003860 storage Methods 0.000 title claims abstract description 17
- 238000004040 coloring Methods 0.000 claims abstract description 288
- 238000005516 engineering process Methods 0.000 claims abstract description 16
- 238000013135 deep learning Methods 0.000 claims abstract description 15
- 238000012549 training Methods 0.000 claims description 31
- 230000015654 memory Effects 0.000 claims description 19
- 238000013507 mapping Methods 0.000 claims description 13
- 238000001514 detection method Methods 0.000 claims description 10
- 238000012545 processing Methods 0.000 abstract description 13
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 12
- 238000004590 computer program Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000002372 labelling Methods 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 239000003086 colorant Substances 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000008485 antagonism Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/001—Texturing; Colouring; Generation of texture or colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The application discloses a video color conversion method, a video color conversion device, electronic equipment and a storage medium, and relates to the field of cloud computing and video processing based on artificial intelligence. The specific implementation scheme is as follows: acquiring an image sequence formed by each frame of image in a video; tracking and coloring each image in the image sequence by adopting a coloring model trained based on a deep learning technology; based on the sequence of images after tracking shading, color converted video is generated. According to the video coloring method and device, coloring continuity between continuous frame images in the video can be guaranteed through tracking coloring, and accuracy of the video after color conversion is improved.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to the field of cloud computing and video processing based on artificial intelligence, and in particular, to a method and apparatus for color conversion of video, an electronic device, and a storage medium.
Background
In the prior art, many videos are black and white. For some surveillance videos, only black and white is used to save storage space. As another example, in the early movie industry development, many precious historical movies or videos were produced, but this technology was limited to the current production technology, and most of them were in the form of black and white videos.
In order to enrich the display effect of the video, the video can be colored into color, and the color conversion of the video is realized. However, the existing basic practice adopts an image color conversion mode to perform independent color conversion on each frame of image in the video, and consistency between continuous frame of images in the video is not considered, so that the accuracy of the color conversion of the video is low.
Disclosure of Invention
In order to solve the technical problems, the application provides a video color conversion method, a video color conversion device, an electronic device and a storage medium.
According to an aspect of the present application, there is provided a color conversion method of a video, wherein the method includes:
acquiring an image sequence formed by each frame of image in a video;
tracking and coloring each image in the image sequence by adopting a coloring model trained based on a deep learning technology;
based on the sequence of images after tracking shading, color converted video is generated.
According to another aspect of the present application, there is provided a color conversion apparatus for video, wherein the apparatus includes:
the acquisition module is used for acquiring an image sequence formed by each frame of image in the video;
the tracking coloring module is used for tracking and coloring each image in the image sequence by adopting a coloring model trained based on a deep learning technology;
and the generation module is used for generating the video after color conversion based on the image sequence after tracking coloring.
According to still another aspect of the present application, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described above.
According to yet another aspect of the present application, there is provided a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method as described above.
According to yet another aspect of the present application, there is provided a computer program product comprising a computer program which, when executed by a processor, implements a method as described above.
According to the technology of the application, an image sequence formed by each frame of image in the video is obtained; tracking and coloring each image in the image sequence by adopting a coloring model trained based on a deep learning technology; based on tracking the sequence of rendered images, color converted video is generated. By tracking coloring, the method and the device can ensure coloring continuity between continuous frame images in the video and improve accuracy of the video after color conversion.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The drawings are for better understanding of the present solution and do not constitute a limitation of the present application. Wherein:
FIG. 1 is a schematic diagram according to a first embodiment of the present application;
FIG. 2 is a schematic diagram according to a second embodiment of the present application;
FIG. 3 is a schematic diagram according to a third embodiment of the present application;
FIG. 4 is a schematic diagram according to a fourth embodiment of the present application;
FIG. 5 is a schematic diagram according to a fifth embodiment of the present application;
fig. 6 is a block diagram of an electronic device for implementing a method of color conversion of video in accordance with an embodiment of the present application.
Detailed Description
Exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
FIG. 1 is a schematic diagram according to a first embodiment of the present application; as shown in fig. 1, the present embodiment provides a video color conversion method, which specifically includes the following steps:
s101, acquiring an image sequence formed by each frame of image in a video;
s102, tracking and coloring each image in an image sequence by adopting a coloring model trained based on a deep learning technology;
s103, generating a video after color conversion based on the image sequence after tracking coloring.
The main implementation body of the video color conversion method in this embodiment is a video color conversion device, which may be an electronic entity, or may be an application that adopts software integration, and the application may perform color conversion on the video.
Firstly, an image sequence formed by each frame of images in a video is acquired, specifically, the video to be converted in color can be analyzed first, and each frame of image is acquired in sequence. And then arranging the images of each frame according to the sequence in the video to form an image sequence. I.e. images of all frames in the video are included in sequence in the image sequence.
Next, in this embodiment, a coloring model trained based on a deep learning technique may be used to track and color each image in the image sequence. The coloring model trained based on the deep learning technology is a neural network model, and tracking coloring of each image in the image sequence can be realized after the model is trained. The trace shading implemented by the shading model of this embodiment differs from the prior art shading of a single image in that: the coloring model may track successive frames in the video such that the coloring of the same object in the images of successive frames is the same. The coloring of the single image does not need to consider the problem of tracking coloring, so the coloring mode of the single image cannot be effectively applied to video.
Therefore, the coloring model based on the training of the deep learning technology in this embodiment not only can realize the independent coloring of each frame of image, but also can realize the tracking coloring, so that the colors of the same object in the continuous frame of images after coloring are the same.
In addition, when the coloring model is trained, countless pieces of training video data can be collected in advance, and each piece of training video data can comprise an uncolored video and a colored video marked by people; corresponding image sequences, namely an uncolored image sequence and a artificially annotated colored image sequence, are then generated based on the uncolored video and the colored video, respectively. During training, the colored image sequence of each training video is input into the coloring model, and the coloring model can color each image in the image sequence to obtain a colored image sequence. And then comparing whether the colored image sequence is consistent with the artificially marked colored image sequence, and if not, adjusting parameters of a coloring model to enable the two to be consistent. The coloring model is trained by adopting countless pieces of training video data, so that the image sequence after coloring treatment of the coloring model is consistent with the artificially marked coloring image sequence all the time, and at the moment, the network parameters of the coloring model can be determined, and further, the coloring model is determined.
Optionally, in the training process of each piece of training video data, after each frame of image in the image sequence is colored, comparing the colored current image with the colored current image marked manually, and if the colored current image is inconsistent with the colored current image marked manually, adjusting parameters of a coloring model, so that coloring of the colored image and the same image marked manually gradually tends to be consistent in the training process, and the training effect is improved.
And finally, tracking and coloring each image in the image sequence by adopting a coloring model, and generating a video after color conversion based on the image sequence obtained after tracking and coloring.
According to the color conversion method of the video, an image sequence formed by each frame of image in the video is obtained; tracking and coloring each image in the image sequence by adopting a coloring model trained based on a deep learning technology; based on tracking the sequence of rendered images, color converted video is generated. According to the embodiment, coloring continuity between continuous frame images in the video can be guaranteed through tracking coloring, and accuracy of the video after color conversion is improved.
FIG. 2 is a schematic diagram according to a second embodiment of the present application; as shown in fig. 2, the method for converting color of video according to the present embodiment is further described in more detail on the basis of the technical scheme of the embodiment shown in fig. 1. As shown in fig. 2, the color conversion method of the video in this embodiment may specifically include the following steps:
s201, acquiring an image sequence formed by each frame of image in a video;
s202, detecting whether a coloring request of a user exists; if so, execute step S203; otherwise, step S207 is performed;
in this embodiment, the user may initiate the coloring request, and at this time, coloring may be performed according to the coloring request of the user, and coloring may be performed on the color of the object that is not defined in the coloring request of the user based on the coloring model. The coloring model of the present embodiment is trained in advance by using a large amount of training data based on deep learning, so that many common coloring theoretic knowledge has been learned. For example, in a sunny day, the sky is blue and the clouds are white. The asian skin is yellow and the hair is black. In addition, the training data of the coloring model can be checked by an aesthetic agent, and the color matching in the coloring video marked in the training data is confirmed to be in accordance with the aesthetic of the masses, so that the color matching has the aesthetic feeling, and a user can see that the mind is happy. After the coloring model is trained by adopting the training data, the coloring model can learn the color matching with aesthetic feeling in the training process so as to prevent color collision in color conversion and avoid some embarrassing color matching.
In this embodiment, the coloring request of the user may carry the identification of the specified coloring object selected by the user and the target color of the specified coloring object. Such as a coloring request that the coat be colored red or that the hat be colored white, etc. The user can directly input the coloring request on the operation interface of the video color conversion device through the man-machine interface module, and at the moment, the identification of the coloring object in the coloring request can be the name of the coloring object. For another example, the user may click directly on a certain object in the video on the operation interface of the color conversion device of the video, and click right to select to color to a certain target color, where the color conversion device of the video may detect the name of the object clicked by the user or the unique identifier of the object in the video, and the target color selected by the user, and may consider that the user initiates the coloring request.
Optionally, in practical application, the user may not initiate a coloring request, and the coloring model performs color conversion on the video according to the learned coloring knowledge.
The step S201 and the step S202 of the present embodiment may not have a precedence order limitation.
S203, obtaining a coloring request of a user; step S204 is executed;
s204, detecting whether a coloring request of a user is hit in the first frame image; if yes, go to step S205; otherwise, step S207 is performed;
for example, it may be detected whether a specified coloring object in the user's coloring request is included in the first frame image, and if so, hit, otherwise miss.
S205, coloring a designated coloring object hitting the coloring request in the first frame image into a target color according to the target color in the coloring request; step S206 is executed;
s206, continuing to color the uncolored object in the first frame image by adopting a coloring model in the coloring model; step S208 is performed;
s207, coloring the first frame image by adopting a coloring model in the coloring model; step S208 is performed;
since the objects included in the first frame image may be very abundant, after the coloring of the object hit in the current first frame image is set to the target color according to the coloring request in step S205, other uncolored objects must still exist in the first frame image, and at this time, the uncolored objects in the first frame image may be continuously colored in step S206. If the first frame image does not hit the coloring request of the user, or if it is not detected that the user initiates the coloring request, the coloring model of the coloring model is directly used to color the first frame image according to step S207.
S208, for the second frame image to the last frame image in the image sequence, tracking coloring the corresponding mapping pixel point in the current image by adopting a tracking coloring model in the coloring model based on the mapping relation between the pixel point of the current image and the pixel point in the previous frame image according to the color value of the colored pixel point in the previous frame image; step S209 is performed;
s209, detecting whether the current image hits the coloring request of the user, if yes, executing step S210; if not, go to step S211; ,
similarly, it may be detected whether the current image includes a specified coloring object in the user's coloring request, if so, hit, otherwise miss.
S210, coloring a designated coloring object hitting the coloring request in the current frame image into a target color according to the target color in the coloring request; step S211 is performed;
s211, coloring the uncolored object in the current image by continuously adopting the coloring model.
After step S208, when the step is performed, other pixels in the current image, which have no mapping relation with the previous frame image, are colored by using the coloring model. After step S209, when the step is executed, the coloring model is used to color the object that does not have a mapping relationship with other pixels in the previous frame image and does not hit the coloring request.
In this embodiment, the coloring model includes a coloring model and a tracking coloring model, where the coloring model may learn to color an image based on training data. The image can also be colored when there is no coloring request. Wherein the coloring model may employ a generative antagonism network (Generative Adversarial Networks; GAN) model. Tracking shading models tracking (Tracking Emerges by Colorizing Videos; TECV) models by shading video may be employed. The TECV model may use a convolutional neural network (Convolutional Neural Networks, CNN) model to obtain all pixel point features of the previous and current images. And constructing a mapping matrix based on the information of each pixel point, so that the pixel point of the current image can be corresponding to the pixel point of the previous frame image. Then, the true color of the pixels of the previous frame image in a certain area can be mapped to the corresponding pixel points of the current image. After the colors of all pixels of the previous frame are mapped to the corresponding positions of the current image, the tracking coloring of the current image is completed. Since not all the pixels in the current image have corresponding pixels in the previous image, for example, there may be newly added objects in the current image, at this time, the coloring model is further required to be continuously adopted to color the uncolored objects in the current image in step S211.
The coloring model and the tracking coloring model of the present embodiment integrally constitute a coloring model, and the coloring model and the tracking coloring model are jointly trained as a whole at the time of training.
In practice, steps S208-S211 may be repeated until all images in the image sequence are rendered.
S212, generating a video after color conversion based on the colored image sequence.
Further optionally, the coloring request of this embodiment may further carry a specified timestamp, which is used to limit coloring of the specified coloring object in the image corresponding to the specified timestamp to the target color. For example, a user initiated coloring request may be to color flowers of a top page in a video to a reddish color.
Step S204 detects whether the first frame image hits the coloring request of the user, which may be to detect whether the timestamp corresponding to the first frame image is a specified timestamp in the coloring request, and detect whether the first frame image includes a specified coloring object in the coloring request of the user; if the time stamp corresponding to the first frame image is determined to be the appointed time stamp in the coloring request, and the first frame image comprises the appointed coloring object in the coloring request of the user, determining that the first frame image hits the coloring request of the user; otherwise, miss. Similarly, step S209 may also be detected in a similar manner. Hit detection in this way can ensure the accuracy of detection.
Alternatively, the coloring request may further carry a specified timestamp range, which is used for limiting coloring of a specified coloring object in the image corresponding to the specified timestamp range to a target color.
For example, step S209 may specifically be to detect whether the timestamp corresponding to the current frame image falls within the specified timestamp range in the coloring request, and detect whether the specified coloring object in the coloring request of the user is included in the current image; if the timestamp corresponding to the current image is determined to be in the range of the appointed timestamp in the coloring request, and the current image comprises the appointed coloring object in the coloring request of the user, determining that the current image hits the coloring request of the user at the moment; otherwise, miss. Similarly, step S204 may also be detected in a similar manner. Similarly, the hit detection in the mode can ensure the detection accuracy.
According to the color conversion method of the video, the video can be tracked and colored by adopting the coloring model and the tracking coloring model in the coloring model, so that the color conversion of the video is realized, the coloring consistency between continuous frame images in the video can be ensured, and the accuracy of the video after the color conversion is improved.
In addition, in the embodiment, the coloring request of the user can be obtained, and the manual intervention can be performed so as to meet the personalized coloring requirement of the user. The manual intervention process can be cooperated with the coloring model to color the video, so that the accuracy of color conversion of the video is effectively ensured.
Further, after a coloring request of a user is obtained, before coloring by adopting a coloring model, detecting whether each frame of image in an image sequence hits the coloring request or not, and coloring the current image according to the coloring request when hitting; and then tracking coloring is carried out by adopting a coloring model so as to color the video according to various coloring requirements, effectively ensure the accuracy of video color conversion and improve the quality of video color conversion.
FIG. 3 is a schematic diagram according to a third embodiment of the present application; as shown in fig. 3, the color conversion apparatus 300 of the video of the present embodiment includes:
an acquiring module 301, configured to acquire an image sequence formed by each frame of image in a video;
the tracking coloring module 302 is configured to perform tracking coloring on each image in the image sequence by using a coloring model trained based on a deep learning technology;
a generating module 303, configured to generate a video after color conversion based on tracking the sequence of images after coloring.
The implementation principle and the technical effect of the video color conversion device 300 of the present embodiment for implementing the video color conversion by using the above modules are the same as those of the above related method embodiments, and detailed descriptions of the above related method embodiments may be referred to and will not be repeated here.
FIG. 4 is a schematic diagram according to a fourth embodiment of the present application; as shown in fig. 4, the video color conversion device 300 of the present embodiment further describes the technical solution of the present application in more detail on the basis of the technical solution of the embodiment shown in fig. 3.
As shown in fig. 4, in the color conversion device 300 of the video of the present embodiment, the tracking coloring module 302 includes:
a first coloring unit 3021 for coloring a first frame image in the image sequence using a coloring model of the coloring models;
a second coloring unit 3022, configured to, for a second frame image to a last frame image in the image sequence, perform tracking coloring on a corresponding mapped pixel point in the current image according to a color value of a pixel point after coloring in the previous frame image by using a tracking coloring model in the coloring model based on a mapping relationship between the pixel point of the current image and the pixel point in the previous frame image;
the first coloring unit 3021 is further configured to continue coloring, by using the coloring model, other pixels in the current image that have no mapping relationship with the previous frame image.
Further alternatively, as shown in fig. 4, the color conversion device 300 of the video of the present embodiment further includes:
a detection module 304, configured to detect whether a coloring request of a user is hit in the first frame image; the coloring request comprises the mark of the selected appointed coloring object and the target color of the appointed coloring object;
the specified coloring module 305 is configured to color, if hit, the specified coloring object in the first frame image to the target color according to the target color.
Further optionally, the detection module 304 is configured to:
it is detected whether a specified coloring object corresponding to an identification of the specified coloring object in the coloring request of the user is included in the first frame image.
Further optionally, the obtaining module 301 is further configured to obtain a coloring request of the user.
Further optionally, the coloring request of the user further includes a designated timestamp;
a detection module 304, configured to detect whether the current image includes a specified coloring object corresponding to an identifier of the specified coloring object in the coloring request of the user, and whether a timestamp of the current image hits the specified timestamp;
the specified coloring module 305 is configured to color, if yes, the specified coloring object in the current image to the target color according to the target color.
The implementation principle and the technical effect of the video color conversion device 300 of the present embodiment for implementing the video color conversion by using the above modules are the same as those of the above related method embodiments, and detailed descriptions of the above related method embodiments may be referred to and will not be repeated here.
FIG. 5 is a schematic diagram according to a fifth embodiment of the present application; the present embodiment provides a video processing platform, as shown in fig. 5, where the video processing platform 500 of the present embodiment may be an application platform of the video color conversion device. As shown in fig. 5, the video processing platform 500 may include a video upload module 501, a labeling module 502, a model training module 503, and a video color conversion device 504. The video color conversion device 504 may employ the video color conversion device of the embodiment shown in fig. 3 or fig. 4.
For example, the video uploading module 501 is configured to upload video to be processed and video of training data. The labeling module 502 is configured to label training data required for training the coloring model, and may be a manually labeled coloring training video, as described above in connection with the embodiment shown in fig. 1. The model training module 503 is configured to respectively obtain an uncolored image sequence and a manually labeled colored image sequence based on the uncolored training video uploaded by the video uploading module 501 and the colored training video labeled by the labeling module 502, and train the colored model. As noted above with respect to the embodiment of FIG. 2, the coloring model may include a coloring model and a tracking coloring model. The training method of the embodiment shown in fig. 1 can be used to train the coloring model. When the color conversion device 504 of the video performs color processing on the video, the video to be processed uploaded by the video uploading module 501 may be obtained, and the coloring model obtained by training by the model training module 503 may be called to perform color conversion processing on the video according to the processing manner of the embodiment shown in fig. 1. Alternatively, the labeling module 502 may also be used to label a user's coloring request, e.g., the user may employ the module to label a target color of a specified coloring object in a video, i.e., to generate the user's coloring request. Correspondingly, the color conversion device 504 of the video may perform color conversion on the video according to the coloring request of the user marked by the marking module 502, and further call the coloring model obtained by training by the model training module 503 to perform color conversion processing on the video according to the processing manner of the embodiment shown in fig. 2. Reference may be made in detail to the description of the related embodiments, which are not repeated here.
According to embodiments of the present application, an electronic device and a readable storage medium are also provided.
As shown in fig. 6, a block diagram of an electronic device implementing a video color conversion method according to an embodiment of the present application is shown. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the application described and/or claimed herein.
As shown in fig. 6, the electronic device includes: one or more processors 601, memory 602, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple electronic devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). One processor 601 is illustrated in fig. 6.
Memory 602 is a non-transitory computer-readable storage medium provided herein. Wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the video color conversion methods provided herein. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to perform the color conversion method of video provided by the present application.
The memory 602 is used as a non-transitory computer readable storage medium for storing non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules (e.g., related modules shown in fig. 4 and 5) corresponding to the video color conversion method in the embodiments of the present application. The processor 601 executes various functional applications of the server and data processing, i.e., implements the color conversion method of video in the above-described method embodiments, by running non-transitory software programs, instructions, and modules stored in the memory 602.
The memory 602 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, at least one application program required for a function; the storage data area may store data created according to the use of an electronic device implementing a color conversion method of video, and the like. In addition, the memory 602 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 602 optionally includes memory remotely located relative to processor 601, which may be connected via a network to an electronic device implementing the video color conversion method. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device implementing the video color conversion method may further include: an input device 603 and an output device 604. The processor 601, memory 602, input device 603 and output device 604 may be connected by a bus or otherwise, for example in fig. 6.
The input device 603 may receive input numeric or character information and generate key signal inputs related to user settings and function controls of the electronic device implementing the color conversion method of video, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointer stick, one or more mouse buttons, a track ball, a joystick, etc. input devices. The output means 604 may include a display device, auxiliary lighting means (e.g., LEDs), tactile feedback means (e.g., vibration motors), and the like. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASIC (application specific integrated circuit), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
These computing programs (also referred to as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), the internet, and blockchain networks.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the application, an image sequence formed by each frame of image in the video is obtained; tracking and coloring each image in the image sequence by adopting a coloring model trained based on a deep learning technology; based on tracking the sequence of rendered images, color converted video is generated. According to the embodiment, coloring continuity between continuous frame images in the video can be guaranteed through tracking coloring, and accuracy of the video after color conversion is improved.
According to the technical scheme, the video can be tracked and colored by adopting the coloring model and the tracking coloring model in the coloring model, so that the color conversion of the video is realized, the coloring consistency between continuous frame images in the video can be ensured, and the accuracy of the video after the color conversion is improved.
Moreover, according to the technical scheme of the embodiment of the application, the coloring request of the user can be obtained, and the manual intervention can be performed so as to meet the personalized coloring requirement of the user. The manual intervention process can be cooperated with the coloring model to color the video, so that the accuracy of color conversion of the video is effectively ensured.
Further, according to the technical scheme of the embodiment of the application, after a coloring request of a user is obtained, before coloring by adopting a coloring model, whether each frame of image in an image sequence hits the coloring request or not is detected, and when the image hits, the current image is colored according to the coloring request; and then tracking coloring is carried out by adopting a coloring model so as to color the video according to various coloring requirements, effectively ensure the accuracy of video color conversion and improve the quality of video color conversion.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, provided that the desired results of the technical solutions disclosed in the present application can be achieved, and are not limited herein.
The above embodiments do not limit the scope of the application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application are intended to be included within the scope of the present application.
Claims (8)
1. A method of color conversion of video, wherein the method comprises:
acquiring an image sequence formed by each frame of image in a video;
tracking and coloring each image in the image sequence by adopting a coloring model trained based on a deep learning technology;
generating a color converted video based on tracking the sequence of rendered images;
wherein, adopt the coloring model based on the training of deep learning technique, carry out the tracking coloring to each image in the image sequence, include:
coloring a first frame of image in the image sequence by adopting a coloring model in the coloring model;
for the second frame image to the last frame image in the image sequence, tracking coloring the corresponding mapped pixel point in the current image according to the color value of the colored pixel point in the previous frame image by adopting a tracking coloring model in the coloring model based on the mapping relation between the pixel point of the current image and the pixel point in the previous frame image;
and continuing to adopt the coloring model to color other pixel points which have no mapping relation with the previous frame image in the current image;
wherein, before coloring the first frame of images in the image sequence with a coloring model of the coloring model, the method further comprises:
detecting whether a coloring request of a user is hit in the first frame image; the coloring request comprises the identification of the selected specified coloring object and the target color of the specified coloring object;
if hit, coloring the designated coloring object in the first frame image into the target color according to the target color;
wherein the coloring request of the user also comprises a designated time stamp;
further, after tracking coloring is performed on the corresponding mapped pixel point in the current image according to the color value of the pixel point colored in the previous frame image, and before the coloring model is continuously adopted to color other pixel points in the current image, which have no mapping relation with the previous frame image, the method further comprises:
detecting whether the current image comprises a specified coloring object corresponding to the identification of the specified coloring object in the coloring request of the user, and whether the timestamp of the current image hits the specified timestamp;
if so, coloring the appointed coloring object in the current image into the target color according to the target color.
2. The method of claim 1, wherein detecting whether the user's coloring request is hit in the first frame image comprises:
detecting whether the first frame image comprises a specified coloring object corresponding to the identification of the specified coloring object in the coloring request of the user.
3. The method of claim 1, wherein prior to detecting whether the user's coloring request is hit in the first frame image, the method further comprises:
and obtaining a coloring request of the user.
4. A color conversion device for video, wherein the device comprises:
the acquisition module is used for acquiring an image sequence formed by each frame of image in the video;
the tracking coloring module is used for tracking and coloring each image in the image sequence by adopting a coloring model trained based on a deep learning technology;
the generation module is used for generating a video after color conversion based on the image sequence after tracking coloring;
wherein the trace shading module comprises:
a first coloring unit configured to color a first frame image in the image sequence using a coloring model in the coloring model;
the second coloring unit is used for tracking and coloring corresponding mapping pixel points in the current image according to the color value of the pixel points colored in the previous frame image by adopting a tracking coloring model in the coloring model based on the mapping relation between the pixel points of the current image and the pixel points in the previous frame image for the second frame image to the last frame image in the image sequence;
the first coloring unit is further configured to continue to use the coloring model to color other pixel points in the current image that have no mapping relationship with the previous frame image;
wherein the apparatus further comprises:
the detection module is used for detecting whether the coloring request of the user is hit in the first frame image; the coloring request comprises the identification of the selected specified coloring object and the target color of the specified coloring object;
a designated coloring module, configured to, if hit, color the designated coloring object in the first frame image to the target color according to the target color;
wherein the coloring request of the user also comprises a designated time stamp;
the detection module is used for detecting whether the current image comprises a specified coloring object corresponding to the identification of the specified coloring object in the coloring request of the user, and whether the timestamp of the current image hits the specified timestamp;
and the specified coloring module is used for coloring the specified coloring object in the current image into the target color according to the target color if the specified coloring object is the target color.
5. The apparatus of claim 4, wherein the detection module is configured to:
detecting whether the first frame image comprises a specified coloring object corresponding to the identification of the specified coloring object in the coloring request of the user.
6. The apparatus of claim 4, wherein the means for obtaining is further configured to obtain a coloring request of the user.
7. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3.
8. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010537912.8A CN111862250B (en) | 2020-06-12 | 2020-06-12 | Video color conversion method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010537912.8A CN111862250B (en) | 2020-06-12 | 2020-06-12 | Video color conversion method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111862250A CN111862250A (en) | 2020-10-30 |
| CN111862250B true CN111862250B (en) | 2023-07-21 |
Family
ID=72986545
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010537912.8A Active CN111862250B (en) | 2020-06-12 | 2020-06-12 | Video color conversion method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111862250B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112884866B (en) * | 2021-01-08 | 2023-06-06 | 北京奇艺世纪科技有限公司 | Coloring method, device, equipment and storage medium for black-and-white video |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104992418A (en) * | 2015-07-07 | 2015-10-21 | 华东理工大学 | Abnormal color correction method applicable to thermal imaging video colorization |
| CN110503725A (en) * | 2019-08-27 | 2019-11-26 | 百度在线网络技术（北京）有限公司 | Method, apparatus, electronic equipment and the computer readable storage medium of image procossing |
| WO2019241346A1 (en) * | 2018-06-13 | 2019-12-19 | Google Llc | Visual tracking by colorization |
| CN111080746A (en) * | 2019-12-10 | 2020-04-28 | 中国科学院计算技术研究所 | Image processing method, image processing device, electronic equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10623709B2 (en) * | 2018-08-31 | 2020-04-14 | Disney Enterprises, Inc. | Video color propagation |
-
2020
- 2020-06-12 CN CN202010537912.8A patent/CN111862250B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104992418A (en) * | 2015-07-07 | 2015-10-21 | 华东理工大学 | Abnormal color correction method applicable to thermal imaging video colorization |
| WO2019241346A1 (en) * | 2018-06-13 | 2019-12-19 | Google Llc | Visual tracking by colorization |
| CN110503725A (en) * | 2019-08-27 | 2019-11-26 | 百度在线网络技术（北京）有限公司 | Method, apparatus, electronic equipment and the computer readable storage medium of image procossing |
| CN111080746A (en) * | 2019-12-10 | 2020-04-28 | 中国科学院计算技术研究所 | Image processing method, image processing device, electronic equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| "Tracking Emerges by Colorizing Videos";Carl Vondrick等;《Computer Vision – ECCV 2018》;第402-419页 * |
| 结合像素流和最优化方法的视频着色系统;吴利杰;丁友东;陈钰;梁欢;陈方灵;;电子测量技术(第06期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111862250A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112541963B (en) | Three-dimensional avatar generation method, three-dimensional avatar generation device, electronic equipment and storage medium | |
| CN111598818B (en) | Training method and device for face fusion model and electronic equipment | |
| CN111563855B (en) | Image processing method and device | |
| CN110806865B (en) | Animation generation method, device, equipment and computer readable storage medium | |
| CN111722245B (en) | Positioning method, positioning device and electronic equipment | |
| US11074437B2 (en) | Method, apparatus, electronic device and storage medium for expression driving | |
| US20210398334A1 (en) | Method for creating image editing model, and electronic device and storage medium thereof | |
| CN111783647A (en) | Training method of face fusion model, face fusion method, device and equipment | |
| CN112149741B (en) | Training method and device for image recognition model, electronic equipment and storage medium | |
| CN111709873B (en) | Training method and device for image conversion model generator | |
| CN111783620A (en) | Expression recognition method, device, equipment and storage medium | |
| CN111968203B (en) | Animation driving method, device, electronic equipment and storage medium | |
| CN112634282B (en) | Image processing method and device and electronic equipment | |
| CN112001248B (en) | Active interaction method, device, electronic equipment and readable storage medium | |
| CN111709875B (en) | Image processing method, device, electronic equipment and storage medium | |
| US11641446B2 (en) | Method for video frame interpolation, and electronic device | |
| CN111539897A (en) | Method and apparatus for generating image conversion model | |
| CN111710008B (en) | Method and device for generating people stream density, electronic equipment and storage medium | |
| CN111862250B (en) | Video color conversion method and device, electronic equipment and storage medium | |
| CN112131414A (en) | Signal lamp image labeling method and device, electronic equipment and road side equipment | |
| CN112508964B (en) | Image segmentation method, device, electronic equipment and storage medium | |
| US20210279928A1 (en) | Method and apparatus for image processing | |
| CN116403285A (en) | Action recognition method, device, electronic equipment and storage medium | |
| CN112508830B (en) | Training method, device, equipment and storage medium of image processing model | |
| CN116167426A (en) | Training method of face key point positioning model and face key point positioning method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |