JP2023504926A - Image processing method, device, electronic device and computer-readable storage medium - Google Patents

Abstract

An image processing method, apparatus, electronic device and computer readable storage medium, wherein the image processing method comprises: a first object in a first video frame image; and overlaying a third object as a foreground image on the first video frame image based on the first object's position in the first video frame image. to obtain a second video frame image, wherein the third object covers the first object in the second video frame image (S102); superimposing the second object as a foreground image on a third object in the second video frame image based on the position of the object in the first video frame image to obtain a third video frame image; (S103). The above method solves the technical problem of the lack of realism in the prior art, which is caused by the fact that the effect overlaid on the object cannot represent the original feature of the object. [Selection diagram] Fig. 1

Description

[Cross reference to related applications]
This disclosure is filed with the Chinese Patent Office on Dec. 18, 2019, with application number 201911306421.6 and titled "Image Processing Method, Apparatus, Electronic Apparatus and Computer Readable Storage Medium". Priority is claimed from a Chinese patent application, the entire content of which is incorporated into this disclosure by reference.

[Technical field]
FIELD OF THE DISCLOSURE The present disclosure relates to the field of image processing, and more particularly to image processing methods, apparatus, electronic devices, and computer-readable storage media.

With the development of computer networks and the popularization of smart phones, general users are not satisfied with expressing their emotions only with monotonous images and characters. Video presents more diverse content and formats and can bring intuitive feeling, so it is deeply favored by users and gradually becomes popular. become a trend. However, while original videos such as selfie videos have a dry expression format, the use of video effects in film and television works is becoming more and more abundant, and the content expression format is also more diverse. can be said to underpin and guarantee successful film and television productions.

However, in the prior art, it is common to create an effect by directly covering the target object (for example, a person's face) with the effect, but the covered part is blocked by the effect, resulting in , so the actual effect of the effect is unnatural and unrealistic.

This Summary of the Invention is provided to introduce concepts in a simplified form that are described in detail in the Detailed Description that follows. This summary of the invention is not intended to mark the key features or essential features of the technical solution to be patented, nor is it intended to limit the scope of the technical solution to be patented. not

According to a first aspect, embodiments of the present disclosure include:
identifying a first object in a first video frame image and a second object located within the first object;
obtaining a second video frame image by overlaying a third object as a foreground image on the first video frame image based on the position of the first object in the first video frame image; wherein the third object covered the first object in the second video frame image;
superimposing the second object as a foreground image on a third object in the second video frame image based on the position of the second object in the first video frame image to produce a third video; and obtaining a frame image.

According to a second aspect, embodiments of the disclosure include:
an object identification module for identifying a first object in a first video frame image and a second object located within said first object;
overlaying a third object as a foreground image on the first video frame image based on the position of the first object in the first video frame image, wherein the third object is the first video frame image; a second video frame image generation module for obtaining a second video frame image covering the object;
superimposing the second object as a foreground image on a third object in the second video frame image based on the position of the second object in the first video frame image to produce a third video; and a third video frame generation module for obtaining frame images.

According to a third aspect, embodiments of the disclosure include:
at least one processor;
a memory communicatively coupled to the at least one processor;
The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the at least one processor to perform any of the first aspects. An electronic device capable of executing an image processing method is provided.

According to a fourth aspect, an embodiment of the present disclosure is characterized by storing computer instructions for causing a computer to perform the image processing method of any of the first aspects. A computer-readable storage medium is provided.

According to a fifth aspect, embodiments of the present disclosure provide a computer program product which, when executed by a computer, causes said computer to perform said image processing method of any of said first aspect.

An image processing method, apparatus, electronic device and computer-readable storage medium are disclosed in embodiments of the present disclosure. The image processing method includes identifying a first object and a second object located within the first object in a first video frame image; overlaying a third object as a foreground image on the first video frame image based on its position in the frame image to obtain a second video frame image, wherein the second video frame image is the second object as a foreground image based on the third object covering the first object in the image and the position of the second object in the first video frame image; superimposing on the third object in the video frame image of , to obtain a third video frame image. The above method solves the technical problem of the lack of realism in the prior art, which is caused by the fact that the effect overlaid on the object cannot represent the original feature of the object.

The above description is merely an overview of the technical solution of the present disclosure, in order to make the technical means of the present disclosure more clearly understood, it can be implemented based on the content of the specification, and the above and other purposes of the present disclosure In order to make the features and advantages clearer and easier to understand, preferred embodiments are described in detail below in conjunction with the accompanying drawings.

The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent with reference to the following detailed description, taken in conjunction with the accompanying drawings. In all the drawings, the same or similar elements are indicated by the same or similar reference numerals. It is to be understood that the accompanying drawings are schematic and elements and components are not necessarily drawn to scale.

4 is a flow chart of an embodiment of an image processing method according to the present disclosure; FIG. 4 is a schematic diagram of calculating positioning points of a person's face in an embodiment of the image processing method according to the present disclosure; 4 is a flowchart of a specific example of step S103 in an embodiment of the image processing method according to the present disclosure; FIG. 4 is a flow chart of an example of step S302 in an embodiment of the image processing method according to the present disclosure; FIG. 1 is a structural schematic diagram of an embodiment of an image processing device according to an embodiment of the present disclosure; FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present disclosure; FIG.

Embodiments of the present disclosure are described in more detail below with reference to the accompanying drawings. Although several embodiments of the disclosure are illustrated in the accompanying drawings, the disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Please understand. Rather, these examples are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and examples of the present disclosure are only illustrative and do not limit the protection scope of the present disclosure.

It should be understood that the steps described in the method embodiments of the present disclosure may be performed in different orders and/or in parallel. It is noted that method embodiments may include additional steps and/or omit performance of the indicated steps. The scope of the disclosure is not limited in this respect.

As used herein, the term "including" and variations thereof is open-ended, meaning "including, but not limited to." The term "based on" means "based at least in part on." The term "one embodiment" means "at least one embodiment" and the term "another embodiment" means "at least one other embodiment" and "some embodiments". The term means "at least some embodiments." Relevant definitions of other terms are provided in the description below.

It should be noted that the terms “first”, “second”, etc. described in this disclosure are merely for distinguishing between different devices, modules or units, and by these devices, modules or units It is not intended to limit the order or interdependencies of the functions performed.

It should be noted that the "one" and "plurality" modifications described in this disclosure are schematic and are not limiting. It is obvious to those skilled in the art that "one or more" should be understood unless specifically stated otherwise.

The names of messages or information exchanged between devices in embodiments of the present disclosure are descriptive only and do not limit the scope of these messages or information.

FIG. 1 is a flow chart of an embodiment of an image processing method according to the present disclosure. This image processing method according to this embodiment may be performed by one image processing device. This image processing device may be implemented as software, or as a combination of software and hardware. This image processing apparatus may be integrated and installed in a device such as an image processing server or an image processing terminal device in an image processing system. As shown in FIG. 1, the method includes the following steps.
Step S101: Identifying a first object in a first video frame image and a second object located within said first object.

In this step, the first video frame image is acquired by an image sensor or memory. Among them, the image sensor refers to various devices capable of capturing images, and typical image sensors are video cameras, camera heads, cameras, and the like. In this embodiment, the image sensor may be a camera head on a terminal device, for example, a front-facing or rear-facing camera head on a smartphone, and the image captured by the camera head may be displayed on the display screen of the mobile phone as it is. can be done. wherein the memory is a local storage or a network storage, the video frame image is obtained from a storage location in the local storage or a storage address in the network storage, and the video frame image in the memory is a pre-captured video frame image file; and can be displayed on the display device of the terminal device by the player of the terminal device.

The first video frame image includes a first object, and the first object may be any object in the first video frame image. Illustratively, said first object is a human face and said second object is part of said first object and is located within said first object. Illustratively, when the first object is a human face, the second object is the five senses of the human face, such as the eyes and mouth.

In this example, the keypoints of the object are used to identify the object. Illustratively, the object may be identified based on a plurality of preset keypoints. When the plurality of keypoints are identified, the object is identified. Optionally, said identifying a first object in said first video frame image and a second object located within said first object comprises said second object in said first video frame image. identifying a plurality of first keypoints of a subject and a plurality of second keypoints of the second subject; and based on the plurality of first keypoints of the first subject, and identifying the second object based on a plurality of second keypoints of the second object, wherein the plurality of second keypoints are edge keypoints of the second object; wherein said edge keypoints identify the contour of said second object; Illustratively, the first object is a human face, and the second object is the eyes and mouth of the human face. The plurality of first keypoints are an eye center keypoint and a nose tip keypoint of a human face, and when the eye center keypoint and the nose tip keypoint are identified, the human face is identified. good too. The plurality of second keypoints are the eye edge keypoints and the mouth edge keypoints, and similarly if the eye edge keypoints and the mouth edge keypoints, the eyes and mouth are identified. be. Illustratively, the first keypoint and the second keypoint can be identified using a deep learning algorithm, and the set of frame images in which the first keypoint and the second keypoint are marked. can be directly used to train a deep learning network to obtain a network model that can regress the first keypoint and the second keypoint, and then input the first video frame image to the network model. and if the first video frame image contains a first object and a second object, the network model outputs the positions of the first keypoint and the second keypoint, whereby the first The first and second objects are directly identified, and the positions of the first and second keypoints are identified. It should be understood that any other keypoint identification algorithm may be used to identify the first and second keypoints and will not be further described here.

Step S102: Overlay a third object as a foreground image on the first video frame image based on the position of the first object in the first video frame image, such that the third object is the A second video frame image is obtained covering the first object.

optionally, said third object is a three-dimensional virtual object, and the area of the two-dimensional projection in said first video frame image of said three-dimensional virtual object is larger than the area of said first object; That is, the third object can completely cover the first object. Exemplarily, the 3D virtual object may be a spherical virtual object such as a 3D orange, a 3D ball, a 3D watermelon. The third object includes a third object positioning point for positioning the position of the third object.

The third object is not limited to the three-dimensional virtual object described above, and may be any object such as a two-dimensional virtual object or a real object. For example, a first video frame image includes a human facial image and an animal facial image, wherein the animal facial image is the third object, the human facial image. It should be understood that it may be overlaid on the image.

In this step, a method of single keypoint alignment may be used to identify locations covered by a third object when overlaying the third object onto the first object. Illustratively, the third object includes a third object positioning point, and the first object is pre-configured with a first object positioning key point corresponding to the third object positioning point. and the first target positioning keypoint may be one of the first keypoints. by aligning the positioning points of the third object with the positions of the positioning key points of the first object when overlaying the third object as a foreground image on the first video frame image; Three objects can be overlaid on the first object.

Optionally, foreground a third object based on said position of said first object in said first video frame image, as alignment using a single keypoint may not be very accurate. Overlaying the first video frame image as an image to obtain a second video frame image comprises: from a plurality of first keypoints of the first object in the first video frame image, the first calculating the positioning point of the first object in one video frame image; and aligning the positioning point of the third object with the positioning point of the first object in the first video frame image. overlaying the third object as a foreground image on the first video frame image to obtain the second video frame image. In this alternative embodiment, the positioning point of the first object is calculated from a plurality of first keypoints, thus the positioning point of the first object is the position of the plurality of first keypoints. Because the information is included, the positioning is more accurate than a single primary keypoint. Exemplarily, as shown in FIG. 2 , the first object is a human face, the plurality of first keypoints are left and right eye center keypoints 201 and 202, and a nose tip keypoint 203; The positioning point of the person's face can be calculated such that the eye center key point and the nose tip key point each occupy 1/3 of the weight, thus calculating the eye center key point and the nose tip key point respectively. point and the tip of the nose keypoint, thus the distortion of the position of a single keypoint has a fairly small effect on the position of the positioning point, and the positioning point of the first object is more stable and accurate.

After obtaining the positioning point of the first object, setting the positioning point of the third object at the position of the positioning point of the first object, and rendering a first video frame image with the third object as the foreground. to obtain a second video frame image.
Step S103: superimposing the second object as a foreground image on a third object in the second video frame image according to the position of the second object in the first video frame image; 3 video frame images are obtained.

A second object has already been identified in step S101, wherein the position of the second object in the first video frame image is the position of the center point of the second object or the plurality of second key points. exemplarily, in identifying the second object, directly identifying the positioning point of the second object as a keypoint, or from the plurality of second keypoints, the calculating a positioning point of a second object; setting a weight value for each second key point involved in the calculation when calculating the positioning point of the second object; is a weighted average of the plurality of second keypoint positions. After obtaining the position of the second object in the first video frame image, the second object is superimposed as a foreground image on the corresponding position of the third object in the second video frame image.

Optionally, if said third object is a 3D object, the 3D object has constant depth (Z-axis information) and the second object is a 2D object and has no depth. Therefore, when the second object is superimposed on the third object, a certain deviation occurs, so that the superimposition result becomes relatively natural when the second object is superimposed on the third object. Therefore, it is necessary to have a certain amount of offset. At this time, as shown in FIG. 3, step S103 may include the following steps.

Step S301: Calculate a first distance between two preset keypoints on the first object.

Step S302: Calculate the offset amount of the second target based on the first distance.

Step S303: Based on the position of the second object in the first video frame image and the amount of offset of the second object, the third object in the second video frame image with the second object as the foreground image. to obtain a third video frame image.

The two preset keypoints on the first object may be any two keypoints on the first object, exemplarily, the two preset keypoints are: Two symmetrical keypoints on the first object, for example, in the above example, if the first object is a human face, the two preset keypoints are the centers of the left and right eyes. It can be a key point. Illustratively, calculating the offset amount of the second object based on the first distance may be multiplying the first distance by one offset factor, and the The offset factor is related to the value of said first distance, e.g., the greater the value of the first distance, the greater the offset factor, and thus the greater the amount of offset of the second object, thereby causing the first This has the effect that the closer the object is to the lens, the more the second object is offset to accommodate the three-dimensional surface of the third object.

The calculation of the offset amount described above is an approximate calculation, and optionally, the offset amount may be calculated more accurately. At this time, as shown in FIG. may include steps.

Step S401: obtaining the yaw angle and pitch angle of the first object, wherein the yaw angle is the horizontal included angle between the orientation of the first object in the first video frame image and the lens shooting direction; A, wherein the pitch angle is a vertical included angle between an orientation in the first video frame image of the first object and a lens shooting direction.

Step S402: Calculate the offset amount of the second object based on the first distance, yaw angle and pitch angle.

ここで、ｓｉｇｎは、符号を取る操作であり、ｄｘは、前記第２の対象のＸ軸方向のオフセット量であり、ｄｙは、前記第２の対象のＹ軸方向のオフセット量であり、ｄｉｓｔ_ｅｙｅは第１の距離であり、ｙａｗは、第１の対象のヨー角であり、ｐｉｔｃｈは、第１の対象のピッチ角であり、θ_１は、第２の対象が前方を向っているときの初期相対ピッチ角であり、θ_２は、第２の対象が前方を向っているときの初期相対ヨー角である。ここで、θ_１とθ_２は、予め設定された角度値であり、３次元の第３の対象の外面は一定のラジアンを有するので、移動する際に、ある程度の減衰が必要である。第１の対象が人の顔で、第２の対象が目と口である場合を例にして、口が第３の対象における上下領域の下半分の領域に位置するため、口の初期相対ピッチ角は大きく、口が第３の対象の左右領域の中間に位置するため、口の初期相対ヨー角は０である。他の第２の対象のθ_１とθ_２は、第３の対象の違いに応じて予め異なる値に設定されてもよく、ここではこれ以上説明しない。 Specifically, the calculation procedure in step S402 can be represented by the following equation:

Here, sign is a sign operation, dx is the offset amount of the second object in the X-axis direction, dy is the offset amount of the second object in the Y-axis direction, and dist _eye is the first distance, yaw is the yaw angle of the first object, pitch is the pitch angle of the first object, and θ ₁ is when the second object is facing forward. and θ ₂ is the initial relative yaw angle when the second object is facing forward. where θ ₁ and θ ₂ are preset angular values, and since the outer surface of the 3D third object has a constant radian, some attenuation is required when moving. Assuming that the first object is a human face and the second object is eyes and mouth, since the mouth is located in the lower half region of the upper and lower regions of the third object, the initial relative pitch of the mouth is The initial relative yaw angle of the mouth is zero because the corners are large and the mouth is located halfway between the left and right regions of the third object. θ ₁ and θ ₂ of other second targets may be set to different values in advance according to different third targets, and will not be further described here.

Optionally, in said step S303, said second object as a foreground image based on said second object's position in said first video frame image and said second object's offset amount. obtaining a third video frame image overlaid on a third object in the video frame image of said first video frame of said second object based on an offset amount of said second object position-shifting an initial positioning point in an image to obtain a target positioning point; rendering the second object as a foreground image at the position of the target positioning point in the second video frame image; obtaining a video frame image of . In the above step, the position of the second object in the first video frame image is defined as an initial positioning point, the initial positioning point includes an X-axis coordinate value and a Y-axis coordinate value, and the X-axis of the initial positioning point. adding an axial coordinate value and a Y-axis coordinate value to the X-axis coordinate value and the Y-axis coordinate value of the offset amount respectively to obtain a target positioning point; as the foreground at the position of the second target positioning point to obtain a third video frame image, superimposing the second object on the third object in the third video frame image; The characteristics of the first object can be realistically reflected, thereby making the effect more realistic.

Optionally, before superimposing said second object as a foreground image onto said third object in said second video frame image, since said third object and said second object may be of different sizes. , further comprising scaling said second object according to a third object. At this time, based on the ratio of the distance between two predetermined keypoints on the third object and the distance between two predetermined keypoints on the first object, the scaling factor of the second object is and thus, if the third object is larger than the first object, enlarge the second object, and if the third object is smaller than the first object, enlarge the Reduce the target of 2. The second object superimposed on the third object in the second video frame image after this step is the scaled second object.

Furthermore, since there may be differences between the colors of the target object and the colors of the 3D virtual object after the target object is overlaid on the 3D virtual object, further color processing may be used to eliminate such differences. By canceling, the effect of superimposing the second object and the third object can be made more natural. Furthermore, after step S103,
The color space of the second object and the third object in the third video frame image is changed from RGB (Red, Green, Blue; red, green, blue) color space to HSV HSV (Hue, Saturation, Value; hue, (saturation, lightness) color space;
replacing H channel values in the HSV color space of the second object with H channel values of the third object;
Converting the color space of the second object and the third object from HSV space to RGB color space to obtain a fourth video frame image.

said step transforming the color of the second object to the color of the third object at said target position if the RGB space is transformed into the HSV space and the original saturation and lightness of the second object are preserved; In this way, it is possible to further increase the degree of fusion between the second object and the third object, make the second object appear to be part of the third object, and make it more realistic. can.

An image processing method, apparatus, electronic device and computer-readable storage medium are disclosed in embodiments of the present disclosure. Among which, the image processing method includes identifying a first object in a first video frame image and a second object located within the first object; overlaying a third object as a foreground image on the first video frame image based on the position in the video frame image of the second video frame with the third object covering the first object obtaining an image and superimposing the second object as a foreground image over a third object in the second video frame image based on the position of the second object in the first video frame image; to obtain a third video frame image. The above method solves the technical problem of the lack of realism caused by the fact that the effect overlaid on the object cannot represent the original feature of the object in the prior art.

Although the steps in the above method embodiments have been described in the above order, the steps in the embodiments of the present disclosure do not necessarily have to be performed in the above order, but in other orders such as reverse, parallel, cross, etc. It will be obvious to those skilled in the art what may be performed, and further, based on the steps described above, those skilled in the art may add other steps, these obvious variations or equivalent substitution schemes are , should be included in the protection scope of the present disclosure and will not be further described here.

FIG. 5 is a structural schematic diagram of an embodiment of an image processing device according to an embodiment of the present disclosure. As shown in FIG. 5, the apparatus 500 includes an object identification module 501 , a second video frame image generation module 502 and a third video frame generation module 503 . Among them
the object identification module 501 is for identifying a first object in a first video frame image and a second object located within said first object;
A second video frame image generation module 502 overlays a third object as a foreground image on the first video frame image based on the position of the first object in the first video frame image. , for obtaining a second video frame image in which the third object covers the first object;
A third video frame generation module 503 generates a third object in the second video frame image with the second object as a foreground image based on the position of the second object in the first video frame image. Superimposed on top to obtain a third video frame image.

Furthermore, the object identification module 501 further:
identifying a plurality of first keypoints of the first object and a plurality of second keypoints of the second object in the first video frame image;
identifying the first object based on a plurality of first keypoints of the first object;
and identifying the second object based on a plurality of second keypoints of the second object that are edge keypoints of the second object.

Further, the second video frame image generation module 502 may include:
calculating a positioning point of the first object in the first video frame image based on a plurality of first keypoints of the first object in the first video frame image;
aligning the positioning point of the third object with the positioning point of the first object in the first video frame image, with the third object as a foreground image on the first video frame image; to obtain the second video frame image.

Additionally, the second video frame image generation module 502 further comprises:
calculating a first distance between two preset keypoints on the first object;
calculating an offset amount of the second target based on the first distance;
above a third object in the second video frame image with the second object as the foreground image based on the position of the second object in the first video frame image and the amount of offset of the second object; and obtaining a third video frame image by superimposing the .

Furthermore, calculating the offset amount of the second target based on the first distance,
obtaining a yaw angle and a pitch angle of the first object, wherein the yaw angle is a horizontal included angle between an orientation of the first object in the first video frame image and a lens shooting direction; a pitch angle is a vertical included angle between an orientation in the first video frame image of the first object and a lens shooting direction;
calculating an offset amount of the second object based on the first distance, yaw angle and pitch angle.

Further, based on the position of the second object in the first video frame image and the offset amount of the second object, the second object in the second video frame image with the second object as the foreground image Obtaining a third video frame image superimposed on the third object
position-shifting an initial positioning point in the first video frame image of the second object to obtain a target positioning point based on the offset amount of the second object;
rendering the second object as a foreground image in the second video frame image at the location of the target positioning point to obtain the third video frame image.

Furthermore, the image processing device 500
converting the color space of the second object and the third object in the third video frame image from the RGB color space to the HSV color space; replacing the H channel values of a third object; and converting the color spaces of the second and third objects from the HSV space to the RGB color space to obtain a fourth video frame image. It further includes a fourth video image generation module that is used.

The apparatus shown in FIG. 5 is capable of carrying out the method of the embodiment shown in FIGS. 1-4, and for those parts not described in detail in this embodiment, the description of the embodiment shown in FIGS. 1-4 is repeated. can be referred to. The implementation procedures and technical effects of the technical solution can be referred to the description of the embodiments shown in FIGS. 1 to 4, and are not further described herein.

6, which shows a structural schematic diagram of an electronic device (eg, terminal device or server in FIG. 1) 600 suitable for implementing an embodiment of the present disclosure. Terminal devices in the embodiments of the present disclosure include, for example, mobile phones, notebook computers, digital broadcast receivers, PDA (personal digital assistant), PAD (tablet), PMP (portable multimedia player), in-vehicle terminals (e.g. , in-vehicle navigation terminals), and fixed terminals such as, for example, digital TVs and desktop computers. The electronic device shown in FIG. 6 is only an example and does not impose any limitation on the functionality and scope of use of the embodiments of the present disclosure.

As shown in FIG. 6, electronic device 600 may include a processing unit (eg, central processing unit, graphics processor, etc.) 601 , which may be randomly accessed from memory 608 or programs stored in read-only memory (ROM) 602 . Various suitable operations and processes may be performed by programs loaded into memory (RAM) 603 . The RAM 603 stores various programs and data necessary for operating the electronic device 600 . Processing unit 601 , ROM 602 and RAM 603 are interconnected via bus 604 . Input/output (I/O) interface 605 is also connected to bus 604 .

Generally, input devices 606 including, for example, touch screens, touch panels, keyboards, mice, camera heads, microphones, accelerometers, gyros, etc., and output devices 607, including, for example, liquid crystal displays (LCDs), speakers, oscillators, etc. Storage devices 608 , including, for example, magnetic tapes, hard disks, etc., and communication devices 609 may be connected to I/O interface 605 . Communication device 609 may allow electronic device 600 to communicate wirelessly or by wire with other devices to exchange data. It should be understood that although FIG. 6 illustrates electronic device 600 with various devices, it is not intended to require that all of the devices shown be implemented or include all of them. More or less devices may alternatively be implemented or included.

In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program resident on a non-transitory computer-readable medium, the computer program having program code for performing the method illustrated in the flowcharts. include. In such embodiments, the computer program may be downloaded and installed from a network by communication device 609, installed from storage device 608, or installed from ROM 602. FIG. When the computer program is executed by the processing unit 601, the above-described functions defined in the method of the embodiments of the present disclosure are performed.

It should be noted that the computer-readable medium in the present disclosure may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the above. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. Computer-readable storage media include, as more specific examples, electrical connections having one or more conductors, portable computer magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable may include, but is not limited to, programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination thereof. do not have. For purposes of this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that is used with or in combination with a command execution system, apparatus, or device. Is possible. In the present disclosure, a computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave and have computer readable program code embodied therein. The data signals so propagated may take many forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium other than a computer readable storage medium used in a command execution system, apparatus or device. or any program used in conjunction therewith may be transmitted, propagated or transmitted. Program code embodied in computer readable media may be transmitted in any suitable medium including, but not limited to, electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof. do not have.

In some embodiments, the client, server may communicate using any now known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and any format or It can be interconnected with a medium of digital data communication (eg, a communication network). Examples of communication networks include local area networks (“LAN”), wide area networks (“WAN”), the Internet (e.g., Internet) and end-to-end networks (e.g., ad hoc end-to-end networks), as well as currently known or any network developed in the future.

The computer-readable medium may be included in the electronic device, or may stand alone without being installed in the electronic device.

The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device instructs identifying a first object and a second object located within said first object; and identifying a third object based on the position of said first object in said first video frame image. obtaining a second video frame image with the third object overlying the first object as a foreground image overlaid on the first video frame image; superimposing the second object as a foreground image on a third object in the second video frame image to obtain a third video frame image based on the position in the video frame image of Let

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or a combination thereof, which may be an object-oriented programming language such as Java , Smalltalk, C++, and also general procedural programming languages such as, but not limited to, the "C" language or similar programming languages. Program code may run entirely on the user computer, partially on the user computer, as a separate software package, partly on the user computer and partly on a remote computer, or completely It can also be run on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user computer by any network, including a local area network (LAN) or wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider). connect via the Internet).

The flowcharts and block diagrams in the accompanying drawings illustrate possible system architectures, functionality, and operation of systems, methods and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram can represent a module, program segment, or portion of code, which implements the specified logical function. contains one or more executable instructions for It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutively shown blocks may be executed substantially in parallel, or they may be executed in the reverse order, depending on the functionality involved. It is noted that each block in the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a dedicated system. Note that it may also be implemented by a combination of hardware and computer instructions.

Units according to embodiments of the present disclosure may be implemented in software or hardware. Here, the name of the unit is in some cases not a limitation on the unit itself.

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, exemplary hardware logic components that can be used are Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Product (ASSP), System on Chip (SOC), Complex Programmable Including, but not limited to, logic devices (CPLDs) and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium and may include a program used by or in combination with a command execution system, apparatus or device. can be remembered. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared media, or semiconductor systems, apparatus or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media are electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory. Including memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination thereof.

An embodiment of the disclosure further provides a computer program product which, when executed by a computer, causes said computer to perform an image processing method according to an embodiment of the disclosure.

According to one or more embodiments of the present disclosure,
identifying a first object in a first video frame image and a second object located within the first object;
obtaining a second video frame image by overlaying a third object as a foreground image on the first video frame image based on the position of the first object in the first video frame image; wherein the third object covered the first object in the second video frame image;
superimposing the second object as a foreground image on a third object in the second video frame image based on the position of the second object in the first video frame image to produce a third video; and obtaining a frame image.
Further, said identifying a first object in said first video frame image and a second object located within said first object comprises:
identifying a plurality of first keypoints of the first object and a plurality of second keypoints of the second object in the first video frame image;
identifying the first object based on a plurality of first keypoints of the first object;
identifying the second object based on a plurality of second keypoints of the second object, wherein the plurality of second keypoints are edge keypoints of the second object; including being

and overlaying a third object as a foreground image on the first video frame image based on the position of the first object in the first video frame image to produce a second video frame. to get the image
calculating a positioning point of the first object in the first video frame image based on a plurality of first keypoints of the first object in the first video frame image;
aligning the positioning point of the third object with the positioning point of the first object in the first video frame image, with the third object as a foreground image on the first video frame image; to obtain the second video frame image.

and superimposing the second object as a foreground image on the third object in the second video frame image based on the position of the second object in the first video frame image. , to obtain the third video frame image,
calculating a first distance between two preset keypoints on the first object;
calculating an offset amount of the second target based on the first distance;
above a third object in the second video frame image with the second object as the foreground image based on the position of the second object in the first video frame image and the amount of offset of the second object; to obtain a third video frame image.

Furthermore, calculating the offset amount of the second target based on the first distance,
obtaining a yaw angle and a pitch angle of the first object, wherein the yaw angle is a horizontal included angle between an orientation of the first object in the first video frame image and a lens shooting direction; a pitch angle is a vertical included angle between an orientation in the first video frame image of the first object and a lens shooting direction;
calculating an offset amount of the second object based on the first distance, yaw angle and pitch angle.

Further, based on the position of the second object in the first video frame image and the offset amount of the second object, the second object in the second video frame image with the second object as the foreground image Obtaining a third video frame image superimposed on the third object
position-shifting an initial positioning point in the first video frame image of the second object to obtain a target positioning point based on the offset amount of the second object;
rendering the second object as a foreground image in the second video frame image at the location of the target positioning point to obtain the third video frame image.

Further, after obtaining the third video frame image,
converting the color space of the second object and the third object in the third video frame image from the RGB color space to the HSV color space;
replacing H channel values in the HSV color space of the second object with H channel values of the third object;
Converting the color space of the second object and the third object from HSV space to RGB color space to obtain a fourth video frame image.

According to one or more embodiments of the present disclosure,
an object identification module for identifying a first object in a first video frame image and a second object located within said first object;
overlaying a third object as a foreground image on the first video frame image based on the position of the first object in the first video frame image, wherein the third object is the first video frame image; a second video frame image generation module for obtaining a second video frame image covering the object;
superimposing the second object as a foreground image on a third object in the second video frame image based on the position of the second object in the first video frame image to produce a third video; and a third video frame generation module for obtaining frame images.

Further, the object identification module further comprises:
identifying a plurality of first keypoints of the first object and a plurality of second keypoints of the second object in the first video frame image;
identifying the first object based on a plurality of first keypoints of the first object;
identifying the second object based on a plurality of second keypoints of the second object, wherein the plurality of second keypoints are edge keypoints of the second object; Used to be and

Further, the second video frame image generation module includes:
calculating a positioning point of the first object in the first video frame image based on a plurality of first keypoints of the first object in the first video frame image;
aligning the positioning point of the third object with the positioning point of the first object in the first video frame image, with the third object as a foreground image on the first video frame image; to obtain the second video frame image.

Further, the second video frame image generation module further comprises:
calculating a first distance between two preset keypoints on the first object;
calculating an offset amount of the second target based on the first distance;
above a third object in the second video frame image with the second object as the foreground image based on the position of the second object in the first video frame image and the amount of offset of the second object; and obtaining a third video frame image by superimposing the .

Furthermore, calculating the offset amount of the second target based on the first distance,
obtaining a yaw angle and a pitch angle of the first object, wherein the yaw angle is a horizontal included angle between an orientation of the first object in the first video frame image and a lens shooting direction; a pitch angle is a vertical included angle between an orientation in the first video frame image of the first object and a lens shooting direction;
calculating an offset amount of the second object based on the first distance, yaw angle and pitch angle.

Further, based on the position of the second object in the first video frame image and the offset amount of the second object, the second object in the second video frame image with the second object as the foreground image Obtaining a third video frame image superimposed on the third object
position-shifting an initial positioning point in the first video frame image of the second object to obtain a target positioning point based on the offset amount of the second object;
rendering the second object as a foreground image in the second video frame image at the location of the target positioning point to obtain the third video frame image.

Furthermore, the image processing device
converting the color space of the second object and the third object in the third video frame image from the RGB color space to the HSV color space; replacing the H channel values of a third object; and converting the color spaces of the second and third objects from the HSV space to the RGB color space to obtain a fourth video frame image. Further includes a fourth video image generation module that is used.

According to one or more embodiments of the disclosure, there is provided a computer program which, when executed by a computer, causes the computer to perform an image processing method according to embodiments of the disclosure.
The above is merely a description of the embodiments and applied technical principles of the present disclosure. The scope of the disclosure according to the present disclosure is not limited to a technical proposal by a specific combination of the above technical features, and other technologies formed by arbitrarily combining the above technical features or equivalent features without departing from the concept of the above disclosure It is obvious to those skilled in the art that the proposal should also be included. For example, technical solutions formed by replacing the above features with technical features having similar functions disclosed in the present disclosure (but not limited to them) are also included.
Also, although each operation has been described in a particular order, this is understood to require that such operations be performed in the specific order shown or in a sequential order. shouldn't. Multitasking and parallel processing can be advantageous in certain environments. Similarly, although some specific implementation details are included in the above description, they should not be construed as limiting the scope of the disclosure. Certain features that are described in the context of a single embodiment can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological logical acts, the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Please understand. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims (17)

identifying a first object in a first video frame image and a second object located within the first object;
obtaining a second video frame image by overlaying a third object as a foreground image on the first video frame image based on the position of the first object in the first video frame image; wherein the third object covered the first object in the second video frame image;
superimposing the second object as a foreground image on a third object in the second video frame image based on the position of the second object in the first video frame image to produce a third video; obtaining a frame image; and an image processing method. Identifying a first object in said first video frame image and a second object located within said first object comprises:
identifying a plurality of first keypoints of the first object and a plurality of second keypoints of the second object in the first video frame image;
identifying the first object based on a plurality of first keypoints of the first object;
identifying the second object based on a plurality of second keypoints of the second object, wherein the plurality of second keypoints are edge keypoints of the second object; The image processing method of claim 1, comprising: Overlaying a third object as a foreground image on the first video frame image based on the position of the first object in the first video frame image to obtain a second video frame image. The thing is
calculating a positioning point of the first object in the first video frame image based on a plurality of first keypoints of the first object in the first video frame image;
aligning the positioning point of the third object with the positioning point of the first object in the first video frame image, with the third object as a foreground image on the first video frame image; 3. The image processing method of claim 2, comprising overlaying on to obtain the second video frame image. superimposing the second object as a foreground image on a third object in the second video frame image based on the position of the second object in the first video frame image; To get the video frame image of
calculating a first distance between two preset keypoints on the first object;
calculating an offset amount of the second target based on the first distance;
above a third object in the second video frame image with the second object as the foreground image based on the position of the second object in the first video frame image and the amount of offset of the second object; obtaining a third video frame image by superimposing on the . Calculating the offset amount of the second target based on the first distance
obtaining a yaw angle and a pitch angle of the first object, wherein the yaw angle is a horizontal included angle between the orientation of the first object in the first video frame image and a lens shooting direction; wherein the pitch angle is a vertical included angle between the orientation of the first object in the first video frame image and the lens shooting direction;
5. The image processing method according to claim 4, comprising calculating an offset amount of said second object based on said first distance, yaw angle and pitch angle. Based on the position of the second object in the first video frame image and the offset amount of the second object, a third image in the second video frame image with the second object as a foreground image. Obtaining a third video frame image superimposed on the object
position-shifting an initial positioning point in the first video frame image of the second object to obtain a target positioning point based on the offset amount of the second object;
6. Rendering the second object as a foreground image in the second video frame image at the position of the target positioning point to obtain the third video frame image. image processing method. After obtaining the third video frame image,
converting the color space of the second object and the third object in the third video frame image from the RGB color space to the HSV color space;
replacing H channel values in the HSV color space of the second object with H channel values of the third object;
Converting the color space of said second and third objects from HSV space to RGB color space to obtain a fourth video frame image. image processing method. an object identification module for identifying a first object in a first video frame image and a second object located within said first object;
overlaying a third object as a foreground image on the first video frame image based on the position of the first object in the first video frame image, wherein the third object is the first video frame image; a second video frame image generation module for obtaining a second video frame image covering the object;
superimposing the second object as a foreground image on a third object in the second video frame image based on the position of the second object in the first video frame image to produce a third video; and a third video frame generation module for obtaining frame images. The object identification module further comprises:
identifying a plurality of first keypoints of the first object and a plurality of second keypoints of the second object in the first video frame image;
identifying the first object based on a plurality of first keypoints of the first object;
identifying the second object based on a plurality of second keypoints of the second object, wherein the plurality of second keypoints are edge keypoints of the second object; 9. The image processing apparatus according to claim 8, used for: The second video frame image generation module comprises:
calculating a positioning point of the first object in the first video frame image based on a plurality of first keypoints of the first object in the first video frame image;
aligning the positioning point of the third object with the positioning point of the first object in the first video frame image, with the third object as a foreground image on the first video frame image; 10. The image processing apparatus of claim 9, comprising overlaying on to obtain the second video frame image. The second video frame image generation module further comprises:
calculating a first distance between two preset keypoints on the first object;
calculating an offset amount of the second target based on the first distance;
above a third object in the second video frame image with the second object as the foreground image based on the position of the second object in the first video frame image and the amount of offset of the second object; 11. The image processing apparatus according to any one of claims 8 to 10, which is used for obtaining a third video frame image by superimposing on the . Calculating the offset amount of the second target based on the first distance
obtaining a yaw angle and a pitch angle of the first object, wherein the yaw angle is a horizontal included angle between the orientation of the first object in the first video frame image and a lens shooting direction; wherein the pitch angle is a vertical included angle between the orientation of the first object in the first video frame image and the lens shooting direction;
12. The image processing apparatus according to claim 11, comprising calculating an offset amount of said second object based on said first distance, yaw angle and pitch angle. Based on the position of the second object in the first video frame image and the offset amount of the second object, a third image in the second video frame image with the second object as a foreground image. Obtaining a third video frame image superimposed on the object
position-shifting an initial positioning point in the first video frame image of the second object to obtain a target positioning point based on the offset amount of the second object;
13. Rendering the second object as a foreground image in the second video frame image at the location of the target positioning point to obtain the third video frame image. image processing device. converting the color space of the second object and the third object in the third video frame image from the RGB color space to the HSV color space; replacing the H channel values of a third object; and converting the color spaces of the second and third objects from the HSV space to the RGB color space to obtain a fourth video frame image. Image processing apparatus according to any one of claims 8 to 13, further comprising a fourth video image generation module used. a memory for storing computer readable instructions;
A processor for executing the computer-readable instructions, the processor realizing the image processing method according to any one of claims 1 to 7 when executed by the processor. used for storing computer readable instructions, said computer readable instructions causing said computer to perform an image processing method according to any one of claims 1 to 7 when said computer readable instructions are executed by said computer; A non-transitory computer-readable storage medium. A computer program that, when executed by a computer, causes the computer to execute the image processing method according to any one of claims 1 to 7.

Images