CN117389451A - Image processing method and device - Google Patents

Abstract

The application discloses an image processing method and a device thereof, wherein the image processing method comprises the following steps: determining an eye region in the first image; acquiring expected image data of eyes; determining eye contour information according to the eye region, and determining eye expected contour information according to the eye expected image data; and determining a transformation relation according to the expected eye contour information and the eye contour information, and adjusting the eye area according to the transformation relation to generate a second image.

Description

Image processing method and device

Technical Field

The application belongs to the technical field of images, and particularly relates to an image processing method and an image processing device.

Background

With the popularity of mobile terminals, more and more users have a need to adjust the eye size of a person's body in an image.

In the related scheme, the processing method for the eye area generally forms an eye adjustment model by training a neural network, then takes an image to be adjusted as input of the eye adjustment model, and outputs a human body image with the eye adjusted by the eye adjustment model. In this way, since the eye region is automatically generated by the eye adjustment model, the generated image has the problems that the eye region is unreal and the personalized requirements of the user cannot be met.

Disclosure of Invention

The application aims to provide an image processing method and an image processing device, which can solve the problems that the eye area in the related technology is unreal and the personalized requirements of users cannot be met.

In a first aspect, an embodiment of the present application provides an image processing method, including: determining an eye region in the first image; acquiring expected image data of eyes; determining eye contour information according to the eye region, and determining eye expected contour information according to the eye expected image data; and determining a transformation relation according to the expected eye contour information and the eye contour information, and adjusting the eye area according to the transformation relation to generate a second image.

In a second aspect, an embodiment of the present application proposes an image processing apparatus including: a first determination module for determining an eye region in the first image; the acquisition module is used for acquiring expected eye image data; the second determining module is used for determining eye contour information according to the eye region and determining eye expected contour information according to the eye expected image data; and the adjusting module is used for determining a transformation relation according to the expected eye contour information and the eye contour information, and adjusting the eye area according to the transformation relation to generate a second image.

In a third aspect, embodiments of the present application provide an electronic device comprising a processor, a memory and a program or instruction stored on the memory and executable on the processor, which program or instruction when executed by the processor implements the steps of the method as in the first aspect.

In a fourth aspect, embodiments of the present application provide a readable storage medium having stored thereon a program or instructions which when executed by a processor perform the steps of the method as in the first aspect.

In a fifth aspect, embodiments of the present application provide a chip comprising a processor and a communication interface coupled to the processor, the processor being configured to execute programs or instructions to implement the method of the first aspect.

The image processing method and the device thereof are used for adjusting the size, the shape and the like of the eye region in the image. Specifically, after the first image that the user needs to adjust is acquired, the eye area in the first image may be identified first, and then the eye desired image data may be acquired, so that the eye area in the first image may be adjusted, for example, the size, shape, etc. of the eye area may be adjusted with reference to the eye desired image data. According to the image processing mode, due to the fact that expected image data of the eyes are referred when the eye area is regulated, on one hand, the regulated eye area is more real, and the generated image of the eyes is more vivid. Meanwhile, the adjusted eye area can meet the personalized requirements of the user. On the other hand, the image processing mode can process the eye area in the image according to the method immediately after the user takes the picture, so as to realize real-time processing of the taken image. Meanwhile, the image processing mode automatically completes adjustment through electronic equipment and the like, and a user does not need to perform any additional operation, so that the eye adjustment of the image is more intelligent and convenient.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Drawings

The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is one of the flow diagrams of an image processing method according to an embodiment of the present application;

FIG. 2 is a second flow chart of an image processing method according to an embodiment of the present application;

FIG. 3 is a third flow chart of an image processing method according to an embodiment of the present application;

FIG. 4 is a fourth flow chart of an image processing method according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a human eye detection frame of a first image of an image processing method of an embodiment of the present application;

FIG. 6 is a schematic diagram of a first image human eye feature point distribution of an image processing method according to an embodiment of the present application;

FIG. 7 is a fifth flow chart of an image processing method according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a human eye detection frame of a desired human eye image of an image processing method of an embodiment of the present application;

fig. 9 is a schematic diagram of a distribution of human eye feature points of a desired human eye image of the image processing method of the embodiment of the present application;

FIG. 10 is a schematic diagram of a human eye transformation area of an image processing method according to an embodiment of the present application;

FIG. 11 is a schematic representation of left and right eye region transforms of a first image of an image processing method of an embodiment of the present application;

fig. 12 is a block diagram of an image processing apparatus according to an embodiment of the present application;

FIG. 13 is a block diagram of an electronic device according to an embodiment of the present application;

fig. 14 is a schematic hardware structure of an electronic device implementing an embodiment of the present application.

Detailed Description

Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the like or similar elements throughout or elements having the same or similar functions. The embodiments described below by referring to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

The image processing method and the device thereof according to the embodiments of the present application are described in detail below with reference to the accompanying drawings.

In an embodiment of the present application, there is provided an image processing method, fig. 1 shows one of flowcharts of the image processing method according to an embodiment of the present application, and as shown in fig. 1, the method includes:

s102, determining an eye region in the first image.

The first image is an image to be adjusted, which is input by a user, for example, a person photo taken by the user, and the like. Of course, the first image may be a processed image after photographing, or the first image may be an portrait. In summary, the form of the first image is not limited, as long as the eye region to be adjusted is included thereon. The eye region in the first image generally includes a left eye region and a right eye region, and when performing image processing, the left eye region and the right eye region may be processed according to the image processing method of the embodiment of the present application, so as to complete processing of all the eye regions in the first image.

S104, acquiring expected eye image data.

The eye expected image data is data of an eye image expected by a user, and expresses the adjustment requirement of the user on the eyes.

S106, determining eye contour information according to the eye region, and determining eye expected contour information according to the eye expected image data.

Wherein the eye contour information substantially represents the size and shape of the eye. Therefore, when the eye region is adjusted according to the eye expected image data, the eye contour information of the eye region and the eye expected contour information of the eye expected image can be determined first, and then the adjustment of the eye region is guided according to the eye expected contour information, so that the contour of the eye region can be adjusted to the expected contour, and the adjustment of the size and the shape of the eye region is realized. Of course, after the outline of the eye is adjusted, the color of the eye region or the like may be adjusted based on the input of the user.

S108, determining a transformation relation according to the expected eye contour information and the eye contour information, and adjusting the eye area according to the transformation relation to generate a second image.

Wherein in this step, the transformation relationship between the eye desired contour information and the eye region contour information is taken as the transformation relationship between the eye region and the eye desired image data, and the image adjustment of the eye region is guided by the transformation relationship.

The image processing method of the embodiment of the application is used for adjusting the size, the shape and the like of the eye region in the image. Specifically, after the first image that the user needs to adjust is acquired, the eye area in the first image may be identified first, and then the eye desired image data may be acquired, so that the eye area in the first image may be adjusted, for example, the size, shape, etc. of the eye area may be adjusted with reference to the eye desired image data. According to the image processing mode, due to the fact that expected image data of the eyes are referred when the eye area is regulated, on one hand, the regulated eye area is more real, and the generated image of the eyes is more vivid. Meanwhile, the adjusted eye area can meet the personalized requirements of the user. On the other hand, the image processing mode can process the eye area in the image according to the method immediately after the user takes the picture, so as to realize real-time processing of the taken image. Meanwhile, the image processing mode automatically completes adjustment through electronic equipment and the like, and a user does not need to perform any additional operation, so that the eye adjustment of the image is more intelligent and convenient.

In an embodiment of the present application, the image processing method further includes:

in the case where the eye region is not included in the first image, the first image is output.

In this embodiment of the present application, when an image of an eye is not recognized in an input image, the first image may be directly output, for example, when a user takes a plurality of photos, there may be a case where an individual photo does not include eyes of a person, for example, a local photo or a landscape photo of other parts of a human body, which are taken, and the like. By this step, when the image input by the user does not include the eye region, the flow of the image processing can be continued downward without the problem of flow interruption or flow complete stop.

In the embodiment of the present application, as shown in fig. 2, step 104 includes:

s202, inputting the first image into the generated artificial intelligence system.

Wherein the generated artificial intelligence, i.e., AIGC (Artificial Intelligence Generated Content) system is capable of automatically generating the eye desired image, thereby forming the eye desired image data. Based on the eye expected images generated by AIGC, the types can be varied, and the personalized requirements of different clients can be met.

S204, when the user does not input the eye expected image data, acquiring the eye expected image data generated by the generating artificial intelligence system according to the first image.

Wherein the user may not input any data while generating the ocular desired image data by the generative artificial intelligence system, in which case the generative artificial intelligence system then directly generates the ocular desired image data according to the internal rules.

S206, when the user inputs the eye expected data, acquiring eye expected image data generated by the generating artificial intelligent system according to the eye expected data and the first image.

Wherein, when generating the eye desired image data by the generating artificial intelligence system, the user can also input some own eye requirements, such as size, color, eye shape, etc., to form the eye desired data. At this time, the generated artificial intelligence system generates the expected eye image data according to the expected eye data and the internal rules formed by the eye demands input by the user, so that the generated expected eye image data can meet the demands of each user, and the personalized generation of the expected eye image data is realized.

In the embodiment of the application, the first image can be input into the generation type artificial intelligence system, so that the generation type artificial intelligence system can be matched with appropriate expected eye image data according to the input eye image, the expected eye image data can be more matched with the image of the eye area in the first image, and the adjusted image can be more natural. Meanwhile, the setting enables a user to select to input eye demands or not according to actual demands, so that more flexible choices are provided for the user, and user experience is improved.

In the above embodiment, as shown in fig. 3, step 108 includes:

s302, determining eye contour information according to a first coordinate set, wherein the first coordinate set comprises coordinate information of a plurality of target points in an eye region.

In determining specific eye contour information, a target point may be determined first, where the target point is a predetermined point, and is generally a point that is critical to the eyes, for example, a point corresponding to left and right eye corners, a point centered above and below the eyes, and the like. The position information of the target points can be determined, and then the position information of the target points can be summarized and recorded as a first coordinate set and then used as outline information of eyes.

S304, determining expected eye contour information according to a second coordinate set, wherein the second coordinate set comprises coordinate information of a plurality of target points in expected eye image data.

Wherein the step of determining the expected eye contour information and the step of determining the eye contour information are the same. The target point is determined first, where the target point is a predetermined point, typically a point that is critical to the eyes, such as a point corresponding to the left and right corners of the eyes, a point centered above and below the eyes, and so on. The position information of the target points can be determined, and then the position information of the target points can be summarized and recorded as a second coordinate set and then used as the expected outline information of the eyes.

Wherein the target points in S302 and S304 are the same. Optionally, the target point includes at least a center point of the eye contour in the eye region, a leftmost point, a topmost point, a rightmost point, a bottommost point of the eye contour.

In the embodiment of the present application, the leftmost, uppermost, rightmost, and lowermost four points represent the upper, lower, left, and right limit positions of the eye region and the eye desired image data, and therefore, the approximate positions and sizes of the eye region and the eye desired image data can be basically determined by the 4 points.

Optionally, the target point further comprises a leftmost point of the eye contour, an intermediate point between the leftmost point, an intermediate point between the rightmost point and the rightmost point, an intermediate point between the rightmost point and the bottommost point, and an intermediate point between the bottommost point and the leftmost point. The shape of the eye outline can be further limited through the arrangement of the middle point, so that the data according to the target point is richer, the eye outline information determined according to the target point is more accurate, and the adjusted eye image is more real.

Optionally, the target point further comprises at least one bisecting point between a leftmost point, a topmost point, at least one bisecting point between a topmost point and a rightmost point, at least one bisecting point between a rightmost point and a bottommost point, and at least one bisecting point between a bottommost point and a leftmost point of the eye contour. That is, the target point includes one or more bisectors, such as a bisector, or a trisection, between any two limit points, in addition to the four limit points, up, down, left, and right.

Optionally, the eye region is equally divided into n parts along the length direction of the eye region by the bisector, and simultaneously the eye region is equally divided into m parts along the width direction of the eye region, and an intersection point of each bisector and a contour line of the eye region is an equal division point. The target point also comprises a central point of the eye area and at least an equal division point, wherein n and m are both more than or equal to 2 and are integers.

In the embodiment of the application, the target point further comprises a center point of the eye region, and the position represents the pupil center of the eye, so that the determination of the eye region has a certain reference meaning. The target point also comprises a plurality of equal dividing points, and the determined outline of the eye can be more close to the actual outline through the equal dividing points, so that the outline of the determined eye area is more accurate, the generated image is more vivid, and the generated image is more close to the original image of the entity.

S306, determining a transformation relation between the first coordinate set and the second coordinate set.

Since the coordinate information of the plurality of target points of the eye region, that is, the first coordinate set, and the coordinate information of the plurality of corresponding target points of the eye desired image, that is, the second coordinate set, are obtained before, the calculation can be performed according to the coordinate information of the target points to determine the affine transformation relationship between the two, that is, the transformation relationship between the two can be determined according to the first coordinate set and the second coordinate set, and then the calculated transformation relationship is used as the transformation relationship between the eye region and the eye desired image data to guide the image adjustment of the eye region.

S308, determining a transformation area according to the eye area and the expected eye image data.

In the process of adjusting the eye region based on the calculated transformation relationship, the transformation region to be transformed may be determined according to the eye region and the eye desired image data, and in general, a region surrounded by the detection frame of the eye region and the detection frame of the eye desired image data may be regarded as a transformation region, which reflects a difference portion between the eye region and the eye desired image data, and thus, points of the region are required to be adjusted, while points of other regions are the same, and thus, adjustment is not required.

S310, determining the point to be transformed of the eye region according to the transformation region. After the transformation area is determined, each pixel point or part of pixel points in the transformation area can be used as points needing transformation.

S312, adjusting the point to be transformed of the eye region according to the transformation relation, and generating a second image.

And transforming each point to be transformed into a point on the second image according to the transformation relation determined before, so as to finish the transformation of all the points and generate a transformed second image.

In the embodiment of the application, since only the points in the transformation area need to be adjusted, the calculated amount can be reduced, and the generation efficiency of the second image can be improved.

In the embodiment of the application, when determining the eye contour information, the first coordinate set needs to be determined first. Wherein the step of determining the first set of coordinates comprises:

step one, generating a first eye detection frame according to an eye region, wherein the eye region is positioned in the first eye detection frame, and four sides of the first eye detection frame are tangent to the contour line of the eye region.

After the eye area is determined, the human eye detection frame data, namely the first detection frame data, can be output through a human eye detection algorithm, and meanwhile, the coordinates of the human eye detection frame are output. The eye region is located in the first eye detection frame, and four sides of the first eye detection frame are tangent to the outline of the eye region, that is, when the first detection frame is a rectangular frame, the first eye detection frame is tangent to the outline of the eye region. The coordinates of the first detection frame may be represented by coordinates of points in the upper left corner and the lower right corner. After the first eye detection frame is output, the first eye detection frame can assist in selecting the target point, so that the coordinates of the target point are obtained.

And secondly, acquiring coordinate information of four points tangent to the contour line of the eye region of the first eye detection frame, and recording the coordinate information into a first coordinate set.

The target point specifically comprises 4 tangent points, wherein the detecting frame is tangent to the contour line of the eye region. These 4 points represent the limit positions of the eye region in the four directions of up, down, left and right, so that the approximate position and size of the eye region can be basically determined through these 4 points.

Third, the center point of the eye region is obtained and marked as a first center point, a first reference line is established by taking the center of the first eye detection frame as a starting point, the first reference line intersects with the first eye detection frame, first rays which form a plurality of target angles with the first reference line are selected, the intersection point of each first ray and the contour line of the eye region is determined and marked as a first intersection point, and the coordinate information of the first center point and/or the coordinate information of each first intersection point are marked into a first coordinate set.

The target point also comprises a central point of the eye region, and the position represents the pupil center of the eye, so that the determination of the eye region has a certain reference meaning. The target point also comprises a plurality of intersection points, and the outline of the eye can be more close to the actual outline through the intersection points, so that the outline of the determined eye area is more accurate, the generated image is more vivid, and the generated image is more close to the original image of the entity.

In the embodiment of the application, when determining the expected outline information of the eye, the second coordinate set needs to be determined first. Wherein the step of determining the second set of coordinates comprises:

step one, generating a second eye detection frame according to eye expected image data, wherein the eye expected image data are located in the second eye detection frame, and four sides of the second eye detection frame are tangent to contour lines of the eye expected image data.

After the eye expected image data is determined, a detection frame of the eye expected image data, namely a second detection frame, can be calculated through a human eye detection algorithm, and meanwhile, the coordinates of the second detection frame are output. The eye expected image data is located in the second eye detection frame, and the 4 sides of the second eye detection frame are tangent to the outline of the eye expected image data, that is, when the second detection frame is a rectangular frame, the outline of the eye expected image data is tangent to the outline of the eye expected image data. The coordinates of the second detection frame may be represented by coordinates of points in the upper left corner and the lower right corner. After the second eye detection frame is output, the second eye detection frame can assist in selecting the target point of the expected eye image data, so that the coordinates of the corresponding target point are obtained.

And secondly, recording coordinates of four points, at which the second eye detection frame is tangent to the contour line of the eye expected image data, into a second coordinate set.

The target point may specifically include 4 tangent points at which the detection frame is tangent to the contour line of the eye desired image data. The 4 points represent the limit positions of the expected eye image data in the up-down, left-right directions, so that the approximate position and the size of the expected eye image data can be basically determined through the 4 points.

Thirdly, acquiring a center point of expected eye image data, marking the center point as a second center point, taking the center of a second eye detection frame as a starting point, establishing a second reference line, intersecting the second reference line with the second eye detection frame, selecting second rays which form a plurality of target angles with the second reference line, determining intersection points of each second ray and contour lines of expected eye image data, marking the intersection points as second intersection points, and recording coordinate information of the second center point and/or coordinate information of each second intersection point into a second coordinate set.

The target point also comprises a center point of the expected eye image data, and the position represents the pupil center of the eye, so that the target point has certain reference significance for determining the expected eye image data. The target point also comprises a plurality of intersection points, and the determined outline of the expected eye is enabled to be closer to an actual outline through the intersection points, so that the outline of the determined expected eye image data is more accurate, the generated image is enabled to be more vivid and is also closer to the original image of the entity.

The image processing method according to the embodiment of the present application will be further described below with reference to fig. 4 to 11 by taking an example of adjusting a human eye.

The image processing method provided by the embodiment of the application is a human eye size personalized adjustment method applied to an image. The basic idea is as follows: based on an AIGC (Artificial Intelligence Generated Content, generating artificial intelligence) system, generating a human eye template image, detecting feature points of the first image and the template image, calculating a matching relation of the feature points, and finally carrying out affine transformation on human eyes in the first image based on the matching relation. The method takes the human eye outline of the template image as traction, directly transforms in the human eye area of the first image, and can be automatically adjusted according to the personalized requirements of the user. As shown in fig. 4, it specifically includes the following steps:

s401, inputting a first image.

S402, detecting human eyes.

S403, judging whether human eyes are contained, directly outputting an image when the human eyes are not included, and directly executing S404 when the human eyes are included.

S404, inputting the first image into an AIGC system.

Denoted as P for the first image _i Human eye information is detected by a human eye detection algorithm. If the first image does not contain human eyes, the first image is directly output. If the first image includes a human eye, the human eye detection algorithm outputs the human Eye detection frame data. The human eye detection frame data comprises the pixel coordinates of the upper left corner of the rectangular frame, and further, the human eye detection frame is divided into a left human eye detection frame and a right human eye detection frame. As shown in fig. 5, the meanings of the symbols in the drawing are as follows:

P _{i_l_tl} (x, y): the left eye of the first image detects the pixel coordinates of the upper left corner of the frame;

P _{i_l_br} (x, y): the lower right corner pixel coordinates of the left eye detection frame of the first image;

P _{i_r_tl} (x, y): the upper left corner pixel coordinates of the right eye detection frame of the first image;

P _{i_r_br} (x, y): the right eye of the first image detects the bottom right corner pixel coordinates of the frame.

The step of detecting the human eye region on the input first image is completed through the steps of S401 to S404.

S405, detecting human eye characteristic points in the first image.

In the human eye frame area of the first image, detecting a left human eye feature point set and a right human eye feature point set through a human eye feature point detection algorithm, wherein the left human eye feature point set is marked as S _{i_l} The right eye feature point set is marked as S _{i_r} . In this embodiment, the distribution of the characteristic points of the human eye is shown in fig. 6. Left eye feature point set S _{i l} Comprises { P ] _l0 ，P _l1 ，P _l2 ，P _l3 ，P _l4 ，P _l5 ，P _l6 ，P _l7 ，P _l8 Characteristic points, right eye characteristic point set S _{i_r} Comprises { P ] _r0 ，P _r1 ，P _r2 ，P _r3 ，P _r4 ，P _r5 ，P _r6 ，P _r7 ，P _r8 Characteristic points. The meaning of each feature point is as follows:

{P _l0 ，P _l2 ，P _l4 ，P _l6 and { P } is _r0 ，，P _r2 ，P _r4 ，P _r6 And respectively represent leftmost, uppermost, rightmost and bottommost position feature points of left and right eyes. These points are the necessary points. The requisite points represent feature points that must be detected by the human eye feature point detection algorithm.

{P _l1 ，P _l3 ，P _l5 ，P _l7 And P _r1 ，P _r3 ，P _r5 ，P _r7 The tangential angles on the left and right eye contours are {45 °,135 °,215 °,315 ° }, respectively, and in the present invention, the tangential angles may be changed to other angle values. { P _l8 And P _r8 And respectively represent center points of pupils of the left and right eyes, which are optional points. The optional points represent feature points that may not be detected. Wherein, at least 3 pairs of non-collinear two-dimensional coordinate points are needed for calculating the affine transformation matrix, and the more the two-dimensional coordinate points are, the more accurate the estimated affine transformation matrix is. The optional points represent feature points which can not be detected, in an actual image, since the eyes may be in an eye-closing/eye-semi-closing state, the pupil center point is not detected, so the center points of the pupils of the left and right eyes are optional points, and the feature points corresponding to the tangential angles do not belong to strong feature points and may not be detected, so the pupil center point is also an optional point.

S406, the AIGC system generates a desired human eye image.

The flow of generating the expected human eye image by the AIGC system is shown in fig. 7, and the specific generating steps are as follows:

s4061, the first image is sent to the AIGC system.

S4062, judging whether the user inputs human eye requirements.

If the AIGC system detects that the user does not input a human eye demand, step S4063 is performed.

S4063, the AIGC system automatically generates a recommended desired human eye image for the user based only on the first image and the internal rules. The image is denoted as P _d 。

If the AIGC system detects that the user has entered a human eye requirement, step S4064 is performed.

S4064, the AIGC system generates a human eye image satisfying the user input demand, called a desired human eye image, from the first image, the human eye demand input by the user, and the internal rule. The image is denoted as P _d 。

S407, detecting the characteristic points of the human eyes of the template image. The step of detecting the characteristic points of the human eyes of the template image specifically comprises the following two steps:

step 1, detecting a human eye frame on an expected human eye image: as with the first image, for the desired human eye image P _d Human eye frame information is detected through the same human eye detection algorithm, and human eye detection frame data is output. As shown in fig. 8, the meanings of the symbols in the drawing are as follows:

P _{d_l_tl} (x, y): the upper left corner pixel coordinates of the left eye detection frame of the expected eye image;

P _{d_l_br} (x, y): the lower right corner pixel coordinates of the left eye detection frame of the eye image are expected;

P _{d_r_tl} (x, y): the upper left corner pixel coordinates of the right eye detection frame of the expected human eye image;

P _{d_r_br} (x, y): the lower right pixel coordinates of the right eye detection frame of the eye image are desired.

And 2, detecting the human eye characteristic points of the expected human eye image.

As with the first image, in the left and right human eye frame areas of the desired human eye image, the feature points of the left and right human eyes are detected by the same human eye feature point detection algorithm. The left eye feature point set is marked as S _{d_l} The right eye feature point set is marked as S _{d_r} . The distribution of the left and right eye feature points of the desired eye image is shown in fig. 9.

S408, matching human eye feature points, and calculating an affine transformation matrix.

And performing characteristic point matching on the characteristic point data of the human eye image obtained in the step and the characteristic point data of the expected human eye image obtained by the template, and respectively calculating the transformation relation between the left eye and the right eye. The transformation relationship is represented by an affine transformation matrix M shown in formula (1).

Wherein, human eye characteristic point matching, calculating affine transformation matrix specifically includes the following steps:

step one: first image left eye feature point set S _{i_l} Set S of left eye feature points of expected eye image _{d_l} Matching the characteristic points, calculating a left eye affine transformation matrix, and marking as M _l 。

Step two: left person of first imageEye feature point set S _{i_r} Set S of left eye feature points of expected eye image _{a_r} Matching the characteristic points, calculating a left eye affine transformation matrix, and marking as M _r 。

Wherein M is _l And M _r All can be calculated according to the affine transformation matrix M described above. Wherein in the above formula (1), m ₀₀ 、m ₀₁ 、m ₁₀ 、m ₁₁ Representing 4 elements, from which 4 elements a transformation relationship can be determined, wherein different elements can represent different transformation relationships, specific transformation relationships including rotation, scaling, reflection, symmetry, etc. For example, m ₀₀ 、m ₀₁ 、m ₁₀ 、m ₁₁ When cos α, -sin α, sina, and cos α are in this order, it means that the image is rotated by α ° around the counterclockwise origin.

In the above formula (1), t _x Representing the amount of translation in the X direction. t is t _y Representing the amount of translation in the Y direction.

S409: the first image is a human eye size transform. The step specifically includes the steps of obtaining a transformation matrix M for the S408 _l And M is as follows _r The method for transforming the left and right eyes of the first image comprises the following steps:

step one: left and right eye transformation regions are determined.

The human eye transformation area takes the minimum bounding box of the human eye detection frame of the first image and the human eye detection frame of the human eye template image, as shown in fig. 10. Wherein the left graph in fig. 10 shows a schematic diagram of the overlapping of the left eye image and the corresponding human eye template image, and the right graph in fig. 10 shows a schematic diagram of the overlapping of the right eye and the corresponding human eye template image. In fig. 10, the large rectangular frame on the outer side is the first image human eye detection frame, the ellipse tangent to the first image human eye detection frame represents the corresponding human eye image, the small rectangular frame on the inner side represents the human eye detection frame of the human eye template image, and the ellipse tangent to the second image human eye detection frame represents the corresponding human eye template image.

Wherein the left upper corner of the left human eye transformation area is marked as T _{l_tl} (x, y), lower right corner seat is marked T _{l_br} (x, y). The upper left corner of the right eye transformation area is marked as T _{r_tl} (x, y), lower right corner seat is marked T _{r_br} (x, y). And determining a left human eye transformation area according to the formula (2). And determining a right human eye transformation area according to the formula (3).

Step two: the first image is affine transformed to left and right human eye transformation regions.

As shown in fig. 11, affine transformation is performed according to equation (4) for each pixel point of the left human eye transformation region of the first image, and the transformed left human eye image is denoted as po_l. For each pixel point of the right eye transformation area of the first image, affine transformation is carried out according to a formula (5), and the transformed left eye image is marked as P _{o_r} 。

Wherein in formula (4), P _{i_1} (x, y) represents the coordinates of the i-th required transformation point in the left-eye transformation area. P (P) _{o_l} (x, y) represents the coordinates of the i-th required transformation point in the left-eye transformation area after transformation. T (T) _{l_tl} X represents T _{l_tl} The abscissa, T, of this point _{l_tl} Y represents T _{l_tl} The ordinate of this point.

Wherein in the formula (5), P _i (x, y) represents the coordinates of the i-th required transformation point in the right-eye transformation area. P (P) _{o_r} (x, y) represents the i-th need in the right eye transformation area Transforming the coordinates of the transformed points. T (T) _{r_tl} X represents T _{r_tl} The abscissa, T, of this point _{r_tl} Y represents T _{r_tl} The ordinate of this point.

S410: outputting the transformed image.

After affine transformation is finished on the left and right eyes of the first image, the image is output, and then the size adjustment of the eyes in the first image is finished.

In the embodiment of the application, according to the matching relation between the key points on the human eye template image and the key points of the actual human eye image, the contour of the actual human eye is guided to be adjusted according to the contour of the human eye template. Compared with the mode of directly attaching the human eye template to the second image in the prior art, the invention only utilizes the outline information of the human eye template generated by the AIGC system to carry out transformation adjustment on the human eyes of the second image based on the information, so that the finally output human eyes are more natural.

As shown in fig. 12, in a second aspect, an embodiment of the present application proposes an image processing apparatus 1200, including: a first determining module 1202 for determining an eye region in the first image; an acquisition module 1204, configured to acquire expected image data of an eye; a second determining module 1206 for determining eye contour information from the eye region and determining eye desired contour information from the eye desired image data; the adjustment module 1208 is configured to determine a transformation relationship according to the expected eye contour information and the eye contour information, and adjust the eye region according to the transformation relationship to generate a second image.

The image processing apparatus 1200 of the embodiment of the present application is used for adjusting the size, shape, and the like of an eye region in an image. Specifically, after the first image that the user needs to adjust is acquired, the eye area in the first image may be identified first, and then the eye desired image data may be acquired, so that the eye area in the first image may be adjusted, for example, the size, shape, etc. of the eye area may be adjusted with reference to the eye desired image data. According to the image processing mode, due to the fact that expected image data of the eyes are referred when the eye area is regulated, on one hand, the regulated eye area is more real, and the generated image of the eyes is more vivid. Meanwhile, the adjusted eye area can meet the personalized requirements of the user. On the other hand, the image processing mode can process the eye area in the image according to the method immediately after the user takes the picture, so as to realize real-time processing of the taken image. Meanwhile, the image processing mode automatically completes adjustment through electronic equipment and the like, and a user does not need to perform any additional operation, so that the eye adjustment of the image is more intelligent and convenient.

As shown in fig. 13, an electronic device 1300 according to some embodiments of the present application includes: memory 1310, the memory 1310 stores a program or instructions, and processor 1320, when the processor 1320 executes the program or instructions, implements the steps of the image processing method provided in any of the aspects of the first aspect.

In the embodiment of the present application, the electronic device 1300 has all the advantages defined by the image processing method because it can implement the steps of the image processing method set forth in any of the embodiments described above.

The electronic device in the embodiment of the application may be an apparatus, or may be a component, an integrated circuit, or a chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. By way of example, the mobile electronic device may be a cell phone, tablet computer, notebook computer, palm computer, vehicle-mounted electronic device, wearable device, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook or personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device may be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., and the embodiments of the present application are not limited in particular.

The electronic device in the embodiment of the application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

The electronic device provided in the embodiment of the present application can implement each process implemented by the embodiment of the image processing method, and in order to avoid repetition, details are not repeated here.

A readable storage medium according to some embodiments of the present application has stored thereon a program or instructions which, when executed, implement the steps of the image processing method provided by any of the aspects of the first aspect.

In the embodiments of the present application, the readable storage medium has all the advantages defined by the image processing method because it can implement the steps of the image processing method set forth in any of the embodiments above.

The processor is a processor in the electronic device in the above embodiment. Readable storage media include computer readable storage media such as Read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), magnetic or optical disks, and the like.

An electronic device according to some embodiments of the present application includes the image processing apparatus 1200 or the readable storage medium provided in any one of the aspects of the second aspect. At this time, the electronic apparatus has all the advantageous effects of the image processing device 1200 or the above-described readable storage medium.

It should be noted that, the electronic device in the embodiment of the present application includes a mobile electronic device, such as a mobile phone, and may also include a non-mobile electronic device, such as a computer, a game console, and the like.

Fig. 14 is a schematic hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 2000 includes, but is not limited to: radio frequency unit 2001, network module 2002, audio output unit 2003, input unit 2004, sensor 2005, display unit 2006, user input unit 2007, interface unit 2008, memory 2009, and processor 2010.

Those skilled in the art will appreciate that the electronic device 2000 may further include a power source 2011 (such as a battery) for powering the various components, where the power source 2011 may be logically connected to the processor 2010 through a power management system to perform functions such as managing charging, discharging, and power consumption. The electronic device structure shown in fig. 14 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than shown, or may combine certain components, or may be arranged in different components, which are not described in detail herein.

Wherein the user input unit 2007 receives a first input;

Processor 2010 generates and stores a corresponding original operation record according to the first input, wherein the original operation record includes at least one original operation node;

the user input unit 2007 receives a second input to a target operation node among the operation nodes;

processor 2010 generates adjusted simulated operational records in response to the second input;

and controlling the electronic equipment to run corresponding programs or functions according to the simulated operation records.

Optionally, the first input includes at least one input step, and each original operation node includes one input step and a corresponding operation result;

wherein, the operation result is: after receiving the input step, the program or function of the electronic device outputs a feedback result according to the input step.

The input unit 2004 acquires a program or function corresponding to the first input;

the memory 2009 records each input step and the corresponding operation result according to the input sequence of the input steps;

the processor 2010 correspondingly stores the program or function corresponding to the first input, the input steps and the operation result in the input order, and forms an original operation record.

Optionally, the display unit 2006 displays an identification associated with the original operation record;

The user input unit 2007 receives a third input of the identification;

the display unit 2006 displays the original operation nodes in the original operation record in the input order in response to the third input.

Optionally, the processor 2010 adjusts the target input step corresponding to the target operation node according to the second input, to obtain an adjusted analog input step;

the processor 2010 controls the electronic equipment to run a program or a function corresponding to the target input step according to the analog input step so as to obtain an analog operation result corresponding to the analog input step;

the processor 2010 generates corresponding simulation operation nodes according to the simulation input steps and the simulation operation results, and generates simulation operation records according to the simulation operation nodes;

the input sequence corresponding to the analog operation node is the same as the input sequence corresponding to the target operation node.

Optionally, the user input unit 2007 receives a running input;

processor 2010, in response to the run input, controls the electronic device to run a corresponding program or function according to the simulated operation record.

Optionally, the processor 2010 separately determines a simulated operation result of each simulated operation node in each of the plurality of simulated operation records;

The display unit 2006 displays corresponding prompt information when any two simulated operation records exist and the simulated operation results of corresponding simulated operation nodes in any two simulated operation records are different.

It should be appreciated that in embodiments of the present application, the input unit 2004 may include a graphics processor (Graphics Processing Unit, GPU) 5082 and a microphone 5084, the graphics processor 5082 processing image data of still pictures or video obtained by an image capturing device (e.g., a camera) in a video capturing mode or an image capturing mode.

The display unit 2006 may include a display panel 5122, and the display panel 5122 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 2007 includes a touch panel 5142 and other input devices 5144. The touch panel 5142 is also referred to as a touch screen. The touch panel 5142 may include two parts of a touch detection device and a touch controller. Other input devices 5144 can include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and so forth, which are not described in detail herein. Memory 2009 may be used to store software programs as well as various data including, but not limited to, application programs and an operating system. Processor 2010 may integrate an application processor with a modem processor, wherein the application processor primarily handles operating systems, user interfaces, applications, etc., and the modem processor primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 2010.

The embodiment of the application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running a program or instructions, each process of the embodiment of the image processing method can be realized, the same technical effect can be achieved, and in order to avoid repetition, the description is omitted here.

It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.

In the description of the present specification, reference to the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiments or examples of the present application is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

While embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.

Claims (10)

1. An image processing method, comprising:

determining an eye region in the first image;

acquiring expected image data of eyes;

determining eye contour information according to the eye region, and determining eye expected contour information according to the eye expected image data;

and determining a transformation relation according to the expected eye contour information and the eye contour information, and adjusting the eye region according to the transformation relation to generate a second image.

2. The image processing method according to claim 1, wherein the step of acquiring the eye desired image data includes:

inputting the first image to a generative artificial intelligence system;

when the user does not input the expected eye data, acquiring the expected eye image data generated by the generated artificial intelligence system according to the first image;

when the user inputs eye expected data, the generated artificial intelligence system obtains the eye expected image data generated according to the eye expected data and the first image.

3. The image processing method according to claim 1, wherein the step of determining eye contour information from the eye region and determining eye desired contour information from the eye desired image data includes:

Determining the eye contour information according to a first coordinate set, wherein the first coordinate set comprises coordinate information of a plurality of target points in the eye region;

and determining the expected eye contour information according to a second coordinate set, wherein the second coordinate set comprises coordinate information of the target points in the expected eye image data.

4. The image processing method according to claim 3, wherein the step of determining a transformation relation from the eye expected contour information and the eye contour information, adjusting the eye region according to the transformation relation, and generating the second image includes:

determining a transformation relationship between the first coordinate set and the second coordinate set;

determining a transformation region from the ocular region and the ocular desired image data;

determining a point to be transformed of the eye region according to the transformation region;

and adjusting the point to be transformed of the eye region according to the transformation relation to generate a second image.

5. The image processing method according to claim 3 or 4, wherein,

the target point at least comprises a center point of an eye contour in the eye region, and leftmost, uppermost, rightmost and lowermost points of the eye contour.

6. An image processing apparatus, comprising:

a first determination module for determining an eye region in the first image;

the acquisition module is used for acquiring expected eye image data;

the second determining module is used for determining eye contour information according to the eye region and determining eye expected contour information according to the eye expected image data;

and the adjusting module is used for determining a transformation relation according to the expected eye contour information and the eye contour information, adjusting the eye region according to the transformation relation and generating a second image.

7. The image processing apparatus of claim 6, wherein the acquisition module is configured to:

inputting the first image to a generative artificial intelligence system;

when the user does not input eye requirements, acquiring the eye expected image data generated by the generating artificial intelligence system according to the first image; and/or

When the user inputs eye expected data, the generated artificial intelligence system obtains the eye expected image data generated according to the eye expected data and the first image.

8. The image processing apparatus of claim 6, wherein the second determination module comprises:

A first sub-module for determining the eye profile information according to a first coordinate set, wherein the first coordinate set comprises coordinate information of a plurality of target points in the eye region;

and the second sub-module is used for determining the expected eye contour information according to a second coordinate set, wherein the second coordinate set comprises coordinate information of the target points in the expected eye image data.

9. The image processing apparatus of claim 8, wherein the adjustment module is configured to:

determining a transformation relationship between the first coordinate set and the second coordinate set;

determining a transformation region from the ocular region and the ocular desired image data;

determining a point to be transformed of the eye region according to the transformation region;

and adjusting the point to be transformed of the eye region according to the transformation relation to generate a second image.

10. The image processing apparatus according to claim 8 or 9, wherein,

the target point at least comprises a center point of an eye contour in the eye region, and leftmost, uppermost, rightmost and lowermost points of the eye contour.