CN117746175A - Model training method, image processing method, device and equipment - Google Patents

Model training method, image processing method, device and equipment Download PDF

Info

Publication number
CN117746175A
CN117746175A CN202310887325.5A CN202310887325A CN117746175A CN 117746175 A CN117746175 A CN 117746175A CN 202310887325 A CN202310887325 A CN 202310887325A CN 117746175 A CN117746175 A CN 117746175A
Authority
CN
China
Prior art keywords
image
area
loss function
detection
image area
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310887325.5A
Other languages
Chinese (zh)
Inventor
刘浪涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Vivo Mobile Communication Hangzhou Co Ltd
Original Assignee
Vivo Mobile Communication Hangzhou Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Vivo Mobile Communication Hangzhou Co Ltd filed Critical Vivo Mobile Communication Hangzhou Co Ltd
Priority to CN202310887325.5A priority Critical patent/CN117746175A/en
Publication of CN117746175A publication Critical patent/CN117746175A/en
Pending legal-status Critical Current

Links

Landscapes

  • Image Analysis (AREA)

Abstract

The application discloses a model training method, an image processing device and model training equipment, and belongs to the technical field of image processing. The method comprises the following steps: acquiring an original data set, wherein the original data set comprises at least two first images, the first images comprise first image areas corresponding to shooting objects, and the first image areas comprise reflection areas; obtaining a training data set based on a first sub-image corresponding to the first image area; training the first detection model based on the training data set to obtain a second detection model; the second detection model is used for detecting a reflection area, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.

Description

Model training method, image processing method, device and equipment
Technical Field
The application belongs to the technical field of image processing, and particularly relates to a model training method, an image processing device and model training equipment.
Background
With the increase of the shooting demands of users, how to shoot images with the effect of enlarging the shooting object, such as large moon images, is a hot topic of current research.
Conventionally, in order to obtain an image having an effect of enlarging a subject, it is a conventional method to extract a subject from a photographed image having a subject, process the subject, and fuse the processed subject into a normally exposed image, thereby obtaining an image having an enlarged subject.
However, for an image having a reflection of a subject, it is necessary to detect the reflection area at the same time, but the current reflection area detection result is not accurate enough. And the above solution may cause a problem that the color of the photographing object region and the color of the reflection region of the photographing object are not consistent, thereby causing image distortion.
Disclosure of Invention
The embodiment of the application aims to provide a model training method, an image processing device and equipment, which can improve the detection precision of a reflection area and improve the color consistency of an image.
In a first aspect, an embodiment of the present application provides a model training method, including:
Acquiring an original data set, wherein the original data set comprises at least two first images, the first images comprise first image areas corresponding to shooting objects, and the first image areas comprise reflection areas;
obtaining a training data set based on a first sub-image corresponding to the first image area;
training the first detection model based on the training data set to obtain a second detection model;
the second detection model is used for detecting a reflection area, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
In a second aspect, an embodiment of the present application provides an image processing method, including:
acquiring a second image, wherein the second image comprises a third image area corresponding to a shooting object, and the third image area comprises a reflection area;
inputting a second sub-image corresponding to the third image area into a second detection model to detect a reflection area, so as to obtain a reflection image area;
migrating the image color information corresponding to the inverted image area to a fourth image area to obtain a third image;
The fourth image area is an image area outside the reflection image area in the second image, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
In a third aspect, an embodiment of the present application provides a model training apparatus, including:
the first acquisition module is used for acquiring an original data set, wherein the original data set comprises at least two first images, the first images comprise first image areas corresponding to shooting objects, and the first image areas comprise reflection areas;
the first determining module is used for obtaining a training data set based on a first sub-image corresponding to the first image area;
the second determining module is used for training the first detection model based on the training data set to obtain a second detection model; the second detection model is used for detecting a reflection area, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
In a fourth aspect, an embodiment of the present application provides an image processing apparatus, including:
the acquisition module is used for acquiring a second image, wherein the second image comprises a third image area corresponding to a shooting object, and the third image area comprises a reflection area;
the determining module is used for inputting a second sub-image corresponding to the third image area into a second detection model to detect a reflection area, so as to obtain a reflection image area;
the color migration module is used for migrating the image color information corresponding to the inverted image area to a fourth image area to obtain a third image; the fourth image area is an image area outside the reflection image area in the second image, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
In a fifth aspect, embodiments of the present application provide an electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the method as described in the first aspect and the steps of the method as described in the second aspect.
In a sixth aspect, embodiments of the present application provide a readable storage medium having stored thereon a program or instructions which, when executed by a processor, implement the steps of the method according to the first aspect and the steps of the method according to the second aspect.
In a seventh aspect, embodiments of the present application provide a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and where the processor is configured to execute a program or instructions to implement the steps of the method according to the first aspect and the method according to the second aspect.
In an eighth aspect, embodiments of the present application provide a computer program product stored in a storage medium, the program product being executable by at least one processor to perform the steps of the method according to the first aspect and the method according to the second aspect.
According to the method, a training data set is obtained according to the first sub-image corresponding to the first image area corresponding to the shooting object in each first image in the obtained original data set, then the training data set is utilized to train a first detection model for detecting the reflection area to obtain a second detection model, and because the loss function corresponding to the second detection model comprises a second loss function for calculating angle deviation information and distance deviation information in addition to a conventional CIOU loss function, the number of measurement factors participating in calculation in the loss function in the second detection model is increased, constraint conditions of the loss function are expanded, calculation accuracy of the second detection model is improved, detection accuracy of the model on the reflection area can be improved, and subsequent image processing based on the reflection area is facilitated.
In this embodiment of the present invention, when performing image processing on a back image area, the back image area may be obtained by acquiring a second image including a third image area corresponding to a photographing object, then inputting a second sub-image corresponding to the third image area into a second detection model to detect the back image area, and then migrating image color information corresponding to the back image area to a fourth image area to obtain a third image.
Drawings
FIG. 1 is a flow diagram of a model training method provided in some embodiments of the present application;
FIG. 2 is a schematic illustration of the determination of a second image region, a vertical region, and a reflection region provided by some embodiments of the present application;
FIG. 3 is a flow chart of an image processing method provided in some embodiments of the present application;
FIG. 4 is a flow chart of an image processing method provided by some embodiments of the present application;
FIG. 5 is a schematic diagram of a model training apparatus shown in some embodiments of the present application;
Fig. 6 is a schematic structural view of an image processing apparatus shown in some embodiments of the present application;
FIG. 7 is a schematic diagram of an electronic device shown in some embodiments of the present application;
fig. 8 is a schematic diagram of a hardware architecture of an electronic device as illustrated in some embodiments of the present application.
Detailed Description
Technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of the protection of the present application.
The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type and not limited to the number of objects, e.g., the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.
As described in the background art, in order to solve the problem that the detection result of the ghost detection model in the prior art is not accurate enough and the color of the ghost area in the image is inconsistent with the color of the non-ghost area, the embodiments of the present application provide a model training method, an image processing device and an apparatus, which are used for obtaining a training dataset according to the obtained first sub-image corresponding to the first image area corresponding to the shooting object in each first image in the original dataset, and then training the first detection model for detecting the ghost area by using the training dataset to obtain a second detection model. When the image processing is carried out on the image reversing area, the image reversing area can be obtained by acquiring a second image comprising a third image area corresponding to a shooting object, then inputting a second sub-image corresponding to the third image area into a second detection model to detect the image reversing area, and then transferring image color information corresponding to the image reversing area to a fourth image area to obtain a third image.
The model training method provided by the embodiment of the application is described in detail below through specific embodiments and application scenes thereof with reference to the accompanying drawings.
The technical scheme of the embodiment of the application can be applied to shooting scenes comprising the back image, so that the colors of a non-back image area and a back image area in a shot image are consistent. For example, for a shooting scene with a moon and a reflection corresponding to the moon, the scheme of the embodiment of the present application may be utilized to obtain a reflection region of the moon, and then, the image of the non-reflection region is processed according to the image data of the reflection region of the moon, so as to obtain an image with consistent colors of the reflection region and the non-reflection region.
Fig. 1 is a schematic flow chart of a model training method provided in the embodiment of the present application, and an execution body of the model training method may be an electronic device, where it should be noted that the execution body is not limited to the embodiment of the present application.
In embodiments of the present application, the electronic device may be, but is not limited to, a personal computer (Personal Computer, PC), a smart phone, a tablet computer, or a personal digital assistant (Personal Digital Assistant, PDA), etc.
As shown in fig. 1, the model training method provided in the embodiment of the present application may include steps 110 to 130.
Step 110, acquiring an original data set, wherein the original data set comprises at least two first images, the first images comprise first image areas corresponding to shooting objects, and the first image areas comprise reflection areas.
The initial data set may be an initial training set for training the first detection model, and the training set for training the first detection model may be obtained by selecting the initial training set. The original data set comprises at least two first images, wherein the first images comprise first image areas corresponding to shooting objects, and the first image areas comprise reflection areas corresponding to reflection of the shooting objects.
The first image includes a non-back image region of the subject and a back image region corresponding to the subject. The first image may be captured by the user in real time by using the image capturing device, or may be obtained from a memory of the electronic device, which is not limited in the embodiment of the present application.
The shooting object described above may be a real shooting object. The first image area may be an image area including a back image area, and the first image area may be an area larger than the back image area. The back image region may be a region in which a back image corresponding to the photographing object is located.
In one example, referring to fig. 2, taking a photographed object as a moon as an example, a first image area 26 corresponding to a moon 22 is provided in a first image 21, and a reflection area 24 of the moon is included in the first image area 26.
In some embodiments of the present application, the first image further includes a second image area corresponding to the shooting object. For example, as shown in fig. 2, the first image further includes a second image area 23 corresponding to the moon 22, and the second image area 23 is a non-reflection area.
In some embodiments of the present application, in order to obtain the first image area, the method referred to above may further comprise:
obtaining second parameter information of a vertical area of the second image area according to the size information of the first image and the first parameter information of the second image area;
and obtaining the first image area according to the second parameter information.
The size information of the first image may include width information and height information of the first image.
The first parameter information is parameter information of a second image area, and the second parameter information is parameter information of a vertical area corresponding to the second image area. The vertical area corresponding to the second image area here is a segment of the second image area in the vertical direction that includes the reflection, and is larger than the reflection area.
In some embodiments of the present application, because the image capturing and the object to be captured are necessarily on the gravity line on the first image, and the gravity information of the electronic device is combined, the image capturing and the object to be captured can be detected in the vertical area corresponding to the second image area, so that the parameter information of the vertical area corresponding to the second image area can be obtained according to the size information of the first image and the first parameter information of the second image area, and then the image capturing and the object to be captured can be detected in the vertical area. Therefore, the region for detecting the reflection is reduced, and the calculation power is improved.
In one example, with continued reference to fig. 2, parameter information for the vertical region 26 corresponding to the second image region may be obtained from the size information of the first image 21 and the first parameter information of the second image region.
For any one of the first parameter information and the second parameter information, the parameter information may include coordinate information of a feature point of the corresponding region, and size information of the corresponding region. The feature point here may be a certain feature point in the boundary of the corresponding region, for example, may be a pixel point in the upper left corner of the corresponding region. The size information here may be width information and height information of the corresponding region.
In one example, with continued reference to fig. 2, the pixel point in the lower left corner of the first image 21 is taken as the origin in fig. 2, and first parameter information is taken as an example, and the first parameter information may include coordinate information of the pixel point in the upper left corner of the second image region 23 in fig. 2, and width information and height information of the second image region 23.
In the embodiment of the application, the second parameter information of the vertical area of the second image area is obtained according to the size information of the first image and the first parameter information of the second image area, and then the first image area is obtained according to the second parameter information, namely, the vertical area is the first image area, so that when the ghost area is detected later, the ghost can be detected in the vertical area, the ghost is not required to be detected on the whole first image, the ghost detection area is reduced, and the calculation power is improved.
In some embodiments of the present application, in order to accurately obtain the second parameter information of the vertical area corresponding to the second image area, the obtaining, according to the size information of the first image and the first parameter information of the second image area, the second parameter information of the vertical area of the second image area may specifically include:
Determining width information of the vertical region according to the width information of the second image region;
obtaining the height information of the characteristic points of the vertical region according to the height information of the first image, the height information of the second image region and the ordinate information of the characteristic points of the second image region;
according to the abscissa information of the feature points of the second image area, obtaining the abscissa information of the feature points of the vertical area;
and obtaining the ordinate information of the feature points of the vertical region according to the ordinate information of the feature points of the second image region and the height information of the second image region.
In some embodiments of the present application, the width information of the vertical region may be determined according to the following formula (1) from the width information of the second image region:
w2＝w1 (1)
where w1 is width information of the second image area, and w2 is width information of the vertical area.
In some embodiments of the present application, the height information of the feature points of the vertical region may be obtained according to the following formula (2) according to the height information of the first image and the height information of the second image region, and the ordinate information of the feature points of the second image region:
h2＝h-h1-y1 (2)
where h2 is the height information of the feature point of the vertical region, h is the height information of the first image, h1 is the height information of the second image region, and y1 is the ordinate information of the feature point of the second image region.
In some embodiments of the present application, according to the abscissa information of the feature points of the second image region, the abscissa information of the feature points of the vertical region may be obtained according to the following formula (3);
x2＝x1 (3)
where x2 is the abscissa information of the feature points of the vertical region, and x1 is the abscissa information of the feature points of the second image region.
In some embodiments of the present application, the ordinate information of the feature points of the vertical region may be obtained according to the following formula (4) according to the ordinate information of the feature points of the second image region and the height information of the second image region:
y2＝y1+h1 (4)
where y2 is ordinate information of the feature point of the vertical region.
In one example, with continued reference to fig. 2, from the first size information h of the first image 21 and the coordinate information (x 1, y 1) of the pixel point at the upper left corner of the second image area 23, the height information h1 of the second image area 23, the width information w1 of the second image area 23, the coordinate information (x 2, y 2) of the pixel point at the upper left corner of the vertical area 26 corresponding to the second image area 23, the height information h2 of the vertical area 26 corresponding to the second image area 23, and the width information w2 of the vertical area 26 corresponding to the second image area 23 may be obtained.
In the embodiment of the application, the width information of the vertical area is determined according to the width information of the second image area, the height information of the feature points of the vertical area is obtained according to the height information of the first image and the height information of the second image area and the ordinate information of the feature points of the second image area, the abscissa information of the feature points of the vertical area is obtained according to the abscissa information of the feature points of the second image area and the ordinate information of the feature points of the second image area, and the ordinate information of the feature points of the vertical area is obtained, so that the second parameter information of the vertical area corresponding to the second image area can be accurately calculated, the first image area containing the ghost area is obtained, the ghost area can be detected in the first image area, and the efficiency and the accuracy of the ghost area detection are improved.
And 120, obtaining a training data set based on the first sub-image corresponding to the first image area.
The first sub-image may be an image corresponding to a first image area, and the first sub-image may include an image formed by the first image area to which the reflection area is not marked, and an image to which the reflection area is marked in the first image area.
The training data set may be a training data set using the first sub-image as the training data set, i.e. the training data set includes an image corresponding to the first image region, and a label image marked out in the first image region.
And 130, training the first detection model based on the training data set to obtain a second detection model.
The second detection model may be a model obtained by training the first detection model using a training data set. The first detection model may be, for example, a support vector machine, a neural network model, a decision tree, etc., which is not limited in the embodiments of the present application.
In some embodiments of the present application, a second detection model may be used to detect the ghost areas, and specifically, as shown in fig. 2, the ghost areas 24 in the first image area 26 are detected using the second detection model. The loss function corresponding to the second detection model may include a first loss function and a second loss function, where the first loss function may include a CIOU loss function, the second loss function may be used to calculate angle deviation information and distance deviation information, and a specific second loss function may be used to calculate angle deviation information and distance deviation information of the predicted reflection region and the actual reflection region.
The second detection model is used for detecting the reflection area, specifically, is used for outputting coordinate information of feature points of the reflection area and size information of the reflection area.
In one example, with continued reference to fig. 2, taking the feature point of the reflection area as the pixel point of the upper left corner of the reflection area as an example, coordinate information (x 3, y 3) of the pixel point of the upper left corner of the reflection area 24, and width information w3 of the reflection area 24, and height information h3 of the reflection area 24 may be obtained based on the second detection model.
The following describes in detail the determination of the first loss function and the second loss function, first the determination of the first loss function:
the model training method related to the above may further include:
respectively acquiring a first area of an overlapping part between a first detection area and a second detection area, a second area of a non-overlapping part between the first detection area and the second detection area, a first distance between a characteristic point of the first detection area and a characteristic point of the second detection area, and a first length of a diagonal line of a third detection area;
calculating a first aspect ratio of the first detection area and an aspect ratio of the second detection area, respectively;
The CIOU loss function is derived from the first area, the second area, the first distance, the first length, the first aspect ratio, and the second aspect ratio.
The first detection region may be a reflection region predicted by the first detection model based on the first image region in the first sub-image, the second detection region is a reflection region marked in the first image region of the first sub-image, and the third detection region is a circumscribed detection region of the first detection region and the second detection region.
The first area is an area of an overlapping portion between the first detection region and the second detection region. The second area is an area of a non-overlapping portion between the first detection region and the second detection region.
The feature point of the first detection region may be any point in the first detection region, for example, may be a pixel point in the upper left corner in the first detection region. The feature point of the second detection region may be any point in the second detection region, for example, may be a pixel point in the upper left corner in the second detection region.
The first distance may be a first distance between the feature point of the first detection region and the feature point of the second detection region.
The first length may be a length of a diagonal of the third detection region.
The first aspect ratio may be a ratio of width information and height information of the first detection region. The second aspect ratio may be a ratio of width information and height information of the second detection area.
The regions in the embodiments of the present application are described by taking rectangles as examples, but the embodiments of the present application are not limited to rectangles, and may be other shapes, for example, circles, stars, and the like, and are not limited to the embodiments of the present application.
In the embodiment of the present application, by respectively acquiring the first area of the overlapping portion between the first detection area and the second detection area, the second area of the non-overlapping portion between the first detection area and the second detection area, the first distance between the feature point of the first detection area and the feature point of the second detection area, and the first length of the diagonal line of the third detection area, the first aspect ratio of the first detection area and the aspect ratio of the second detection area are calculated respectively, and according to the first area, the second area, the first distance, the first length, the first aspect ratio, and the second aspect ratio, the CIOU loss function can be accurately obtained, and further the second detection model can be accurately obtained, and further the accuracy of the detection of the ghost area is improved.
In some embodiments of the present application, to further accurately obtain the CIOU loss function, the obtaining the CIOU loss function according to the first area, the second area, the first distance, the first length, the first aspect ratio, and the second aspect ratio may specifically include:
calculating a first ratio of the first area to the second area;
calculating a second ratio based on the first distance and the first length;
and obtaining the CIOU loss function according to the first ratio, the second ratio, the first aspect ratio and the second aspect ratio.
Wherein the first ratio may be a ratio of the first area to the second area. The second ratio may be a ratio of the first distance to the first length.
In some embodiments of the present application, the CIOU loss function may be derived according to the following equation (5) from the first ratio, the second ratio, the first aspect ratio, and the second aspect ratio:
wherein, loss 1 For the first loss function value, IOU is a first ratio, iou=s 1 /S 2 ，S 1 Is a first area, S 2 Is a second area; p is the first distance, c is the first length, v is the aspect ratio similarity of the first detection region and the second detection region, and a is the coefficient of v. The closer the aspect ratio between the first detection area and the second detection area, the closer v is to 0, and a is to 0, otherwise the closer v is to 1, and a is to 0.5.a can be set according to the needs of the user, and is not limited in the embodiment of the application.
Wherein S is 1 ＝(min(x 1 +w 1 ,x 2 +w 2 )-max(x 1 ,x 2 ))×(min(y 1 +h 1 ,y 2 +h 2 )-max(y 1 ,y 2 ))；S 2 ＝w 1 ×h 1 +w 2 ×h 2 -S 1 ；(x 1 ,y 1 ,w 1 ,h 1 ) Coordinate information of the first detection region, (x) 1 ,y 1 ) The coordinates of the feature point of the first detection region may specifically be the coordinates of the pixel point at the upper left corner of the first detection region, w 1 For the width of the first detection region, h 1 For the height of the first detection region, (x) 2 ,y 2 ,w 2 ,h 2 ) For the coordinate information of the second detection region, (x) 2 ,y 2 ) Is the coordinates of the feature points of the second detection region,specifically, the coordinates of the pixel point at the upper left corner of the second detection area, w 2 For the width of the second detection region, h 2 Is the height of the second detection region.
In the above-mentioned formula (5),c 2 ＝(min(x 1 +w 1 ,x 2 +w 2 )-min(x 1 ,x 2 )) 2 +(max(y 1 +h 1 ,y 2 +h 2 )-min(y 1 ,y 2 )) 2 ，/> for the first aspect ratio->Is the second aspect ratio.
In the embodiment of the application, the CIOU loss function can be accurately obtained by calculating the first ratio of the first area to the second area, then calculating the second ratio according to the first distance and the first length, and according to the first ratio, the second ratio, the first aspect ratio and the second aspect ratio, so that the first detection model is accurately trained, and the prediction accuracy of the second detection model is improved.
The determination of the second loss function is described below, and the method referred to above may further comprise:
respectively obtaining a distance association loss value and an angle association loss value;
And obtaining a second loss function based on the distance correlation loss value, the first weight corresponding to the distance correlation loss value, the angle correlation loss value and the second weight corresponding to the angle correlation loss value.
The distance correlation loss value is used for representing distance deviation information, and specifically may be a distance deviation between a line connecting the center of the first detection area and the center of the second image area and a line connecting the center of the second detection area and the center of the second image area. The distance association loss value may be determined based on a second distance between the first detection region and the second image region, and a third distance between the second detection region and the second image region.
The angle correlation loss value is used for representing angle deviation information, and specifically, may be an angle deviation of a line connecting the center of the first detection area and the center of the second image area and a line connecting the center of the second detection area and the center of the second image area. The angle association loss value may be determined based on coordinate information of the feature point of the first detection region, coordinate information of the feature point of the second detection region, the second distance, and the third distance.
The first weight and the second weight may be the weights of the distance association loss value and the angle association loss value, respectively, and the first weight and the second weight may be set according to the user requirement, which is not limited in the embodiment of the present application.
In some embodiments of the present application, from the second distance between the first detection region and the second image region, and the third distance between the second detection region and the second image region, a distance-associated loss value may be determined according to the following formula (6):
wherein, loss d For distance-associated loss value, d 13 At the time of the second distance from the first distance,d 23 for a third distance, +>(x 3 ,y 3 ) The coordinates of the feature points of the second image region may be specifically the pixel points of the upper left corner of the second image regionCoordinates.
Based on the coordinate information of the feature point of the first detection area, the coordinate information of the feature point of the second detection area, the second distance, and the third distance, the angle deviation between the line connecting the center of the first detection area and the center of the second image area and the line connecting the center of the second detection area and the center of the second image area can be determined according to the following formula (7):
wherein, in the formula (7), θ is the angle deviation between the line connecting the center of the first detection area and the center of the second image area and the line connecting the center of the second detection area and the center of the second image area, (x) 1 ,y 1 ) The coordinates of the feature point of the first detection region may specifically be coordinates of the center point of the first detection region, (x 2 ,y 2 ) The coordinates of the feature point of the second detection region may specifically be coordinates of the center point of the second detection region, (x 3 ,y 3 ) The coordinates of the feature point of the second image area may specifically be coordinates of a center point of the second image area.
After obtaining the angle deviation between the connecting line of the center of the first detection area and the center of the second image area and the connecting line of the center of the second detection area and the center of the second image area, obtaining an angle association loss value according to the following formula (8):
Loss angle ＝(1-cosθ)/2 (8)
wherein, loss angle The loss value is associated for the angle.
After obtaining the distance-associated loss value and the distance-associated loss value, a second loss function value may be obtained according to the following formula (9) based on the distance-associated loss value, the first weight corresponding to the distance-associated loss value, the angle-associated loss value, and the second weight corresponding to the angle-associated loss value:
Loss 2 ＝α×Loss d +β×Loss angle (9)
wherein, loss 2 As a second loss functionThe value α is a first weight and β is a second weight.
In some embodiments of the present application, the first weight and the second weight may be taken according to a manner that when an angle deviation between a line connecting a center of the first detection area and a center of the second image area and a line connecting a center of the second detection area and a center of the second image area is smaller, the predicted reflection area is considered to be within a certain tolerant angle range, and at this time, the predicted distance information is considered in an important way, the weight of the distance loss function value is increased, the weight of the angle loss function value is reduced, that is, the first weight is increased, and the second weight is reduced. If the angle deviation between the connecting line of the center of the first detection area and the center of the second image area and the connecting line of the center of the second detection area and the center of the second image area is larger, the predicted reflection area is considered to be beyond a certain tolerance angle range, the difference in distance is not important, the deviation in angle is considered, the weight of the angle loss function value is increased, the weight of the distance loss function value is reduced, namely the second weight is increased, and the first weight is reduced.
In the embodiment of the application, the second loss function value can be accurately obtained through the obtained distance association loss value, the angle association loss value and the first weight corresponding to the distance association loss value and the second weight corresponding to the angle association loss value, so that the first detection model is accurately trained, and the prediction accuracy of the second detection model is improved.
In some embodiments of the present application, in order to obtain an image with a magnified object, in the prior art, the object is extracted from a captured image with the object, then the object is processed, and then the processed object is fused into a normally exposed image, so as to obtain the image with the magnified object. However, the above-described scheme is directed to an image having a back image of an object, and causes a problem in that colors of the object and the back image of the object are not uniform, resulting in distortion of the image.
In order to solve the above-mentioned problems, the present embodiment provides an image processing method, which may first detect a reflection area using the second detection model obtained in the above-mentioned embodiment, and then correct the color of the shooting object area according to the color of the reflection area, so as to ensure that the color of the non-reflection area is consistent with the color of the reflection area.
As shown in fig. 3, the image processing method provided in the embodiment of the present application may include steps 310 to 330.
Step 310, a second image is acquired, wherein the second image includes a third image area corresponding to the shooting object, and the third image area includes a reflection area.
The second image may be an image including a non-back image region to be color corrected, in which a third image region corresponding to the subject is included, and the third image region may include a back image region, and may be a region larger than the back image region, as in the first image region 26 of fig. 2.
The second image here is an image to be processed by the user, and is different from the first image in the above embodiment, which is an image used for training the first detection model, and is an image detected by using the second detection model obtained by training in the above embodiment.
It should be noted that, the determining manner of the third image area is determined according to the size information of the second image and the parameter information of the vertical area of the image area corresponding to the shooting object in the second image, and the determining manner is consistent with the determining manner of the first image area in the foregoing embodiment, that is, the second parameter information of the first image area 26 may be determined according to the size information of the first image 21 and the first parameter information of the second image area 23 in fig. 2, so that the process of determining the first image area according to the second parameter information is not repeated herein.
And 320, inputting a second sub-image corresponding to the third image area into a second detection model to detect the image reflection area, so as to obtain the image reflection area.
The second sub-image may be an image including a third image area, and the image is input into the second detection model for detecting a reflection area, so as to obtain a reflection image area, such as an area 24 in fig. 2.
And 330, migrating the image color information corresponding to the inverted image area to a fourth image area to obtain a third image.
Wherein the fourth image area may be an image area other than the ghost image area in the second image, i.e. the non-reflection area.
In some embodiments of the present application, the loss function corresponding to the second detection model may include a first loss function and a second loss function, where the first loss function includes a CIOU loss function, and the second loss function is used to calculate the angle deviation information and the distance deviation information. The specific determining process of the first loss function and the second loss function is identical to the determining process of the first loss function and the second loss function in the above embodiment, and will not be described herein.
The third image may be an image in which the color of the obtained back image region matches the color of the fourth image region after transferring the image color information corresponding to the back image region to the fourth image region.
In the embodiment of the application, the image area can be obtained by acquiring the first image including the third image area corresponding to the shooting object, then inputting the second sub-image corresponding to the third image area into the second detection model to detect the image area, and then transferring the image color information corresponding to the image area to the fourth image area to obtain the third image.
In some embodiments of the present application, in order to accurately improve the consistency of the color information of the non-reflection area and the color information of the reflection image area, the step 330 may specifically include:
acquiring pixel values of all pixel points in the inverted image area;
determining pixel points with pixel values within a threshold range from all the pixel points to obtain migration pixel points;
and migrating the color information of the migrated pixel points to a fourth image area to obtain a third image.
The threshold range may be a pixel point for performing color migration that is to be satisfied by a pixel value of each pixel point in the pre-set back image area.
The shift pixel point may be a pixel point whose pixel value selected from the pixel points in the back image area is within a threshold range.
In some embodiments of the present application, since there are usually too bright or too dark pixels in the image area of the reflection, the accuracy of color correction may be affected, so that the pixel values of the pixels in the image area of the reflection may be screened from the too dark pixels and the overexposure pixels, specifically, two thresholds may be set, and these two thresholds form a threshold range, the pixels with the pixel values within the threshold range are screened out, so as to obtain a migration pixel, and then the color information of the migration pixel is migrated into a fourth area, so as to obtain the third image.
In the embodiment of the application, the pixel points with the pixel values within the threshold range are determined from the pixel points in the inverted image area to obtain the migration pixel points, then the color information of the migration pixel points is migrated to the fourth image area to obtain the third image, so that the over-dark and over-exposure pixel points in the inverted image area are removed, only the effective points are reserved to participate in the calculation of the color migration, the correction of the color of the fourth image area is accurately realized, and the consistency of the color information of the inverted image area and the color information of the fourth image area is further improved.
In some embodiments of the present application, the second image may be in RGB color format, but in the color correction process, it is necessary to convert it into LAB color space and then perform color correction. How to convert the first image from the RGB color space to the LAB color space belongs to the prior art, and is not described here.
In order to further accurately realize the correction of the color of the non-reflection area, the transferring the color information of the transferred pixel point to the fourth image area to obtain the third image may specifically include:
calculating a first average value of pixel values of all pixel points in the fourth image area in each channel of the LAB channel, and a first standard deviation of pixel values of all pixel points in the fourth image area in each channel of the LAB channel;
calculating a second average value of pixel values of the migration pixel points in each of the LAB channels, and a second standard deviation of pixel values of the migration pixel points in each of the LAB channels;
and processing the pixel points in the fourth image area according to the first mean value, the first standard deviation, the second mean value and the second standard deviation to obtain a third image.
The first average value may be an average value of pixel values of each pixel point in the fourth image area in each channel in the LAB channel.
The first standard deviation may be a standard deviation of pixel values of the pixels in the fourth image area for each of the LAB channels.
The second average value may be an average value of pixel values of the shifted pixel points in each of the LAB channels.
The second standard deviation may be a standard deviation of pixel values of the shifted pixel points for each of the LAB channels.
In some embodiments of the present application, the first average value of the pixel values of the pixels in the fourth image region in each of the LAB channels may be calculated according to the following formula (10):
wherein m is the number of pixel rows of the fourth image area, m is the number of pixel columns of the fourth image area,and->Average value of pixel values of each pixel point in the fourth image area in each LAB channel, (pixel) L ，pixel A ，pixel B ) And (3) pixel values of each channel in the LAB channel for each pixel point in the fourth image area.
The first standard deviation of the pixel values of the pixels in the fourth image area for each of the LAB channels may be calculated according to the following equation (11):
wherein,and->The pixel values of the pixels in the fourth image region are standard deviations of the channels in the LAB channel, respectively.
The second average of the pixel values of the migrated pixel points in each of the LAB channels may be calculated according to the following equation (12):
Wherein n is the number of pixel rows in the fourth image area, n is the number of pixel columns in the fourth image area, and->The average value of the pixel values of the migration pixel points in each LAB channel, (pixel) L ，pixel A ，pixel B ) And the pixel value of each channel in the LAB channel is used for each migration pixel point.
The second standard deviation of the pixel values of the migrated pixel points in each of the LAB channels may be calculated according to the following equation (13):
wherein,and the standard deviation of the pixel values of the migration pixel points in each of the LAB channels respectively.
In some embodiments of the present application, after calculating the first mean, the first standard deviation, the second mean, and the second standard deviation according to the formulas (10) - (13), the pixel points of the fourth image area in the first image may be processed according to the first mean, the first standard deviation, the second mean, and the second standard deviation, so as to obtain the third image.
In the embodiment of the present application, the first average value of the pixel values of each pixel point in the fourth image area in each LAB channel is calculated, the first standard deviation of the pixel values of each pixel point in the fourth image area in each LAB channel is calculated, the second average value of the pixel values of the migration pixel points in each LAB channel is calculated, and the second standard deviation of the pixel values of the migration pixel points in each LAB channel is calculated, and then the pixel points in the fourth image area are processed according to the first average value, the first standard deviation, the second average value and the second standard deviation, so that the pixel points of the fourth image area can be accurately processed, and the color correction of the fourth image area can be further accurately realized.
In some embodiments of the present application, in order to further accurately implement correction of the color of the non-reflection area, the processing, according to the first mean value, the first standard deviation, the second mean value, and the second standard deviation, the pixel points in the fourth image area to obtain the third image may specifically include:
for each pixel point in the fourth image area, the following process is performed to obtain a third image:
calculating a difference value between a pixel value of the pixel point and the first average value to obtain a first difference value;
calculating the ratio of the first standard deviation to the second standard deviation to obtain a first ratio;
and processing the pixel points in the fourth image area based on the first difference value, the first ratio and the second average value to obtain a third image.
Wherein, for each pixel point in the fourth image area, the first difference value may be a difference value between a pixel value of the pixel point and the first average value.
The first ratio may be a ratio of the first standard deviation and the second standard deviation.
In some embodiments of the present application, after obtaining the first difference value, the first ratio value, and the second average value, the pixel points in the fourth image area may be processed based on the first difference value, the first ratio value, and the second average value to obtain a third image, specifically, for each pixel point in the fourth image area, the pixel value of the pixel point is subtracted from the first average value to obtain the first difference value, and then the first ratio value is multiplied by the second average value, and then the color information of the migrated pixel point may be migrated into the fourth image area, and specifically may be as shown in formula (14):
In some embodiments of the present application, after the third image is obtained, the third image is in the LAB color space, and since the third image viewed by the user needs to be in the RGB color space, the third image needs to be reconverted from the LAB color space to the RGB color space.
In the embodiment of the application, for each pixel point in the fourth image area, calculating the difference between the pixel value of the pixel point and the first average value to obtain a first difference value, calculating the ratio of the first standard deviation to the second standard deviation to obtain a first ratio value, and processing the pixel point in the fourth image area based on the first difference value, the first ratio value and the second average value to obtain a third image, so that the pixel point in the fourth image area can be accurately processed to further accurately realize the correction of the color of the fourth image area and keep the consistency of the color of the fourth image area and the color of the inverted image area.
In some embodiments of the present application, in order to more clearly understand the technical solutions of the embodiments of the present application, another implementation manner of the image processing method is further provided in the embodiments of the present application, and fig. 4 is a schematic flow chart of an image processing method provided in the embodiments of the present application, and as shown in fig. 4, the image processing method provided in the embodiments of the present application may include steps 410 to 430.
Step 410, training the first detection model to obtain a second detection model.
In some embodiments of the present application, the training process of the first detection model may refer to the process from step 110 to step 130 in the above-mentioned embodiment of the model training method, which is not described herein.
And step 420, inputting a second sub-image corresponding to a third image area in the second image into a second detection model to detect the reflection area, and obtaining the reflection image area.
Step 420 corresponds to step 320 in the above-described embodiment of the image processing method, and is not described herein.
Step 430, color migration.
In some embodiments of the present application, specific color migration may refer to the procedure of step 330 in the above-mentioned image processing method embodiment, which is not described herein.
According to the model training method provided by the embodiment of the application, the execution subject can be a model training device. In the embodiment of the present application, a model training device executes a model training method as an example, and the model training device provided in the embodiment of the present application is described.
Fig. 5 is a schematic diagram showing a structure of a model training apparatus according to an exemplary embodiment.
As shown in fig. 5, the model training apparatus 500 may include:
A first obtaining module 510, configured to obtain an original data set, where the original data set includes at least two first images, where the first images include a first image area corresponding to a shooting object, and the first image area includes a reflection area;
a first determining module 520, configured to obtain a training data set based on a first sub-image corresponding to the first image area;
a second determining module 530, configured to train the first detection model based on the training data set, to obtain a second detection model; the second detection model is used for detecting a reflection area, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
In the embodiment of the application, a training data set is obtained according to the obtained first sub-image corresponding to the first image area corresponding to the shooting object in each first image in the original data set, then the training data set is utilized to train the first detection model for detecting the reflection area, and a second detection model is obtained.
In some embodiments of the present application, the apparatus referred to above may further comprise:
a second obtaining module, configured to obtain a first area of an overlapping portion between a first detection area and a second detection area, a second area of a non-overlapping portion between the first detection area and the second detection area, a first distance between a feature point of the first detection area and a feature point of the second detection area, and a first length of a diagonal line of a third detection area, where the first detection area is a reflection area in the first sub-image predicted based on the first detection model, the second detection area is a reflection area marked in the first sub-image, and the third detection area is an external detection area of the first detection area and the second detection area;
a calculating module for calculating a first aspect ratio of the first detection area and an aspect ratio of the second detection area, respectively;
and a third determining module, configured to obtain the CIOU loss function according to the first area, the second area, the first distance, the first length, the first aspect ratio, and the second aspect ratio.
In some embodiments of the present application, the third determining module may specifically be configured to:
calculating a first ratio of the first area to the second area;
calculating a second ratio based on the first distance and the first length;
and obtaining the CIOU loss function according to the first ratio, the second ratio, the first aspect ratio and the second aspect ratio.
In some embodiments of the present application, the first image may further include a second image area corresponding to the photographed object, and the apparatus referred to above may further include:
a third obtaining module, configured to obtain a distance association loss value and an angle association loss value, where the distance association loss value is used to represent the distance deviation information, the distance association loss value is determined based on a second distance between the first detection area and the second image area, and a third distance between the second detection area and the second image area, the angle association loss value is used to represent the angle deviation information, and the angle association loss value is determined based on coordinate information of a feature point of the first detection area, coordinate information of a feature point of the second detection area, the second distance, and the third distance;
And a fourth determining module, configured to obtain the second loss function based on the distance-associated loss value, a first weight corresponding to the distance-associated loss value, the angle-associated loss value, and a second weight corresponding to the angle-associated loss value.
In some embodiments of the present application, the apparatus referred to above may further comprise:
a fifth determining module, configured to obtain second parameter information of a vertical area of the second image area according to the size information of the first image and the first parameter information of the second image area;
and a sixth determining module, configured to obtain the first image area according to the second parameter information.
According to the image processing method provided by the embodiment of the application, the execution subject can be an image processing device. In the embodiment of the present application, an image processing apparatus provided in the embodiment of the present application will be described by taking an example in which the image processing apparatus executes an image processing method.
Fig. 6 is a schematic structural view of an image processing apparatus according to an exemplary embodiment.
As shown in fig. 6, the image processing apparatus 600 may include:
an obtaining module 610, configured to obtain a second image, where the second image includes a third image area corresponding to a shooting object, and the third image area includes a reflection area;
A determining module 620, configured to input a second sub-image corresponding to the third image area into a second detection model to perform a detection of a reflection area, so as to obtain a reflection image area;
the color migration module 630 is configured to migrate the image color information corresponding to the inverted image area to a fourth image area, so as to obtain a third image; the fourth image area is an image area outside the reflection image area in the second image, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
In the embodiment of the application, the image area can be obtained by acquiring the first image including the third image area corresponding to the shooting object, then inputting the second sub-image corresponding to the third image area into the second detection model to detect the image area, and then transferring the image color information corresponding to the image area to the fourth image area to obtain the third image.
In some embodiments of the present application, the color migration module 630 may specifically include:
an acquisition unit configured to acquire pixel values of each pixel point in the back image area;
the first determining unit is used for determining pixel points with pixel values within a threshold range from the pixel points to obtain migration pixel points;
and the second determining unit is used for migrating the color information of the migration pixel points to the fourth image area to obtain a third image.
In some embodiments of the present application, the second determining unit may specifically be configured to:
calculating a first average value of pixel values of all pixel points in the fourth image area in each channel of the LAB channel, and a first standard deviation of pixel values of all pixel points in the fourth image area in each channel of the LAB channel;
calculating a second average value of the pixel values of the migration pixel points in each of the LAB channels, and a second standard deviation of the pixel values of the migration pixel points in each of the LAB channels;
and processing the pixel points in the fourth image area according to the first mean value, the first standard deviation, the second mean value and the second standard deviation to obtain a third image.
The model training device and the image processing device in the embodiments of the present application may be an electronic device, or may be a component in an electronic device, for example, an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices than a terminal. By way of example, the electronic device may be a mobile phone, tablet computer, notebook computer, palm computer, vehicle-mounted electronic device, mobile internet appliance (Mobile Internet Device, MID), augmented reality (augmented reality, AR)/Virtual Reality (VR) device, robot, wearable device, ultra-mobile personal computer, UMPC, netbook or personal digital assistant (personal digital assistant, PDA), etc., but may also be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., and the embodiments of the present application are not limited in particular.
The model training apparatus and the image processing apparatus in the embodiments of the present application may be an apparatus having an operating system. The operating system may be an Android operating system, an ios operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.
The model training device and the image processing device provided in the embodiments of the present application can respectively implement each process implemented by the method embodiments of fig. 1 and fig. 3, so as to achieve the same technical effect, and in order to avoid repetition, no description is repeated here.
Optionally, as shown in fig. 7, the embodiment of the present application further provides an electronic device 700, including a processor 701 and a memory 702, where the memory 702 stores a program or an instruction that can be executed on the processor 701, and the program or the instruction implements each step of the foregoing model training method and the image processing method embodiment when executed by the processor 701, and the steps achieve the same technical effects, so that repetition is avoided and redundant description is omitted herein.
The electronic device in the embodiment of the application includes the mobile electronic device and the non-mobile electronic device described above.
Fig. 8 is a schematic hardware structure of an electronic device implementing an embodiment of the present application.
The electronic device 800 includes, but is not limited to: radio frequency unit 801, network module 802, audio output unit 803, input unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, and processor 810.
Those skilled in the art will appreciate that the electronic device 800 may also include a power source (e.g., a battery) for powering the various components, which may be logically connected to the processor 810 by a power management system to perform functions such as managing charge, discharge, and power consumption by the power management system. The electronic device structure shown in fig. 8 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than shown, or may combine certain components, or may be arranged in different components, which are not described in detail herein.
In one case, when the electronic device 800 performs the model training method shown in fig. 1 described above, the following functions may be implemented by the components:
the processor 810 is configured to obtain an original data set, where the original data set includes at least two first images, the first images include a first image area corresponding to a shooting object, and the first image area includes a reflection area; obtaining a training data set based on a first sub-image corresponding to the first image area; training the first detection model based on the training data set to obtain a second detection model; the second detection model is used for detecting a reflection area, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
In this way, a training data set is obtained according to the first sub-image corresponding to the first image area corresponding to the shooting object in each first image in the obtained original data set, then the training data set is utilized to train the first detection model for detecting the reflection area to obtain the second detection model, and as the loss function corresponding to the second detection model comprises the second loss function for calculating the angle deviation information and the distance deviation information in addition to the conventional CIOU loss function, the number of measurement factors participating in calculation in the loss function in the second detection model is increased, the constraint condition of the loss function is expanded, the calculation accuracy of the second detection model is further improved, the detection accuracy of the model on the reflection area can be improved, and the subsequent image processing based on the reflection area is facilitated. When the image processing is carried out on the image reversing area, the image reversing area can be obtained by acquiring a second image comprising a third image area corresponding to a shooting object, then inputting a second sub-image corresponding to the third image area into a second detection model to detect the image reversing area, and then transferring image color information corresponding to the image reversing area to a fourth image area to obtain a third image.
In some embodiments, the processor 810 is further configured to obtain a first area of an overlapping portion between a first detection region and a second detection region, a second area of a non-overlapping portion between the first detection region and the second detection region, a first distance between a feature point of the first detection region and a feature point of the second detection region, and a first length of a diagonal line of a third detection region, where the first detection region is a reflection region in the first sub-image predicted based on the first detection model, the second detection region is a reflection region marked in the first sub-image, and the third detection region is a circumscribed detection region of the first detection region and the second detection region; calculating a first aspect ratio of the first detection region and an aspect ratio of the second detection region, respectively; and obtaining the CIOU loss function according to the first area, the second area, the first distance, the first length, the first aspect ratio and the second aspect ratio.
Thus, by respectively obtaining the first area of the overlapping portion between the first detection region and the second detection region, the second area of the non-overlapping portion between the first detection region and the second detection region, the first distance between the feature point of the first detection region and the feature point of the second detection region, and the first length of the diagonal line of the third detection region, the first aspect ratio of the first detection region and the aspect ratio of the second detection region are respectively calculated, and according to the first area, the second area, the first distance, the first length, the first aspect ratio and the second aspect ratio, the CIOU loss function can be accurately obtained, the second detection model can be accurately obtained, and the accuracy of detection of the ghost region can be further improved.
In some embodiments, processor 810 is further configured to calculate a first ratio of the first area and the second area; calculating a second ratio based on the first distance and the first length; and obtaining the CIOU loss function according to the first ratio, the second ratio, the first aspect ratio and the second aspect ratio.
Therefore, the CIOU loss function can be accurately obtained by calculating the first ratio of the first area to the second area, then calculating the second ratio according to the first distance and the first length, and according to the first ratio, the second ratio, the first aspect ratio and the second aspect ratio, so that the first detection model is accurately trained, and the prediction accuracy of the second detection model is improved.
In some embodiments, the processor 810 is further configured to obtain a distance-associated loss value and an angle-associated loss value, respectively, wherein the distance-associated loss value is used to characterize the distance deviation information, the distance-associated loss value is determined based on a second distance between the first detection region and the second image region, and a third distance between the second detection region and the second image region, the angle-associated loss value is used to characterize the angle deviation information, and the angle-associated loss value is determined based on coordinate information of feature points of the first detection region, coordinate information of feature points of the second detection region, the second distance, and the third distance; and obtaining the second loss function based on the distance correlation loss value, the first weight corresponding to the distance correlation loss value, the angle correlation loss value and the second weight corresponding to the angle correlation loss value.
Therefore, the second loss function value can be accurately obtained through the obtained distance correlation loss value, the angle correlation loss value, the first weight corresponding to the distance correlation loss value and the second weight corresponding to the angle correlation loss value, and then the first detection model is accurately trained, so that the prediction accuracy of the second detection model is improved.
In some embodiments, the processor 810 is further configured to obtain second parameter information of a vertical area of the second image area according to the size information of the first image and the first parameter information of the second image area; and obtaining the first image area according to the second parameter information.
In this way, the second parameter information of the vertical area of the second image area is obtained according to the size information of the first image and the first parameter information of the second image area, and then the first image area is obtained according to the second parameter information, namely the vertical area is the first image area, so that when the ghost area is detected later, the ghost can be detected in the vertical area, the ghost is not required to be detected on the whole first image, the ghost detection area is reduced, and the calculation power is improved.
In one case, when the electronic device 800 performs the image processing method shown in fig. 3 described above, the respective components can realize the following functions:
a processor 810, configured to acquire a second image, where the second image includes a third image area corresponding to a shooting object, and the third image area includes a reflection area; inputting a second sub-image corresponding to the third image area into a second detection model to detect a reflection area, so as to obtain a reflection image area; migrating the image color information corresponding to the inverted image area to a fourth image area to obtain a third image; the fourth image area is an image area outside the reflection image area in the second image, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
In this way, by acquiring the first image including the third image area corresponding to the photographing object, then inputting the second sub-image corresponding to the third image area into the second detection model to detect the back image area, the back image area can be obtained, and then transferring the image color information corresponding to the back image area to the fourth image area, so as to obtain the third image, since the color of the fourth image area is transferred from the color of the back image area, the color of the fourth image area in the third image is consistent with the color of the back image area, and the third image is not distorted.
In some embodiments, the processor 810 is further configured to obtain a pixel value of each pixel point in the inverted image area; determining pixel points with pixel values within a threshold range from the pixel points to obtain migration pixel points; and migrating the color information of the migrated pixel points to the fourth image area to obtain a third image.
In this way, the pixel points with pixel values within the threshold range are determined from all the pixel points in the inverted image area to obtain the migration pixel points, then the color information of the migration pixel points is migrated to the fourth image area to obtain the third image, so that the over-dark and over-exposed pixel points in the inverted image area are removed, only the effective points are reserved to participate in the calculation of the color migration, the correction of the color of the fourth image area is accurately realized, and the consistency of the color information of the inverted image area and the color information of the fourth image area is further improved.
In some embodiments, the processor 810 is further configured to calculate a first average value of pixel values of the pixels in the fourth image area in each of the LAB channels, and a first standard deviation of pixel values of the pixels in the fourth image area in each of the LAB channels; calculating a second average value of the pixel values of the migration pixel points in each of the LAB channels, and a second standard deviation of the pixel values of the migration pixel points in each of the LAB channels; and processing the pixel points in the fourth image area according to the first mean value, the first standard deviation, the second mean value and the second standard deviation to obtain a third image.
In this way, by calculating the first average value of the pixel value of each pixel point in the fourth image area in each channel of the LAB channel, the first standard deviation of the pixel value of each pixel point in the fourth image area in each channel of the LAB channel, the second average value of the pixel value of the migration pixel point in each channel of the LAB channel, and the second standard deviation of the pixel value of the migration pixel point in each channel of the LAB channel, then processing the pixel points in the fourth image area according to the first average value, the first standard deviation, the second average value and the second standard deviation, a third image can be obtained, and thus the pixel points in the fourth image area can be accurately processed to further accurately realize the color correction of the fourth image area.
It should be appreciated that in embodiments of the present application, the input unit 804 may include a graphics processor (Graphics Processing Unit, GPU) 8041 and a microphone 8042, with the graphics processor 8041 processing image data of still pictures or video obtained by an image capturing device (e.g., a camera) in a video capturing mode or an image capturing mode. The display unit 806 may include a display panel 8061, and the display panel 8061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. Touch panel 8071, also referred to as a touch screen. The touch panel 8071 may include two parts, a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and so forth, which are not described in detail herein.
The memory 809 can be used to store software programs as well as various data. The memory 809 may mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area may store an operating system, application programs or instructions (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like. Further, the memory 809 may include volatile memory or nonvolatile memory, or the memory 809 may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM), static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (Double Data Rate SDRAM), enhanced SDRAM (ESDRAM), synchronous DRAM (sync linkDRAM, SLDRAM), and Direct RAM (DRRAM). Memory 809 in embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.
The processor 810 may include one or more processing units; optionally, the processor 810 integrates an application processor that primarily processes operations involving an operating system, user interface, application programs, etc., and a modem processor that primarily processes wireless communication signals, such as a baseband processor. It will be appreciated that the modem processor described above may not be integrated into the processor 810.
The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the embodiment of the model training method and each process of the embodiment of the image processing method, and the same technical effect can be achieved, so that repetition is avoided, and no further description is given here.
Wherein the processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes computer readable storage medium such as computer readable memory ROM, random access memory RAM, magnetic or optical disk, etc.
The embodiment of the application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is configured to run a program or instructions, implement each process of the model training method embodiment and each process of the image processing method embodiment, and achieve the same technical effect, so that repetition is avoided, and no further description is provided here.
It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.
The embodiments of the present application provide a computer program product stored in a storage medium, where the program product is executed by at least one processor to implement the respective processes of the model training method embodiment and the respective processes of the image processing method embodiment, and achieve the same technical effects, and are not repeated herein.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a computer software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods described in the embodiments of the present application.
The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those of ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are also within the protection of the present application.

Claims (11)

1. A method of model training, the method comprising:
acquiring an original data set, wherein the original data set comprises at least two first images, the first images comprise first image areas corresponding to shooting objects, and the first image areas comprise reflection areas;
obtaining a training data set based on a first sub-image corresponding to the first image area;
training the first detection model based on the training data set to obtain a second detection model;
the second detection model is used for detecting a reflection area, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
2. The method according to claim 1, wherein the method further comprises:
respectively acquiring a first area of an overlapping part between a first detection area and a second detection area, a second area of a non-overlapping part between the first detection area and the second detection area, a first distance between a characteristic point of the first detection area and a characteristic point of the second detection area, and a first length of a diagonal line of a third detection area, wherein the first detection area is a reflection area in the first sub-image predicted based on the first detection model, the second detection area is a reflection area marked in the first sub-image, and the third detection area is an external detection area of the first detection area and the second detection area;
Calculating a first aspect ratio of the first detection region and an aspect ratio of the second detection region, respectively;
and obtaining the CIOU loss function according to the first area, the second area, the first distance, the first length, the first aspect ratio and the second aspect ratio.
3. The method of claim 2, wherein the deriving the CIOU loss function from the first area, the second area, the first distance, the first length, the first aspect ratio, and the second aspect ratio comprises:
calculating a first ratio of the first area to the second area;
calculating a second ratio based on the first distance and the first length;
and obtaining the CIOU loss function according to the first ratio, the second ratio, the first aspect ratio and the second aspect ratio.
4. The method of claim 1, wherein the first image further comprises a second image region corresponding to the subject, the method further comprising:
respectively acquiring a distance correlation loss value and an angle correlation loss value, wherein the distance correlation loss value is used for representing the distance deviation information, the distance correlation loss value is determined based on a second distance between the first detection area and the second image area and a third distance between the second detection area and the second image area, the angle correlation loss value is used for representing the angle deviation information, and the angle correlation loss value is determined based on coordinate information of characteristic points of the first detection area, coordinate information of characteristic points of the second detection area, the second distance and the third distance;
And obtaining the second loss function based on the distance correlation loss value, the first weight corresponding to the distance correlation loss value, the angle correlation loss value and the second weight corresponding to the angle correlation loss value.
5. The method according to claim 4, wherein the method further comprises:
obtaining second parameter information of a vertical area of the second image area according to the size information of the first image and the first parameter information of the second image area;
and obtaining the first image area according to the second parameter information.
6. An image processing method, the method comprising:
acquiring a second image, wherein the second image comprises a third image area corresponding to a shooting object, and the third image area comprises a reflection area;
inputting a second sub-image corresponding to the third image area into a second detection model to detect a reflection area, so as to obtain a reflection image area;
migrating the image color information corresponding to the inverted image area to a fourth image area to obtain a third image;
the fourth image area is an image area outside the reflection image area in the second image, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
7. The method according to claim 6, wherein the transferring the image color information corresponding to the back image area to the fourth image area to obtain the third image includes:
acquiring pixel values of all pixel points in the inverted image area;
determining pixel points with pixel values within a threshold range from the pixel points to obtain migration pixel points;
and migrating the color information of the migrated pixel points to the fourth image area to obtain a third image.
8. The method of claim 7, wherein migrating the color information of the migrated pixel point to the fourth image area to obtain a third image, comprises:
calculating a first average value of pixel values of all pixel points in the fourth image area in each channel of the LAB channel, and a first standard deviation of pixel values of all pixel points in the fourth image area in each channel of the LAB channel;
calculating a second average value of the pixel values of the migration pixel points in each of the LAB channels, and a second standard deviation of the pixel values of the migration pixel points in each of the LAB channels;
and processing the pixel points in the fourth image area according to the first mean value, the first standard deviation, the second mean value and the second standard deviation to obtain a third image.
9. A model training apparatus, the apparatus comprising:
the first acquisition module is used for acquiring an original data set, wherein the original data set comprises at least two first images, the first images comprise first image areas corresponding to shooting objects, and the first image areas comprise reflection areas;
the first determining module is used for obtaining a training data set based on a first sub-image corresponding to the first image area;
the second determining module is used for training the first detection model based on the training data set to obtain a second detection model; the second detection model is used for detecting a reflection area, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
10. An image processing apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring a second image, wherein the second image comprises a third image area corresponding to a shooting object, and the third image area comprises a reflection area;
The determining module is used for inputting a second sub-image corresponding to the third image area into a second detection model to detect a reflection area, so as to obtain a reflection image area;
the color migration module is used for migrating the image color information corresponding to the inverted image area to a fourth image area to obtain a third image; the fourth image area is an image area outside the reflection image area in the second image, the loss function corresponding to the second detection model comprises a first loss function and a second loss function, the first loss function comprises a CIOU loss function, and the second loss function is used for calculating angle deviation information and distance deviation information.
11. An electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the model training method of any one of claims 1-5, and the steps of the image processing method of any one of claims 6-8.
CN202310887325.5A 2023-07-19 2023-07-19 Model training method, image processing method, device and equipment Pending CN117746175A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310887325.5A CN117746175A (en) 2023-07-19 2023-07-19 Model training method, image processing method, device and equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310887325.5A CN117746175A (en) 2023-07-19 2023-07-19 Model training method, image processing method, device and equipment

Publications (1)

Publication Number Publication Date
CN117746175A true CN117746175A (en) 2024-03-22

Family

ID=90281891

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310887325.5A Pending CN117746175A (en) 2023-07-19 2023-07-19 Model training method, image processing method, device and equipment

Country Status (1)

Country Link
CN (1) CN117746175A (en)

Similar Documents

Publication Publication Date Title
WO2019134516A1 (en) Method and device for generating panoramic image, storage medium, and electronic apparatus
WO2018176925A1 (en) Hdr image generation method and apparatus
US20110211233A1 (en) Image processing device, image processing method and computer program
CN111556336B (en) Multimedia file processing method, device, terminal equipment and medium
CN111028276A (en) Image alignment method and device, storage medium and electronic equipment
US20240163566A1 (en) Exposure compensation method and apparatus, and electronic device
CN112258404A (en) Image processing method, image processing device, electronic equipment and storage medium
CN112532881A (en) Image processing method and device and electronic equipment
CN116612015A (en) Model training method, image mole pattern removing method and device and electronic equipment
CN112333385B (en) Electronic anti-shake control method and device
CN113989387A (en) Camera shooting parameter adjusting method and device and electronic equipment
CN117746175A (en) Model training method, image processing method, device and equipment
CN115623313A (en) Image processing method, image processing apparatus, electronic device, and storage medium
CN116261043A (en) Focusing distance determining method, device, electronic equipment and readable storage medium
CN115439386A (en) Image fusion method and device, electronic equipment and storage medium
CN113129227A (en) Image processing method, image processing device, computer equipment and storage medium
CN114782261B (en) Image processing method and device, electronic equipment and readable storage medium
CN112367470B (en) Image processing method and device and electronic equipment
CN111353929A (en) Image processing method and device and electronic equipment
CN115423823A (en) Image processing method and device
CN117750215A (en) Shooting parameter updating method and electronic equipment
CN116416141A (en) Image processing method and device, equipment and storage medium
CN113034611A (en) Operation evaluation method, operation evaluation device, electronic apparatus, and medium
CN116342992A (en) Image processing method and electronic device
CN115883986A (en) Image processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination