CN110213605B - Image correction method, device and equipment - Google Patents
Image correction method, device and equipment Download PDFInfo
- Publication number
- CN110213605B CN110213605B CN201910453576.6A CN201910453576A CN110213605B CN 110213605 B CN110213605 B CN 110213605B CN 201910453576 A CN201910453576 A CN 201910453576A CN 110213605 B CN110213605 B CN 110213605B
- Authority
- CN
- China
- Prior art keywords
- image
- rotation angle
- rotation
- rotating
- degrees
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/24—Aligning, centring, orientation detection or correction of the image
- G06V10/242—Aligning, centring, orientation detection or correction of the image by image rotation, e.g. by 90 degrees
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/65—Transmission of management data between client and server
- H04N21/658—Transmission by the client directed to the server
Abstract
The application discloses an image correction method, device and equipment. The method comprises the following steps: training an image set with a known rotation angle as a sample to obtain a rotation angle recognition model, receiving an acquired image to be detected, inputting the image to be detected into the rotation angle recognition model for recognition to obtain a rotation angle of the image to be detected, and performing rotation correction on the image to be detected according to the rotation angle to obtain a target forward image corresponding to the image to be detected, wherein the rotation angle of the target forward image is 0 degree. After the scheme is adopted, the live broadcast platform can directly identify the forward image, so that the content in the image can be identified, then the identified content is subjected to operations such as beautifying, special effect adding or dubbing processing, the live broadcast effect is enhanced, and the user viscosity of the live broadcast platform is improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to an image correction method, apparatus, and device.
Background
With the development of internet technology, live broadcast platforms are more and more popular, users can directly conduct live broadcast on the platforms through cameras, and users conducting live broadcast are called as anchor broadcasts. Live content is also becoming more diverse. The current popular live broadcast types include game live broadcast, sports event live broadcast, show live broadcast, outdoor live broadcast, E-commerce live broadcast and the like.
In order to make live broadcast attract more viewers, the existing live broadcast platform can also process images collected by a camera, for example, the collected images can be subjected to processing such as beautifying, special effect adding or dubbing operation, which all require that images input by the camera are forward.
At present, the placing position of a camera of some users during live broadcasting may not be forward, so that the finally acquired image is not forward and has a rotation angle, when the acquired image is processed on a live broadcasting platform, the acquired image cannot be identified due to the fact that the rotation angle of the image is unknown, the acquired image cannot be subjected to beautifying, special effect adding or dubbing operation and the like, and the live broadcasting effect is influenced.
Disclosure of Invention
The embodiment of the application provides an image correction method, an image correction device and image correction equipment, which are used for solving the problem that when an image collected in the prior art is processed on a live broadcast platform, the image cannot be identified due to the fact that the rotation angle of the image is unknown, and the live broadcast effect is influenced.
An image correction method provided by an embodiment of the application includes:
training by taking an image set with a known rotation angle as a sample to obtain a rotation angle identification model;
receiving an acquired image to be detected;
inputting the image to be detected into the rotation angle identification model for identification to obtain the rotation angle of the image to be detected;
and carrying out rotation correction on the image to be detected according to the rotation angle to obtain a target forward image corresponding to the image to be detected, wherein the rotation angle of the target forward image is 0 degree.
Optionally, the training with the image set with the known rotation angle as a sample to obtain the rotation angle identification model specifically includes:
acquiring a preset number of forward images;
respectively rotating the forward images according to a preset rotation rule to obtain a rotating image set corresponding to the forward images and comprising different rotation angles of the forward images, wherein each rotating image in the rotating image set carries a rotation angle identifier;
classifying the rotating images in the rotating image set according to the rotating angle identification to obtain rotating image subsets with different rotating angles;
and inputting the rotating image subsets with different rotating angles into a convolutional neural network as a sample for training to obtain a rotating angle identification model.
Optionally, after the training is performed by using the image set with the known rotation angle as a sample to obtain the rotation angle identification model, the method further includes:
inputting the test image set into the rotation angle identification model for identification to obtain the test rotation angle of each test image in the test image set;
judging whether the test rotation angle of each test image is consistent with the actual rotation angle of each test image to obtain a judgment result;
determining the identification accuracy of the rotation angle identification model according to the judgment result;
and if the identification accuracy is smaller than the preset accuracy, retraining the rotation angle identification model until the identification accuracy of the newly trained rotation angle identification model is not smaller than the preset accuracy.
Optionally, the determining the identification accuracy of the rotation angle identification model according to the judgment result specifically includes:
determining the number of target images with the testing rotation angles of the testing images consistent with the actual rotation angles of the testing images according to the judgment result;
and determining the identification accuracy of the rotation angle identification model according to the number of the target images and the total number of the test images.
Optionally, the preset rotation rule is to respectively rotate the images by 90 degrees, 180 degrees, and 270 degrees, and the forward images are respectively rotated according to the preset rotation rule to obtain a rotation image set corresponding to the forward images and including different rotation angles of the forward images themselves, where each rotation image in the rotation image set carries a rotation angle identifier, and the method specifically includes:
respectively rotating the forward image by 90 degrees, 180 degrees and 270 degrees according to a preset rotation rule to obtain a rotated image set with rotation angles of 0 degree, 90 degrees, 180 degrees and 270 degrees corresponding to the forward image, wherein each rotated image in the rotated image set carries a rotation angle identifier, and the rotated image with the rotation angle of 0 degree is the forward image;
after the forward image is respectively rotated by 90 degrees, 180 degrees and 270 degrees according to a preset rotation rule to obtain a rotated image set of which the rotation angles corresponding to the forward image are respectively 0 degree, 90 degrees, 180 degrees and 270 degrees, the method further includes:
classifying the rotating images in the rotating image set according to the rotating angle identification to obtain rotating image subsets with rotating angles of 0 degree, 90 degrees, 180 degrees and 270 degrees;
and inputting the rotation image subsets with the rotation angles of 0 degree, 90 degrees, 180 degrees and 270 degrees into a convolutional neural network as samples for training to obtain a rotation angle identification model.
An image correction device provided by an embodiment of the application includes:
the training module is used for training an image set with a known rotation angle as a sample to obtain a rotation angle identification model;
the receiving module is used for receiving the collected image to be detected;
the identification module is used for inputting the image to be detected into the rotation angle identification model for identification to obtain the rotation angle of the image to be detected;
and the correcting module is used for performing rotation correction on the image to be detected according to the rotation angle to obtain a target forward image corresponding to the image to be detected, wherein the rotation angle of the target forward image is 0 degree.
Optionally, the training module specifically includes:
the device comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring forward images with preset quantity;
the rotating unit is used for respectively rotating the forward images according to a preset rotating rule to obtain a rotating image set corresponding to the forward images and comprising different rotating angles of the forward images, wherein each rotating image in the rotating image set carries a rotating angle identifier;
the classification unit is used for classifying all the rotating images in the rotating image set according to the rotating angle identification to obtain rotating image subsets with different rotating angles;
and the input unit is used for inputting the rotating image subsets with different rotating angles into a convolutional neural network as samples for training to obtain a rotating angle identification model.
Optionally, after the training module, the method further includes:
the obtaining module is used for inputting the test image set into the rotation angle recognition model for recognition to obtain the test rotation angle of each test image in the test image set;
the consistency judging module is used for judging whether the test rotation angle of each test image is consistent with the actual rotation angle of each test image to obtain a judgment result;
the accuracy determining module is used for determining the identification accuracy of the rotation angle identification model according to the judgment result;
and the retraining module is used for retraining the rotation angle recognition model until the recognition accuracy of the newly trained rotation angle recognition model is not less than the preset accuracy if the recognition accuracy is less than the preset accuracy.
An embodiment of the present application provides an image correction apparatus, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor,
the instructions are executable by the at least one processor to enable the at least one processor to perform the image correction method.
An embodiment of the present application provides a computer-readable storage medium having instructions stored thereon, which when executed by a processor implement the image correction method.
The embodiment of the application provides an image correction method, device and equipment, the image to be detected is identified by utilizing a trained rotation angle identification model, after the rotation angle of the image to be detected is determined, the image to be detected is subjected to rotation correction, the image to be detected is converted into forward, a live broadcast platform can directly identify the forward image, the content in the image can be identified, then, the identified content is subjected to operations such as beautifying, special effect adding or dubbing, the live broadcast effect is enhanced, and further the user viscosity of the live broadcast platform is improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a schematic view of a common placement mode of a camera provided in an embodiment of the present application;
fig. 2 is a schematic flowchart of an image correction method according to an embodiment of the present disclosure;
FIG. 3 is a schematic diagram illustrating an image rotation correction according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram illustrating a specific rotation manner of an image provided in an embodiment of the present application;
fig. 5 is an application schematic diagram of an image correction apparatus deployed in a client application according to an embodiment of the present application;
fig. 6 is a schematic view of an application of an image correction apparatus deployed in a camera firmware according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an image correction apparatus according to an embodiment of the present disclosure;
fig. 8 is a schematic structural diagram of an image correction apparatus according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail and completely with reference to the following specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
As shown in fig. 1, a schematic view of a common placing mode of a camera provided in an embodiment of the present specification may specifically include: put the camera upset forward, put the camera upset from top to bottom, rotate the camera 90 degrees and rotate these several kinds of modes of putting of 270 degrees with the camera. For several placement modes of the camera in fig. 2, only the first placement mode is correct, and for other placement modes, because the output image is not forward, the application program of the client cannot automatically determine the rotation angle of the image, which may cause that the captured image cannot be recognized and cannot be processed because the deflection angle of the image is unknown when the captured image is processed on the live broadcast platform.
For example, users who are performing live broadcasting are generally divided into two types, the first type is a small white user who has just started using the live broadcasting function, and the other type is a main broadcasting user who often uses the live broadcasting function.
For a small white user, the camera may be placed in the second camera placement manner in fig. 1 inadvertently, resulting in an inverted image output by the camera. Then, when the live broadcast is carried out, the function of turning over the live broadcast platform from top to bottom is just started. The inverted image output by the original camera is turned into a forward image after the function of turning up and down in the live broadcast platform, so that a user in small white can mistakenly place the camera without any problem. But in fact, before the live broadcast platform turns over up and down, the live broadcast platform can carry out face recognition on the inverted image sent by the camera. For an inverted image, the live platform cannot recognize the face, so that although the image is positive after the up-down turning function in the live platform, the related special effect fails, for example, the special effect function related to face recognition (such as advanced shaping and beautifying) fails.
For the anchor user, they want to realize the effect of vertical screen live broadcast similar to a mobile phone on the live broadcast platform, and therefore the camera can be intentionally placed in a third or fourth camera placing mode in fig. 1. A transverse image is shot, and then the live broadcast platform is selectively turned over for 90 degrees, so that the effect of vertical screen display of the mobile phone can be achieved. However, as for the above reasons, since the image entering the live platform is not forward, the related special effects may be disabled, for example, the special effects function related to face recognition (such as advanced shaping and beautifying) may be disabled.
Fig. 2 is a schematic flowchart of an image correction method according to an embodiment of the present application, and from a program perspective, an execution subject of the flowchart may be an application program deployed on a client. The method specifically comprises the following steps:
step S201, training an image set with a known rotation angle as a sample to obtain a rotation angle identification model.
Before obtaining the rotation angle recognition model, a sample needs to be determined first, and the sample is used as an input to train the model. The mode of determining the samples is various, a plurality of video records of the main broadcast can be directly selected from the video records of the live broadcast platform, and one screenshot of the video records can be respectively intercepted. The video of one anchor can be captured once every preset time, namely, a plurality of images with different actions are captured by one anchor. Wherein the images in the live platform are all forward. And the system can also be used for collecting in places such as schools, squares, markets and the like with dense personnel to obtain a sample image set. In order to improve the identification accuracy of the trained rotation angle identification model, a sufficient number of samples need to be ensured, and the number of forward image sample sets can be 5000-. In the present embodiment, the number of forward image samples employed is 10000.
The forward image may then be rotated, for example, by 90 degrees, to obtain a new image, and the newly obtained image is marked with a rotation angle of 90 degrees. And then training the image before rotation and the image after rotation with the rotation mark as samples to obtain a rotation angle identification model.
In addition, the rotation angle of each image can be recorded in a local file correspondingly, and when training is carried out, the image before rotation, the image after rotation and the local file recording the rotation angle are used as samples to be trained, so that the rotation angle recognition model can be obtained.
Wherein a convolutional neural network can be used to train an image set of known rotation angles.
Step S202, receiving the collected image to be detected.
After the rotation angle recognition model is trained, the image can be recognized, and the rotation angle of the image can be determined. Before the identification, the application program on the client needs to receive the image to be detected collected by the image collection device. The image acquisition device may include a camera, a video camera, a scanner, a video capture card, and the like, and in the embodiment of the present application, a USB camera is used. Namely, an application program on the client receives the image to be detected acquired by the USB camera.
In addition, the image can be obtained from the image stored in the local of the client or the image stored in the cloud.
And S203, inputting the image to be detected into the rotation angle identification model for identification to obtain the rotation angle of the image to be detected.
After the image to be detected is input into the rotation angle identification model for identification, the rotation angle identification model can extract the characteristics of the image in the image to be detected, then the extracted characteristics are respectively compared with the characteristics of the images with different rotation angles, and the rotation angle is closest to the image characteristic of which rotation angle, so that the rotation angle of the image to be detected is considered to be how many degrees. For example, if the image to be detected is closest to an image feature rotated by 90 degrees, the image to be detected is considered to be rotated by 90 degrees. And if the image to be detected is closest to the image characteristics of 180-degree transformation, the image to be detected is considered to be rotated by 180 degrees.
And S204, performing rotation correction on the image to be detected according to the rotation angle to obtain a target forward image corresponding to the image to be detected, wherein the rotation angle of the target forward image is 0 degree.
And acquiring the determined rotation angle of the image, performing rotation correction on the image to be detected according to the determined rotation angle, and correcting the image to be detected to a forward image corresponding to the image to be detected, namely an image with the rotation angle of 0 degree.
For example, as shown in fig. 3, an image rotation correction schematic diagram provided in the embodiment of the present application may specifically include:
when the rotation angle of the image to be detected is determined to be 90 degrees, rotation correction is carried out on the image to be detected according to the rotation angle of 90 degrees, namely the image to be detected is reversely rotated by 90 degrees, and the image with the rotation angle of 0 degree corresponding to the image to be detected is obtained.
And for the image to be detected with the rotation angle of 180 degrees, performing rotation correction on the image to be detected according to the rotation angle of 180 degrees, namely rotating the image to be detected by 180 degrees in the reverse direction and rotating the image to be detected by 180 degrees in the forward direction to obtain the image with the rotation angle of 0 degree corresponding to the image to be detected.
For the image to be detected with the rotation angle of 270 degrees, in order to reduce the rotation angle, the image to be detected may be rotated and corrected in a forward direction by 90 degrees, so as to obtain an image with the rotation angle of 0 degree corresponding to the image to be detected.
After the scheme is adopted, the image to be detected is identified by utilizing the trained rotation angle identification model, after the rotation angle of the image to be detected is determined, the image to be detected is subjected to rotation correction, the image to be detected is converted into the forward direction, the live broadcast platform can directly identify the image of the forward direction, the content in the image can be identified, then, the identified content is subjected to operations such as beautifying, special effect adding or dubbing processing, the live broadcast effect is enhanced, and the user viscosity of the live broadcast platform is improved.
Based on the method of fig. 2, the embodiments of the present specification also provide some specific implementations of the method, which are described below.
In a specific embodiment, the training with the image set with the known rotation angle as a sample to obtain the rotation angle recognition model specifically may include:
a preset number of forward images are acquired.
Wherein the preset number of forward images may be in the range of 5000-.
And respectively rotating the forward images according to a preset rotation rule to obtain a rotation image set corresponding to the forward images and comprising different rotation angles of the forward images, wherein each rotation image in the rotation image set carries a rotation angle identifier.
The rotation angle identifier can be various types of identifiers, for example, the corresponding angle numerical identifier can be directly used, such as 90 degrees rotated by 90 degrees or 90 degrees rotated by 180 degrees or 180 degrees rotated by 180 degrees. It may also be identified by letters, such as letter a for 90 degrees rotation, letter b for 180 degrees rotation, etc. The above description is only a few specific embodiments, and other ways of indicating the rotation angle are also within the scope of the present application.
The pre-stored rotation rule is an angle by which the forward image is to be rotated. For example, the forward direction image may be rotated by 90 degrees, 180 degrees, and 270 degrees, or the forward direction image may be rotated by 45 degrees, 90 degrees, and 135 degrees, respectively. The above description is only a few specific embodiments, and other degrees of rotation rules and rotation modes are also within the scope of the present application.
And classifying the rotating images in the rotating image set according to the rotating angle identification to obtain rotating image subsets with different rotating angles.
And inputting the rotating image subsets with different rotating angles into a convolutional neural network as a sample for training to obtain a rotating angle identification model.
For example, each rotated image in the rotated image set is classified according to the pre-stored rotation rule, and four rotated image subsets with rotation angles of 0 degrees, 90 degrees, 180 degrees, and 270 degrees are obtained. And then inputting the four rotary image subsets as samples into a convolutional neural network for training, wherein the convolutional neural network determines the characteristics of the rotary image at the corresponding angle according to the characteristics of the four rotary image subset samples, and then obtains a rotary angle identification model according to the characteristics training of the rotary image at the corresponding angle. For example, for a rotated image subset with a rotation angle of 90 degrees, the convolutional neural network determines the characteristics of the rotated image with a rotation angle of 90 degrees according to the samples of the image subset with a rotation angle of 90 degrees. And the training process of the image subset samples of other rotation angles is the same, and then a rotation angle identification model is obtained according to the determined characteristics of the rotation image with the rotation angle of 0 degree, the rotation image with the rotation angle of 90 degrees, the rotation image with the rotation angle of 180 degrees and the rotation image with the rotation angle of 270 degrees.
When some users perform live broadcasting, the actual rotation angle of the camera may not be the exact rotation angle set by the users. In order to improve the identification accuracy, a threshold range is set according to the rotation angles in the preset rotation rule, a plurality of rotation angles are selected selectively in the threshold range, the forward image is rotated according to the selected plurality of rotation angles, and the rotated image is input into the convolutional neural network as a sample to be trained. For example, a threshold range of ± 15 degrees may be set. Assuming that the rotation angle defined in the preset rule is 90 degrees, when the forward image is rotated, the forward image may be rotated by 90 degrees, or the forward image may be rotated by 75 to 105 degrees such as 75 degrees, 80 degrees, 100 degrees, and 105 degrees, to obtain a rotated image. And then, the images in the rotation range are classified into an image set with a rotation angle of 90 degrees by default, and the image set is input into a convolutional neural network for training, so that the identification accuracy of the model is improved.
In a specific embodiment, after the training with the image set with the known rotation angle as a sample to obtain the rotation angle identification model, the method further includes:
and inputting the test image set into the rotation angle identification model for identification to obtain the test rotation angle of each test image in the test image set.
And judging whether the test rotation angle of each test image is consistent with the actual rotation angle of each test image to obtain a judgment result.
After the rotation angle recognition model is trained, the trained rotation angle recognition model can be tested by utilizing the test image set, and whether the trained rotation angle recognition model meets the recognition requirement is tested. The test image set may be a standard image set, i.e. an image set with a known rotation angle. And inputting the image set with the known rotation angle into the trained rotation angle recognition model to obtain the recognition result of each test image in the image set, and judging whether the rotation angle recognition model meets the recognition requirement or not according to the recognition result.
There are various ways to determine whether the rotation angle recognition model meets the recognition requirement, and the recognition accuracy of the rotation angle recognition model, the duration of the rotation angle recognition model obtaining the recognition result, or the stability of the rotation angle recognition model can be viewed.
If the judgment is carried out according to the identification accuracy of the rotation angle identification model, the identification accuracy of the rotation angle identification model can be determined according to the judgment result, and if the identification accuracy is smaller than the preset accuracy, the rotation angle identification model is retrained until the identification accuracy of the newly trained rotation angle identification model is not smaller than the preset accuracy.
The preset accuracy rate can be 90%, if the identification accuracy rate is less than 90%, the trained rotation angle identification model is considered to be unavailable, the rotation angle identification model needs to be adjusted again, training is performed again after adjustment, and the identification accuracy rate is tested again after training is completed until the identification accuracy rate of the newly trained rotation angle identification model is not less than the preset accuracy rate.
If the determination is made according to the duration of the recognition result of the rotation angle recognition model, it may be determined that the rotation angle recognition model may not be available according to the duration of the obtained recognition result. If the time length of the obtained recognition result exceeds the preset time length, the recognition speed of the rotation angle recognition model obtained through training is too slow and unavailable. And if the time length of the obtained recognition result does not exceed the preset time length, the rotation angle recognition model obtained through training is available. Wherein, the preset time length can range from 0 to 60 seconds. In a specific real-time mode, the preset time period is 30 seconds.
If the judgment is performed according to the stability of the rotation angle identification model, the unavailability of the rotation angle identification model can be judged according to the stability of the obtained identification result. The stability of the rotation angle recognition model represents whether the accuracy of the multiple recognition results is stable, for example, if the accuracy of each recognition result is 95% when the multiple recognition is performed by using the rotation angle recognition model, the stability of the rotation angle recognition model is high, and the rotation angle recognition model is available. If the accuracy of the recognition result is 95%, 70%, or 80%, the accuracy of each recognition result is different, the stability of the rotation angle recognition model is low, and the rotation angle recognition model is not usable. The preset stability range can be set to be 80% -90%, and the accuracy error is within a range of +/-2%, so that the accuracy is considered to be the same. For example, the 95% accuracy and the 96% accuracy are regarded as the same accuracy, and the stability of the multiple groups of rotation angle recognition models is obtained by dividing the times of the same accuracy by the total detection times. And if the determined values of the group with the highest stability exceed the preset stability, the rotation angle recognition model is considered to be available, and if the determined stabilities do not exceed the preset stability, the rotation angle recognition model is considered to be unavailable.
If the rotation angle recognition model is judged to be unavailable, the specific module causing the error in the rotation angle recognition model can be determined according to a pre-stored backward conduction algorithm, then the specific module causing the error can be adjusted according to a preset BP algorithm, and the training test is carried out again after the adjustment is finished until the rotation angle recognition model is available.
In a specific embodiment, the determining, according to the determination result, the identification accuracy of the rotation angle identification model may specifically include:
and determining the number of target images with the testing rotation angles of the testing images consistent with the actual rotation angles of the testing images according to the judgment result. And determining the identification accuracy of the rotation angle identification model according to the number of the target images and the total number of the test images.
And judging the identification accuracy of the rotation angle identification model according to the condition that the identified test rotation angle in the identification result is consistent with the actual rotation angle of the test image. For example, the number of target images for which the determined test rotation angle coincides with the actual rotation angle of the test image is 95, the total number of test images is 100, and the recognition accuracy is 95/100 equal to 95%.
In a specific embodiment, as shown in fig. 4, a schematic diagram of a specific image rotation manner provided in an embodiment of the present application may specifically include: the preset rotation rules are respectively 90-degree, 180-degree and 270-degree rotation of an image, the forward image is respectively rotated according to the preset rotation rules to obtain a rotation image set corresponding to the forward image and comprising different rotation angles of the forward image, wherein each rotation image in the rotation image set carries a rotation angle identifier, the forward image is respectively rotated by 90-degree, 180-degree and 270-degree according to the preset rotation rules to obtain a rotation image set corresponding to the forward image and respectively having rotation angles of 0-degree, 90-degree, 180-degree and 270-degree, and each rotation image in the rotation image set carries a rotation angle identifier.
After the forward image is respectively rotated by 90 degrees, 180 degrees and 270 degrees according to a preset rotation rule to obtain a rotated image set of which the rotation angles corresponding to the forward image are respectively 0 degree, 90 degrees, 180 degrees and 270 degrees, the method further includes:
and classifying the rotating images in the rotating image set according to the rotating angle identification to obtain rotating image subsets with rotating angles of 0 degree, 90 degrees, 180 degrees and 270 degrees.
And inputting the rotation image subsets with the rotation angles of 0 degree, 90 degrees, 180 degrees and 270 degrees into a convolutional neural network as samples for training to obtain a rotation angle identification model.
The image modes shot under several camera placing modes which may be adopted by a user are used as samples to be trained to obtain the rotation angle recognition model, so that the rotation angle recognition model obtained through training can automatically recognize the turning angle of the image, and a basis is provided for subsequent turning processing.
As shown in fig. 5, an application diagram of an image correction apparatus deployed in a client application program provided in an embodiment of the present disclosure may specifically include:
the client application program can be a YY companion application program or a YY broadcast application program.
The client receives the image to be detected which is acquired and sent by the camera, then the image is identified by using a rotation angle identification model in the image correction device, the rotation angle of the image to be detected is determined, then the image to be detected is subjected to rotation correction according to the determined rotation angle of the image to be detected, and the image to be detected is converted into a forward image. That is, before inputting into the YY companion application program and performing face recognition, it is ensured that the image to be detected is forward. And then inputting the image to be detected converted into the positive direction into a YY companion application program for face recognition, and then performing face beautifying processing according to the recognized face. And after the processing is finished, outputting the data to a virtual camera, and displaying the data on a live broadcast interface for live broadcast.
As shown in fig. 6, an application diagram of an image correction apparatus deployed in a camera firmware provided in an embodiment of the present disclosure may specifically include:
there is also a CPU (also called image processor, or embedded CPU) inside the camera, and the software running on the CPU is called firmware, similar to the application running on the computer terminal.
After an image is shot by a lens sensor of the camera, a rotation angle recognition model in a built-in image correction device carries out camera posture recognition, then rotation correction is carried out on the image to be detected according to a recognition result, the image to be detected is sent to an image processing module for processing after the correction is finished, the image to be detected is converted into a forward image, and finally, the forward image is output through a USB.
For the camera, the trained rotation angle recognition model can be deployed in the camera firmware, namely, a posture sensor is simulated, and the posture of the camera can be detected in real time, so that when the camera is used for turnover shooting, the shot image can be automatically corrected from the inside, and the output image is ensured to be forward. In addition, an attitude sensor chip is not required to be additionally added, so that the cost is saved under the condition of achieving the same function.
Based on the same idea, the embodiment of the present specification further provides a device corresponding to the method.
Fig. 7 is a schematic structural diagram of an image correction apparatus corresponding to fig. 2 provided in an embodiment of the present disclosure. As shown in fig. 7, the apparatus may include:
the training module 701 is configured to train an image set with a known rotation angle as a sample to obtain a rotation angle identification model.
A receiving module 702, configured to receive the acquired image to be detected.
The identifying module 703 is configured to input the image to be detected into the rotation angle identifying model for identification, so as to obtain a rotation angle of the image to be detected.
And the correcting module 704 is configured to perform rotation correction on the image to be detected according to the rotation angle to obtain a target forward image corresponding to the image to be detected, where the rotation angle of the target forward image is 0 degree.
The device in fig. 7 recognizes the image to be detected by using the trained rotation angle recognition model, after the rotation angle of the image to be detected is determined, the image to be detected is corrected in a rotating mode, the image to be detected is converted into the forward direction, so that the live broadcast platform can directly recognize the forward image, the content in the image can be recognized, then the recognized content is subjected to operations such as beautifying, special effect adding or dubbing, the live broadcast effect is enhanced, and the user viscosity of the live broadcast platform is improved.
In a specific embodiment, the training module may specifically include:
the device comprises an acquisition unit, a display unit and a display unit, wherein the acquisition unit is used for acquiring a preset number of forward images.
And the rotating unit is used for respectively rotating the forward images according to a preset rotating rule to obtain a rotating image set corresponding to the forward images and comprising different rotating angles of the forward images, wherein each rotating image in the rotating image set carries a rotating angle identifier.
And the classification unit is used for classifying the rotating images in the rotating image set according to the rotating angle identification to obtain rotating image subsets with different rotating angles.
And the input unit is used for inputting the rotating image subsets with different rotating angles into a convolutional neural network as samples for training to obtain a rotating angle identification model.
In a specific embodiment, after the training module, the method further comprises:
and the obtaining module is used for inputting the test image set into the rotation angle identification model for identification to obtain the test rotation angle of each test image in the test image set.
And the consistency judging module is used for judging whether the test rotation angle of each test image is consistent with the actual rotation angle of each test image to obtain a judgment result.
And the accuracy determining module is used for determining the identification accuracy of the rotation angle identification model according to the judgment result.
And the retraining module is used for retraining the rotation angle recognition model until the recognition accuracy of the newly trained rotation angle recognition model is not less than the preset accuracy if the recognition accuracy is less than the preset accuracy.
In a specific embodiment, the accuracy determining module may specifically include:
and the consistency judging unit is used for determining the number of target images with the testing rotation angles of the testing images consistent with the actual rotation angles of the testing images according to the judging result.
And the accuracy determining unit is used for determining the identification accuracy of the rotation angle identification model according to the number of the target images and the total number of the test images.
In a specific embodiment, the preset rotation rule is that 90 degrees, 180 degrees and 270 degrees of rotation are performed on the image, and the rotation unit may be further configured to:
and respectively rotating the forward image by 90 degrees, 180 degrees and 270 degrees according to a preset rotation rule to obtain a rotated image set with the rotation angles corresponding to the forward image being 0 degree, 90 degrees, 180 degrees and 270 degrees, wherein each rotated image in the rotated image set carries a rotation angle identifier.
After the forward image is respectively rotated by 90 degrees, 180 degrees and 270 degrees according to a preset rotation rule to obtain a rotated image set of which the rotation angles corresponding to the forward image are respectively 0 degree, 90 degrees, 180 degrees and 270 degrees, the method further includes:
and classifying the rotating images in the rotating image set according to the rotating angle identification to obtain rotating image subsets with rotating angles of 0 degree, 90 degrees, 180 degrees and 270 degrees.
And inputting the rotation image subsets with the rotation angles of 0 degree, 90 degrees, 180 degrees and 270 degrees into a convolutional neural network as samples for training to obtain a rotation angle identification model.
Fig. 8 is a schematic structural diagram of an image correction apparatus corresponding to fig. 2 provided in an embodiment of the present specification. As shown in fig. 8, the apparatus 800 may include:
at least one processor 810; and the number of the first and second groups,
a memory 830 communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory 830 stores instructions 820 that can be executed by the at least one processor 810, and the instructions are executed by the at least one processor 810, so that the at least one processor 810 can implement the embodiment of the image correction method, for specific function implementation, please refer to the description in the method embodiment, which is not repeated herein.
Based on the same idea, embodiments of the present specification further provide a computer-readable storage medium, where instructions are stored on the computer-readable storage medium, and when executed by a processor, the instructions may implement the image correction method described above.
In addition, other identical elements exist. As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. An image correction method, comprising:
capturing forward live broadcast images of one or more anchor broadcasters in a live broadcast platform video, obtaining an image set with a known rotation angle based on rotation of the forward live broadcast images at different angles, and training the image set with the known rotation angle as a sample to obtain a rotation angle identification model;
receiving a live broadcast image shot by a camera arranged at a certain angle when a user live broadcast as an image to be detected;
inputting the image to be detected into the rotation angle identification model for identification to obtain the rotation angle of the image to be detected; the rotation angle represents the angle difference between the current placing angle and the forward placing angle of the camera;
correcting the image to be detected to be a target forward image with a rotation angle of 0 degrees in a rotating mode according to the rotation angle;
and inputting the target forward image into a live broadcast platform, so that the live broadcast platform performs face recognition based on the target forward image, performs related effect processing on the recognized face and displays the processed face on a live broadcast interface.
2. The method as claimed in claim 1, wherein said obtaining a set of images of known rotation angles based on rotating said forward live image by different angles comprises:
respectively rotating the forward live broadcast images according to a preset rotation rule to obtain a rotation image set corresponding to the forward live broadcast images and comprising different rotation angles of the forward live broadcast images, wherein each rotation image in the rotation image set carries a rotation angle identification;
classifying the rotating images in the rotating image set according to the rotating angle identification to obtain rotating image subsets with different rotating angles;
and inputting the rotating image subsets with different rotating angles into a convolutional neural network as samples for training to obtain a rotating angle identification model.
3. The method of claim 1, wherein after training the image set with the known rotation angle as a sample to obtain a rotation angle recognition model, further comprising:
inputting a test image set into the rotation angle identification model for identification to obtain a test rotation angle of each test image in the test image set;
judging whether the test rotation angle of each test image is consistent with the actual rotation angle of each test image to obtain a judgment result;
determining the identification accuracy of the rotation angle identification model according to the judgment result;
and if the identification accuracy is smaller than the preset accuracy, retraining the rotation angle identification model until the identification accuracy of the newly trained rotation angle identification model is not smaller than the preset accuracy.
4. The method according to claim 3, wherein the determining the recognition accuracy of the rotation angle recognition model according to the determination result specifically includes:
determining the number of target images with the testing rotation angles of the testing images consistent with the actual rotation angles of the testing images according to the judgment result;
and determining the identification accuracy of the rotation angle identification model according to the number of the target images and the total number of the test images.
5. The method according to claim 2, wherein the preset rotation rule is that images are respectively rotated by 90 degrees, 180 degrees, and 270 degrees, and the forward live broadcast images are respectively rotated according to the preset rotation rule to obtain a set of rotated images corresponding to the forward live broadcast images and including different rotation angles of the forward live broadcast images, where each rotated image in the set of rotated images carries a rotation angle identifier, and specifically includes:
respectively rotating the forward live broadcast images by 90 degrees, 180 degrees and 270 degrees according to a preset rotation rule to obtain a rotation image set with rotation angles of 0 degree, 90 degrees, 180 degrees and 270 degrees corresponding to the forward live broadcast images, wherein each rotation image in the rotation image set carries a rotation angle identifier, and the rotation image with the rotation angle of 0 degree is the forward live broadcast image;
after the forward live broadcast image is respectively rotated by 90 degrees, 180 degrees and 270 degrees according to a preset rotation rule to obtain a rotated image set of which the corresponding rotation angles of the forward live broadcast image are respectively 0 degree, 90 degrees, 180 degrees and 270 degrees, the method further comprises the following steps:
classifying the rotating images in the rotating image set according to the rotating angle identification to obtain rotating image subsets with rotating angles of 0 degree, 90 degrees, 180 degrees and 270 degrees;
and inputting the rotation image subsets with the rotation angles of 0 degree, 90 degrees, 180 degrees and 270 degrees into a convolutional neural network as samples for training to obtain a rotation angle identification model.
6. An image correction apparatus characterized by comprising:
the system comprises a training module, a live broadcast platform video recording module and a video recognition module, wherein the training module is used for intercepting one or more forward live broadcast images of a main broadcast in the live broadcast platform video recording, obtaining an image set with a known rotation angle based on different angles of rotation of the forward live broadcast images, and training the image set with the known rotation angle as a sample to obtain a rotation angle recognition model;
the receiving module is used for receiving a live broadcast image shot by a camera which is arranged at a certain angle when a user live broadcast, and the live broadcast image is used as an image to be detected;
the identification module is used for inputting the image to be detected into the rotation angle identification model for identification to obtain the rotation angle of the image to be detected; the rotation angle represents the angle difference between the current placing angle and the forward placing angle of the camera;
and the correcting module is used for correcting the image to be detected into a target forward image with the rotation angle of 0 degrees according to the rotation angle, and inputting the target forward image into the live broadcast platform so that the live broadcast platform can perform face recognition based on the target forward image and display the recognized face on a live broadcast interface after performing related effect processing on the recognized face.
7. The apparatus of claim 6, wherein the training module specifically comprises:
the rotating unit is used for respectively rotating the forward live broadcast images according to a preset rotating rule to obtain a rotating image set corresponding to the forward live broadcast images and comprising different rotating angles of the forward live broadcast images, wherein each rotating image in the rotating image set carries a rotating angle identification;
the classification unit is used for classifying all the rotating images in the rotating image set according to the rotating angle identification to obtain rotating image subsets with different rotating angles;
and the input unit is used for inputting the rotating image subsets with different rotating angles into a convolutional neural network as samples for training to obtain a rotating angle identification model.
8. The apparatus of claim 6, further comprising, after the training module:
the obtaining module is used for inputting the test image set into the rotation angle recognition model for recognition to obtain the test rotation angle of each test image in the test image set;
the consistency judging module is used for judging whether the test rotation angle of each test image is consistent with the actual rotation angle of each test image to obtain a judgment result;
the accuracy determining module is used for determining the identification accuracy of the rotation angle identification model according to the judgment result;
and the retraining module is used for retraining the rotation angle recognition model until the recognition accuracy of the newly trained rotation angle recognition model is not less than the preset accuracy if the recognition accuracy is less than the preset accuracy.
9. An image correction apparatus characterized by comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor,
the instructions being executable by the at least one processor to enable the at least one processor to perform the steps of the image correction method as claimed in any one of claims 1-5.
10. A computer readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement the steps of the image correction method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910453576.6A CN110213605B (en) | 2019-05-28 | 2019-05-28 | Image correction method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910453576.6A CN110213605B (en) | 2019-05-28 | 2019-05-28 | Image correction method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110213605A CN110213605A (en) | 2019-09-06 |
| CN110213605B true CN110213605B (en) | 2022-09-13 |
Family
ID=67789192
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910453576.6A Active CN110213605B (en) | 2019-05-28 | 2019-05-28 | Image correction method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110213605B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111563407B (en) * | 2020-02-24 | 2024-02-02 | 北京乐学帮网络技术有限公司 | Model training method, picture direction correcting method and device |
| CN111539357B (en) * | 2020-04-28 | 2024-01-23 | 睿魔智能科技（深圳）有限公司 | Human body recognition model, training method, training system, storage medium and training equipment thereof |
| CN115941859A (en) * | 2021-08-16 | 2023-04-07 | 广州视源电子科技股份有限公司 | Image direction adjusting method and device, storage medium and electronic equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104077577A (en) * | 2014-07-03 | 2014-10-01 | 浙江大学 | Trademark detection method based on convolutional neural network |
| CN109583445A (en) * | 2018-11-26 | 2019-04-05 | 平安科技（深圳）有限公司 | Character image correction processing method, device, equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6959109B2 (en) * | 2002-06-20 | 2005-10-25 | Identix Incorporated | System and method for pose-angle estimation |
| CN107977668A (en) * | 2017-07-28 | 2018-05-01 | 北京物灵智能科技有限公司 | A kind of robot graphics' recognition methods and system |
-
2019
- 2019-05-28 CN CN201910453576.6A patent/CN110213605B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104077577A (en) * | 2014-07-03 | 2014-10-01 | 浙江大学 | Trademark detection method based on convolutional neural network |
| CN109583445A (en) * | 2018-11-26 | 2019-04-05 | 平安科技（深圳）有限公司 | Character image correction processing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110213605A (en) | 2019-09-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107105310B (en) | Human image replacing method and device in video live broadcast and recording and broadcasting system | |
| WO2020151489A1 (en) | Living body detection method based on facial recognition, and electronic device and storage medium | |
| CN110213605B (en) | Image correction method, device and equipment | |
| CN110232369B (en) | Face recognition method and electronic equipment | |
| CN110645986B (en) | Positioning method and device, terminal and storage medium | |
| CN110705405B (en) | Target labeling method and device | |
| TW201911130A (en) | Method and device for remake image recognition | |
| WO2021095693A1 (en) | Information processing device, information processing method, and program | |
| US9826208B2 (en) | Method and system for generating weights for use in white balancing an image | |
| CN110493595B (en) | Camera detection method and device, storage medium and electronic device | |
| WO2013165565A1 (en) | Method of detecting a main subject in an image | |
| CN111401238B (en) | Method and device for detecting character close-up fragments in video | |
| CN109284700B (en) | Method, storage medium, device and system for detecting multiple faces in image | |
| CN110991297A (en) | Target positioning method and system based on scene monitoring | |
| CN105678301B (en) | method, system and device for automatically identifying and segmenting text image | |
| CN107832598B (en) | Unlocking control method and related product | |
| CN106327531A (en) | Panorama video identification method and device, and video playing method and device | |
| CN111027450A (en) | Bank card information identification method and device, computer equipment and storage medium | |
| CN111476056B (en) | Target object identification method, device, terminal equipment and computer storage medium | |
| CN116958795A (en) | Method and device for identifying flip image, electronic equipment and storage medium | |
| CN111444822A (en) | Object recognition method and apparatus, storage medium, and electronic apparatus | |
| CN110569707A (en) | identity recognition method and electronic equipment | |
| US11734790B2 (en) | Method and apparatus for recognizing landmark in panoramic image and non-transitory computer-readable medium | |
| CN111611973B (en) | Target user identification method, device and storage medium | |
| CN110430424B (en) | FOV testing method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210111 Address after: 511442 3108, 79 Wanbo 2nd Road, Nancun Town, Panyu District, Guangzhou City, Guangdong Province Applicant after: GUANGZHOU CUBESILI INFORMATION TECHNOLOGY Co.,Ltd. Address before: 511442 28 / F, building B-1, Wanda Plaza North, Wanbo business district, 79 Wanbo 2nd Road, Nancun Town, Panyu District, Guangzhou City, Guangdong Province Applicant before: GUANGZHOU HUADUO NETWORK TECHNOLOGY Co.,Ltd. |
|
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20190906 Assignee: GUANGZHOU HUADUO NETWORK TECHNOLOGY Co.,Ltd. Assignor: GUANGZHOU CUBESILI INFORMATION TECHNOLOGY Co.,Ltd. Contract record no.: X2021440000054 Denomination of invention: Image correction method, device and equipment License type: Common License Record date: 20210208 |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |