CN109741250B - Image processing method and device, storage medium and electronic equipment - Google Patents
Image processing method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN109741250B CN109741250B CN201910008670.0A CN201910008670A CN109741250B CN 109741250 B CN109741250 B CN 109741250B CN 201910008670 A CN201910008670 A CN 201910008670A CN 109741250 B CN109741250 B CN 109741250B
- Authority
- CN
- China
- Prior art keywords
- image
- network
- images
- barrel
- confidence value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses an image processing method and device, a storage medium and electronic equipment, and relates to the technical field of image processing. The image processing method comprises the following steps: determining a plurality of groups of images as training samples; each group of images in the plurality of groups of images comprises an original image and a barrel-shaped image corresponding to the original image; training a generation network and a discrimination network in a pair of antagonistic neural networks by using a training sample to determine a trained generation network; inputting the image to be processed into a trained generation network to determine a barrel-shaped image corresponding to the image to be processed; and displaying the barrel-shaped image corresponding to the image to be processed on a screen of the virtual reality equipment. The present disclosure may enable predistortion processing of images on virtual reality devices.
Description
Technical Field
The present disclosure relates to the field of image processing technology, and in particular, to an image processing method, an image processing apparatus, a storage medium, and an electronic device.
Background
With the continuous progress of image processing technology, virtual Reality (VR) technology is gradually appearing in the field of view of people, especially in the aspect of games, and has been rapidly developed.
In order for a user to visually possess a realistic sense of immersion, a virtual reality device (e.g., a head-mounted stereoscopic display, or simply a head-mounted display) should cover the visual range of the human eye as much as possible. Therefore, a magnifying lens with a specific spherical radian needs to be configured on the virtual reality device. In this case, when a conventional image is projected into the human eye using the magnifying lens, the image is distorted.
Currently, for the case of distortion of an image, distortion parameters of a lens in a virtual reality device are generally required to be calculated and corrected. In addition, in software implementation, interpolation operation is required to solve the problem of image distortion. However, for the current processing mode, on one hand, errors may exist when solving distortion parameters, so that the image received by human eyes is abnormal; on the other hand, when scene rendering, the coordinates of the image on the screen need to be continuously determined, and the time consumption is long; in yet another aspect, errors may exist in lens installation in a virtual reality device, resulting in errors in calculated screen coordinate points.
It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
An object of the present disclosure is to provide an image processing method, an image processing apparatus, a storage medium, and an electronic device, which further overcome, at least to some extent, the problems of abnormal image display on a screen of a virtual reality device and slow image processing speed due to limitations and drawbacks of the related art.
According to an aspect of the present disclosure, there is provided an image processing method including: determining a plurality of groups of images as training samples; wherein each set of images in the plurality of sets of images comprises an original image and a barrel-shaped image corresponding to the original image; training a generation network and a discrimination network in a pair of antagonistic neural networks by using the training sample to determine a trained generation network; inputting an image to be processed into the trained generation network to determine a barrel image corresponding to the image to be processed; and displaying the barrel-shaped image corresponding to the image to be processed on a screen of the virtual reality equipment.
Optionally, determining the plurality of sets of images as training samples comprises: determining an original image; calculating lens distortion parameters of the virtual reality equipment; converting the original image into a barrel image corresponding to the original image by using the lens distortion parameters; wherein the original image and the barrel image are used as a set of images to be used as training samples.
Optionally, training the generation network and the discrimination network in the pair of antagonistic neural networks using the training samples includes: inputting original images in the plurality of groups of images into a generation network of an antagonistic neural network to determine an intermediate image corresponding to the original images; inputting barrel-shaped images which are in the same group with the original images into a discrimination network of the antagonistic neural network to determine a first confidence value; inputting the intermediate image into the discrimination network to determine a second confidence value; determining a loss of the generating network using the second confidence value, and determining a loss of the discriminating network using the first confidence value and the second confidence value, so as to train the generating network and the discriminating network in the antagonistic neural network.
Optionally, determining the loss of the generated network using the second confidence value comprises: a loss of the generated network is determined based on a cross entropy between the second confidence value and 1.
Optionally, determining the loss of the discrimination network using the first confidence value and the second confidence value includes: determining a loss of the discrimination network based on a cross entropy between the first confidence value and 1 and a cross entropy between the second confidence value and 0.
Optionally, the image processing method further includes: saving the trained generation network as a model file in a predetermined format; and under the condition that the image to be processed is acquired, loading the model file so as to determine a barrel-shaped image corresponding to the image to be processed.
Optionally, saving the trained generation network as a model file in a predetermined format includes: verifying the trained generation network by adopting the verification image; and if the verification result meets a preset condition, saving the trained generation network as a model file in a preset format.
According to one aspect of the present disclosure, there is provided an image processing apparatus including a training sample determination module, a network training module, an image determination module, and an image display module.
Specifically, the training sample determining module is used for determining a plurality of groups of images as training samples; wherein each set of images in the plurality of sets of images comprises an original image and a barrel-shaped image corresponding to the original image; the network training module is used for training the generation network and the discrimination network in the pair of antagonistic neural networks by using the training sample so as to determine a trained generation network; the image determining module is used for inputting an image to be processed into the trained generating network so as to determine a barrel-shaped image corresponding to the image to be processed; and the image display module is used for displaying the barrel-shaped image corresponding to the image to be processed on a screen of the virtual reality equipment.
Optionally, the training sample determining module includes an original image determining unit, a parameter calculating unit, and a barrel image determining unit.
Specifically, the original image determining unit is used for determining an original image; the parameter calculation unit is used for calculating lens distortion parameters of the virtual reality equipment; the barrel-shaped image determining unit is used for converting the original image into a barrel-shaped image corresponding to the original image by utilizing the lens distortion parameters; wherein the original image and the barrel image are used as a set of images to be used as training samples.
Optionally, the network training module includes an intermediate image determining unit, a first confidence value determining unit, a second confidence value determining unit, and a network training unit.
Specifically, the intermediate image determining unit is used for inputting original images in the plurality of groups of images into a generation network of the antagonistic neural network so as to determine intermediate images corresponding to the original images; the first confidence value determining unit is used for inputting barrel-shaped images which are in the same group with the original images into a discrimination network of the countermeasure neural network so as to determine a first confidence value; the second confidence value determining unit is used for inputting the intermediate image into the discrimination network to determine a second confidence value; the network training unit is used for determining the loss of the generating network by using the second confidence value and determining the loss of the judging network by using the first confidence value and the second confidence value so as to train the generating network and the judging network in the countermeasure neural network.
Optionally, the network training unit comprises a first loss determination unit.
Specifically, the first loss determination unit is configured to determine a loss of the generating network based on a cross entropy between the second confidence value and 1.
Optionally, the network training unit comprises a second loss determination unit.
Specifically, the second loss determining unit is configured to determine a loss of the discrimination network based on a cross entropy between the first confidence value and 1 and a cross entropy between the second confidence value and 0.
Optionally, the image processing apparatus further comprises a network save module.
Specifically, the network storage module is used for storing the trained generation network as a model file in a preset format; and under the condition that the image to be processed is acquired, loading the model file so as to determine a barrel-shaped image corresponding to the image to be processed.
Optionally, the network saving module includes a network verification unit and a network saving unit.
Specifically, the network verification unit is used for verifying the trained generation network by adopting the verification image; the network storage unit is used for storing the trained generation network as a model file in a preset format if the verification result meets a preset condition.
According to an aspect of the present disclosure, there is provided a storage medium having stored thereon a computer program which, when executed by a processor, implements the image processing method of any one of the above.
According to one aspect of the present disclosure, there is provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image processing method of any one of the above via execution of the executable instructions.
In the technical schemes provided by some embodiments of the present disclosure, a barrel-shaped image corresponding to an image is generated by using an antagonistic neural network so as to be displayed on a screen of a virtual reality device, on one hand, the scheme of the present disclosure does not need to calculate distortion parameters of a lens of the virtual reality device, does not need to perform interpolation operation, solves the problem of errors in the calculation process, and can greatly improve the rendering speed compared with the related art; on the other hand, the method can be used for different virtual reality devices, and has universality.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort. In the drawings:
fig. 1 shows a schematic diagram of a related art scheme of processing image distortion;
fig. 2 schematically illustrates a flowchart of an image processing method according to an exemplary embodiment of the present disclosure;
FIG. 3 shows a schematic diagram of an original image and a barrel image in a training sample according to an exemplary embodiment of the present disclosure;
FIG. 4 illustrates a schematic diagram of a scheme for acquiring training samples according to an exemplary embodiment of the present disclosure;
fig. 5 schematically illustrates an acquired original image and a corresponding barrel image according to an exemplary embodiment of the present disclosure;
FIG. 6 schematically illustrates an effect diagram of generating a barrel image using the trained generation network of the present disclosure, according to an example embodiment of the present disclosure;
FIG. 7 shows a schematic diagram of an antagonistic neural network for implementing an image processing method according to an exemplary embodiment of the present disclosure;
fig. 8 schematically illustrates a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure;
FIG. 9 schematically illustrates a block diagram of a training sample determination module according to an example embodiment of the present disclosure;
FIG. 10 schematically illustrates a block diagram of a network training module according to an exemplary embodiment of the present disclosure;
FIG. 11 schematically illustrates a block diagram of a network training unit according to an exemplary embodiment of the present disclosure;
FIG. 12 schematically illustrates a block diagram of a network training unit according to another exemplary embodiment of the present disclosure;
fig. 13 schematically illustrates a block diagram of an image processing apparatus according to another exemplary embodiment of the present disclosure;
fig. 14 schematically illustrates a block diagram of a network save module according to an exemplary embodiment of the present disclosure;
FIG. 15 shows a schematic diagram of a storage medium according to an exemplary embodiment of the present disclosure; and
fig. 16 schematically illustrates a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the aspects of the disclosure may be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.
The flow diagrams depicted in the figures are exemplary only and not necessarily all steps are included. For example, some steps may be decomposed, and some steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.
In the process of carrying out distortion processing on an image in the related technology, whether solving distortion parameters of a lens or solving a corresponding relation between coordinates of points on a screen and points on an imaging image of the lens, the coordinates of a certain point on the screen and coordinates of corresponding points after passing through the lens are finally converted. Referring to fig. 1, it is necessary to calculate the coordinates (x, y) of a certain point on the screen and the corresponding coordinate point FOV after passing through the lens. In addition, during rendering, a camera in a scene searches a coordinate point FOV, corresponding (x, y) is determined according to the coordinate point FOV, and then a pixel, which is needed to be displayed on a screen, of a certain point in the scene is determined.
However, this approach is prone to computational errors and requires a continuous determination of the coordinates of the image, which is time consuming.
In view of the above, the present disclosure provides an image processing method and apparatus to solve the above-mentioned problems. It should be noted that the image processing method of the present disclosure may be implemented by a processing device, which may be, for example, a virtual environment processor, deployed in a virtual reality device. In this case, the image processing apparatus of the present disclosure may be located within the processing apparatus. However, without being limited thereto, the image processing method of the present disclosure may also be implemented by a remote server.
Fig. 2 schematically shows a flowchart of an image processing method of an exemplary embodiment of the present disclosure. Referring to fig. 2, the image processing method may include the steps of:
s22, determining a plurality of groups of images as training samples; wherein each of the plurality of sets of images includes an original image and a barrel image corresponding to the original image.
In an exemplary embodiment of the present disclosure, the lens distortion parameters of the virtual reality device may be utilized to determine training samples for the model used in the present disclosure.
First, an original image may be determined. The original image can be, for example, an image of a certain frame in a game scene, and the source, resolution, size, color temperature and other conditions of the original image are not particularly limited in the present disclosure; next, lens distortion parameters of the virtual reality device may be calculated. Specifically, the lens distortion parameters of the virtual reality device can be determined by adopting a method for determining the lens distortion parameters in the related technology, for example, the lens distortion parameters can be determined by adopting the principle of constant cross ratio in perspective projection imaging; the calculated lens distortion parameters may then be used to convert the original image into a barrel image corresponding to the original image.
Fig. 3 shows a schematic diagram of a determined original image 31 and a barrel image 32 corresponding to the original image 31.
The determined original image and barrel image are used as a group of images to be used as training samples. By continually acquiring raw images and determining corresponding barrel images to determine multiple sets of images, multiple sets of images can be used as a training set for the model used in the present disclosure to train the model.
In addition, for the acquisition of training samples, the VR machine with better distortion solving at present can be adopted, and the pictures (original images) shot by the same frame of camera and the images (barrel-shaped images) displayed on the screen can be saved as the training samples.
Fig. 4 shows a scheme for obtaining training samples. A barrel image is displayed on the screen 41, and the camera 43 can acquire an image (original image) of the barrel image obtained via the lens 42. The training set used in the present disclosure may be determined by constantly replacing the images on the screen and acquiring the corresponding images with the camera.
Referring to fig. 5, an image 51 shows the determined true original image, and an image 52 shows a barrel image corresponding to the image 51. Wherein the images 51 and 52 are the set of images described above.
S24, training the generation network and the discrimination network in the pair of antagonistic neural networks by using the training sample so as to determine the trained generation network.
Firstly, in one aspect, a barrel image corresponding to an original image in a training sample can be input into a discrimination network of a pair of antagonistic neural networks to determine a first confidence value; on the other hand, the original image may be input to a generation network of the antagonistic neural network to determine an intermediate image corresponding to the original image, and the intermediate image may be input to a discrimination network to determine the second confidence value. Wherein the terms "first", "second" are merely for distinguishing confidence values and should not be taken as limitations of the present disclosure.
The first confidence value and the second confidence value can be used for adjusting parameters of the antagonism neural network, so as to achieve the process of generating the network and judging the network in the antagonism neural network.
The second confidence value may be used to determine a loss of the generated network, and additionally, the first confidence value and the second confidence value may be used to determine a loss of the discriminating network. Specifically, the loss of the generated network may be determined based on the cross entropy between the second confidence value and 1; determining a loss of the discrimination network based on the cross entropy between the first confidence value and 1 and the cross entropy between the second confidence value and 0.
For example, the original image may be a 3-channel image of size 960×960, and 50 such images may be input at a time by an input node of the generation network. After the original image is subjected to the generation network, the output intermediate image is 3 channels, and may have a size of 960×1080.
The size of the image input to the discrimination network may be the same as the size of the output image of the generation network, and the number of output nodes of the discrimination network is 1, which is used for characterizing the confidence.
The generation network may be a 17-layer convolutional neural network, activated with a ReLU activation function, and gradient descent optimization with an Adam-type Optimizer (Optimizer). In addition, the discrimination network can be a 12-layer convolutional neural network, the full-connection part can be optimized by adopting a Dropout layer, and the last layer of the discrimination network can be activated by adopting a Sigmoid activation function.
According to some embodiments of the present disclosure, the trained generation network may be saved as a model file in a predetermined format (e.g.,. Pb format) in order to load the call.
According to further embodiments of the present disclosure, after training the generation network with the training set, the trained generation network may be validated with the validation image. That is, the trained generation network may be validated using the validation set. And if the verification result meets a preset condition, saving the trained generation network as a model file in a preset format. For example, in the case where the confidence value is 0 to 1, the preset condition may be that the confidence value obtained after verification of the verification set is greater than 0.95.
S26, inputting the image to be processed into a trained generation network to determine a barrel-shaped image corresponding to the image to be processed.
In an exemplary embodiment of the present disclosure, the image to be processed may be an image that is currently desired to be presented. In some scenarios, the image to be processed may be, for example, an image that the user is currently required to show while playing the VR game. However, it should be understood that the image to be processed of the present disclosure may also be any image that needs to be converted into a barrel image.
Referring to fig. 6, the image to be processed 61 is input to the generation network 60, and the result output by the generation network 60 is a corresponding barrel image 62.
And S28, displaying the barrel-shaped image corresponding to the image to be processed on a screen of the virtual reality equipment.
After determining the barrel image corresponding to the image to be processed in step S26, the barrel image may be displayed on the screen of the virtual reality device, so that the user knows the current virtual scene. For example, in the context of VR skiing games, a user may perform a corresponding skiing action based on a current virtual scene known to the human eye.
The antagonistic neural network of the present disclosure will be described below with reference to fig. 7. Wherein the antagonistic neural network includes a generation network (G network) and a discrimination network (D network).
Image a (i.e., the original image above) is input into the G network to obtain the and image a_out; inputting the image a_out into the D network to obtain a confidence value d_out_a; image B is a barrel image corresponding to image a, which is input to the D network to obtain a confidence value d_out_b. In this case, the loss of the G network may be determined according to the cross entropy between d_out_a and 1, and in addition, the loss of the D network may be determined according to the cross entropy between d_out_b and 1 and the cross entropy between d_out_a and 0.
The above-described images a and B are only a set of images for training the antagonistic neural network. The trained generation network is determined through a training process of the plurality of groups of images. The trained generation network may then be utilized to convert the image to be processed into a corresponding barrel image.
In addition, the number of nodes and the connection relationship of the G network and the D network in fig. 7 are only examples, and the present disclosure does not particularly limit the number of nodes, the connection relationship of nodes, and the number of network layers of the generation network and the discrimination network of the countermeasure neural network.
In summary, according to the image processing method disclosed by the disclosure, on one hand, distortion parameters of a lens of a virtual reality device do not need to be calculated, interpolation operation does not need to be performed, the problem that errors exist in the calculation process is solved, and compared with the related art, the rendering speed can be greatly improved; on the other hand, the method can be used for different virtual reality devices, and has universality.
It should be noted that although the steps of the methods in the present disclosure are depicted in the accompanying drawings in a particular order, this does not require or imply that the steps must be performed in that particular order, or that all illustrated steps be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform, etc.
Further, an image processing apparatus is also provided in the present exemplary embodiment.
Fig. 8 schematically shows a block diagram of an image processing apparatus of an exemplary embodiment of the present disclosure. Referring to fig. 8, an image processing apparatus 8 according to an exemplary embodiment of the present disclosure may include a training sample determination module 81, a network training module 83, an image determination module 85, and an image display module 87.
Specifically, the training sample determining module 81 may be configured to determine a plurality of sets of images as training samples; wherein each set of images in the plurality of sets of images comprises an original image and a barrel-shaped image corresponding to the original image; the network training module 83 may be configured to train the generation network and the discrimination network in the pair of antagonistic neural networks using the training samples to determine a trained generation network; the image determination module 85 may be configured to input an image to be processed into the trained generation network to determine a barrel image corresponding to the image to be processed; the image display module 87 may be configured to display a barrel image corresponding to the image to be processed on a screen of a virtual reality device.
According to the image processing device disclosed by the invention, on one hand, distortion parameters of the lens of the virtual reality equipment are not required to be calculated, interpolation operation is not required, the problem of errors in the calculation process is solved, and compared with the related technology, the rendering speed can be greatly improved; on the other hand, the method can be used for different virtual reality devices, and has universality.
According to an exemplary embodiment of the present disclosure, referring to fig. 9, the training sample determination module 81 may include an original image determination unit 901, a parameter calculation unit 903, and a barrel image determination unit 905.
Specifically, the original image determining unit 901 may be configured to determine an original image; the parameter calculation unit 903 may be used to calculate lens distortion parameters of the virtual reality device; the barrel image determining unit 905 may be configured to convert the original image into a barrel image corresponding to the original image using the lens distortion parameters; wherein the original image and the barrel image are used as a set of images to be used as training samples.
According to an exemplary embodiment of the present disclosure, referring to fig. 10, the network training module 83 may include an intermediate image determination unit 101, a first confidence value determination unit 103, a second confidence value determination unit 105, and a network training unit 107.
Specifically, the intermediate image determining unit 101 may be configured to input an original image of the plurality of sets of images into a generation network of an antagonistic neural network to determine an intermediate image corresponding to the original image; the first confidence value determining unit 103 may be configured to input barrel images that are the same group as the original image into a discrimination network of the antagonistic neural network to determine a first confidence value; the second confidence value determining unit 105 may be configured to input the intermediate image into the discrimination network to determine a second confidence value; the network training unit 107 may be configured to determine a loss of the generating network using the second confidence value and to determine a loss of the discriminating network using the first confidence value and the second confidence value, so as to train the generating network and the discriminating network in the antagonistic neural network.
According to an exemplary embodiment of the present disclosure, referring to fig. 11, the network training unit 107 may include a first loss determination unit 111.
Specifically, the first loss determination unit 111 may be configured to determine a loss of the generated network based on a cross entropy between the second confidence value and 1.
According to an exemplary embodiment of the present disclosure, referring to fig. 12, the network training unit 107 may further include a second loss determination unit 121.
Specifically, the second loss determination unit 121 may be configured to determine the loss of the discrimination network based on the cross entropy between the first confidence value and 1 and the cross entropy between the second confidence value and 0.
According to an exemplary embodiment of the present disclosure, referring to fig. 13, the image processing apparatus 13 may further include a network saving module 131, compared to the image processing apparatus 8.
Specifically, the network saving module 131 may be configured to save the trained generated network as a model file in a predetermined format; and under the condition that the image to be processed is acquired, loading the model file so as to determine a barrel-shaped image corresponding to the image to be processed.
According to an exemplary embodiment of the present disclosure, referring to fig. 14, the network saving module 131 may include a network authentication unit 141 and a network saving unit 143.
In particular, the network verification unit 141 may be configured to verify the trained generation network using the verification image; the network saving unit 143 may be configured to save the trained generation network as a model file in a predetermined format if the result of the verification satisfies a preset condition.
Since each functional module of the program execution performance analysis device according to the embodiment of the present invention is the same as that of the above-described method embodiment of the present invention, a detailed description thereof will be omitted.
In an exemplary embodiment of the present disclosure, a computer-readable storage medium having stored thereon a program product capable of implementing the method described above in the present specification is also provided. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the invention as described in the "exemplary methods" section of this specification, when said program product is run on the terminal device.
Referring to fig. 15, a program product 1500 for implementing the above-described method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.
Those skilled in the art will appreciate that the various aspects of the invention may be implemented as a system, method, or program product. Accordingly, aspects of the invention may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.
An electronic device 1600 according to such an embodiment of the invention is described below with reference to fig. 16. The electronic device 1600 shown in fig. 16 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.
As shown in fig. 16, the electronic device 1600 is embodied in the form of a general purpose computing device. The components of the electronic device 1600 may include, but are not limited to: the at least one processing unit 1610, the at least one memory unit 1620, a bus 1630 connecting the different system components (including the memory unit 1620 and the processing unit 1610), and a display unit 1640.
Wherein the storage unit stores program code that is executable by the processing unit 1610 such that the processing unit 1610 performs steps according to various exemplary embodiments of the present invention described in the above-described "exemplary methods" section of the present specification. For example, the processing unit 1610 may perform steps S22 to S28 as shown in fig. 2.
The memory unit 1620 may include readable media in the form of volatile memory units, such as Random Access Memory (RAM) 16201 and/or cache memory 16202, and may further include Read Only Memory (ROM) 16203.
The storage unit 1620 may also include a program/utility 16204 having a set (at least one) of program modules 16205, such program modules 16205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.
Furthermore, the above-described drawings are only schematic illustrations of processes included in the method according to the exemplary embodiment of the present invention, and are not intended to be limiting. It will be readily appreciated that the processes shown in the above figures do not indicate or limit the temporal order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, among a plurality of modules.
It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (9)
1. An image processing method, comprising:
determining a plurality of groups of images as training samples, wherein each group of images in the plurality of groups of images comprises an original image and a barrel-shaped image corresponding to the original image;
training a generation network and a discrimination network in a pair of antagonistic neural networks by using the training sample to determine a trained generation network;
inputting an image to be processed into the trained generation network to determine a barrel image corresponding to the image to be processed;
displaying a barrel-shaped image corresponding to the image to be processed on a screen of virtual reality equipment;
determining multiple sets of images as training samples includes:
determining an original image;
calculating lens distortion parameters of the virtual reality equipment;
converting the original image into a barrel image corresponding to the original image by using the lens distortion parameters;
wherein the original image and the barrel image are used as a set of images to be used as training samples.
2. The image processing method according to claim 1, wherein training the generation network and the discrimination network in the pair of antagonistic neural networks using the training samples includes:
inputting original images in the plurality of groups of images into a generation network of an antagonistic neural network to determine an intermediate image corresponding to the original images;
inputting barrel-shaped images which are in the same group with the original images into a discrimination network of the antagonistic neural network to determine a first confidence value;
inputting the intermediate image into the discrimination network to determine a second confidence value;
determining a loss of the generating network using the second confidence value, and determining a loss of the discriminating network using the first confidence value and the second confidence value, so as to train the generating network and the discriminating network in the antagonistic neural network.
3. The image processing method of claim 2, wherein determining the loss of the generated network using the second confidence value comprises:
a loss of the generated network is determined based on a cross entropy between the second confidence value and 1.
4. The image processing method of claim 3, wherein determining the loss of the discrimination network using the first confidence value and the second confidence value comprises:
determining a loss of the discrimination network based on a cross entropy between the first confidence value and 1 and a cross entropy between the second confidence value and 0.
5. The image processing method according to claim 1, characterized in that the image processing method further comprises:
saving the trained generation network as a model file in a predetermined format;
and under the condition that the image to be processed is acquired, loading the model file so as to determine a barrel-shaped image corresponding to the image to be processed.
6. The image processing method according to claim 5, wherein saving the trained generation network as a model file in a predetermined format comprises:
verifying the trained generation network by adopting the verification image;
and if the verification result meets a preset condition, saving the trained generation network as a model file in a preset format.
7. An image processing apparatus, comprising:
the training sample determining module is used for determining a plurality of groups of images as training samples; each set of images in the plurality of sets of images comprises an original image and a barrel-shaped image corresponding to the original image;
the network training module is used for training the generation network and the discrimination network in the pair of antagonistic neural networks by utilizing the training sample so as to determine a trained generation network;
the image determining module is used for inputting an image to be processed into the trained generating network so as to determine a barrel-shaped image corresponding to the image to be processed;
the image display module is used for displaying the barrel-shaped image corresponding to the image to be processed on a screen of the virtual reality equipment;
the training sample determination module includes:
an original image determining unit configured to determine an original image;
a parameter calculation unit for calculating a lens distortion parameter of the virtual reality device;
a barrel image determining unit for converting the original image into a barrel image corresponding to the original image using the lens distortion parameters;
wherein the original image and the barrel image are used as a set of images to be used as training samples.
8. A storage medium having stored thereon a computer program, which when executed by a processor implements the image processing method of any of claims 1 to 6.
9. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the image processing method of any one of claims 1 to 6 via execution of the executable instructions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910008670.0A CN109741250B (en) | 2019-01-04 | 2019-01-04 | Image processing method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910008670.0A CN109741250B (en) | 2019-01-04 | 2019-01-04 | Image processing method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109741250A CN109741250A (en) | 2019-05-10 |
| CN109741250B true CN109741250B (en) | 2023-06-16 |
Family
ID=66363519
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910008670.0A Active CN109741250B (en) | 2019-01-04 | 2019-01-04 | Image processing method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109741250B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110347305A (en) | 2019-05-30 | 2019-10-18 | 华为技术有限公司 | A kind of VR multi-display method and electronic equipment |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105869142A (en) * | 2015-12-21 | 2016-08-17 | 乐视致新电子科技（天津）有限公司 | Method and device for testing imaging distortion of virtual reality helmets |
| CN107154027B (en) * | 2017-04-17 | 2020-07-07 | 深圳大学 | Compensation method and device for restoration of distorted image |
| CN107451965B (en) * | 2017-07-24 | 2019-10-18 | 深圳市智美达科技股份有限公司 | Distort face image correcting method, device, computer equipment and storage medium |
| CN107945133B (en) * | 2017-11-30 | 2022-08-05 | 北京小米移动软件有限公司 | Image processing method and device |
| CN109120854B (en) * | 2018-10-09 | 2020-10-23 | 北京旷视科技有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
-
2019
- 2019-01-04 CN CN201910008670.0A patent/CN109741250B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109741250A (en) | 2019-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109816589B (en) | Method and apparatus for generating cartoon style conversion model | |
| CN112989904B (en) | Method for generating style image, method, device, equipment and medium for training model | |
| CN108830235B (en) | Method and apparatus for generating information | |
| CN111476871B (en) | Method and device for generating video | |
| CN111654746B (en) | Video frame insertion method and device, electronic equipment and storage medium | |
| CN109829432B (en) | Method and apparatus for generating information | |
| CN108765340B (en) | Blurred image processing method and device and terminal equipment | |
| CN108416744B (en) | Image processing method, device, equipment and computer readable storage medium | |
| KR20210029089A (en) | System and metho for video processsing with enhanced temporal consistency | |
| CN112785507A (en) | Image processing method and device, storage medium and terminal | |
| KR102552532B1 (en) | Method and device for image rendering | |
| EP3872760A2 (en) | Method and apparatus of training depth estimation network, and method and apparatus of estimating depth of image | |
| CN114073071B (en) | Video frame inserting method and device and computer readable storage medium | |
| US20230143452A1 (en) | Method and apparatus for generating image, electronic device and storage medium | |
| CN111105375A (en) | Image generation method, model training method and device thereof, and electronic equipment | |
| CN113379877A (en) | Face video generation method and device, electronic equipment and storage medium | |
| CN111199540A (en) | Image quality evaluation method, image quality evaluation device, electronic device, and storage medium | |
| CN111951192A (en) | Shot image processing method and shooting equipment | |
| CN111177167B (en) | Augmented reality map updating method, device, system, storage and equipment | |
| CN115272565A (en) | Head three-dimensional model reconstruction method and electronic equipment | |
| CN107622241A (en) | Display methods and device for mobile device | |
| CN107798675A (en) | The detection method and device of smear in display image | |
| CN109741250B (en) | Image processing method and device, storage medium and electronic equipment | |
| CN113516697A (en) | Image registration method and device, electronic equipment and computer-readable storage medium | |
| CN109816791B (en) | Method and apparatus for generating information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |