CN109741250B - Image processing method and device, storage medium and electronic equipment - Google Patents

Image processing method and device, storage medium and electronic equipment Download PDF

Info

Publication number
CN109741250B
CN109741250B CN201910008670.0A CN201910008670A CN109741250B CN 109741250 B CN109741250 B CN 109741250B CN 201910008670 A CN201910008670 A CN 201910008670A CN 109741250 B CN109741250 B CN 109741250B
Authority
CN
China
Prior art keywords
image
network
images
barrel
confidence value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910008670.0A
Other languages
Chinese (zh)
Other versions
CN109741250A (en
Inventor
薛鸿臻
张�浩
陈丽莉
孙建康
马福强
张硕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BOE Technology Group Co Ltd
Beijing BOE Optoelectronics Technology Co Ltd
Original Assignee
BOE Technology Group Co Ltd
Beijing BOE Optoelectronics Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by BOE Technology Group Co Ltd, Beijing BOE Optoelectronics Technology Co Ltd filed Critical BOE Technology Group Co Ltd
Priority to CN201910008670.0A priority Critical patent/CN109741250B/en
Publication of CN109741250A publication Critical patent/CN109741250A/en
Application granted granted Critical
Publication of CN109741250B publication Critical patent/CN109741250B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02TCLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00Road transport of goods or passengers
    • Y02T10/10Internal combustion engine [ICE] based vehicles
    • Y02T10/40Engine management systems

Abstract

The invention discloses an image processing method and device, a storage medium and electronic equipment, and relates to the technical field of image processing. The image processing method comprises the following steps: determining a plurality of groups of images as training samples; each group of images in the plurality of groups of images comprises an original image and a barrel-shaped image corresponding to the original image; training a generation network and a discrimination network in a pair of antagonistic neural networks by using a training sample to determine a trained generation network; inputting the image to be processed into a trained generation network to determine a barrel-shaped image corresponding to the image to be processed; and displaying the barrel-shaped image corresponding to the image to be processed on a screen of the virtual reality equipment. The present disclosure may enable predistortion processing of images on virtual reality devices.

Description

Image processing method and device, storage medium and electronic equipment
Technical Field
The present disclosure relates to the field of image processing technology, and in particular, to an image processing method, an image processing apparatus, a storage medium, and an electronic device.
Background
With the continuous progress of image processing technology, virtual Reality (VR) technology is gradually appearing in the field of view of people, especially in the aspect of games, and has been rapidly developed.
In order for a user to visually possess a realistic sense of immersion, a virtual reality device (e.g., a head-mounted stereoscopic display, or simply a head-mounted display) should cover the visual range of the human eye as much as possible. Therefore, a magnifying lens with a specific spherical radian needs to be configured on the virtual reality device. In this case, when a conventional image is projected into the human eye using the magnifying lens, the image is distorted.
Currently, for the case of distortion of an image, distortion parameters of a lens in a virtual reality device are generally required to be calculated and corrected. In addition, in software implementation, interpolation operation is required to solve the problem of image distortion. However, for the current processing mode, on one hand, errors may exist when solving distortion parameters, so that the image received by human eyes is abnormal; on the other hand, when scene rendering, the coordinates of the image on the screen need to be continuously determined, and the time consumption is long; in yet another aspect, errors may exist in lens installation in a virtual reality device, resulting in errors in calculated screen coordinate points.
It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
An object of the present disclosure is to provide an image processing method, an image processing apparatus, a storage medium, and an electronic device, which further overcome, at least to some extent, the problems of abnormal image display on a screen of a virtual reality device and slow image processing speed due to limitations and drawbacks of the related art.
According to an aspect of the present disclosure, there is provided an image processing method including: determining a plurality of groups of images as training samples; wherein each set of images in the plurality of sets of images comprises an original image and a barrel-shaped image corresponding to the original image; training a generation network and a discrimination network in a pair of antagonistic neural networks by using the training sample to determine a trained generation network; inputting an image to be processed into the trained generation network to determine a barrel image corresponding to the image to be processed; and displaying the barrel-shaped image corresponding to the image to be processed on a screen of the virtual reality equipment.
Optionally, determining the plurality of sets of images as training samples comprises: determining an original image; calculating lens distortion parameters of the virtual reality equipment; converting the original image into a barrel image corresponding to the original image by using the lens distortion parameters; wherein the original image and the barrel image are used as a set of images to be used as training samples.
Optionally, training the generation network and the discrimination network in the pair of antagonistic neural networks using the training samples includes: inputting original images in the plurality of groups of images into a generation network of an antagonistic neural network to determine an intermediate image corresponding to the original images; inputting barrel-shaped images which are in the same group with the original images into a discrimination network of the antagonistic neural network to determine a first confidence value; inputting the intermediate image into the discrimination network to determine a second confidence value; determining a loss of the generating network using the second confidence value, and determining a loss of the discriminating network using the first confidence value and the second confidence value, so as to train the generating network and the discriminating network in the antagonistic neural network.
Optionally, determining the loss of the generated network using the second confidence value comprises: a loss of the generated network is determined based on a cross entropy between the second confidence value and 1.
Optionally, determining the loss of the discrimination network using the first confidence value and the second confidence value includes: determining a loss of the discrimination network based on a cross entropy between the first confidence value and 1 and a cross entropy between the second confidence value and 0.
Optionally, the image processing method further includes: saving the trained generation network as a model file in a predetermined format; and under the condition that the image to be processed is acquired, loading the model file so as to determine a barrel-shaped image corresponding to the image to be processed.
Optionally, saving the trained generation network as a model file in a predetermined format includes: verifying the trained generation network by adopting the verification image; and if the verification result meets a preset condition, saving the trained generation network as a model file in a preset format.
According to one aspect of the present disclosure, there is provided an image processing apparatus including a training sample determination module, a network training module, an image determination module, and an image display module.
Specifically, the training sample determining module is used for determining a plurality of groups of images as training samples; wherein each set of images in the plurality of sets of images comprises an original image and a barrel-shaped image corresponding to the original image; the network training module is used for training the generation network and the discrimination network in the pair of antagonistic neural networks by using the training sample so as to determine a trained generation network; the image determining module is used for inputting an image to be processed into the trained generating network so as to determine a barrel-shaped image corresponding to the image to be processed; and the image display module is used for displaying the barrel-shaped image corresponding to the image to be processed on a screen of the virtual reality equipment.
Optionally, the training sample determining module includes an original image determining unit, a parameter calculating unit, and a barrel image determining unit.
Specifically, the original image determining unit is used for determining an original image; the parameter calculation unit is used for calculating lens distortion parameters of the virtual reality equipment; the barrel-shaped image determining unit is used for converting the original image into a barrel-shaped image corresponding to the original image by utilizing the lens distortion parameters; wherein the original image and the barrel image are used as a set of images to be used as training samples.
Optionally, the network training module includes an intermediate image determining unit, a first confidence value determining unit, a second confidence value determining unit, and a network training unit.
Specifically, the intermediate image determining unit is used for inputting original images in the plurality of groups of images into a generation network of the antagonistic neural network so as to determine intermediate images corresponding to the original images; the first confidence value determining unit is used for inputting barrel-shaped images which are in the same group with the original images into a discrimination network of the countermeasure neural network so as to determine a first confidence value; the second confidence value determining unit is used for inputting the intermediate image into the discrimination network to determine a second confidence value; the network training unit is used for determining the loss of the generating network by using the second confidence value and determining the loss of the judging network by using the first confidence value and the second confidence value so as to train the generating network and the judging network in the countermeasure neural network.
Optionally, the network training unit comprises a first loss determination unit.
Specifically, the first loss determination unit is configured to determine a loss of the generating network based on a cross entropy between the second confidence value and 1.
Optionally, the network training unit comprises a second loss determination unit.
Specifically, the second loss determining unit is configured to determine a loss of the discrimination network based on a cross entropy between the first confidence value and 1 and a cross entropy between the second confidence value and 0.
Optionally, the image processing apparatus further comprises a network save module.
Specifically, the network storage module is used for storing the trained generation network as a model file in a preset format; and under the condition that the image to be processed is acquired, loading the model file so as to determine a barrel-shaped image corresponding to the image to be processed.
Optionally, the network saving module includes a network verification unit and a network saving unit.
Specifically, the network verification unit is used for verifying the trained generation network by adopting the verification image; the network storage unit is used for storing the trained generation network as a model file in a preset format if the verification result meets a preset condition.
According to an aspect of the present disclosure, there is provided a storage medium having stored thereon a computer program which, when executed by a processor, implements the image processing method of any one of the above.
According to one aspect of the present disclosure, there is provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image processing method of any one of the above via execution of the executable instructions.
In the technical schemes provided by some embodiments of the present disclosure, a barrel-shaped image corresponding to an image is generated by using an antagonistic neural network so as to be displayed on a screen of a virtual reality device, on one hand, the scheme of the present disclosure does not need to calculate distortion parameters of a lens of the virtual reality device, does not need to perform interpolation operation, solves the problem of errors in the calculation process, and can greatly improve the rendering speed compared with the related art; on the other hand, the method can be used for different virtual reality devices, and has universality.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort. In the drawings:
fig. 1 shows a schematic diagram of a related art scheme of processing image distortion;
fig. 2 schematically illustrates a flowchart of an image processing method according to an exemplary embodiment of the present disclosure;
FIG. 3 shows a schematic diagram of an original image and a barrel image in a training sample according to an exemplary embodiment of the present disclosure;
FIG. 4 illustrates a schematic diagram of a scheme for acquiring training samples according to an exemplary embodiment of the present disclosure;
fig. 5 schematically illustrates an acquired original image and a corresponding barrel image according to an exemplary embodiment of the present disclosure;
FIG. 6 schematically illustrates an effect diagram of generating a barrel image using the trained generation network of the present disclosure, according to an example embodiment of the present disclosure;
FIG. 7 shows a schematic diagram of an antagonistic neural network for implementing an image processing method according to an exemplary embodiment of the present disclosure;
fig. 8 schematically illustrates a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure;
FIG. 9 schematically illustrates a block diagram of a training sample determination module according to an example embodiment of the present disclosure;
FIG. 10 schematically illustrates a block diagram of a network training module according to an exemplary embodiment of the present disclosure;
FIG. 11 schematically illustrates a block diagram of a network training unit according to an exemplary embodiment of the present disclosure;
FIG. 12 schematically illustrates a block diagram of a network training unit according to another exemplary embodiment of the present disclosure;
fig. 13 schematically illustrates a block diagram of an image processing apparatus according to another exemplary embodiment of the present disclosure;
fig. 14 schematically illustrates a block diagram of a network save module according to an exemplary embodiment of the present disclosure;
FIG. 15 shows a schematic diagram of a storage medium according to an exemplary embodiment of the present disclosure; and
fig. 16 schematically illustrates a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the aspects of the disclosure may be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.
The flow diagrams depicted in the figures are exemplary only and not necessarily all steps are included. For example, some steps may be decomposed, and some steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.
In the process of carrying out distortion processing on an image in the related technology, whether solving distortion parameters of a lens or solving a corresponding relation between coordinates of points on a screen and points on an imaging image of the lens, the coordinates of a certain point on the screen and coordinates of corresponding points after passing through the lens are finally converted. Referring to fig. 1, it is necessary to calculate the coordinates (x, y) of a certain point on the screen and the corresponding coordinate point FOV after passing through the lens. In addition, during rendering, a camera in a scene searches a coordinate point FOV, corresponding (x, y) is determined according to the coordinate point FOV, and then a pixel, which is needed to be displayed on a screen, of a certain point in the scene is determined.
However, this approach is prone to computational errors and requires a continuous determination of the coordinates of the image, which is time consuming.
In view of the above, the present disclosure provides an image processing method and apparatus to solve the above-mentioned problems. It should be noted that the image processing method of the present disclosure may be implemented by a processing device, which may be, for example, a virtual environment processor, deployed in a virtual reality device. In this case, the image processing apparatus of the present disclosure may be located within the processing apparatus. However, without being limited thereto, the image processing method of the present disclosure may also be implemented by a remote server.
Fig. 2 schematically shows a flowchart of an image processing method of an exemplary embodiment of the present disclosure. Referring to fig. 2, the image processing method may include the steps of:
s22, determining a plurality of groups of images as training samples; wherein each of the plurality of sets of images includes an original image and a barrel image corresponding to the original image.
In an exemplary embodiment of the present disclosure, the lens distortion parameters of the virtual reality device may be utilized to determine training samples for the model used in the present disclosure.
First, an original image may be determined. The original image can be, for example, an image of a certain frame in a game scene, and the source, resolution, size, color temperature and other conditions of the original image are not particularly limited in the present disclosure; next, lens distortion parameters of the virtual reality device may be calculated. Specifically, the lens distortion parameters of the virtual reality device can be determined by adopting a method for determining the lens distortion parameters in the related technology, for example, the lens distortion parameters can be determined by adopting the principle of constant cross ratio in perspective projection imaging; the calculated lens distortion parameters may then be used to convert the original image into a barrel image corresponding to the original image.
Fig. 3 shows a schematic diagram of a determined original image 31 and a barrel image 32 corresponding to the original image 31.
The determined original image and barrel image are used as a group of images to be used as training samples. By continually acquiring raw images and determining corresponding barrel images to determine multiple sets of images, multiple sets of images can be used as a training set for the model used in the present disclosure to train the model.
In addition, for the acquisition of training samples, the VR machine with better distortion solving at present can be adopted, and the pictures (original images) shot by the same frame of camera and the images (barrel-shaped images) displayed on the screen can be saved as the training samples.
Fig. 4 shows a scheme for obtaining training samples. A barrel image is displayed on the screen 41, and the camera 43 can acquire an image (original image) of the barrel image obtained via the lens 42. The training set used in the present disclosure may be determined by constantly replacing the images on the screen and acquiring the corresponding images with the camera.
Referring to fig. 5, an image 51 shows the determined true original image, and an image 52 shows a barrel image corresponding to the image 51. Wherein the images 51 and 52 are the set of images described above.
S24, training the generation network and the discrimination network in the pair of antagonistic neural networks by using the training sample so as to determine the trained generation network.
Firstly, in one aspect, a barrel image corresponding to an original image in a training sample can be input into a discrimination network of a pair of antagonistic neural networks to determine a first confidence value; on the other hand, the original image may be input to a generation network of the antagonistic neural network to determine an intermediate image corresponding to the original image, and the intermediate image may be input to a discrimination network to determine the second confidence value. Wherein the terms "first", "second" are merely for distinguishing confidence values and should not be taken as limitations of the present disclosure.
The first confidence value and the second confidence value can be used for adjusting parameters of the antagonism neural network, so as to achieve the process of generating the network and judging the network in the antagonism neural network.
The second confidence value may be used to determine a loss of the generated network, and additionally, the first confidence value and the second confidence value may be used to determine a loss of the discriminating network. Specifically, the loss of the generated network may be determined based on the cross entropy between the second confidence value and 1; determining a loss of the discrimination network based on the cross entropy between the first confidence value and 1 and the cross entropy between the second confidence value and 0.
For example, the original image may be a 3-channel image of size 960×960, and 50 such images may be input at a time by an input node of the generation network. After the original image is subjected to the generation network, the output intermediate image is 3 channels, and may have a size of 960×1080.
The size of the image input to the discrimination network may be the same as the size of the output image of the generation network, and the number of output nodes of the discrimination network is 1, which is used for characterizing the confidence.
The generation network may be a 17-layer convolutional neural network, activated with a ReLU activation function, and gradient descent optimization with an Adam-type Optimizer (Optimizer). In addition, the discrimination network can be a 12-layer convolutional neural network, the full-connection part can be optimized by adopting a Dropout layer, and the last layer of the discrimination network can be activated by adopting a Sigmoid activation function.
According to some embodiments of the present disclosure, the trained generation network may be saved as a model file in a predetermined format (e.g.,. Pb format) in order to load the call.
According to further embodiments of the present disclosure, after training the generation network with the training set, the trained generation network may be validated with the validation image. That is, the trained generation network may be validated using the validation set. And if the verification result meets a preset condition, saving the trained generation network as a model file in a preset format. For example, in the case where the confidence value is 0 to 1, the preset condition may be that the confidence value obtained after verification of the verification set is greater than 0.95.
S26, inputting the image to be processed into a trained generation network to determine a barrel-shaped image corresponding to the image to be processed.
In an exemplary embodiment of the present disclosure, the image to be processed may be an image that is currently desired to be presented. In some scenarios, the image to be processed may be, for example, an image that the user is currently required to show while playing the VR game. However, it should be understood that the image to be processed of the present disclosure may also be any image that needs to be converted into a barrel image.
Referring to fig. 6, the image to be processed 61 is input to the generation network 60, and the result output by the generation network 60 is a corresponding barrel image 62.
And S28, displaying the barrel-shaped image corresponding to the image to be processed on a screen of the virtual reality equipment.
After determining the barrel image corresponding to the image to be processed in step S26, the barrel image may be displayed on the screen of the virtual reality device, so that the user knows the current virtual scene. For example, in the context of VR skiing games, a user may perform a corresponding skiing action based on a current virtual scene known to the human eye.
The antagonistic neural network of the present disclosure will be described below with reference to fig. 7. Wherein the antagonistic neural network includes a generation network (G network) and a discrimination network (D network).
Image a (i.e., the original image above) is input into the G network to obtain the and image a_out; inputting the image a_out into the D network to obtain a confidence value d_out_a; image B is a barrel image corresponding to image a, which is input to the D network to obtain a confidence value d_out_b. In this case, the loss of the G network may be determined according to the cross entropy between d_out_a and 1, and in addition, the loss of the D network may be determined according to the cross entropy between d_out_b and 1 and the cross entropy between d_out_a and 0.
The above-described images a and B are only a set of images for training the antagonistic neural network. The trained generation network is determined through a training process of the plurality of groups of images. The trained generation network may then be utilized to convert the image to be processed into a corresponding barrel image.
In addition, the number of nodes and the connection relationship of the G network and the D network in fig. 7 are only examples, and the present disclosure does not particularly limit the number of nodes, the connection relationship of nodes, and the number of network layers of the generation network and the discrimination network of the countermeasure neural network.
In summary, according to the image processing method disclosed by the disclosure, on one hand, distortion parameters of a lens of a virtual reality device do not need to be calculated, interpolation operation does not need to be performed, the problem that errors exist in the calculation process is solved, and compared with the related art, the rendering speed can be greatly improved; on the other hand, the method can be used for different virtual reality devices, and has universality.
It should be noted that although the steps of the methods in the present disclosure are depicted in the accompanying drawings in a particular order, this does not require or imply that the steps must be performed in that particular order, or that all illustrated steps be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform, etc.
Further, an image processing apparatus is also provided in the present exemplary embodiment.
Fig. 8 schematically shows a block diagram of an image processing apparatus of an exemplary embodiment of the present disclosure. Referring to fig. 8, an image processing apparatus 8 according to an exemplary embodiment of the present disclosure may include a training sample determination module 81, a network training module 83, an image determination module 85, and an image display module 87.
Specifically, the training sample determining module 81 may be configured to determine a plurality of sets of images as training samples; wherein each set of images in the plurality of sets of images comprises an original image and a barrel-shaped image corresponding to the original image; the network training module 83 may be configured to train the generation network and the discrimination network in the pair of antagonistic neural networks using the training samples to determine a trained generation network; the image determination module 85 may be configured to input an image to be processed into the trained generation network to determine a barrel image corresponding to the image to be processed; the image display module 87 may be configured to display a barrel image corresponding to the image to be processed on a screen of a virtual reality device.
According to the image processing device disclosed by the invention, on one hand, distortion parameters of the lens of the virtual reality equipment are not required to be calculated, interpolation operation is not required, the problem of errors in the calculation process is solved, and compared with the related technology, the rendering speed can be greatly improved; on the other hand, the method can be used for different virtual reality devices, and has universality.
According to an exemplary embodiment of the present disclosure, referring to fig. 9, the training sample determination module 81 may include an original image determination unit 901, a parameter calculation unit 903, and a barrel image determination unit 905.
Specifically, the original image determining unit 901 may be configured to determine an original image; the parameter calculation unit 903 may be used to calculate lens distortion parameters of the virtual reality device; the barrel image determining unit 905 may be configured to convert the original image into a barrel image corresponding to the original image using the lens distortion parameters; wherein the original image and the barrel image are used as a set of images to be used as training samples.
According to an exemplary embodiment of the present disclosure, referring to fig. 10, the network training module 83 may include an intermediate image determination unit 101, a first confidence value determination unit 103, a second confidence value determination unit 105, and a network training unit 107.
Specifically, the intermediate image determining unit 101 may be configured to input an original image of the plurality of sets of images into a generation network of an antagonistic neural network to determine an intermediate image corresponding to the original image; the first confidence value determining unit 103 may be configured to input barrel images that are the same group as the original image into a discrimination network of the antagonistic neural network to determine a first confidence value; the second confidence value determining unit 105 may be configured to input the intermediate image into the discrimination network to determine a second confidence value; the network training unit 107 may be configured to determine a loss of the generating network using the second confidence value and to determine a loss of the discriminating network using the first confidence value and the second confidence value, so as to train the generating network and the discriminating network in the antagonistic neural network.
According to an exemplary embodiment of the present disclosure, referring to fig. 11, the network training unit 107 may include a first loss determination unit 111.
Specifically, the first loss determination unit 111 may be configured to determine a loss of the generated network based on a cross entropy between the second confidence value and 1.
According to an exemplary embodiment of the present disclosure, referring to fig. 12, the network training unit 107 may further include a second loss determination unit 121.
Specifically, the second loss determination unit 121 may be configured to determine the loss of the discrimination network based on the cross entropy between the first confidence value and 1 and the cross entropy between the second confidence value and 0.
According to an exemplary embodiment of the present disclosure, referring to fig. 13, the image processing apparatus 13 may further include a network saving module 131, compared to the image processing apparatus 8.
Specifically, the network saving module 131 may be configured to save the trained generated network as a model file in a predetermined format; and under the condition that the image to be processed is acquired, loading the model file so as to determine a barrel-shaped image corresponding to the image to be processed.
According to an exemplary embodiment of the present disclosure, referring to fig. 14, the network saving module 131 may include a network authentication unit 141 and a network saving unit 143.
In particular, the network verification unit 141 may be configured to verify the trained generation network using the verification image; the network saving unit 143 may be configured to save the trained generation network as a model file in a predetermined format if the result of the verification satisfies a preset condition.
Since each functional module of the program execution performance analysis device according to the embodiment of the present invention is the same as that of the above-described method embodiment of the present invention, a detailed description thereof will be omitted.
In an exemplary embodiment of the present disclosure, a computer-readable storage medium having stored thereon a program product capable of implementing the method described above in the present specification is also provided. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the invention as described in the "exemplary methods" section of this specification, when said program product is run on the terminal device.
Referring to fig. 15, a program product 1500 for implementing the above-described method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.
Those skilled in the art will appreciate that the various aspects of the invention may be implemented as a system, method, or program product. Accordingly, aspects of the invention may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.
An electronic device 1600 according to such an embodiment of the invention is described below with reference to fig. 16. The electronic device 1600 shown in fig. 16 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.
As shown in fig. 16, the electronic device 1600 is embodied in the form of a general purpose computing device. The components of the electronic device 1600 may include, but are not limited to: the at least one processing unit 1610, the at least one memory unit 1620, a bus 1630 connecting the different system components (including the memory unit 1620 and the processing unit 1610), and a display unit 1640.
Wherein the storage unit stores program code that is executable by the processing unit 1610 such that the processing unit 1610 performs steps according to various exemplary embodiments of the present invention described in the above-described "exemplary methods" section of the present specification. For example, the processing unit 1610 may perform steps S22 to S28 as shown in fig. 2.
The memory unit 1620 may include readable media in the form of volatile memory units, such as Random Access Memory (RAM) 16201 and/or cache memory 16202, and may further include Read Only Memory (ROM) 16203.
The storage unit 1620 may also include a program/utility 16204 having a set (at least one) of program modules 16205, such program modules 16205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
Bus 1630 may be a local bus representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or using any of a variety of bus architectures.
Electronic device 1600 may also communicate with one or more external devices 1700 (e.g., keyboard, pointing device, bluetooth device, etc.), as well as with one or more devices that enable a user to interact with the electronic device 1600, and/or with any device (e.g., router, modem, etc.) that enables the electronic device 1600 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 1650. Also, electronic device 1600 can communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through network adapter 1660. As shown, network adapter 1660 communicates with other modules of electronic device 1600 over bus 1630. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 1600, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.
Furthermore, the above-described drawings are only schematic illustrations of processes included in the method according to the exemplary embodiment of the present invention, and are not intended to be limiting. It will be readily appreciated that the processes shown in the above figures do not indicate or limit the temporal order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, among a plurality of modules.
It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (9)

1. An image processing method, comprising:
determining a plurality of groups of images as training samples, wherein each group of images in the plurality of groups of images comprises an original image and a barrel-shaped image corresponding to the original image;
training a generation network and a discrimination network in a pair of antagonistic neural networks by using the training sample to determine a trained generation network;
inputting an image to be processed into the trained generation network to determine a barrel image corresponding to the image to be processed;
displaying a barrel-shaped image corresponding to the image to be processed on a screen of virtual reality equipment;
determining multiple sets of images as training samples includes:
determining an original image;
calculating lens distortion parameters of the virtual reality equipment;
converting the original image into a barrel image corresponding to the original image by using the lens distortion parameters;
wherein the original image and the barrel image are used as a set of images to be used as training samples.
2. The image processing method according to claim 1, wherein training the generation network and the discrimination network in the pair of antagonistic neural networks using the training samples includes:
inputting original images in the plurality of groups of images into a generation network of an antagonistic neural network to determine an intermediate image corresponding to the original images;
inputting barrel-shaped images which are in the same group with the original images into a discrimination network of the antagonistic neural network to determine a first confidence value;
inputting the intermediate image into the discrimination network to determine a second confidence value;
determining a loss of the generating network using the second confidence value, and determining a loss of the discriminating network using the first confidence value and the second confidence value, so as to train the generating network and the discriminating network in the antagonistic neural network.
3. The image processing method of claim 2, wherein determining the loss of the generated network using the second confidence value comprises:
a loss of the generated network is determined based on a cross entropy between the second confidence value and 1.
4. The image processing method of claim 3, wherein determining the loss of the discrimination network using the first confidence value and the second confidence value comprises:
determining a loss of the discrimination network based on a cross entropy between the first confidence value and 1 and a cross entropy between the second confidence value and 0.
5. The image processing method according to claim 1, characterized in that the image processing method further comprises:
saving the trained generation network as a model file in a predetermined format;
and under the condition that the image to be processed is acquired, loading the model file so as to determine a barrel-shaped image corresponding to the image to be processed.
6. The image processing method according to claim 5, wherein saving the trained generation network as a model file in a predetermined format comprises:
verifying the trained generation network by adopting the verification image;
and if the verification result meets a preset condition, saving the trained generation network as a model file in a preset format.
7. An image processing apparatus, comprising:
the training sample determining module is used for determining a plurality of groups of images as training samples; each set of images in the plurality of sets of images comprises an original image and a barrel-shaped image corresponding to the original image;
the network training module is used for training the generation network and the discrimination network in the pair of antagonistic neural networks by utilizing the training sample so as to determine a trained generation network;
the image determining module is used for inputting an image to be processed into the trained generating network so as to determine a barrel-shaped image corresponding to the image to be processed;
the image display module is used for displaying the barrel-shaped image corresponding to the image to be processed on a screen of the virtual reality equipment;
the training sample determination module includes:
an original image determining unit configured to determine an original image;
a parameter calculation unit for calculating a lens distortion parameter of the virtual reality device;
a barrel image determining unit for converting the original image into a barrel image corresponding to the original image using the lens distortion parameters;
wherein the original image and the barrel image are used as a set of images to be used as training samples.
8. A storage medium having stored thereon a computer program, which when executed by a processor implements the image processing method of any of claims 1 to 6.
9. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the image processing method of any one of claims 1 to 6 via execution of the executable instructions.
CN201910008670.0A 2019-01-04 2019-01-04 Image processing method and device, storage medium and electronic equipment Active CN109741250B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910008670.0A CN109741250B (en) 2019-01-04 2019-01-04 Image processing method and device, storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910008670.0A CN109741250B (en) 2019-01-04 2019-01-04 Image processing method and device, storage medium and electronic equipment

Publications (2)

Publication Number Publication Date
CN109741250A CN109741250A (en) 2019-05-10
CN109741250B true CN109741250B (en) 2023-06-16

Family

ID=66363519

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910008670.0A Active CN109741250B (en) 2019-01-04 2019-01-04 Image processing method and device, storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN109741250B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110347305A (en) * 2019-05-30 2019-10-18 华为技术有限公司 A kind of VR multi-display method and electronic equipment

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105869142A (en) * 2015-12-21 2016-08-17 乐视致新电子科技(天津)有限公司 Method and device for testing imaging distortion of virtual reality helmets
CN107154027B (en) * 2017-04-17 2020-07-07 深圳大学 Compensation method and device for restoration of distorted image
CN107451965B (en) * 2017-07-24 2019-10-18 深圳市智美达科技股份有限公司 Distort face image correcting method, device, computer equipment and storage medium
CN107945133B (en) * 2017-11-30 2022-08-05 北京小米移动软件有限公司 Image processing method and device
CN109120854B (en) * 2018-10-09 2020-10-23 北京旷视科技有限公司 Image processing method, image processing device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN109741250A (en) 2019-05-10

Similar Documents

Publication Publication Date Title
CN109816589B (en) Method and apparatus for generating cartoon style conversion model
US11455830B2 (en) Face recognition method and apparatus, electronic device, and storage medium
CN112989904B (en) Method for generating style image, method, device, equipment and medium for training model
CN108830235B (en) Method and apparatus for generating information
CN111476871B (en) Method and device for generating video
CN109829432B (en) Method and apparatus for generating information
CN108765340B (en) Blurred image processing method and device and terminal equipment
CN108416744B (en) Image processing method, device, equipment and computer readable storage medium
KR102552532B1 (en) Method and device for image rendering
CN112785507A (en) Image processing method and device, storage medium and terminal
CN114073071B (en) Video frame inserting method and device and computer readable storage medium
CN110059623B (en) Method and apparatus for generating information
US20230143452A1 (en) Method and apparatus for generating image, electronic device and storage medium
CN107315729A (en) For the data processing method of chart, medium, device and computing device
CN110288532B (en) Method, apparatus, device and computer readable storage medium for generating whole body image
CN107622241A (en) Display methods and device for mobile device
CN115272565A (en) Head three-dimensional model reconstruction method and electronic equipment
CN107798675A (en) The detection method and device of smear in display image
CN111199540A (en) Image quality evaluation method, image quality evaluation device, electronic device, and storage medium
CN109741250B (en) Image processing method and device, storage medium and electronic equipment
CN113379877A (en) Face video generation method and device, electronic equipment and storage medium
CN113902636A (en) Image deblurring method and device, computer readable medium and electronic equipment
CN111177167B (en) Augmented reality map updating method, device, system, storage and equipment
CN109816791B (en) Method and apparatus for generating information
CN111784726A (en) Image matting method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant