CN112070888B - Image generation method, device, equipment and computer readable medium - Google Patents
Image generation method, device, equipment and computer readable medium Download PDFInfo
- Publication number
- CN112070888B CN112070888B CN202010936339.8A CN202010936339A CN112070888B CN 112070888 B CN112070888 B CN 112070888B CN 202010936339 A CN202010936339 A CN 202010936339A CN 112070888 B CN112070888 B CN 112070888B
- Authority
- CN
- China
- Prior art keywords
- image
- network
- sample
- feature data
- feature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 239000000463 material Substances 0.000 claims abstract description 39
- 238000009877 rendering Methods 0.000 claims abstract description 24
- 239000000523 sample Substances 0.000 claims description 102
- 238000000605 extraction Methods 0.000 claims description 98
- 238000005286 illumination Methods 0.000 claims description 51
- 238000013075 data extraction Methods 0.000 claims description 29
- 238000012549 training Methods 0.000 claims description 24
- 238000012545 processing Methods 0.000 claims description 12
- 230000004044 response Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 11
- 230000000694 effects Effects 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000013527 convolutional neural network Methods 0.000 description 6
- 230000002457 bidirectional effect Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 238000005315 distribution function Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/506—Illumination models
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Image Analysis (AREA)
Abstract
Embodiments of the present disclosure disclose an image generation method, apparatus, electronic device, and computer-readable medium. One embodiment of the method comprises the following steps: encoding the acquired image to be processed displaying the target object to obtain hidden variables; extracting feature data contained in the hidden variables, wherein the feature data comprises material feature data; rendering is carried out based on the characteristic data, and a rendered image is obtained. According to the embodiment, the material characteristic data of the target object displayed in the given image can be obtained, and the three-dimensional shape of the target object displayed in the given image and the material used by the object can be displayed together, so that the rendered image has a more real effect.
Description
Technical Field
Embodiments of the present disclosure relate to the field of computer technology, and in particular, to an image generation method, apparatus, device, and computer readable medium.
Background
With the development of technology, AR (augmented reality ) greatly enriches people's lives. The related art can model a three-dimensional shape of an object displayed in an image, and it is difficult to model a block of a material used for the object.
Disclosure of Invention
The disclosure is in part intended to introduce concepts in a simplified form that are further described below in the detailed description. The disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Some embodiments of the present disclosure propose image generation methods, apparatuses, devices, and computer readable media to solve the technical problems mentioned in the background section above.
In a first aspect, some embodiments of the present disclosure provide an image generation method, the method comprising: encoding the acquired image to be processed displaying the target object to obtain hidden variables; extracting feature data contained in the hidden variables, wherein the feature data comprises material feature data; rendering is carried out based on the characteristic data, and a rendered image is obtained.
In a second aspect, some embodiments of the present disclosure provide an image generating apparatus, the apparatus comprising: the coding unit is configured to code the acquired image to be processed, which is displayed with the target object, so as to obtain hidden variables; an extracting unit configured to extract feature data included in the hidden variable, wherein the feature data includes texture feature data; and the rendering unit is configured to render based on the characteristic data to obtain a rendered image.
In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method as described in any of the implementations of the first aspect.
In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program when executed by a processor implements a method as described in any of the implementations of the first aspect.
One of the above embodiments of the present disclosure has the following advantageous effects: the material characteristic data of the target object displayed in the given image can be obtained, and the three-dimensional shape of the target object displayed in the given image and the material used by the object can be displayed together, so that the rendered image has a more real effect.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.
FIG. 1 is a schematic illustration of one application scenario of an image generation method of some embodiments of the present disclosure;
FIG. 2 is a flow chart of some embodiments of an image generation method according to the present disclosure;
FIG. 3 is a flow chart of further embodiments of an image generation method according to the present disclosure;
FIG. 4 is a schematic structural view of some embodiments of an image generation apparatus according to the present disclosure;
fig. 5 is a schematic structural diagram of an electronic device suitable for use in implementing some embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings. Embodiments of the present disclosure and features of embodiments may be combined with each other without conflict.
It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.
It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.
The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 shows a schematic diagram of one application scenario in which the image generation method of some embodiments of the present disclosure may be applied.
In the application scenario shown in fig. 1, first, the computing device 101 encodes an acquired image 102 to be processed, on which a target object is displayed, to obtain a hidden variable 103 of the image to be processed. Thereafter, the computing device 101 may extract the feature data 104 included in the hidden variable 103, where the feature data includes texture feature data. Finally, computing device 101 renders 105 using the above-described feature data, resulting in a rendered image 106.
The computing device 101 may be hardware or software. When the computing device is hardware, it may be implemented as a distributed cluster of multiple servers or electronic devices, or as a single server or single electronic device. When the computing device is embodied as software, it may be implemented as a plurality of software or software modules, for example, to provide distributed services, or as a single software or software module. The present invention is not particularly limited herein.
It should be understood that the number of computing devices 101 in fig. 1 is merely illustrative. There may be any number of computing devices 101 as desired for an implementation.
With further reference to fig. 2, a flow 200 of some embodiments of an image generation method is shown. The flow 200 of the image generation method comprises the steps of:
and step 201, encoding the acquired image to be processed displaying the target object to obtain hidden variables.
In some embodiments, the execution subject of the image generation method performs image encoding on the image to be processed through various image encoding algorithms (e.g., huffman encoding, predictive encoding), so as to obtain hidden variables. The image coding can be to transform and combine the image to be processed according to a certain rule. Image coding can reduce the associated redundancy of the image pixels to be processed, enabling as much information as possible to be represented with as little data as possible. The hidden variable is the data after the image to be processed is coded.
Alternatively, the execution subject of the above-described image generation method may input the image to be processed into an encoder structure of a convolutional neural network (for example, an encoder structure of SegNet). After the image to be processed is given, the encoder structure learns to obtain hidden variables of the image to be processed through a convolutional neural network.
Step 202, extracting feature data contained in the hidden variables, wherein the feature data comprises material feature data.
In some embodiments, the above feature data may include: depth feature data, illumination feature data, normal vector feature data, color feature data, and texture feature data. The above feature data can be divided into two types, one in the form of a feature map (e.g., depth feature data) and one in the form of a feature vector (e.g., illumination feature data).
The illumination characteristic data can be represented by spherical harmonic illumination or ambient illumination. Wherein ambient illumination is a simplified global illumination model using a constant color with a small number of values added to the final color of the illuminated point of the light on the object. In three-dimensional scenes, ambient light maps are typically used to represent illumination in various directions. The various directions of the ambient light map need to be sampled to determine the lighting conditions of a point. Spherical harmonic illumination is a simplification of illumination. Because the illumination is different in all directions for points in space, the spherical harmonic illumination completely restores the illumination of a point in space by recording the illumination in all directions on the surrounding sphere. The material characteristic data are parameters of a bidirectional reflection distribution function, and the parameters of the function can be obtained by a solution optimization method. The bidirectional reflection function is a generation algorithm of the surface color of the object after receiving illumination, and represents the surface texture information of the object.
In some embodiments, the execution subject of the image generating method may extract the feature data contained in the hidden variable through a convolutional neural network. Wherein the convolutional neural network may include, but is not limited to: FCN network (Fully Convolutional Networks, full convolutional network), resNet network (Deep Residual Networks, residual network), dense convolutional network (Densenet, dense convolutional neural network).
And 203, rendering based on the characteristic data to obtain a rendered image.
In some embodiments, the execution body may render using depth, illumination, normal vector, color, and texture feature data to obtain a rendered image representing texture information.
One of the above embodiments of the present disclosure has the following advantageous effects: firstly, the execution main body of the image generation method encodes the image to be processed to obtain hidden variables which represent as much information as possible with as little data as possible, thereby facilitating the extraction of the subsequent characteristic data. Then, the execution body extracts the characteristic data containing the material information and renders the characteristic data containing the material information, so that the three-dimensional shape of the target object displayed in the given image and the material used by the object can be displayed together, and the rendered image has a more real effect.
With further reference to FIG. 3, a flow 300 of further embodiments of an image generation method is shown. The flow 300 of the image generation method comprises the steps of:
step 301, performing mask processing on the obtained image to be processed, on which the target object is displayed, to obtain a target object image.
In some embodiments, the execution body of the image generating method performs mask processing on the image to be processed to obtain an image for highlighting the target object. The mask processing is to multiply the target object area mask with the image to be processed to obtain the target object image. Wherein the pixel values of the image in the target object area remain unchanged, and the pixel values of the image outside the target object area are all 0. Wherein the mask is a template of an image filter. For example, an image is pixel filtered by a matrix of the same size as the image, which is a mask, and then the target object is highlighted.
Step 301 may highlight the image of the target object, ensure that the extracted texture feature data only contains the texture feature data of the target object, and reduce the influence of the scene information on the texture feature data of the target object.
Step 302, inputting the target object image into a coding network in a pre-trained feature data extraction model to obtain the hidden variable, wherein the feature data extraction model comprises the coding network and at least one feature extraction sub-network.
In an alternative implementation of some embodiments, the pre-trained feature data extraction model may be obtained by training: obtaining a sample set, wherein a sample in the sample set comprises a sample image displaying a target object and sample characteristic data corresponding to the sample image, and the sample characteristic data comprises sample material characteristic data; selecting samples from the sample set, and performing the following training steps: inputting a sample image in the sample into an initial model to generate prediction characteristic data of the sample image; rendering the image based on the prediction characteristic data to obtain a predicted image; analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value; analyzing the predicted image and the sample image to determine an image loss value; taking the weighted results of the characteristic loss value and the image loss value as a sample total loss value according to preset characteristic weight and image weight, and comparing the sample total loss value with a target value; determining whether the initial model is trained according to the comparison result; in response to determining that the initial model training is complete, the initial model is determined to be the feature data extraction model.
In an alternative implementation of some embodiments, the pre-trained feature data extraction model may further include the steps of: and in response to determining that the initial model is not trained, adjusting relevant parameters in the initial model, and re-selecting samples from the sample set, using the adjusted initial model as the initial model, and continuing to execute the training step.
In some embodiments, the coding network model may include, but is not limited to: VGG network (Visual Geometry Group, deep convolutional neural network), res net network (Deep Residual Networks, residual network).
And 303, inputting the hidden variables into the at least one feature extraction sub-network respectively, and taking the result output by the at least one feature extraction sub-network as the feature data, wherein the feature data comprises material feature data.
In an alternative implementation of some embodiments, the at least one feature extraction sub-network includes: an illumination characteristic extraction sub-network, a color characteristic extraction sub-network, a material characteristic extraction sub-network, a normal vector characteristic extraction sub-network and a depth characteristic extraction sub-network. Thus, step 303 may proceed as follows:
And the first step, inputting the hidden variable into the illumination characteristic extraction sub-network to obtain illumination characteristic data.
The illumination characteristic data are 3-order spherical harmonic illumination. Wherein, spherical harmonic illumination is a simplification of illumination. Because the illumination of all directions is different for one point in space, when the spherical harmonic illumination is used for rendering, the illumination of all directions on the surrounding spherical surface needs to be recorded to determine the illumination condition of one point in space, so that the rendering has certain difficulty. In practice, the spherical harmonics can be used to represent illumination in all directions of the surrounding sphere. Wherein, the spherical harmonic function can represent a complex spherical function by using a simple spherical harmonic base and corresponding coefficients. Specifically, the spherical function is restored by multiplying the simple spherical harmonic basis function by the corresponding coefficient, so that the spherical harmonic illumination is approximately represented. As an example, the spherical harmonic illumination is approximately replaced with a 3-order spherical harmonic illumination. Thereby, the complexity in the rendering process is reduced.
And secondly, inputting the hidden variable into the color feature extraction sub-network to obtain color feature data.
And thirdly, inputting the hidden variable into the material characteristic extraction sub-network to obtain material characteristic data.
Wherein the material characteristic data are parameters of a bidirectional reflection distribution function. The bidirectional reflection function is a generation algorithm of the surface color of the object after receiving illumination, and represents the surface material information of the object.
The parameters of the bidirectional reflection distribution function are solved through the material characteristic extraction sub-network, so that the characteristic extraction efficiency is improved.
And step four, inputting the hidden variable into the normal vector feature extraction sub-network to obtain normal vector feature data.
And fifthly, inputting the hidden variable into the depth feature extraction sub-network to obtain depth feature data.
And step 304, rendering is carried out based on the characteristic data, and a rendered image is obtained.
In some embodiments, the execution body renders the extracted depth, texture, normal vector, illumination, and color to obtain a rendered image containing texture information.
As can be seen in fig. 3, the flow 300 of the image generation method in some embodiments corresponding to fig. 3 highlights the implementation of extracting depth, texture, color, normal vector, and illumination in the target object image using the pre-trained feature data extraction model, as compared to the description of some embodiments corresponding to fig. 2. The pre-trained feature data extraction model greatly reduces the complexity of extracting feature data of the target object image. Meanwhile, 3-order spherical harmonic illumination replaces spherical harmonic illumination, so that the image rendering process is easier to realize.
With further reference to fig. 4, as an implementation of the method shown in the above figures, the present disclosure provides some embodiments of an image generation apparatus, which correspond to those method embodiments shown in fig. 2, and which are particularly applicable in various electronic devices.
As shown in fig. 4, the image generating apparatus 400 of some embodiments includes: an encoding unit 401 configured to encode the acquired image to be processed, on which the target object is displayed, to obtain a hidden variable; an extracting unit 402 configured to extract feature data included in the hidden variable, wherein the feature data includes texture feature data; and a rendering unit 403 configured to render based on the above feature data, resulting in a rendered image.
In alternative implementations of some embodiments, the encoding unit of apparatus 400 may be further configured to: performing mask processing on the image to be processed to obtain a target object image; inputting the target object image into a coding network in a pre-trained characteristic data extraction model to obtain the hidden variable, wherein the characteristic data extraction model comprises the coding network and at least one characteristic extraction sub-network.
In alternative implementations of some embodiments, the extraction unit of apparatus 400 may be further configured to: and respectively inputting the hidden variables into the at least one feature extraction sub-network, and taking the result output by the at least one feature extraction sub-network as the feature data.
In an alternative implementation of some embodiments, the at least one feature extraction sub-network of apparatus 400 includes: an illumination feature extraction sub-network, a color feature extraction sub-network, a material feature extraction sub-network, a normal vector feature extraction sub-network and a depth feature extraction sub-network; and the extraction unit is further configured to: inputting the hidden variable into the illumination characteristic extraction sub-network to obtain illumination characteristic data; inputting the hidden variable into the color feature extraction sub-network to obtain color feature data; inputting the hidden variable into the material characteristic extraction sub-network to obtain material characteristic data; inputting the hidden variable into the normal vector feature extraction sub-network to obtain normal vector feature data; and inputting the hidden variable into the depth feature extraction sub-network to obtain depth feature data.
In an alternative implementation of some embodiments, the feature data extraction model of the apparatus 400 is trained by: an acquisition unit configured to acquire a sample set, wherein a sample in the sample set includes a sample image in which a target object is displayed and sample feature data corresponding to the sample image, and the sample feature data includes sample texture feature data; a training unit configured to select samples from the sample set, and to perform the training steps of: inputting a sample image in the sample into an initial model to generate prediction characteristic data of the sample image; rendering the image based on the prediction characteristic data to obtain a predicted image; analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value; analyzing the predicted image and the sample image to determine an image loss value; according to preset characteristic weight and image weight, taking the weighted results of the characteristic loss value and the image loss value as a sample total loss value, and comparing the sample total loss value with a target value; determining whether the initial model is trained according to the comparison result; in response to determining that the initial model training is complete, the initial model is determined to be the feature data extraction model.
In alternative implementations of some embodiments, the apparatus 400 may further include: and an adjusting unit configured to adjust relevant parameters in the initial model in response to determining that the initial model is not trained, and to reselect samples from the sample set, and to continue performing the training step using the adjusted initial model as the initial model.
It will be appreciated that the elements described in the apparatus 400 correspond to the various steps in the method described with reference to fig. 2. Thus, the operations, features and resulting benefits described above with respect to the method are equally applicable to the apparatus 400 and the units contained therein, and are not described in detail herein.
Referring now to fig. 5, a schematic diagram of an electronic device (e.g., server or terminal device of fig. 1) 500 suitable for use in implementing some embodiments of the present disclosure is shown. The electronic devices in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), car terminals (e.g., car navigation terminals), and the like, as well as stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 5 is merely an example and should not impose any limitations on the functionality and scope of use of embodiments of the present disclosure.
As shown in fig. 5, the electronic device 500 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 501, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage means 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data required for the operation of the electronic apparatus 500 are also stored. The processing device 501, the ROM 502, and the RAM503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
In general, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 507 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 508 including, for example, magnetic tape, hard disk, etc.; and communication means 509. The communication means 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. While fig. 5 shows an electronic device 500 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead. Each block shown in fig. 5 may represent one device or a plurality of devices as needed.
In particular, according to some embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via the communications device 509, or from the storage device 508, or from the ROM 502. The above-described functions defined in the methods of some embodiments of the present disclosure are performed when the computer program is executed by the processing device 501.
It should be noted that the computer readable medium according to some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, the computer-readable signal medium may comprise a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: encoding the acquired image to be processed displaying the target object to obtain hidden variables; extracting feature data contained in the hidden variables, wherein the feature data comprises material feature data; rendering is carried out based on the characteristic data, and a rendered image is obtained.
Computer program code for carrying out operations for some embodiments of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in some embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The described units may also be provided in a processor, for example, described as: a processor includes an encoding unit, an extracting unit, and a rendering unit. The names of these units do not limit the unit itself in some cases, and for example, the encoding unit may also be described as "a unit that encodes an acquired image to be processed, in which a target object is displayed, to obtain hidden variables".
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
According to one or more embodiments of the present disclosure, there is provided an image generation method including: encoding the acquired image to be processed displaying the target object to obtain hidden variables; extracting feature data contained in the hidden variables, wherein the feature data comprises material feature data; rendering is carried out based on the characteristic data, and a rendered image is obtained.
According to one or more embodiments of the present disclosure, the encoding the acquired image to be processed, in which the target object is displayed, to obtain the hidden variable includes: performing mask processing on the image to be processed to obtain a target object image; inputting the target object image into a coding network in a pre-trained characteristic data extraction model to obtain the hidden variable, wherein the characteristic data extraction model comprises the coding network and at least one characteristic extraction sub-network.
According to one or more embodiments of the present disclosure, the extracting feature data included in the hidden variable includes: and respectively inputting the hidden variables into the at least one feature extraction sub-network, and taking the result output by the at least one feature extraction sub-network as the feature data.
According to one or more embodiments of the present disclosure, the at least one feature extraction sub-network includes: an illumination feature extraction sub-network, a color feature extraction sub-network, a material feature extraction sub-network, a normal vector feature extraction sub-network and a depth feature extraction sub-network; and inputting the hidden variables into the at least one feature extraction sub-network, respectively, and using the result output by the at least one feature extraction sub-network as the feature data, wherein the method comprises the following steps: inputting the hidden variable into the illumination characteristic extraction sub-network to obtain illumination characteristic data; inputting the hidden variable into the color feature extraction sub-network to obtain color feature data; inputting the hidden variable into the material characteristic extraction sub-network to obtain material characteristic data; inputting the hidden variable into the normal vector feature extraction sub-network to obtain normal vector feature data; and inputting the hidden variable into the depth feature extraction sub-network to obtain depth feature data.
According to one or more embodiments of the present disclosure, the above-described feature data extraction model is trained by: obtaining a sample set, wherein a sample in the sample set comprises a sample image displayed with a target object and sample characteristic data corresponding to the sample image, and the sample characteristic data comprises sample material characteristic data; selecting samples from the sample set, and performing the following training steps: inputting a sample image in the sample into an initial model to generate prediction characteristic data of the sample image; rendering the image based on the prediction characteristic data to obtain a predicted image; analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value; analyzing the predicted image and the sample image to determine an image loss value; according to preset characteristic weight and image weight, taking the weighted results of the characteristic loss value and the image loss value as a sample total loss value, and comparing the sample total loss value with a target value; determining whether the initial model is trained according to the comparison result; in response to determining that the initial model training is complete, the initial model is determined to be the feature data extraction model.
According to one or more embodiments of the present disclosure, the above method further comprises: and in response to determining that the initial model is not trained, adjusting relevant parameters in the initial model, and re-selecting samples from the sample set, using the adjusted initial model as the initial model, and continuing to execute the training step.
According to one or more embodiments of the present disclosure, there is provided an image generating apparatus including: the coding unit is configured to code the acquired image to be processed, which is displayed with the target object, so as to obtain hidden variables; an extracting unit configured to extract feature data included in the hidden variable, wherein the feature data includes texture feature data; and the rendering unit is configured to render based on the characteristic data to obtain a rendered image.
According to one or more embodiments of the present disclosure, the above-described encoding unit is further configured to: performing mask processing on the image to be processed to obtain a target object image; inputting the target object image into a coding network in a pre-trained characteristic data extraction model to obtain the hidden variable, wherein the characteristic data extraction model comprises the coding network and at least one characteristic extraction sub-network.
According to one or more embodiments of the present disclosure, the above-described extraction unit is further configured to: and respectively inputting the hidden variables into the at least one feature extraction sub-network, and taking the result output by the at least one feature extraction sub-network as the feature data.
According to one or more embodiments of the present disclosure, the at least one feature extraction sub-network includes: an illumination feature extraction sub-network, a color feature extraction sub-network, a material feature extraction sub-network, a normal vector feature extraction sub-network and a depth feature extraction sub-network; and the extraction unit is further configured to: inputting the hidden variable into the illumination characteristic extraction sub-network to obtain illumination characteristic data; inputting the hidden variable into the color feature extraction sub-network to obtain color feature data; inputting the hidden variable into the material characteristic extraction sub-network to obtain material characteristic data; inputting the hidden variable into the normal vector feature extraction sub-network to obtain normal vector feature data; and inputting the hidden variable into the depth feature extraction sub-network to obtain depth feature data.
According to one or more embodiments of the present disclosure, the above-described feature data extraction model is trained by: an acquisition unit configured to acquire a sample set, wherein a sample in the sample set includes a sample image in which a target object is displayed and sample feature data corresponding to the sample image, and the sample feature data includes sample texture feature data; a training unit configured to select samples from the sample set, and to perform the training steps of: inputting a sample image in the sample into an initial model to generate prediction characteristic data of the sample image; rendering the image based on the prediction characteristic data to obtain a predicted image; analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value; analyzing the predicted image and the sample image to determine an image loss value; according to preset characteristic weight and image weight, taking the weighted results of the characteristic loss value and the image loss value as a sample total loss value, and comparing the sample total loss value with a target value; determining whether the initial model is trained according to the comparison result; in response to determining that the initial model training is complete, the initial model is determined to be the feature data extraction model.
According to one or more embodiments of the present disclosure, the above-described apparatus further includes: and an adjusting unit configured to adjust relevant parameters in the initial model in response to determining that the initial model is not trained, and to reselect samples from the sample set, and to continue performing the training step using the adjusted initial model as the initial model.
According to one or more embodiments of the present disclosure, there is provided an electronic device including: one or more processors; and a storage device having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to implement a method as described above.
According to one or more embodiments of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon a computer program, wherein the program when executed by a processor implements a method as any of the above.
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above technical features, but encompasses other technical features formed by any combination of the above technical features or their equivalents without departing from the spirit of the invention. Such as the above-described features, are mutually substituted with (but not limited to) the features having similar functions disclosed in the embodiments of the present disclosure.
Claims (12)
1. An image generation method applied to modeling of a three-dimensional shape of an object displayed in an image, comprising:
encoding the acquired image to be processed displaying the target object to obtain hidden variables;
extracting feature data contained in the hidden variables, wherein the feature data comprises material feature data;
rendering is carried out based on the characteristic data, and a rendered image is obtained;
the rendering image can display the three-dimensional shape of the target object and the used materials;
extracting the feature data by using a pre-trained feature data extraction model, wherein the feature data extraction model is obtained by training the following steps:
acquiring a sample set, wherein a sample in the sample set comprises a sample image displayed with a target object and sample characteristic data corresponding to the sample image, and the sample characteristic data comprises sample material characteristic data;
selecting a sample from the sample set and performing the training steps of:
inputting a sample image in the sample into an initial model to generate prediction characteristic data of the sample image;
rendering an image based on the prediction feature data to obtain a predicted image;
Analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value;
analyzing the predicted image and the sample image to determine an image loss value;
according to preset feature weights and image weights, taking the weighted results of the feature loss values and the image loss values as sample total loss values, and comparing the sample total loss values with target values;
determining whether the initial model is trained according to the comparison result;
determining the initial model as a feature data extraction model in response to determining that the initial model training is complete;
the feature data extraction model includes an encoding network and at least one feature extraction sub-network, the at least one feature extraction sub-network comprising: the system comprises an illumination characteristic extraction sub-network, a color characteristic extraction sub-network, a material characteristic extraction sub-network, a normal vector characteristic extraction sub-network and a depth characteristic extraction sub-network.
2. The method according to claim 1, wherein the encoding the acquired image to be processed, in which the target object is displayed, to obtain the hidden variable includes:
performing mask processing on the image to be processed to obtain a target object image;
Inputting the target object image into a coding network in a pre-trained characteristic data extraction model to obtain the hidden variable.
3. The method of claim 2, wherein the extracting feature data contained in the hidden variable comprises:
and respectively inputting the hidden variables into the at least one feature extraction sub-network, and taking the result output by the at least one feature extraction sub-network as the feature data.
4. A method according to claim 3, further comprising:
inputting the hidden variables into the at least one feature extraction sub-network respectively, and taking the result output by the at least one feature extraction sub-network as the feature data, wherein the method comprises the following steps:
inputting the hidden variable into the illumination characteristic extraction sub-network to obtain illumination characteristic data;
inputting the hidden variable into the color feature extraction sub-network to obtain color feature data;
inputting the hidden variable into the material characteristic extraction sub-network to obtain material characteristic data;
inputting the hidden variable into the normal vector feature extraction sub-network to obtain normal vector feature data;
and inputting the hidden variable into the depth feature extraction sub-network to obtain depth feature data.
5. The method of claim 1, wherein the method further comprises:
and in response to determining that the initial model is not trained, adjusting relevant parameters in the initial model, and re-selecting samples from the sample set, continuing to perform the training step by using the adjusted initial model as the initial model.
6. An image generating apparatus for modeling a three-dimensional shape of an object displayed in an image, comprising:
the coding unit is configured to code the acquired image to be processed, which is displayed with the target object, so as to obtain hidden variables;
an extraction unit configured to extract feature data contained in the hidden variable, wherein the feature data includes texture feature data;
a rendering unit configured to render based on the feature data, resulting in a rendered image;
the rendering image can display the three-dimensional shape of the target object and the used materials;
extracting the feature data by using a pre-trained feature data extraction model, wherein the feature data extraction model is obtained by training the following steps:
an acquisition unit configured to acquire a sample set, wherein a sample in the sample set includes a sample image displaying a target object and sample feature data corresponding to the sample image, and the sample feature data includes sample texture feature data;
A training unit configured to select samples from the set of samples, and to perform the training steps of: inputting a sample image in the sample into an initial model to generate prediction characteristic data of the sample image; rendering an image based on the prediction feature data to obtain a predicted image; analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value; analyzing the predicted image and the sample image to determine an image loss value; according to preset feature weights and image weights, taking the weighted results of the feature loss values and the image loss values as sample total loss values, and comparing the sample total loss values with target values; determining whether the initial model is trained according to the comparison result; determining the initial model as a feature data extraction model in response to determining that the initial model training is complete;
the feature data extraction model includes an encoding network and at least one feature extraction sub-network, the at least one feature extraction sub-network comprising: the system comprises an illumination characteristic extraction sub-network, a color characteristic extraction sub-network, a material characteristic extraction sub-network, a normal vector characteristic extraction sub-network and a depth characteristic extraction sub-network.
7. The apparatus of claim 6, wherein the encoding unit is further configured to:
performing mask processing on the image to be processed to obtain a target object image;
inputting the target object image into a coding network in a pre-trained characteristic data extraction model to obtain the hidden variable.
8. The apparatus of claim 7, wherein the extraction unit is further configured to:
and respectively inputting the hidden variables into the at least one feature extraction sub-network, and taking the result output by the at least one feature extraction sub-network as the feature data.
9. The device according to claim 8,
the extraction unit is further configured to:
inputting the hidden variable into the illumination characteristic extraction sub-network to obtain illumination characteristic data;
inputting the hidden variable into the color feature extraction sub-network to obtain color feature data;
inputting the hidden variable into the material characteristic extraction sub-network to obtain material characteristic data;
inputting the hidden variable into the normal vector feature extraction sub-network to obtain normal vector feature data;
and inputting the hidden variable into the depth feature extraction sub-network to obtain depth feature data.
10. The apparatus of claim 6, wherein the apparatus further comprises:
and an adjustment unit configured to adjust relevant parameters in the initial model in response to determining that the initial model is not trained, and to reselect samples from the sample set, and to continue performing the training step using the adjusted initial model as the initial model.
11. An electronic device, comprising:
one or more processors;
a storage means for storing one or more programs;
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-5.
12. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010936339.8A CN112070888B (en) | 2020-09-08 | 2020-09-08 | Image generation method, device, equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010936339.8A CN112070888B (en) | 2020-09-08 | 2020-09-08 | Image generation method, device, equipment and computer readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112070888A CN112070888A (en) | 2020-12-11 |
| CN112070888B true CN112070888B (en) | 2024-04-05 |
Family
ID=73664454
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010936339.8A Active CN112070888B (en) | 2020-09-08 | 2020-09-08 | Image generation method, device, equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112070888B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972112B (en) * | 2022-06-17 | 2024-05-14 | 如你所视(北京)科技有限公司 | Method, apparatus, device and medium for image inverse rendering |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107564049A (en) * | 2017-09-08 | 2018-01-09 | 北京达佳互联信息技术有限公司 | Faceform's method for reconstructing, device and storage medium, computer equipment |
| CN108875935A (en) * | 2018-06-11 | 2018-11-23 | 兰州理工大学 | Based on the natural image target materials visual signature mapping method for generating confrontation network |
| CN109636741A (en) * | 2018-11-14 | 2019-04-16 | 建湖云飞数据科技有限公司 | A kind of image denoising processing method |
| CN109753578A (en) * | 2019-01-25 | 2019-05-14 | 浙江理工大学 | A kind of image search method based on exposure mask selection convolution feature |
| WO2019227479A1 (en) * | 2018-06-01 | 2019-12-05 | 华为技术有限公司 | Method and apparatus for generating face rotation image |
-
2020
- 2020-09-08 CN CN202010936339.8A patent/CN112070888B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107564049A (en) * | 2017-09-08 | 2018-01-09 | 北京达佳互联信息技术有限公司 | Faceform's method for reconstructing, device and storage medium, computer equipment |
| WO2019227479A1 (en) * | 2018-06-01 | 2019-12-05 | 华为技术有限公司 | Method and apparatus for generating face rotation image |
| CN108875935A (en) * | 2018-06-11 | 2018-11-23 | 兰州理工大学 | Based on the natural image target materials visual signature mapping method for generating confrontation network |
| CN109636741A (en) * | 2018-11-14 | 2019-04-16 | 建湖云飞数据科技有限公司 | A kind of image denoising processing method |
| CN109753578A (en) * | 2019-01-25 | 2019-05-14 | 浙江理工大学 | A kind of image search method based on exposure mask selection convolution feature |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112070888A (en) | 2020-12-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113689372B (en) | Image processing method, apparatus, storage medium, and program product | |
| CN111915480B (en) | Method, apparatus, device and computer readable medium for generating feature extraction network | |
| CN112419179B (en) | Method, apparatus, device and computer readable medium for repairing image | |
| CN112330788A (en) | Image processing method, image processing device, readable medium and electronic equipment | |
| CN113449070A (en) | Multimodal data retrieval method, device, medium and electronic equipment | |
| CN113034648A (en) | Image processing method, device, equipment and storage medium | |
| CN114863214A (en) | Image generation model training method, image generation device, image generation medium, and image generation device | |
| CN112418249A (en) | Mask image generation method and device, electronic equipment and computer readable medium | |
| CN114693876B (en) | Digital person generation method, device, storage medium and electronic equipment | |
| CN114066722B (en) | Method and device for acquiring image and electronic equipment | |
| CN114399814A (en) | Deep learning-based obstruction removal and three-dimensional reconstruction method | |
| CN112070888B (en) | Image generation method, device, equipment and computer readable medium | |
| CN112241744B (en) | Image color migration method, device, equipment and computer readable medium | |
| CN112258622B (en) | Image processing method and device, readable medium and electronic equipment | |
| CN113034670A (en) | Training method and device for image reconstruction model and electronic equipment | |
| CN115546766B (en) | Lane line generation method, lane line generation device, electronic device, and computer-readable medium | |
| CN114418835B (en) | Image processing method, device, equipment and medium | |
| CN116030321A (en) | Method, device, electronic equipment and storage medium for generating image | |
| CN112418233B (en) | Image processing method and device, readable medium and electronic equipment | |
| CN114399590A (en) | Face occlusion removal and three-dimensional model generation method based on face analysis graph | |
| CN114723600A (en) | Method, device, equipment, storage medium and program product for generating cosmetic special effect | |
| CN113066166A (en) | Image processing method and device and electronic equipment | |
| CN116527993A (en) | Video processing method, apparatus, electronic device, storage medium and program product | |
| CN114419298A (en) | Virtual object generation method, device, equipment and storage medium | |
| CN112488947A (en) | Model training and image processing method, device, equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Tiktok vision (Beijing) Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: BEIJING BYTEDANCE NETWORK TECHNOLOGY Co.,Ltd. Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Douyin Vision Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: Tiktok vision (Beijing) Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |