WO2023181904A1 - Information processing device, information processing method, and recording medium - Google Patents
Information processing device, information processing method, and recording medium Download PDFInfo
- Publication number
- WO2023181904A1 WO2023181904A1 PCT/JP2023/008482 JP2023008482W WO2023181904A1 WO 2023181904 A1 WO2023181904 A1 WO 2023181904A1 JP 2023008482 W JP2023008482 W JP 2023008482W WO 2023181904 A1 WO2023181904 A1 WO 2023181904A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- rgb
- information
- data
- synthesis
- Prior art date
Links
- 230000010365 information processing Effects 0.000 title claims abstract description 35
- 238000003672 processing method Methods 0.000 title claims description 6
- 238000000034 method Methods 0.000 claims abstract description 139
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 89
- 238000003786 synthesis reaction Methods 0.000 claims abstract description 89
- 238000012545 processing Methods 0.000 claims abstract description 82
- 230000008569 process Effects 0.000 claims abstract description 57
- 230000008859 change Effects 0.000 claims abstract description 13
- 239000002131 composite material Substances 0.000 claims description 36
- 239000000203 mixture Substances 0.000 claims description 26
- 238000005286 illumination Methods 0.000 claims description 17
- 230000003287 optical effect Effects 0.000 claims description 11
- 230000005855 radiation Effects 0.000 claims description 10
- 238000005259 measurement Methods 0.000 claims description 6
- 238000009877 rendering Methods 0.000 claims description 5
- 230000003247 decreasing effect Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 39
- 230000006870 function Effects 0.000 description 10
- 238000001308 synthesis method Methods 0.000 description 10
- 230000000694 effects Effects 0.000 description 7
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000012937 correction Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 239000000470 constituent Substances 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 239000000654 additive Substances 0.000 description 1
- 230000000996 additive effect Effects 0.000 description 1
- 230000000740 bleeding effect Effects 0.000 description 1
- 238000005282 brightening Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 230000001151 other effect Effects 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
Definitions
- the present disclosure relates to an information processing device, an information processing method, and a recording medium.
- the synthesis quality of the ADR method and the QI method has advantages and disadvantages depending on the situation of the actual background, so in many cases, the user needs to manually select which method to use depending on the content of the background.
- the algorithms of these methods assume that the shadow area cast by CG on the real image is only a planar shape, so it is difficult to apply shadows to areas where many three-dimensional shapes intersect in a complex manner and cause occlusion. When applied, there is also the problem that it is strongly influenced by noise and is reflected as an artifact.
- the present disclosure proposes an information processing device, an information processing method, and a recording medium that can further improve the quality of synthesizing CG data into a three-dimensional space representing a real space.
- an information processing device inputs an RGB image, a depth image, and a spherical image and converts CG data into a three-dimensional space representing a real space corresponding to the RGB image.
- a compositing processing unit is provided that performs compositing processing for arranging.
- the synthesis processing unit generates a first rendered image using only the real-life information measured from the input, and a second rendered image using the real-life information and the CG data, and A first composite image by a first method based on the difference in shading between the rendered image and the second rendered image and a second composite image by a second method based on the ratio of the shading are generated, A shadow impact image representing a change in optical radiation energy between the case of using only the information and the case of using the live-action information and the CG data is generated, and the first composite image is generated using the shadow impact image. and the second composite image are linearly combined.
- FIG. 1 is a schematic explanatory diagram of an image synthesis method according to an embodiment of the present disclosure.
- FIG. 1 is a block diagram illustrating a configuration example of an image synthesis device according to an embodiment of the present disclosure.
- FIG. 2 is a block diagram showing an example of the configuration of a composition processing section.
- FIG. 3 is a diagram showing an example of an output image.
- FIG. 3 is a diagram showing an example of an RGB image. It is a figure showing an example of CG data. It is a figure showing an example of a depth image. It is a figure showing an example of a PY image.
- FIG. 3 is a diagram showing an example of a VI image. It is a figure showing an example of a mask image.
- FIG. 1 is a block diagram illustrating a configuration example of an image synthesis device according to an embodiment of the present disclosure.
- FIG. 2 is a block diagram showing an example of the configuration of a composition processing section.
- FIG. 3 is a diagram showing an example
- FIG. 3 is a diagram showing an example of an RPY image. It is a figure showing an example of an RVI image.
- FIG. 3 is a diagram showing an example of a shadow impact image. It is a figure showing an example of an ADR image.
- FIG. 2 is a diagram (part 1) showing an example of a QI image.
- FIG. 2 is a diagram (part 2) showing an example of a QI image.
- FIG. 2 is a diagram (part 1) showing an example of a light gap image.
- FIG. 2 is a diagram (part 2) showing an example of a light gap image.
- FIG. 7 is a diagram showing the effect of correction using a light gap image.
- 3 is a flowchart illustrating a processing procedure executed by the image compositing device.
- FIG. 2 is a hardware configuration diagram showing an example of a computer that implements the functions of an image synthesis device.
- the information processing device is the image composition device 10.
- the information processing method according to the embodiment of the present disclosure is an image synthesis method.
- This information is obtained by processing the RGB image L RGB captured in real space, the depth image I Dep , and the spherical image I Sep using general DCC (Digital Content Creation) tools such as Maya and Blender. be able to.
- DCC Digital Content Creation
- Both the ADR method and the QI method generate three images, a PY image L PY , a VI image L VI , and a mask image M, using the aforementioned geometry, reflectance, illumination map, and CG data to be synthesized.
- PY image L PY is an image rendered using only actual photographic information measured from a depth image.
- VI image L VI is an image rendered using both real-shot information and CG data to be synthesized.
- the mask image M is an image indicating an area where CG data to be synthesized exists.
- the RGB image L RGB , the PY image L PY , the VI image L VI and the mask image M are input to the synthesis algorithm in the ADR method and the QI method, and each of these algorithms has advantages and disadvantages in terms of their structure.
- an ADR image L ADR is calculated based on the following equation (1).
- a QI image L QI is calculated based on the following equation (3).
- the QI method can add shadows to the real background by multiplying the input RGB image L RGB by the ratio of shadows L VI /L PY produced by the compositing process of the CG data D Cg . can.
- the second term of equation (3) is actually operated by the following equation (4) in order to prevent the synthesis result from becoming unstable due to division by zero.
- ⁇ is a constant to prevent division by zero.
- L QI is limited to a maximum of 1.01 times L PY .
- the QI method has a problem in that there is an upper limit to the range in which a live photograph can be brightened by, for example, combining processing of emitting CG data DCg .
- equation (3) due to the nature of equation (3), if the input depth image IDep is of low quality and has a large error with respect to the true value, an operation extremely similar to division by zero will occur in the second term of equation (4). There is a high possibility that this will occur, and in that case, there is a problem that it will be reflected in the QI image LQI as an artifact.
- such a region is referred to as a light-gap region.
- the light gap region occurs when the distance measurement quality of the depth image IDep is low quality and the three-dimensional object shielding environment in real space cannot be accurately acquired.
- the following equation (7) automatically holds true unless a virtual object that emits light is included in the CG data D Cg .
- Equation (8) Equation (8) below holds true, and the desired output cannot be expected in the region where shading is to be added to the RGB image L RGB by the compositing process.
- Equation (9) Equation (9) below holds true, causing a phenomenon similar to that of the ADR method.
- the technical issues of both the ADR method and the QI method are summarized as follows.
- the ADR method there are restrictions on the expression of darkening the RGB image L RGB by the compositing process of the CG data D Cg .
- the ADR method tends to cause artifacts due to the low quality ranging quality of reflectance.
- artifacts in the light gap region become noticeable.
- the QI method there are restrictions on the expression of brightening the RGB image L RGB by the synthesis process of the CG data D Cg .
- the QI method tends to cause artifacts due to the low ranging quality of the depth image IDep .
- artifacts in the light gap region become noticeable.
- FIG. 1 is a schematic explanatory diagram of an image synthesis method according to an embodiment of the present disclosure.
- the image synthesis device 10 inputs the RGB image L RGB , the depth image I Dep , and the spherical image I Sep and converts the RGB image L RGB into an RGB image L RGB .
- a synthesis process is performed to arrange the CG data D Cg in a three-dimensional space representing the corresponding real space.
- the image compositing device 10 combines the PY image LPY , which is a rendered image using only the actual photographic information measured from the input, and the aforementioned photographic information and CG data DCG .
- a VI image LVI which is the rendered image used, is generated (steps S1-1, S1-2).
- PY image L PY corresponds to an example of a "first rendered image.”
- VI image L VI corresponds to an example of a "second rendered image.”
- the image synthesis device 10 also generates an ADR image L ADR, which is a synthesized image based on the ADR method based on the difference in shading between the PY image L PY and a VI image L VI , and a QI image, which is a synthesized image based on the QI method based on the ratio of the shadings.
- Images L and QI are generated (steps S2-1 and S2-2).
- ADR image L ADR corresponds to an example of a "first composite image.”
- QI image L QI corresponds to an example of a "second composite image.”
- the image synthesis device 10 also generates a shadow impact image SI representing a change in light radiation energy between the case where only the above-mentioned real-time photograph information is used and the case where the above-mentioned live-action information and CG data DCg are used. Step S3).
- the image synthesis device 10 also performs a linear combination of the ADR image LADR and the QI image LQI using the shadow impact image S I to generate an LC image L LC (step S4).
- the RGB image L RGB , the depth image I Dep , and the spherical image I Sep are input, and CG is created in a three-dimensional space representing the real space corresponding to the RGB image L RGB .
- the ADR image L ADR generated by the ADR method and the QI image L QI generated by the QI method are linearly combined using the shadow impact image S I generated by the new algorithm of the present disclosure, and The advantages of the and QI methods are adaptively mixed for each image pixel.
- the image synthesis method according to the embodiment of the present disclosure it is possible to further improve the synthesis quality of CG data DCg with respect to a three-dimensional space representing a real space.
- a configuration example of the image synthesis apparatus 10 to which the image synthesis method according to the embodiment of the present disclosure is applied will be described in more detail.
- FIG. 2 is a block diagram illustrating a configuration example of the image synthesis device 10 according to the embodiment of the present disclosure. Note that FIG. 2 and FIG. 3 shown later show only the constituent elements necessary for explaining the features of the embodiment of the present disclosure, and descriptions of general constituent elements are omitted.
- each component illustrated in FIGS. 2 and 3 is functionally conceptual, and does not necessarily need to be physically configured as illustrated.
- the specific form of distributing/integrating each block is not limited to what is shown in the diagram, and all or part of the blocks can be functionally or physically distributed/integrated in arbitrary units depending on various loads and usage conditions. It is possible to configure them in an integrated manner.
- the image synthesis device 10 includes a storage section 11 and a control section 12.
- the storage unit 11 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a ROM (Read Only Memory), or a flash memory, or a storage device such as a hard disk or an optical disk.
- the storage unit 11 stores geometry information 11a, reflectance information 11b, illumination map information 11c, and DCC tool program 11d.
- the geometry information 11a is information corresponding to the above-mentioned geometry.
- the reflectance information 11b is information corresponding to the above-mentioned reflectance.
- the illumination map information 11c is information corresponding to the aforementioned illumination map.
- the DCC tool program 11d is program data of the DCC tool.
- the control unit 12 is a controller, and includes, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. This is realized by executing the information processing program according to the embodiment using the RAM as a work area. Further, the control unit 12 can be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
- ASIC Application Specific Integrated Circuit
- FPGA Field Programmable Gate Array
- the control unit 12 includes an acquisition unit 12a, a conversion unit 12b, a composition processing unit 12c, and an output unit 12d, and realizes or executes information processing functions and operations described below.
- the acquisition unit 12a acquires the RGB image L RGB , the depth image I Dep , the spherical image I Sep , and the CG data D Cg .
- the converter 12b converts the RGB image LRGB , the depth image IDep , and the spherical image ISep into geometry information 11a, reflectance information 11b, and illumination map information 11c that can be read by the DCC tool.
- the synthesis processing unit 12c receives the converted geometry information 11a, reflectance information 11b, illumination map information 11c, CG data D , and RGB image L as input, and generates a three-dimensional space representing a real space corresponding to the RGB image L. A compositing process is performed to arrange CG data D Cg .
- FIG. 3 is a block diagram showing a configuration example of the composition processing section 12c.
- the composition processing section 12c includes a first generation section 12ca, a second generation section 12cb, a third generation section 12cc, a fourth generation section 12cd, a fifth generation section 12ce, and an output image. It has a generation unit 12cf.
- the first generation unit 12ca reads the converted geometry information 11a, reflectance information 11b, illumination map information 11c, and CG data D Cg using the DCC tool, and generates a PY image L PY , a VI image L VI , and a mask image within the DCC tool. M, generate an RPY image R PY and an RVI image R VI .
- the RPY image R PY and the RVI image R VI are images that are input when generating the shadow impact image S I.
- the second generation unit 12cb generates an ADR image L ADR and a QI image L QI using the generated PY image L PY , VI image L VI , mask image M, and RGB image L RGB .
- the third generation unit 12cc receives the RPY image RPY and the RVI image RVI as input and generates a shadow impact image SI in parallel with the second generation unit 12cb.
- the fourth generation unit 12cd receives the ADR image L ADR , the QI image L QI , and the shadow impact image S I and generates the LC image L LC .
- the fifth generation unit 12ce receives the RGB image L RGB , the PY image L PY , and the shadow impact image S I in parallel with the fourth generation unit 12 cd and generates a light gap image w g .
- the output image generation unit 12cf receives the VI image L VI , LC image L LC , and light gap image w g as input, generates an output image L end , and outputs it.
- the output unit 12d outputs the output image L end generated by the composition processing unit 12c to an external device such as a display device.
- FIG. 4 is a diagram showing an example of the output image L end .
- FIG. 5 is a diagram showing an example of an RGB image L RGB .
- FIG. 6 is a diagram showing an example of CG data DCg .
- FIG. 7 is a diagram showing an example of the depth image IDep .
- the output image L end shown in FIG. 4 is finally output from the composition processing unit 12c after undergoing composition processing.
- FIG. 4 shows an example in which the mannequin in the rear is a mannequin that exists in real space, and the person in the foreground is CG. Note that, below, the figures are appropriately simplified in order to make the explanation easier to understand. Therefore, the examples shown below do not limit the synthesis quality of the synthesis processing according to the embodiments of the present disclosure.
- the M1 portion in the diagram of FIG. 4 will be described later.
- the RGB image L RGB obtained by the obtaining unit 12a is as shown in FIG. 5.
- the M2 portion in the diagram of FIG. 5 will be described later.
- the CG data D Cg similarly obtained by the obtaining unit 12a is as shown in FIG.
- the depth image IDep acquired by the acquisition unit 12a is as shown in FIG. Note that the spherical image I Sep acquired by the acquisition unit 12a is omitted here.
- the synthesis processing unit 12c generates a PY image L PY , a VI image L VI , a mask image M, and an RPY image R PY based on these RGB image L RGB , CG data D Cg , depth image I Dep , and spherical image I Sep , generates an RVI image RVI .
- FIG. 8 is a diagram showing an example of the PY image LPY .
- FIG. 9 is a diagram showing an example of the VI image LVI.
- FIG. 10 is a diagram showing an example of the mask image M.
- FIG. 11 is a diagram showing an example of the RPY image RPY .
- FIG. 12 is a diagram showing an example of the RVI image RVI .
- FIG. 13 is a diagram showing an example of a shadow impact image SI .
- the PY image L PY becomes as shown in FIG. 8.
- the VI image LVI becomes as shown in FIG.
- the mask image M becomes as shown in FIG.
- a new algorithm generates a shadow impact image SI , which can be called a weighting function image that detects areas in which the ADR method and the QI method are respectively good.
- the above equation (11) is the intensity of illumination incident on the object plane from the direction vector ⁇ i to the three-dimensional position x in real space corresponding to the pixel in the image, and is the intensity of the illumination that enters the object plane from the direction vector ⁇ i , and Calculated from I Sep.
- Equation ( 12) shows that when looking at the direction vector ⁇ i from the three-dimensional position Visibility) function.
- the approximation R has no color component and is gray scaled.
- the RPY image R PY obtained as a result of calculating irradiance using only the distance-measured real-photo information, and the result of calculating irradiance using both the distance-measured real-photo information and the CG data D Cg to be synthesized.
- the resulting RVI images RVI are respectively generated.
- the RPY image RPY becomes as shown in FIG. 11.
- the RVI image RVI becomes as shown in FIG.
- composition processing unit 12c generates a shadow impact image S1 using the RVI image RVI and the RPY image RPY according to the following equation (13).
- dilate() is a general dilation process applied to the entire image in order to reduce noise that cannot be canceled out due to the light gap.
- the synthesis processing unit 12c performs a linear combination of the ADR image L ADR and the QI image L QI using the shadow impact image S I according to the following equation (14) to generate an LC image L LC .
- FIG. 14 is a diagram showing an example of the ADR image L ADR .
- FIG. 15 is a diagram (part 1) showing an example of QI image L QI .
- FIG. 16 is a diagram (part 2) showing an example of QI image L QI .
- FIG. 14 is an ADR image LADR in which the person is a CG image.
- the ADR image L ADR color disharmony or the like may appear in the vicinity of the step of the pillar, as shown, for example, in the M41 part of the M4 part in the real space.
- color blurring or the like may appear at the boundary of the object, as shown in the M42 section of the M4 section, for example.
- FIG. 15 is a QI image LQI when the person is a CG image.
- inappropriate noise may appear in the QI image LQI in a portion where geometry noise is large (lower end of the image), as shown, for example, in the M51 portion of the M5 portion in real space.
- FIG. 16 is a QI image LQI in which the person is a real object and a CG of a character holding a light-emitting sword is synthesized next to the person.
- a situation may occur in which an area that should be brightened does not become bright due to reflection from the CG light-emitting sword, as shown in the M61 part of the M6 part in the real space, for example.
- the synthesis processing unit 12c combines the ADR image L ADR and the QI image using the shadow impact image S I so that the defects of the ADR image L ADR and the QI image L QI are eliminated.
- a linear combination of LQI is performed to generate an LC image LLC . Therefore, the LC image LLC is generated as an image in which such defects are eliminated.
- the compositing processing unit 12c uses the shadow impact image SI to remove artifacts in the light gap region.
- a light gap image w g indicating a light gap region is calculated using the following equation (15).
- saturate() is a function that limits the input range to 0 to 1
- opening() is a morphological operation for denoising.
- the synthesis processing unit 12c preferentially allocates the VI image LVI to the light gap area, that is, the area where wg is high, by linear combination, and outputs the final synthesis result from which artifacts have been removed.
- Image L end is obtained as shown in equation (16) below.
- FIG. 17 is a diagram (part 1) showing an example of the light gap image wg .
- FIG. 18 is a diagram (part 2) showing an example of the light gap image wg .
- FIG. 17 corresponds to the case where the synthesis process of the output image L end shown in FIG. 4 is performed. In such a case, the light gap image wg will be as shown in FIG. 17.
- the M7 section shown in FIG. 17 corresponds to the M2 section of the RGB image LRGB shown in FIG. Furthermore, the M7 section corresponds to the M3 section of the PY image LPY shown in FIG.
- FIG. 18 shows these M2 portion, M3 portion, and M7 portion arranged side by side for comparison. As shown in FIG. 18, it can be seen that the rendering result of the PY image LPY is significantly different from the RGB image L RGB in the space between the right arm and torso of the mannequin in the image.
- the composition processing unit 12c detects this portion as a light gap image wg , as shown in FIG.
- FIG. 19 is a diagram showing the effect of correction using the light gap image wg .
- FIG. 19 shows the M1 portion of the output image L end shown in FIG. 4, the M1 ADR portion of the ADR image L ADR corresponding to the M1 portion, and the M1 QI portion of the QI image L QI corresponding to the M1 portion. They are arranged side by side for comparison.
- the shadow impact image S I has a low pixel value in an area where the light radiant energy has increased due to the synthesis process of the CG data D Cg , and has a high pixel value in an area where the light radiant energy has decreased.
- RVI image RVI and the RPY image RPY which are irradiance images.
- general CG methods also consider the direction of normal lines in space, but normal lines are susceptible to noise when the depth image IDep is of low quality. Therefore, by not using normal information to generate the RVI image RVI and the RPY image RPY , it is possible to generate a filter that is not affected by noise originating from the low-quality depth image IDep .
- the light gap region is defined using the above equation (15), and calculation and detection are performed.
- the range measurement error expressed by the difference between the input RGB image L RGB and the PY image L PY is raised to the power of ⁇ p to identify areas where the range measurement error is significantly large. can be detected.
- the light gap image w g is not a binary image, and the light gap changes smoothly from a low region to a high region, so the light gap image w g does not have sharp edges.
- the synthesis process can be performed so that the joint of the blended images is not noticeable.
- FIG. 20 is a flowchart showing the processing procedure executed by the image synthesis device 10.
- the acquisition unit 12a first acquires an RGB image L RGB , a depth image I Dep , a spherical image I Sep , and CG data D Cg (step S101). Then, the conversion unit 12b converts the RGB image L RGB , the depth image I Dep , and the spherical image I Sep so that they can be read into the DCC tool (step S102). Then, the composition processing unit 12c reads the converted data and the CG data DCg using the DCC tool (step S103).
- the synthesis processing unit 12c generates a PY image L PY , a VI image L VI , a mask image M, an RPY image R PY , and an RVI image R VI within the DCC tool (step S104).
- the synthesis processing unit 12c generates the ADR image L ADR and the QI image L QI using the PY image L PY , the VI image L VI , the mask image M, and the RGB image L RGB (step S105).
- the synthesis processing unit 12c generates a shadow impact image S I using the RPY image R PY and the RVI image R VI (step S106).
- the synthesis processing unit 12c When steps S105 and S106 are completed, the synthesis processing unit 12c generates an LC image LLC using the ADR image L ADR , the QI image L QI , and the shadow impact image S I (step S107).
- the synthesis processing unit 12c generates a light gap image w g using the shadow impact image S I , the RGB image L RGB , and the PY image L PY (step S108).
- the synthesis processing unit 12c When steps S107 and S108 are completed, the synthesis processing unit 12c generates an output image L end using the LC image L LC and the light gap image w g (step S109). Then, the output unit 12d outputs the output image L end (step S110), and the process ends.
- FIG. 21 is a flowchart showing the processing procedure when outputting a still image.
- a user photographs an RGB image L RGB with an RGB camera (step S201). Further, the user photographs a depth image I Dep with a depth camera (step S202). Further, the user photographs a spherical image I Sep with a spherical camera (step S203). Further, the user creates CG data DCg using the DCC tool (step S204).
- the image synthesis device 10 inputs each data acquired in steps S201 to S204 and executes the image synthesis process shown in FIG. 20 (step S205).
- the display device displays the output image L end output from the image synthesis device 10 as a still image (step S206), and the process ends.
- FIG. 22 is a flowchart showing the processing procedure when outputting a moving image. Note that steps S301 to S304 in FIG. 22 are the same as steps S201 to S204 shown in FIG. 21, so the description thereof will be omitted here.
- the user updates the CG data DCg according to the frame number (step S305).
- the image synthesis device 10 inputs each data acquired in steps S301 to S305, and executes the image synthesis process shown in FIG. 20 (step S306).
- the image synthesis device 10 determines whether the output image L end outputted by the image synthesis process has reached a predetermined number of frames (step S307). Here, if the predetermined number of frames has not been reached (step S307, No), the frame number is updated (step S308), and the processing from step S305 is repeated.
- step S307 if the predetermined number of frames has been reached (step S307, Yes), for example, the display device combines the output images L end of each frame in time series and displays it as a moving image (step S309). Then, the process ends.
- each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings.
- the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.
- FIG. 23 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the image synthesis apparatus 10.
- Computer 1000 has CPU 1100, RAM 1200, ROM 1300, HDD (Hard Disk Drive) 1400, communication interface 1500, and input/output interface 1600. Each part of computer 1000 is connected by bus 1050.
- the CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each part. For example, the CPU 1100 loads programs stored in the ROM 1300 or HDD 1400 into the RAM 1200, and executes processes corresponding to various programs.
- the ROM 1300 stores boot programs such as BIOS (Basic Input Output System) that are executed by the CPU 1100 when the computer 1000 is started, programs that depend on the hardware of the computer 1000, and the like.
- BIOS Basic Input Output System
- the HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by the programs.
- HDD 1400 is a recording medium that records an information processing program according to an embodiment of the present disclosure, which is an example of program data 1450.
- Communication interface 1500 is an interface for connecting computer 1000 to external network 1550 (for example, network N).
- CPU 1100 receives data from other devices or transmits data generated by CPU 1100 to other devices via communication interface 1500.
- the input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000.
- the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input/output interface 1600. Further, the CPU 1100 transmits data to an output device such as a display, speaker, or printer via an input/output interface 1600.
- the input/output interface 1600 may function as a media interface that reads programs and the like recorded on a predetermined recording medium.
- Media includes, for example, optical recording media such as DVD (Digital Versatile Disc) and PD (Phase change rewritable disk), magneto-optical recording media such as MO (Magneto-Optical disk), tape media, magnetic recording media, semiconductor memory, etc. It is.
- the CPU 1100 of the computer 1000 realizes the functions of the control unit 12 by executing a program loaded onto the RAM 1200.
- the HDD 1400 stores an information processing program according to the present disclosure and data in the storage unit 11. Note that although the CPU 1100 reads and executes the program data 1450 from the HDD 1400, as another example, these programs may be obtained from another device via the external network 1550.
- the image synthesis device 10 (corresponding to an example of an "information processing device") generates an RGB image L RGB , a depth image I Dep , and a spherical image I Sep.
- a composition processing unit 12c is provided which performs composition processing of arranging CG data DCg in a three-dimensional space representing a real space corresponding to the RGB image L RGB as an input.
- the synthesis processing unit 12c generates a PY image L PY (corresponding to an example of a "first rendered image") using only the real shot information measured from the above input, and a VI using the above real shot information and CG data D Cg .
- An image LVI (corresponding to an example of a "second rendered image”) is generated.
- the synthesis processing unit 12c generates an ADR image L ADR (“first composite image”) based on the ADR method (corresponding to an example of the “first method”) based on the difference in shading between the PY image L PY and the VI image L VI . ) and a QI image L QI (corresponding to an example of a "second composite image”) by the QI method (corresponding to an example of a "second method”) based on the shading ratio.
- first composite image (“first composite image”) based on the ADR method (corresponding to an example of the “first method”) based on the difference in shading between the PY image L PY and the VI image L VI . ) and a QI image L QI (corresponding to an example of a "second composite image”) by the QI method (corresponding to an example of a "second method”) based on the shading ratio.
- composition processing unit 12c generates a shadow impact image S I representing a change in optical radiation energy between the case where only the above-mentioned live-action information is used and the case where the above-mentioned live-action information and CG data DCg are used, A linear combination of the ADR image L ADR and the QI image L QI is performed using the shadow impact image S I.
- the present technology can also have the following configuration.
- a composition processing unit that receives an RGB image, a depth image, and a spherical image as input and performs a composition process of arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
- the synthesis processing section is Generating a first rendered image using only the live-action information measured from the input, and a second rendering image using the live-action information and the CG data, Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings.
- the synthesis processing section is The shadow impact image is generated so that areas where light radiant energy has increased through the synthesis process have low pixel values, and areas where light radiant energy has decreased have high pixel values, and the areas with low pixel values have low pixel values. assigning the first composite image and the second composite image to the high pixel value area, respectively;
- the information processing device according to (1) above.
- the synthesis processing section is The shadow impact image is created such that the low pixel value area corresponds to the area of the RGB image that becomes brighter due to the compositing process, and the high pixel value area corresponds to an area of the RGB image that becomes dark due to the compositing process. generate, The information processing device according to (2) above.
- the synthesis processing section at least includes: generating the shadow impact image based on information regarding illumination calculated from the distance-measured spherical image and information regarding visibility of the CG data; The information processing device according to (1), (2) or (3) above.
- the synthesis processing section is generating the shadow impact image based only on the illumination information and the visibility information; The information processing device according to (4) above.
- the synthesis processing section is using the shadow impact image to remove artifacts occurring in the compositing process; The information processing device according to any one of (1) to (5) above.
- the synthesis processing section is By generating a light gap image indicating a light gap region in which the first rendered image is much darker than the RGB image, and preferentially allocating the second rendered image to the light gap region by linear combination. remove artifacts, The information processing device according to (6) above.
- the synthesis processing section is An area where the distance measurement error expressed by the difference between the first rendered image and the RGB image is extremely large is detected as the light gap area, and the area where the artifacts are removed is determined by multiplying the area by the shadow impact image. limit, The information processing device according to (7) above.
- (9) performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image; including;
- Performing the above-mentioned compositing process includes: Generating a first rendered image using only the live-action information measured from the input, and a second rendered image using the live-action information and the CG data; Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings.
- Information processing methods further including: (10) performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image; make the computer run
- Performing the above-mentioned compositing process includes: generating a first rendered image using only live-action information measured from the input, and a second rendered image using the live-action information and the CG data; Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings.
- a computer-readable recording medium having recorded thereon an information processing program that causes the computer to further execute the following.
- Image synthesis device 11 Storage unit 11a Geometry information 11b Reflectance information 11c Illumination map information 11d DCC tool program 12 Control unit 12a Acquisition unit 12b Conversion unit 12c Synthesis processing unit 12ca First generation unit 12cb Second generation unit 12cc Third generation unit 12cd Fourth generation section 12ce Fifth generation section 12cf Output image generation section 12d Output section
Landscapes
- Engineering & Computer Science (AREA)
- Computer Graphics (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
An image synthesis device (10), which corresponds to an example of an information processing device, is provided with a synthesis processing unit (12c) that uses an RGB image (LRGB), a depth image (IDep), and an omnidirectional image (ISep) as inputs to perform a synthesis process for disposing CG data (DCg) in a three-dimensional space representing a real space corresponding to the RGB image (LRGB). The synthesis processing unit (12c): generates a PY image (LPY) using only captured image information measured from the inputs, and a VI image (LVI) using the captured image information and the CG data (DCg); generates an ADR image (LADR) using an ADR method based on the difference between the shades of the PY image (LPY) and the VI image (LVI), and a QI image (LQI) using a QI method based on the ratio between the shades; generates a shadow impact image (SI) representing the change in light radiant energy between when only the captured image information is used and when the captured image information and the CG data (DCg) are used; and linearly combines the ADR image (L)ADR) and the QI image (LQI) using the shadow impact image (SI).
Description
本開示は、情報処理装置、情報処理方法および記録媒体に関する。
The present disclosure relates to an information processing device, an information processing method, and a recording medium.
従来、VFX(visual effects)をはじめとする、CG(computer graphics)または合成処理によって実写映像を加工・編集する技術は、映画やテレビドラマなどの映像分野において、現実にはない画面効果を実現する作品の普及に貢献している。
Traditionally, technology that processes and edits live-action footage using CG (computer graphics) or compositing processing, including VFX (visual effects), has been used to create screen effects that do not exist in reality in the video field such as movies and TV dramas. Contributing to the spread of the work.
このようなVFXの現場では、合成するCGの仮想オブジェクトが実写映像上であたかも存在すると視聴者に感じさせるために、仮想オブジェクトの見た目とその合成処理によって生じる陰影の変化までも忠実に再現するような合成処理が求められる。
At such VFX sites, in order to make the viewer feel that the CG virtual object to be synthesized exists in the live-action video, it is necessary to faithfully reproduce the appearance of the virtual object and even the changes in shadows caused by the compositing process. A unique synthesis process is required.
従来、このような陰影の変化の計算を高品位に行ううえで、ADR(Additive differential rendering)法や、QI(Quotient image)法といった技術が知られている(例えば、非特許文献1,2参照)。
Conventionally, techniques such as the ADR (Additive Differential Rendering) method and the QI (Quotient Image) method have been known to perform high-quality calculations of such shadow changes (for example, see Non-Patent Documents 1 and 2). ).
しかしながら、上述した従来技術には、実空間を表す3次元空間に対するCGデータの合成品質をより向上させるうえで、さらなる改善の余地がある。
However, the above-mentioned conventional technology has room for further improvement in improving the quality of synthesizing CG data with respect to a three-dimensional space representing a real space.
例えば、ADR法やQI法は、実写背景の状況によって合成品質が一長一短であるため、多くの場合、背景の内容に応じてどちらを採用するかユーザが手動で選択する必要がある。加えて、これらの手法のアルゴリズムは、CGが実写に対して落とす陰影の領域は平面形状のみであることを想定しているため、多くの3次元形状が複雑に交差して遮蔽が生じる領域に適用すると、ノイズの影響を強く受け、アーチファクトとして反映されるという問題点もある。
For example, the synthesis quality of the ADR method and the QI method has advantages and disadvantages depending on the situation of the actual background, so in many cases, the user needs to manually select which method to use depending on the content of the background. In addition, the algorithms of these methods assume that the shadow area cast by CG on the real image is only a planar shape, so it is difficult to apply shadows to areas where many three-dimensional shapes intersect in a complex manner and cause occlusion. When applied, there is also the problem that it is strongly influenced by noise and is reflected as an artifact.
そこで、本開示では、実空間を表す3次元空間に対するCGデータの合成品質をより向上させることができる情報処理装置、情報処理方法および記録媒体を提案する。
Therefore, the present disclosure proposes an information processing device, an information processing method, and a recording medium that can further improve the quality of synthesizing CG data into a three-dimensional space representing a real space.
上記の課題を解決するために、本開示に係る一形態の情報処理装置は、RGB画像、デプス画像および全天球画像を入力として前記RGB画像に対応する実空間を表す3次元空間にCGデータを配置する合成処理を行う合成処理部を備える。前記合成処理部は、前記入力から測距された実写情報のみを用いた第1のレンダリング画像と、前記実写情報および前記CGデータを用いた第2のレンダリング画像とを生成し、前記第1のレンダリング画像および前記第2のレンダリング画像の陰影の差に基づく第1の手法による第1の合成画像と、前記陰影の比率に基づく第2の手法による第2の合成画像とを生成し、前記実写情報のみを用いた場合と、前記実写情報および前記CGデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成し、前記シャドウインパクト画像を用いて前記第1の合成画像と前記第2の合成画像との線形結合を行う。
In order to solve the above problems, an information processing device according to an embodiment of the present disclosure inputs an RGB image, a depth image, and a spherical image and converts CG data into a three-dimensional space representing a real space corresponding to the RGB image. A compositing processing unit is provided that performs compositing processing for arranging. The synthesis processing unit generates a first rendered image using only the real-life information measured from the input, and a second rendered image using the real-life information and the CG data, and A first composite image by a first method based on the difference in shading between the rendered image and the second rendered image and a second composite image by a second method based on the ratio of the shading are generated, A shadow impact image representing a change in optical radiation energy between the case of using only the information and the case of using the live-action information and the CG data is generated, and the first composite image is generated using the shadow impact image. and the second composite image are linearly combined.
以下に、本開示の実施形態について図面に基づいて詳細に説明する。なお、以下の各実施形態において、同一の部位には同一の符号を付することにより重複する説明を省略する。
Below, embodiments of the present disclosure will be described in detail based on the drawings. In addition, in each of the following embodiments, the same portions are given the same reference numerals and redundant explanations will be omitted.
また、以下では、本開示の実施形態に係る情報処理装置が、画像合成装置10であるものとする。また、以下では、本開示の実施形態に係る情報処理方法が、画像合成方法であるものとする。
Furthermore, in the following description, it is assumed that the information processing device according to the embodiment of the present disclosure is the image composition device 10. Further, in the following, it is assumed that the information processing method according to the embodiment of the present disclosure is an image synthesis method.
また、以下に示す項目順序に従って本開示を説明する。
1.概要
1-1.ADR法とQI法の技術的課題
1-2.本開示の実施形態に係る画像合成方法の概要
2.画像合成装置の構成
3.合成処理部が実行する合成処理の詳細
4.変形例
5.ハードウェア構成
6.むすび Further, the present disclosure will be described according to the order of items shown below.
1. Overview 1-1. Technical issues of ADR method and QI method 1-2. Outline of image synthesis method according to embodiment of the present disclosure 2. Configuration of image synthesis device 3. Details of the compositing process executed by the compositing processing unit 4. Modification example 5. Hardware configuration 6. Conclusion
1.概要
1-1.ADR法とQI法の技術的課題
1-2.本開示の実施形態に係る画像合成方法の概要
2.画像合成装置の構成
3.合成処理部が実行する合成処理の詳細
4.変形例
5.ハードウェア構成
6.むすび Further, the present disclosure will be described according to the order of items shown below.
1. Overview 1-1. Technical issues of ADR method and QI method 1-2. Outline of image synthesis method according to embodiment of the present disclosure 2. Configuration of image synthesis device 3. Details of the compositing process executed by the compositing processing unit 4. Modification example 5. Hardware configuration 6. Conclusion
<<1.概要>>
<1-1.ADR法とQI法の技術的課題>
まず、本開示の実施形態に係る画像合成方法の説明に先立って、ADR法とQI法の技術的課題について詳細に述べておく。 <<1. Overview >>
<1-1. Technical issues of ADR method and QI method>
First, prior to explaining the image synthesis method according to the embodiment of the present disclosure, technical issues of the ADR method and the QI method will be described in detail.
<1-1.ADR法とQI法の技術的課題>
まず、本開示の実施形態に係る画像合成方法の説明に先立って、ADR法とQI法の技術的課題について詳細に述べておく。 <<1. Overview >>
<1-1. Technical issues of ADR method and QI method>
First, prior to explaining the image synthesis method according to the embodiment of the present disclosure, technical issues of the ADR method and the QI method will be described in detail.
本開示の実施形態の先行技術であるADR法やQI法で合成処理を行うためには、合成対象のCGデータDCgに加え、実空間の3次元的形状であるジオメトリ、ジオメトリに対応する反射率、実空間の照明マップが入力として必要である。
In order to perform synthesis processing using the ADR method or QI method, which is the prior art of the embodiment of the present disclosure, in addition to the CG data D Cg to be synthesized, geometry that is a three-dimensional shape in real space, and reflections corresponding to the geometry are required. A real-space illumination map is required as input.
これらの情報は、MayaやBlender等の一般的なDCC(Digital Content Creation)ツールを用い、実空間を撮像したRGB画像LRGB、デプス画像IDep、全天球画像ISepを加工することで得ることができる。
This information is obtained by processing the RGB image L RGB captured in real space, the depth image I Dep , and the spherical image I Sep using general DCC (Digital Content Creation) tools such as Maya and Blender. be able to.
なお、こうして得られるジオメトリ、および反射率の各品質は、低品位であるという前提の下、以下説明する。ADR法、QI法はともに、前述のジオメトリ、反射率、照明マップおよび合成対象のCGデータを用いて、PY画像LPY、VI画像LVIおよびマスク画像Mの3枚の画像を生成する。
Note that the following description will be made on the premise that the quality of the geometry and reflectance obtained in this way are of low quality. Both the ADR method and the QI method generate three images, a PY image L PY , a VI image L VI , and a mask image M, using the aforementioned geometry, reflectance, illumination map, and CG data to be synthesized.
PY画像LPYは、デプス画像から測距した実写情報のみでレンダリングを行った画像である。VI画像LVIは、実写情報と合成処理するCGデータの両方を用いてレンダリングを行った画像である。マスク画像Mは、合成処理するCGデータが存在する領域を示す画像である。これらの画像は、前述のDCCツールでレンダリングすることで生成可能である。
PY image L PY is an image rendered using only actual photographic information measured from a depth image. VI image L VI is an image rendered using both real-shot information and CG data to be synthesized. The mask image M is an image indicating an area where CG data to be synthesized exists. These images can be generated by rendering with the DCC tool described above.
RGB画像LRGB、PY画像LPY、VI画像LVIおよびマスク画像Mは、ADR法およびQI法における合成アルゴリズムの入力であり、それらのアルゴリズムの構造上、それぞれ利点と欠点を有する。
The RGB image L RGB , the PY image L PY , the VI image L VI and the mask image M are input to the synthesis algorithm in the ADR method and the QI method, and each of these algorithms has advantages and disadvantages in terms of their structure.
まず、ADR法では、以下の式(1)に基づいてADR画像LADRを計算する。
First, in the ADR method, an ADR image L ADR is calculated based on the following equation (1).
式(1)の第一項は、CGデータDCgが存在する領域、すなわちM=1である場合は、VI画像LVIをそのまま表示することを表している。また、式(1)の第二項は、入力されるRGB画像LRGBに対し、CGデータDCgの合成処理により生じる陰影の変化LVI-LPYを加えることを意味する。
The first term of equation (1) indicates that in the area where the CG data D Cg exists, that is, when M=1, the VI image L VI is displayed as is. Furthermore, the second term in equation (1) means adding a change in shading L VI −L PY caused by the compositing process of the CG data D Cg to the input RGB image L RGB .
これにより、測距品質が十分に高くない場合でも、陰影の差分の情報だけをVI画像LVIから抽出し、実写背景に対し効果的に高品位な陰影を付加することができる。
As a result, even if the ranging quality is not sufficiently high, only the information on the difference in shadows can be extracted from the VI image LVI , and high-quality shadows can be effectively added to the real background.
しかし、VI画像LVIの画素値は非負である。すなわちLVI≧0であるため、ADR法の第二項においては、以下の式(2)が成り立つ。
However, the pixel values of the VI image LVI are non-negative. That is, since L VI ≧0, the following formula (2) holds true in the second term of the ADR method.
つまり、ADR法には、LRGB-LPYが正である場合には、LADRのピクセル値をLRGB-LPY未満にすることができず、陰影により実写を暗くできる範囲に下限が存在するという問題がある。加えて、上記の式(1)の性質上、入力の反射率が低品位であり、真値に対して誤差が大きい場合、RGB画像LRGBとPY画像LPYの色分布が乖離し、アーチファクトとしてADR画像LADRに反映されるという問題もある。
In other words, in the ADR method, if L RGB - L PY is positive, the L ADR pixel value cannot be made less than L RGB - L PY , and there is a lower limit to the range in which the actual photograph can be darkened by shading. There is a problem with doing so. In addition, due to the nature of equation (1) above, if the input reflectance is of low quality and has a large error with respect to the true value, the color distributions of the RGB image L RGB and the PY image L PY will diverge, causing artifacts. There is also the problem that this is reflected in the ADR image LADR .
次に、QI法では、以下の式(3)に基づいてQI画像LQIを計算する。
Next, in the QI method, a QI image L QI is calculated based on the following equation (3).
QI法もADR法と同様に、入力されるRGB画像LRGBに対し、CGデータDCgの合成処理により生じる陰影の比率LVI/LPYを掛けることで、実写背景に陰影を付加することができる。
Similar to the ADR method, the QI method can add shadows to the real background by multiplying the input RGB image L RGB by the ratio of shadows L VI /L PY produced by the compositing process of the CG data D Cg . can.
また、QI法において、式(3)の第二項は、ゼロ除算による合成結果の不安定化を防止するために、実際は以下の式(4)で運用される。
Furthermore, in the QI method, the second term of equation (3) is actually operated by the following equation (4) in order to prevent the synthesis result from becoming unstable due to division by zero.
ここで、εはゼロ除算防止のための定数である。このとき、LQIはLPYの最大1.01倍に制限される。このため、QI法には、例えば発光するCGデータDCgの合成処理により実写を明るくできる範囲には上限が存在するという問題がある。加えて、式(3)の性質上、入力のデプス画像IDepが低品位であり、真値に対して誤差が大きい場合、式(4)の第二項においてゼロ除算にきわめて近い演算が発生する可能性が高く、その場合アーチファクトとしてQI画像LQIに反映されるという問題もある。
Here, ε is a constant to prevent division by zero. At this time, L QI is limited to a maximum of 1.01 times L PY . For this reason, the QI method has a problem in that there is an upper limit to the range in which a live photograph can be brightened by, for example, combining processing of emitting CG data DCg . In addition, due to the nature of equation (3), if the input depth image IDep is of low quality and has a large error with respect to the true value, an operation extremely similar to division by zero will occur in the second term of equation (4). There is a high possibility that this will occur, and in that case, there is a problem that it will be reflected in the QI image LQI as an artifact.
最後に、ADR法とQI法の両者で共通した技術的課題を示す。上記の式(1),式(3)の合成処理では、実空間のジオメトリ、反射率、照明マップの測距品質が高く、画像全体にわたって下記の式(5)が成り立つことを前提として、高品位な陰影の合成を保証している。
Finally, we will show technical issues common to both the ADR method and the QI method. In the synthesis process of equations (1) and (3) above, we assume that the real space geometry, reflectance, and illumination map have high distance measurement quality, and that equation (5) below holds true over the entire image. It guarantees high-quality shading composition.
しかし、前提として、ジオメトリおよび反射率の各測距品質が低い場合、式(5)が成り立たない領域が画像中に存在する場合がある。このような領域中で、特に合成品質に影響を及ぼすのは、PY画像LPYがRGB画像LRGBよりも極めて暗くなる、すなわち下記の式(6)となる領域である。
However, as a premise, if the ranging qualities of geometry and reflectance are low, there may be an area in the image where equation (5) does not hold. Among these regions, the region where the PY image L PY is much darker than the RGB image L RGB , that is, the region where the following equation (6) is satisfied, particularly affects the synthesis quality.
本開示の実施形態では、このような領域をライトギャップ(Light-gap)領域と呼ぶ。ライトギャップ領域は、上記前提のように、デプス画像IDepの測距品質が低品位であり、実空間の3次元的な物体の遮蔽環境が正確に取得できない場合に生じる。ライトギャップ領域に対してCGデータDCgの合成処理を行う場合、CGデータDCg中に発光する仮想オブジェクトが含まれていなければ、自動的に下記の式(7)が成り立つ。
In embodiments of the present disclosure, such a region is referred to as a light-gap region. As stated above, the light gap region occurs when the distance measurement quality of the depth image IDep is low quality and the three-dimensional object shielding environment in real space cannot be accurately acquired. When performing compositing processing of CG data D Cg on the light gap region, the following equation (7) automatically holds true unless a virtual object that emits light is included in the CG data D Cg .
ゆえに、ADR法の式(1)の第二項においては、下記の式(8)が成り立ち、合成処理によりRGB画像LRGBに陰影を付加すべき領域において、所望の出力を期待することができない。
Therefore, in the second term of Equation (1) of the ADR method, Equation (8) below holds true, and the desired output cannot be expected in the region where shading is to be added to the RGB image L RGB by the compositing process. .
同様に、QI法の式(4)の第二項においても、下記の式(9)が成り立ち、ADR法と同様の現象が引き起こされる。
Similarly, in the second term of Equation (4) of the QI method, Equation (9) below holds true, causing a phenomenon similar to that of the ADR method.
このように、ライトギャップ領域では、合成処理によりRGB画像LRGBに陰影を付加することができず、その結果アーチファクトとしてADR画像LADRやQI画像LQIに反映されるという技術的課題がある。
As described above, in the light gap region, there is a technical problem that shading cannot be added to the RGB image L RGB by the synthesis process, and as a result, it is reflected in the ADR image L ADR and the QI image L QI as an artifact.
以上を総括し、ADR法とQI法の両者の技術的課題をまとめると、次の通りである。まずADR法は、CGデータDCgの合成処理によってRGB画像LRGBを暗くする表現に制約が存在する。また、ADR法は、反射率の低品位な測距品質に対し、アーチファクトが生じやすい。また、ADR法は、ライトギャップ領域におけるアーチファクトが顕著となる。
Summarizing the above, the technical issues of both the ADR method and the QI method are summarized as follows. First, in the ADR method, there are restrictions on the expression of darkening the RGB image L RGB by the compositing process of the CG data D Cg . In addition, the ADR method tends to cause artifacts due to the low quality ranging quality of reflectance. Furthermore, in the ADR method, artifacts in the light gap region become noticeable.
一方、QI法は、CGデータDCgの合成処理によってRGB画像LRGBを明るくする表現に制約が存在する。また、QI法は、デプス画像IDepの低品位な測距品質に対し、アーチファクトが生じやすい。また、QI法は、ライトギャップ領域におけるアーチファクトが顕著となる。
On the other hand, in the QI method, there are restrictions on the expression of brightening the RGB image L RGB by the synthesis process of the CG data D Cg . In addition, the QI method tends to cause artifacts due to the low ranging quality of the depth image IDep . Furthermore, in the QI method, artifacts in the light gap region become noticeable.
<1-2.本開示の実施形態に係る画像合成方法の概要>
図1は、本開示の実施形態に係る画像合成方法の概要説明図である。上記の技術的課題に対し、本開示の実施形態に係る画像合成方法では、画像合成装置10が、RGB画像LRGB、デプス画像IDepおよび全天球画像ISepを入力としてRGB画像LRGBに対応する実空間を表す3次元空間にCGデータDCgを配置する合成処理を行う。 <1-2. Overview of image synthesis method according to embodiment of the present disclosure>
FIG. 1 is a schematic explanatory diagram of an image synthesis method according to an embodiment of the present disclosure. In order to solve the above technical problem, in the image synthesis method according to the embodiment of the present disclosure, theimage synthesis device 10 inputs the RGB image L RGB , the depth image I Dep , and the spherical image I Sep and converts the RGB image L RGB into an RGB image L RGB . A synthesis process is performed to arrange the CG data D Cg in a three-dimensional space representing the corresponding real space.
図1は、本開示の実施形態に係る画像合成方法の概要説明図である。上記の技術的課題に対し、本開示の実施形態に係る画像合成方法では、画像合成装置10が、RGB画像LRGB、デプス画像IDepおよび全天球画像ISepを入力としてRGB画像LRGBに対応する実空間を表す3次元空間にCGデータDCgを配置する合成処理を行う。 <1-2. Overview of image synthesis method according to embodiment of the present disclosure>
FIG. 1 is a schematic explanatory diagram of an image synthesis method according to an embodiment of the present disclosure. In order to solve the above technical problem, in the image synthesis method according to the embodiment of the present disclosure, the
図1に示すように、かかる合成処理において、画像合成装置10は、上記入力から測距された実写情報のみを用いたレンダリング画像であるPY画像LPYと、上記実写情報およびCGデータDCgを用いたレンダリング画像であるVI画像LVIとを生成する(ステップS1-1,S1-2)。PY画像LPYは、「第1のレンダリング画像」の一例に相当する。VI画像LVIは、「第2のレンダリング画像」の一例に相当する。
As shown in FIG. 1, in this compositing process, the image compositing device 10 combines the PY image LPY , which is a rendered image using only the actual photographic information measured from the input, and the aforementioned photographic information and CG data DCG . A VI image LVI , which is the rendered image used, is generated (steps S1-1, S1-2). PY image L PY corresponds to an example of a "first rendered image." VI image L VI corresponds to an example of a "second rendered image."
また、画像合成装置10は、PY画像LPYおよびVI画像LVIの陰影の差に基づくADR法による合成画像であるADR画像LADRと、上記陰影の比率に基づくQI法による合成画像であるQI画像LQIとを生成する(ステップS2-1,S2-2)。ADR画像LADRは、「第1の合成画像」の一例に相当する。QI画像LQIは、「第2の合成画像」の一例に相当する。
The image synthesis device 10 also generates an ADR image L ADR, which is a synthesized image based on the ADR method based on the difference in shading between the PY image L PY and a VI image L VI , and a QI image, which is a synthesized image based on the QI method based on the ratio of the shadings. Images L and QI are generated (steps S2-1 and S2-2). ADR image L ADR corresponds to an example of a "first composite image." QI image L QI corresponds to an example of a "second composite image."
また、画像合成装置10は、上記実写情報のみを用いた場合と、上記実写情報およびCGデータDCgを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像SIを生成する(ステップS3)。また、画像合成装置10は、シャドウインパクト画像SIを用いてADR画像LADRとQI画像LQIとの線形結合を行い、LC画像LLCを生成する(ステップS4)。
The image synthesis device 10 also generates a shadow impact image SI representing a change in light radiation energy between the case where only the above-mentioned real-time photograph information is used and the case where the above-mentioned live-action information and CG data DCg are used. Step S3). The image synthesis device 10 also performs a linear combination of the ADR image LADR and the QI image LQI using the shadow impact image S I to generate an LC image L LC (step S4).
すなわち、本開示の実施形態に係る画像合成方法では、RGB画像LRGB、デプス画像IDep、全天球画像ISepを入力として、RGB画像LRGBに対応する実空間を表す3次元空間にCGデータDCgを配置する合成処理を行う。その際、ADR法により生成されるADR画像LADRと、QI法により生成されるQI画像LQIを、本開示の新規アルゴリズムにより生成するシャドウインパクト画像SIを用いて線形結合を行い、ADR法とQI法の長所を画像ピクセルごとに適応的に混合する。
That is, in the image synthesis method according to the embodiment of the present disclosure, the RGB image L RGB , the depth image I Dep , and the spherical image I Sep are input, and CG is created in a three-dimensional space representing the real space corresponding to the RGB image L RGB . Performs synthesis processing to arrange data D Cg . At that time, the ADR image L ADR generated by the ADR method and the QI image L QI generated by the QI method are linearly combined using the shadow impact image S I generated by the new algorithm of the present disclosure, and The advantages of the and QI methods are adaptively mixed for each image pixel.
これにより、ユーザは、ADR画像LADRとQI画像LQIの手動的選択を行う必要がなくなる。さらに、シャドウインパクト画像SIを用いて遮蔽が生じる領域のアーチファクトを解消し、多くの3次元形状が複雑に交差して遮蔽が生じる実空間での自動的な合成処理を実現することができる。
This eliminates the need for the user to manually select the ADR image L ADR and the QI image L QI . Furthermore, it is possible to eliminate artifacts in areas where occlusion occurs using the shadow impact image SI , and to realize automatic synthesis processing in real space where many three-dimensional shapes intersect in a complex manner and occlusion occurs.
すなわち、本開示の実施形態に係る画像合成方法によれば、実空間を表す3次元空間に対するCGデータDCgの合成品質をより向上させることができる。以下、本開示の実施形態に係る画像合成方法を適用した画像合成装置10の構成例について、より具体的に説明する。
That is, according to the image synthesis method according to the embodiment of the present disclosure, it is possible to further improve the synthesis quality of CG data DCg with respect to a three-dimensional space representing a real space. Hereinafter, a configuration example of the image synthesis apparatus 10 to which the image synthesis method according to the embodiment of the present disclosure is applied will be described in more detail.
<<2.画像合成装置の構成>>
図2は、本開示の実施形態に係る画像合成装置10の構成例を示すブロック図である。なお、図2および後に示す図3では、本開示の実施形態の特徴を説明するために必要な構成要素のみを表しており、一般的な構成要素についての記載を省略している。 <<2. Configuration of image synthesis device >>
FIG. 2 is a block diagram illustrating a configuration example of theimage synthesis device 10 according to the embodiment of the present disclosure. Note that FIG. 2 and FIG. 3 shown later show only the constituent elements necessary for explaining the features of the embodiment of the present disclosure, and descriptions of general constituent elements are omitted.
図2は、本開示の実施形態に係る画像合成装置10の構成例を示すブロック図である。なお、図2および後に示す図3では、本開示の実施形態の特徴を説明するために必要な構成要素のみを表しており、一般的な構成要素についての記載を省略している。 <<2. Configuration of image synthesis device >>
FIG. 2 is a block diagram illustrating a configuration example of the
換言すれば、図2および図3に図示される各構成要素は機能概念的なものであり、必ずしも物理的に図示の如く構成されていることを要しない。例えば、各ブロックの分散・統合の具体的形態は図示のものに限られず、その全部または一部を、各種の負荷や使用状況などに応じて、任意の単位で機能的または物理的に分散・統合して構成することが可能である。
In other words, each component illustrated in FIGS. 2 and 3 is functionally conceptual, and does not necessarily need to be physically configured as illustrated. For example, the specific form of distributing/integrating each block is not limited to what is shown in the diagram, and all or part of the blocks can be functionally or physically distributed/integrated in arbitrary units depending on various loads and usage conditions. It is possible to configure them in an integrated manner.
また、図2および図3を用いた説明では、既に説明済みの構成要素については、説明を簡略するか、省略する場合がある。
Furthermore, in the explanation using FIGS. 2 and 3, the explanation of components that have already been explained may be simplified or omitted.
図2に示すように、画像合成装置10は、記憶部11と、制御部12とを有する。記憶部11は、例えば、RAM(Random Access Memory)、ROM(Read Only Memory)、フラッシュメモリ(Flash Memory)等の半導体メモリ素子、または、ハードディスク、光ディスク等の記憶装置によって実現される。
As shown in FIG. 2, the image synthesis device 10 includes a storage section 11 and a control section 12. The storage unit 11 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a ROM (Read Only Memory), or a flash memory, or a storage device such as a hard disk or an optical disk.
図2に示す例では、記憶部11は、ジオメトリ情報11aと、反射率情報11bと、照明マップ情報11cと、DCCツールプログラム11dを記憶する。ジオメトリ情報11aは、前述のジオメトリに対応する情報である。反射率情報11bは、前述の反射率に対応する情報である。照明マップ情報11cは、前述の照明マップに対応する情報である。DCCツールプログラム11dは、DCCツールのプログラムデータである。
In the example shown in FIG. 2, the storage unit 11 stores geometry information 11a, reflectance information 11b, illumination map information 11c, and DCC tool program 11d. The geometry information 11a is information corresponding to the above-mentioned geometry. The reflectance information 11b is information corresponding to the above-mentioned reflectance. The illumination map information 11c is information corresponding to the aforementioned illumination map. The DCC tool program 11d is program data of the DCC tool.
制御部12は、コントローラ(controller)であり、例えば、CPU(Central Processing Unit)やMPU(Micro Processing Unit)、GPU(Graphics Processing Unit)等によって、記憶部11に記憶された図示略の本開示の実施形態に係る情報処理プログラムがRAMを作業領域として実行されることにより実現される。また、制御部12は、例えば、ASIC(Application Specific Integrated Circuit)やFPGA(Field Programmable Gate Array)等の集積回路により実現することができる。
The control unit 12 is a controller, and includes, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. This is realized by executing the information processing program according to the embodiment using the RAM as a work area. Further, the control unit 12 can be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
制御部12は、取得部12aと、変換部12bと、合成処理部12cと、出力部12dとを有し、以下に説明する情報処理の機能や作用を実現または実行する。
The control unit 12 includes an acquisition unit 12a, a conversion unit 12b, a composition processing unit 12c, and an output unit 12d, and realizes or executes information processing functions and operations described below.
取得部12aは、RGB画像LRGB、デプス画像IDep、全天球画像ISepおよびCGデータDCgを取得する。
The acquisition unit 12a acquires the RGB image L RGB , the depth image I Dep , the spherical image I Sep , and the CG data D Cg .
変換部12bは、RGB画像LRGB、デプス画像IDep、全天球画像ISepをDCCツールが読み込み可能なジオメトリ情報11a、反射率情報11bおよび照明マップ情報11cに変換する。
The converter 12b converts the RGB image LRGB , the depth image IDep , and the spherical image ISep into geometry information 11a, reflectance information 11b, and illumination map information 11c that can be read by the DCC tool.
合成処理部12cは、変換されたジオメトリ情報11a、反射率情報11b、照明マップ情報11c、CGデータDCgおよびRGB画像LRGBを入力として、RGB画像LRGBに対応する実空間を表す3次元空間にCGデータDCgを配置する合成処理を行う。
The synthesis processing unit 12c receives the converted geometry information 11a, reflectance information 11b, illumination map information 11c, CG data D , and RGB image L as input, and generates a three-dimensional space representing a real space corresponding to the RGB image L. A compositing process is performed to arrange CG data D Cg .
図3は、合成処理部12cの構成例を示すブロック図である。図3に示すように、合成処理部12cは、第1生成部12caと、第2生成部12cbと、第3生成部12ccと、第4生成部12cdと、第5生成部12ceと、出力画像生成部12cfとを有する。
FIG. 3 is a block diagram showing a configuration example of the composition processing section 12c. As shown in FIG. 3, the composition processing section 12c includes a first generation section 12ca, a second generation section 12cb, a third generation section 12cc, a fourth generation section 12cd, a fifth generation section 12ce, and an output image. It has a generation unit 12cf.
第1生成部12caは、変換されたジオメトリ情報11a、反射率情報11b、照明マップ情報11cおよびCGデータDCgをDCCツールで読み込み、DCCツール内でPY画像LPY、VI画像LVI、マスク画像M、RPY画像RPY、RVI画像RVIを生成する。RPY画像RPY、RVI画像RVIは、シャドウインパクト画像SIを生成する際の入力となる画像である。
The first generation unit 12ca reads the converted geometry information 11a, reflectance information 11b, illumination map information 11c, and CG data D Cg using the DCC tool, and generates a PY image L PY , a VI image L VI , and a mask image within the DCC tool. M, generate an RPY image R PY and an RVI image R VI . The RPY image R PY and the RVI image R VI are images that are input when generating the shadow impact image S I.
第2生成部12cbは、生成されたPY画像LPY、VI画像LVI、マスク画像M、RGB画像LRGBを用いて、ADR画像LADRおよびQI画像LQIを生成する。第3生成部12ccは、第2生成部12cbに並行して、RPY画像RPYとRVI画像RVIを入力として、シャドウインパクト画像SIを生成する。
The second generation unit 12cb generates an ADR image L ADR and a QI image L QI using the generated PY image L PY , VI image L VI , mask image M, and RGB image L RGB . The third generation unit 12cc receives the RPY image RPY and the RVI image RVI as input and generates a shadow impact image SI in parallel with the second generation unit 12cb.
そして、第4生成部12cdは、ADR画像LADR、QI画像LQI、シャドウインパクト画像SIを入力として、LC画像LLCを生成する。第5生成部12ceは、第4生成部12cdに並行して、RGB画像LRGB、PY画像LPY、シャドウインパクト画像SIを入力として、ライトギャップ画像wgを生成する。最後に、出力画像生成部12cfは、VI画像LVI、LC画像LLC、ライトギャップ画像wgを入力として、出力画像Lendを生成し、出力する。
Then, the fourth generation unit 12cd receives the ADR image L ADR , the QI image L QI , and the shadow impact image S I and generates the LC image L LC . The fifth generation unit 12ce receives the RGB image L RGB , the PY image L PY , and the shadow impact image S I in parallel with the fourth generation unit 12 cd and generates a light gap image w g . Finally, the output image generation unit 12cf receives the VI image L VI , LC image L LC , and light gap image w g as input, generates an output image L end , and outputs it.
かかる合成処理部12cが実行する合成処理のより詳細な内容については、図4以降を用いた説明で後述する。
More detailed contents of the compositing process executed by the compositing processing unit 12c will be described later in the explanation using FIG. 4 and subsequent figures.
図3の説明に戻る。出力部12dは、合成処理部12cによって生成された出力画像Lendを表示装置等の外部装置へ出力する。
Returning to the explanation of FIG. 3. The output unit 12d outputs the output image L end generated by the composition processing unit 12c to an external device such as a display device.
<<3.合成処理部が実行する合成処理の詳細>>
次に、合成処理部12cが実行する合成処理の詳細について、図4~図19を用いて各画像の例を挙げつつ説明する。 <<3. Details of the compositing process executed by the compositing processing unit >>
Next, the details of the compositing process executed by thecompositing processing unit 12c will be explained while giving examples of each image using FIGS. 4 to 19.
次に、合成処理部12cが実行する合成処理の詳細について、図4~図19を用いて各画像の例を挙げつつ説明する。 <<3. Details of the compositing process executed by the compositing processing unit >>
Next, the details of the compositing process executed by the
図4は、出力画像Lendの一例を示す図である。また、図5は、RGB画像LRGBの一例を示す図である。また、図6は、CGデータDCgの一例を示す図である。また、図7は、デプス画像IDepの一例を示す図である。以下では、図4に示す出力画像Lendが、合成処理を経て合成処理部12cから最終的に出力される例を挙げる。
FIG. 4 is a diagram showing an example of the output image L end . Further, FIG. 5 is a diagram showing an example of an RGB image L RGB . Further, FIG. 6 is a diagram showing an example of CG data DCg . Further, FIG. 7 is a diagram showing an example of the depth image IDep . In the following, an example will be given in which the output image L end shown in FIG. 4 is finally output from the composition processing unit 12c after undergoing composition processing.
図4は、後方に存在するマネキンが実空間に存在するマネキンであり、手前の人物がCGである例を示している。なお、以下では、説明を分かりやすくするために図を適宜模式化している。したがって、以下に示す例は、本開示の実施形態に係る合成処理の合成品質を限定するものではない。
FIG. 4 shows an example in which the mannequin in the rear is a mannequin that exists in real space, and the person in the foreground is CG. Note that, below, the figures are appropriately simplified in order to make the explanation easier to understand. Therefore, the examples shown below do not limit the synthesis quality of the synthesis processing according to the embodiments of the present disclosure.
また、図4の図中のM1部については後述する。図4に示す出力画像Lendの合成処理を行う場合、取得部12aが取得するRGB画像LRGBは、図5に示すようなものとなる。なお、図5の図中のM2部については後述する。
Further, the M1 portion in the diagram of FIG. 4 will be described later. When performing the synthesis process of the output image L end shown in FIG. 4, the RGB image L RGB obtained by the obtaining unit 12a is as shown in FIG. 5. Note that the M2 portion in the diagram of FIG. 5 will be described later.
また、同じく取得部12aが取得するCGデータDCgは、図6に示すようなものとなる。同様に、取得部12aが取得するデプス画像IDepは、図7に示すようなものとなる。なお、取得部12aが取得する全天球画像ISepについては、ここでは省略した。
Further, the CG data D Cg similarly obtained by the obtaining unit 12a is as shown in FIG. Similarly, the depth image IDep acquired by the acquisition unit 12a is as shown in FIG. Note that the spherical image I Sep acquired by the acquisition unit 12a is omitted here.
合成処理部12cは、これらRGB画像LRGB、CGデータDCg、デプス画像IDepおよび全天球画像ISepに基づいて、PY画像LPY、VI画像LVI、マスク画像M、RPY画像RPY、RVI画像RVIを生成する。
The synthesis processing unit 12c generates a PY image L PY , a VI image L VI , a mask image M, and an RPY image R PY based on these RGB image L RGB , CG data D Cg , depth image I Dep , and spherical image I Sep , generates an RVI image RVI .
図8は、PY画像LPYの一例を示す図である。また、図9は、VI画像LVIの一例を示す図である。また、図10は、マスク画像Mの一例を示す図である。また、図11は、RPY画像RPYの一例を示す図である。また、図12は、RVI画像RVIの一例を示す図である。また、図13は、シャドウインパクト画像SIの一例を示す図である。
FIG. 8 is a diagram showing an example of the PY image LPY . Further, FIG. 9 is a diagram showing an example of the VI image LVI. Moreover, FIG. 10 is a diagram showing an example of the mask image M. Further, FIG. 11 is a diagram showing an example of the RPY image RPY . Further, FIG. 12 is a diagram showing an example of the RVI image RVI . Further, FIG. 13 is a diagram showing an example of a shadow impact image SI .
図4に示す出力画像Lendの合成処理を行う場合、PY画像LPYは、図8に示すようなものとなる。同様に、VI画像LVIは、図9に示すようなものとなる。同様に、マスク画像Mは、図10に示すようなものとなる。
When performing the synthesis process of the output image L end shown in FIG. 4, the PY image L PY becomes as shown in FIG. 8. Similarly, the VI image LVI becomes as shown in FIG. Similarly, the mask image M becomes as shown in FIG.
ところで、本開示の実施形態では、2段階の合成処理によってADR法とQI法の技術的課題を克服する。まず1段階目の合成処理では、ADR法とQI法がそれぞれ得意とする領域を検出する重み関数画像とも言えるシャドウインパクト画像SIを新規のアルゴリズムによって生成する。
By the way, in the embodiment of the present disclosure, the technical problems of the ADR method and the QI method are overcome by a two-step synthesis process. First, in the first stage of synthesis processing, a new algorithm generates a shadow impact image SI , which can be called a weighting function image that detects areas in which the ADR method and the QI method are respectively good.
ここで、シャドウインパクト画像SIの生成アルゴリズムについて説明する。本アルゴリズムでは、以下の式(10)により定義される、単位面積当たりの光放射エネルギーである放射照度の近似値Rを計算する。
Here, the generation algorithm of the shadow impact image SI will be explained. In this algorithm, an approximate value R of irradiance, which is the optical radiation energy per unit area, is calculated by the following equation (10).
ここで、上記の式(11)は、画像中のピクセルに対応する実空間の3次元位置xに方向ベクトルωiから物体面に入射する照明の強さであり、測距した全天球画像ISepから計算される。
Here, the above equation (11) is the intensity of illumination incident on the object plane from the direction vector ω i to the three-dimensional position x in real space corresponding to the pixel in the image, and is the intensity of the illumination that enters the object plane from the direction vector ω i , and Calculated from I Sep.
また、上記の式(12)は、3次元位置xから方向ベクトルωiを見たときに、合成処理を行うCGデータDCgが存在すれば0を、存在しなければ1を返す可視性(Visibility)関数である。加えて、近似値Rはカラー要素を持たず、グレースケール化されている。
In addition, the above equation ( 12) shows that when looking at the direction vector ω i from the three-dimensional position Visibility) function. In addition, the approximation R has no color component and is gray scaled.
本アルゴリズムでは、測距した実写情報のみを用いて放射照度を計算した結果得られるRPY画像RPY、測距した実写情報と合成処理するCGデータDCgの両方を用いて放射照度を計算した結果得られるRVI画像RVIをそれぞれ生成する。
In this algorithm, the RPY image R PY obtained as a result of calculating irradiance using only the distance-measured real-photo information, and the result of calculating irradiance using both the distance-measured real-photo information and the CG data D Cg to be synthesized. The resulting RVI images RVI are respectively generated.
図4に示す出力画像Lendの合成処理を行う場合、RPY画像RPYは、図11に示すようなものとなる。同様に、RVI画像RVIは、図12に示すようなものとなる。
When performing the synthesis process of the output image L end shown in FIG. 4, the RPY image RPY becomes as shown in FIG. 11. Similarly, the RVI image RVI becomes as shown in FIG.
そして、合成処理部12cは、RVI画像RVIとRPY画像RPYを用いてシャドウインパクト画像SIを以下の式(13)によって生成する。
Then, the composition processing unit 12c generates a shadow impact image S1 using the RVI image RVI and the RPY image RPY according to the following equation (13).
ここで、dilate()はライトギャップにより相殺しきれないノイズを軽減するために、画像全体に適用される一般的な膨張処理である。そして、合成処理部12cは、シャドウインパクト画像SIを用いて、以下の式(14)によってADR画像LADRとQI画像LQIの線形結合を行い、LC画像LLCを生成する。
Here, dilate() is a general dilation process applied to the entire image in order to reduce noise that cannot be canceled out due to the light gap. Then, the synthesis processing unit 12c performs a linear combination of the ADR image L ADR and the QI image L QI using the shadow impact image S I according to the following equation (14) to generate an LC image L LC .
図4に示す出力画像Lendの合成処理を行う場合、シャドウインパクト画像SIは、図13に示すようなものとなる。
When performing the synthesis process of the output image L end shown in FIG. 4, the shadow impact image S I becomes as shown in FIG. 13.
なお、説明が前後するが、ここで、ADR画像LADRに現れるADR法の欠点、および、QI画像LQIに現れるQI法の欠点のそれぞれの具体例を示しておく。図14は、ADR画像LADRの一例を示す図である。また、図15は、QI画像LQIの一例を示す図(その1)である。また、図16は、QI画像LQIの一例を示す図(その2)である。
Although the explanation is complicated, specific examples of the drawbacks of the ADR method that appear in the ADR image L ADR and the drawbacks of the QI method that appear in the QI image L QI will be shown here. FIG. 14 is a diagram showing an example of the ADR image L ADR . Further, FIG. 15 is a diagram (part 1) showing an example of QI image L QI . Further, FIG. 16 is a diagram (part 2) showing an example of QI image L QI .
図14は、図4と同様に、人物がCGである場合のADR画像LADRであるものとする。かかる場合、ADR画像LADRには、例えば実空間中のM4部の中のM41部に示すように、柱の段差付近での色の不調和等が現れることがある。
As with FIG. 4, FIG. 14 is an ADR image LADR in which the person is a CG image. In such a case, in the ADR image L ADR , color disharmony or the like may appear in the vicinity of the step of the pillar, as shown, for example, in the M41 part of the M4 part in the real space.
また、ADR画像LADRには、例えば同じくM4部の中のM42部に示すように、物体の境界での色のにじみ等が現れることがある。
Further, in the ADR image LADR , color blurring or the like may appear at the boundary of the object, as shown in the M42 section of the M4 section, for example.
また、図15は、図14と同様に、人物がCGである場合のQI画像LQIであるものとする。かかる場合、QI画像LQIには、例えば実空間中のM5部の中のM51部に示すように、ジオメトリノイズが大きい部分(画像の下端部)で、不適当なノイズが走ることがある。
Furthermore, similarly to FIG. 14, it is assumed that FIG. 15 is a QI image LQI when the person is a CG image. In such a case, inappropriate noise may appear in the QI image LQI in a portion where geometry noise is large (lower end of the image), as shown, for example, in the M51 portion of the M5 portion in real space.
また、図16は、図14および図15とは逆に人物が実物体であり、人物の横に発光する剣を持ったキャラクタのCGを合成する場合のQI画像LQIであるものとする。かかる場合、QI画像LQIには、例えば実空間中のM6部の中のM61部に示すように、CGの発光する剣による照り返しで明るくなるべき領域が明るくならないといった事態が起こることがある。
Furthermore, in contrast to FIGS. 14 and 15, FIG. 16 is a QI image LQI in which the person is a real object and a CG of a character holding a light-emitting sword is synthesized next to the person. In such a case, in the QI image LQI , a situation may occur in which an area that should be brightened does not become bright due to reflection from the CG light-emitting sword, as shown in the M61 part of the M6 part in the real space, for example.
前述の1段階目の合成処理では、合成処理部12cは、こうしたADR画像LADRおよびQI画像LQIの欠点が解消されるように、シャドウインパクト画像SIを用いてADR画像LADRとQI画像LQIの線形結合を行い、LC画像LLCを生成する。したがって、LC画像LLCは、こうした欠点が解消された画像として生成されることとなる。
In the above-mentioned first stage synthesis processing, the synthesis processing unit 12c combines the ADR image L ADR and the QI image using the shadow impact image S I so that the defects of the ADR image L ADR and the QI image L QI are eliminated. A linear combination of LQI is performed to generate an LC image LLC . Therefore, the LC image LLC is generated as an image in which such defects are eliminated.
次に、2段階目の合成処理では、合成処理部12cは、シャドウインパクト画像SIを用いてライトギャップ領域におけるアーチファクトを除去する。本アルゴリズムでは、ライトギャップ領域を示すライトギャップ画像wgを以下の式(15)により計算する。
Next, in the second stage of compositing processing, the compositing processing unit 12c uses the shadow impact image SI to remove artifacts in the light gap region. In this algorithm, a light gap image w g indicating a light gap region is calculated using the following equation (15).
ここで、saturate()は入力を0から1に範囲制限する関数であり、opening()はデノイズのためのモルフォロジー演算である。また、λg、λpはハイパーパラメータであり、本開示の実施形態では全ての合成実行において経験的にλg=2.0、λp=2.0に設定する。
Here, saturate() is a function that limits the input range to 0 to 1, and opening() is a morphological operation for denoising. Furthermore, λ g and λ p are hyperparameters, and in the embodiment of the present disclosure, they are empirically set to λ g =2.0 and λ p =2.0 in all synthesis executions.
そして、最後に、合成処理部12cは、ライトギャップ領域、すなわちwgが高い領域にVI画像LVIを線形結合によって優先的に割り当てることで、アーチファクトが除去された最終的な合成結果である出力画像Lendを以下の式(16)のように得る。
Finally, the synthesis processing unit 12c preferentially allocates the VI image LVI to the light gap area, that is, the area where wg is high, by linear combination, and outputs the final synthesis result from which artifacts have been removed. Image L end is obtained as shown in equation (16) below.
図17は、ライトギャップ画像wgの一例を示す図(その1)である。また、図18は、ライトギャップ画像wgの一例を示す図(その2)である。
FIG. 17 is a diagram (part 1) showing an example of the light gap image wg . Further, FIG. 18 is a diagram (part 2) showing an example of the light gap image wg .
図17は、図4に示す出力画像Lendの合成処理を行う場合に対応している。かかる場合、ライトギャップ画像wgは、図17に示すようなものとなる。
FIG. 17 corresponds to the case where the synthesis process of the output image L end shown in FIG. 4 is performed. In such a case, the light gap image wg will be as shown in FIG. 17.
図17に示すM7部は、図5に示したRGB画像LRGBのM2部に対応している。また、M7部は、図8に示したPY画像LPYのM3部に対応している。
The M7 section shown in FIG. 17 corresponds to the M2 section of the RGB image LRGB shown in FIG. Furthermore, the M7 section corresponds to the M3 section of the PY image LPY shown in FIG.
これらM2部,M3部,M7部を比較のために並べたものが図18である。図18に示すように、画像中のマネキンの右腕と胴体の間の空間において、PY画像LPYのレンダリング結果は、RGB画像LRGBと大きく異なることが分かる。合成処理部12cは、図18に示すように、この部分をライトギャップ画像wgとして検出する。
FIG. 18 shows these M2 portion, M3 portion, and M7 portion arranged side by side for comparison. As shown in FIG. 18, it can be seen that the rendering result of the PY image LPY is significantly different from the RGB image L RGB in the space between the right arm and torso of the mannequin in the image. The composition processing unit 12c detects this portion as a light gap image wg , as shown in FIG.
また、図19は、ライトギャップ画像wgを用いた補正の効果を示す図である。図19は、図4に示した出力画像LendのM1部、かかるM1部に対応するADR画像LADRのM1ADR部、および、同様にM1部に対応するQI画像LQIのM1QI部を比較のために並べたものとなっている。
Further, FIG. 19 is a diagram showing the effect of correction using the light gap image wg . FIG. 19 shows the M1 portion of the output image L end shown in FIG. 4, the M1 ADR portion of the ADR image L ADR corresponding to the M1 portion, and the M1 QI portion of the QI image L QI corresponding to the M1 portion. They are arranged side by side for comparison.
図18と同様に、図19に示すように、ライトギャップが大きいマネキンの右腕と胴体の間の空間では、ADR画像LADRもQI画像LQIもアーチファクトが大きいことが分かる。例えば、ADR画像LADRは影となるべき部分が明る過ぎることが分かる。また、QI画像LQIは色のにじみや不調和があることが分かる。
Similar to FIG. 18, as shown in FIG. 19, it can be seen that in the space between the mannequin's right arm and torso where the light gap is large, artifacts are large in both the ADR image L ADR and the QI image L QI . For example, it can be seen that in the ADR image L ADR , the portion that should be a shadow is too bright. It can also be seen that the QI image LQI has color bleeding and disharmony.
これに対し、出力画像Lendは、ライトギャップ画像wgを用いた補正の効果により、ADR画像LADRやQI画像LQIに見られたアーチファクトが解消されていることが分かる。
On the other hand, it can be seen that in the output image L end , the artifacts observed in the ADR image LADR and the QI image L QI are eliminated due to the effect of the correction using the light gap image w g .
以上のような本開示の実施形態に係る合成処理のアルゴリズムによる効果をまとめる。まず、シャドウインパクト画像SIは、CGデータDCgの合成処理により光放射エネルギーが増加した領域では低い画素値を、光放射エネルギーが減少した領域では高い画素値を持つ。
The effects of the synthesis processing algorithm according to the embodiment of the present disclosure as described above will be summarized. First, the shadow impact image S I has a low pixel value in an area where the light radiant energy has increased due to the synthesis process of the CG data D Cg , and has a high pixel value in an area where the light radiant energy has decreased.
これは、CGデータDCgの合成処理によりRGB画像LRGBが明るくなる場合と、暗くなる場合とにそれぞれ対応する。ゆえに、シャドウインパクトが低い領域にはADR画像LADRを、シャドウインパクトが高い領域にはQI画像LQIを割り当てることで、両手法の長所を自動的に折衷した合成処理が可能となる。
This corresponds to a case where the RGB image L RGB becomes brighter and a case where it becomes darker due to the composition processing of the CG data DCg . Therefore, by assigning the ADR image L ADR to an area with a low shadow impact and the QI image L QI to an area with a high shadow impact, it is possible to automatically combine the advantages of both methods.
また、放射照度画像であるRVI画像RVIとRPY画像RPYの生成には、照明と可視性のみの情報を用いている。一般的なCGの手法は、これに加えて空間中の法線の向きも考慮するが、法線はデプス画像IDepが低品位である場合にノイズの影響を受けやすい。そのため、RVI画像RVIとRPY画像RPYの生成に法線の情報を用いないことで、低品位なデプス画像IDepに由来するノイズの影響を受けないフィルタを生成することができる。
Further, information only about illumination and visibility is used to generate the RVI image RVI and the RPY image RPY , which are irradiance images. In addition to this, general CG methods also consider the direction of normal lines in space, but normal lines are susceptible to noise when the depth image IDep is of low quality. Therefore, by not using normal information to generate the RVI image RVI and the RPY image RPY , it is possible to generate a filter that is not affected by noise originating from the low-quality depth image IDep .
次に、シャドウインパクト画像SIを用いたアーチファクトの除去効果を述べる。本アルゴリズムでは、上記の式(15)によってライトギャップ領域の定義を行い、計算および検出を行う。式(15)中のopening関数の引数では、入力RGB画像LRGBとPY画像LPYとの差分により表現される測距誤差をλpによって累乗することで、測距誤差が顕著に大きい領域を検出することができる。
Next, the effect of removing artifacts using the shadow impact image SI will be described. In this algorithm, the light gap region is defined using the above equation (15), and calculation and detection are performed. In the argument of the opening function in equation (15), the range measurement error expressed by the difference between the input RGB image L RGB and the PY image L PY is raised to the power of λ p to identify areas where the range measurement error is significantly large. can be detected.
さらに、式(15)中でシャドウインパクト画像SIを掛けることで、RGB画像LRGBにおいて合成処理により画素値の変化が起きる可能性がある領域にのみアーチファクトの除去処理を限定することができる。
Furthermore, by multiplying the shadow impact image S I in Equation (15), it is possible to limit the artifact removal process to only areas where there is a possibility that a change in pixel value will occur due to the composition process in the RGB image L RGB .
また、ライトギャップ画像wgは2値画像ではなく、ライトギャップが低い領域から高い領域になめらかに変化するため、ライトギャップ画像wgは急峻なエッジを持たない。その結果、上記の式(16)中でVI画像LVIとLC画像LLCを線形結合させてブレンドする際に、ブレンドした画像の接合部が目立たないように合成処理を行うことができる。
Further, the light gap image w g is not a binary image, and the light gap changes smoothly from a low region to a high region, so the light gap image w g does not have sharp edges. As a result, when the VI image L VI and the LC image L LC are linearly combined and blended in the above equation (16), the synthesis process can be performed so that the joint of the blended images is not noticeable.
次に、画像合成装置10が実行する処理手順について、図20を用いて説明する。図20は、画像合成装置10が実行する処理手順を示すフローチャートである。
Next, the processing procedure executed by the image synthesis device 10 will be explained using FIG. 20. FIG. 20 is a flowchart showing the processing procedure executed by the image synthesis device 10.
図20に示すように、まず取得部12aが、RGB画像LRGB、デプス画像IDep、全天球画像ISep、CGデータDCgを取得する(ステップS101)。そして、変換部12bが、RGB画像LRGB、デプス画像IDep、全天球画像ISepをDCCツールに読み込み可能に変換する(ステップS102)。そして、合成処理部12cが、変換後データおよびCGデータDCgをDCCツールで読み込む(ステップS103)。
As shown in FIG. 20, the acquisition unit 12a first acquires an RGB image L RGB , a depth image I Dep , a spherical image I Sep , and CG data D Cg (step S101). Then, the conversion unit 12b converts the RGB image L RGB , the depth image I Dep , and the spherical image I Sep so that they can be read into the DCC tool (step S102). Then, the composition processing unit 12c reads the converted data and the CG data DCg using the DCC tool (step S103).
つづいて、合成処理部12cは、DCCツール内でPY画像LPY、VI画像LVI、マスク画像M、RPY画像RPY、RVI画像RVIを生成する(ステップS104)。
Subsequently, the synthesis processing unit 12c generates a PY image L PY , a VI image L VI , a mask image M, an RPY image R PY , and an RVI image R VI within the DCC tool (step S104).
そして、合成処理部12cは、PY画像LPY、VI画像LVI、マスク画像M、RGB画像LRGBを用いてADR画像LADR、QI画像LQIを生成する(ステップS105)。
Then, the synthesis processing unit 12c generates the ADR image L ADR and the QI image L QI using the PY image L PY , the VI image L VI , the mask image M, and the RGB image L RGB ( step S105).
また、これと並列に、合成処理部12cは、RPY画像RPY、RVI画像RVIを用いてシャドウインパクト画像SIを生成する(ステップS106)。
Further, in parallel with this, the synthesis processing unit 12c generates a shadow impact image S I using the RPY image R PY and the RVI image R VI (step S106).
ステップS105,S106が終了すると、合成処理部12cは、ADR画像LADR、QI画像LQI、シャドウインパクト画像SIを用いてLC画像LLCを生成する(ステップS107)。
When steps S105 and S106 are completed, the synthesis processing unit 12c generates an LC image LLC using the ADR image L ADR , the QI image L QI , and the shadow impact image S I (step S107).
また、これと並列に、合成処理部12cは、シャドウインパクト画像SI、RGB画像LRGB、PY画像LPYを用いてライトギャップ画像wgを生成する(ステップS108)。
Further, in parallel with this, the synthesis processing unit 12c generates a light gap image w g using the shadow impact image S I , the RGB image L RGB , and the PY image L PY (step S108).
ステップS107,S108が終了すると、合成処理部12cは、LC画像LLC、ライトギャップ画像wgを用いて出力画像Lendを生成する(ステップS109)。そして、出力部12dが出力画像Lendを出力し(ステップS110)、処理を終了する。
When steps S107 and S108 are completed, the synthesis processing unit 12c generates an output image L end using the LC image L LC and the light gap image w g (step S109). Then, the output unit 12d outputs the output image L end (step S110), and the process ends.
次に、図21は、静止画を出力する場合の処理手順を示すフローチャートである。まず、ユーザがRGBカメラでRGB画像LRGBを撮影する(ステップS201)。また、ユーザは、デプスカメラでデプス画像IDepを撮影する(ステップS202)。また、ユーザは、全天球カメラで全天球画像ISepを撮影する(ステップS203)。また、ユーザは、DCCツールでCGデータDCgを作成する(ステップS204)。
Next, FIG. 21 is a flowchart showing the processing procedure when outputting a still image. First, a user photographs an RGB image L RGB with an RGB camera (step S201). Further, the user photographs a depth image I Dep with a depth camera (step S202). Further, the user photographs a spherical image I Sep with a spherical camera (step S203). Further, the user creates CG data DCg using the DCC tool (step S204).
そして、画像合成装置10は、ステップS201~S204で取得された各データを入力し、図20に示した画像合成処理を実行する(ステップS205)。
Then, the image synthesis device 10 inputs each data acquired in steps S201 to S204 and executes the image synthesis process shown in FIG. 20 (step S205).
そして、例えば表示装置が、画像合成装置10から出力された出力画像Lendを静止画として表示し(ステップS206)、処理を終了する。
Then, for example, the display device displays the output image L end output from the image synthesis device 10 as a still image (step S206), and the process ends.
次に、図22は、動画を出力する場合の処理手順を示すフローチャートである。なお、図22のステップS301~S304は、図21に示したステップS201~S204と同様であるため、ここでの説明を省略する。
Next, FIG. 22 is a flowchart showing the processing procedure when outputting a moving image. Note that steps S301 to S304 in FIG. 22 are the same as steps S201 to S204 shown in FIG. 21, so the description thereof will be omitted here.
つづいて、ユーザはフレーム番号に応じてCGデータDCgを更新する(ステップS305)。そして、画像合成装置10は、ステップS301~S305で取得された各データを入力し、図20に示した画像合成処理を実行する(ステップS306)。
Subsequently, the user updates the CG data DCg according to the frame number (step S305). Then, the image synthesis device 10 inputs each data acquired in steps S301 to S305, and executes the image synthesis process shown in FIG. 20 (step S306).
そして、画像合成装置10は、画像合成処理によって出力された出力画像Lendが所定のフレーム数に到達したか否かを判定する(ステップS307)。ここで、所定のフレーム数に到達していなければ(ステップS307,No)、フレーム番号を更新し(ステップS308)、ステップS305からの処理を繰り返す。
Then, the image synthesis device 10 determines whether the output image L end outputted by the image synthesis process has reached a predetermined number of frames (step S307). Here, if the predetermined number of frames has not been reached (step S307, No), the frame number is updated (step S308), and the processing from step S305 is repeated.
一方、所定のフレーム数に到達していれば(ステップS307,Yes)、例えば表示装置が、各フレームの出力画像Lendを時系列に結合し、動画として表示する(ステップS309)。そして、処理を終了する。
On the other hand, if the predetermined number of frames has been reached (step S307, Yes), for example, the display device combines the output images L end of each frame in time series and displays it as a moving image (step S309). Then, the process ends.
<<4.変形例>>
上述した本開示の実施形態において説明した各処理のうち、自動的に行われるものとして説明した処理の全部又は一部を手動的に行うこともでき、あるいは、手動的に行われるものとして説明した処理の全部又は一部を公知の方法で自動的に行うこともできる。この他、上記文書中や図面中で示した処理手順、具体的名称、各種のデータやパラメータを含む情報については、特記する場合を除いて任意に変更することができる。例えば、各図に示した各種情報は、図示した情報に限られない。 <<4. Modified example >>
Among the processes described in the embodiments of the present disclosure described above, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed manually. All or part of the processing can also be performed automatically using known methods. In addition, information including the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.
上述した本開示の実施形態において説明した各処理のうち、自動的に行われるものとして説明した処理の全部又は一部を手動的に行うこともでき、あるいは、手動的に行われるものとして説明した処理の全部又は一部を公知の方法で自動的に行うこともできる。この他、上記文書中や図面中で示した処理手順、具体的名称、各種のデータやパラメータを含む情報については、特記する場合を除いて任意に変更することができる。例えば、各図に示した各種情報は、図示した情報に限られない。 <<4. Modified example >>
Among the processes described in the embodiments of the present disclosure described above, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed manually. All or part of the processing can also be performed automatically using known methods. In addition, information including the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.
また、図示した各装置の各構成要素は機能概念的なものであり、必ずしも物理的に図示の如く構成されていることを要しない。すなわち、各装置の分散・統合の具体的形態は図示のものに限られず、その全部又は一部を、各種の負荷や使用状況などに応じて、任意の単位で機能的又は物理的に分散・統合して構成することができる。
Furthermore, each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. In other words, the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.
また、上述した本開示の実施形態は、処理内容を矛盾させない領域で適宜組み合わせることが可能である。また、本実施形態のシーケンス図或いはフローチャートに示された各ステップは、適宜順序を変更することが可能である。
Furthermore, the embodiments of the present disclosure described above can be combined as appropriate in areas where the processing contents do not conflict. Further, the order of each step shown in the sequence diagram or flowchart of this embodiment can be changed as appropriate.
<<5.ハードウェア構成>>
また、上述してきた本開示の実施形態に係る画像合成装置10は、例えば図23に示すような構成のコンピュータ1000によって実現される。図23は、画像合成装置10の機能を実現するコンピュータ1000の一例を示すハードウェア構成図である。コンピュータ1000は、CPU1100、RAM1200、ROM1300、HDD(Hard Disk Drive)1400、通信インターフェイス1500、及び入出力インターフェイス1600を有する。コンピュータ1000の各部は、バス1050によって接続される。 <<5. Hardware configuration >>
Further, theimage synthesis apparatus 10 according to the embodiment of the present disclosure described above is realized by, for example, a computer 1000 having a configuration as shown in FIG. 23. FIG. 23 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the image synthesis apparatus 10. Computer 1000 has CPU 1100, RAM 1200, ROM 1300, HDD (Hard Disk Drive) 1400, communication interface 1500, and input/output interface 1600. Each part of computer 1000 is connected by bus 1050.
また、上述してきた本開示の実施形態に係る画像合成装置10は、例えば図23に示すような構成のコンピュータ1000によって実現される。図23は、画像合成装置10の機能を実現するコンピュータ1000の一例を示すハードウェア構成図である。コンピュータ1000は、CPU1100、RAM1200、ROM1300、HDD(Hard Disk Drive)1400、通信インターフェイス1500、及び入出力インターフェイス1600を有する。コンピュータ1000の各部は、バス1050によって接続される。 <<5. Hardware configuration >>
Further, the
CPU1100は、ROM1300又はHDD1400に格納されたプログラムに基づいて動作し、各部の制御を行う。例えば、CPU1100は、ROM1300又はHDD1400に格納されたプログラムをRAM1200に展開し、各種プログラムに対応した処理を実行する。
The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each part. For example, the CPU 1100 loads programs stored in the ROM 1300 or HDD 1400 into the RAM 1200, and executes processes corresponding to various programs.
ROM1300は、コンピュータ1000の起動時にCPU1100によって実行されるBIOS(Basic Input Output System)等のブートプログラムや、コンピュータ1000のハードウェアに依存するプログラム等を格納する。
The ROM 1300 stores boot programs such as BIOS (Basic Input Output System) that are executed by the CPU 1100 when the computer 1000 is started, programs that depend on the hardware of the computer 1000, and the like.
HDD1400は、CPU1100によって実行されるプログラム、及び、かかるプログラムによって使用されるデータ等を非一時的に記録する、コンピュータが読み取り可能な記録媒体である。具体的には、HDD1400は、プログラムデータ1450の一例である本開示の実施形態に係る情報処理プログラムを記録する記録媒体である。
The HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by the programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to an embodiment of the present disclosure, which is an example of program data 1450.
通信インターフェイス1500は、コンピュータ1000が外部ネットワーク1550(例えばネットワークN)と接続するためのインターフェイスである。例えば、CPU1100は、通信インターフェイス1500を介して、他の機器からデータを受信したり、CPU1100が生成したデータを他の機器へ送信したりする。
Communication interface 1500 is an interface for connecting computer 1000 to external network 1550 (for example, network N). For example, CPU 1100 receives data from other devices or transmits data generated by CPU 1100 to other devices via communication interface 1500.
入出力インターフェイス1600は、入出力デバイス1650とコンピュータ1000とを接続するためのインターフェイスである。例えば、CPU1100は、入出力インターフェイス1600を介して、キーボードやマウス等の入力デバイスからデータを受信する。また、CPU1100は、入出力インターフェイス1600を介して、ディスプレイやスピーカーやプリンタ等の出力デバイスにデータを送信する。また、入出力インターフェイス1600は、所定の記録媒体(メディア)に記録されたプログラム等を読み取るメディアインターフェイスとして機能してもよい。メディアとは、例えばDVD(Digital Versatile Disc)、PD(Phase change rewritable Disk)等の光学記録媒体、MO(Magneto-Optical disk)等の光磁気記録媒体、テープ媒体、磁気記録媒体、または半導体メモリ等である。
The input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input/output interface 1600. Further, the CPU 1100 transmits data to an output device such as a display, speaker, or printer via an input/output interface 1600. Furthermore, the input/output interface 1600 may function as a media interface that reads programs and the like recorded on a predetermined recording medium. Media includes, for example, optical recording media such as DVD (Digital Versatile Disc) and PD (Phase change rewritable disk), magneto-optical recording media such as MO (Magneto-Optical disk), tape media, magnetic recording media, semiconductor memory, etc. It is.
例えば、コンピュータ1000が本開示の実施形態に係る画像合成装置10として機能する場合、コンピュータ1000のCPU1100は、RAM1200上にロードされたプログラムを実行することにより、制御部12の機能を実現する。また、HDD1400には、本開示に係る情報処理プログラムや、記憶部11内のデータが格納される。なお、CPU1100は、プログラムデータ1450をHDD1400から読み取って実行するが、他の例として、外部ネットワーク1550を介して、他の装置からこれらのプログラムを取得してもよい。
For example, when the computer 1000 functions as the image composition device 10 according to the embodiment of the present disclosure, the CPU 1100 of the computer 1000 realizes the functions of the control unit 12 by executing a program loaded onto the RAM 1200. Further, the HDD 1400 stores an information processing program according to the present disclosure and data in the storage unit 11. Note that although the CPU 1100 reads and executes the program data 1450 from the HDD 1400, as another example, these programs may be obtained from another device via the external network 1550.
<<6.むすび>>
以上説明したように、本開示の一実施形態によれば、画像合成装置10(「情報処理装置」の一例に相当)は、RGB画像LRGB、デプス画像IDepおよび全天球画像ISepを入力としてRGB画像LRGBに対応する実空間を表す3次元空間にCGデータDCgを配置する合成処理を行う合成処理部12cを備える。 <<6. Conclusion >>
As described above, according to an embodiment of the present disclosure, the image synthesis device 10 (corresponding to an example of an "information processing device") generates an RGB image L RGB , a depth image I Dep , and a spherical image I Sep. Acomposition processing unit 12c is provided which performs composition processing of arranging CG data DCg in a three-dimensional space representing a real space corresponding to the RGB image L RGB as an input.
以上説明したように、本開示の一実施形態によれば、画像合成装置10(「情報処理装置」の一例に相当)は、RGB画像LRGB、デプス画像IDepおよび全天球画像ISepを入力としてRGB画像LRGBに対応する実空間を表す3次元空間にCGデータDCgを配置する合成処理を行う合成処理部12cを備える。 <<6. Conclusion >>
As described above, according to an embodiment of the present disclosure, the image synthesis device 10 (corresponding to an example of an "information processing device") generates an RGB image L RGB , a depth image I Dep , and a spherical image I Sep. A
合成処理部12cは、上記入力から測距された実写情報のみを用いたPY画像LPY(「第1のレンダリング画像」の一例に相当)と、上記実写情報およびCGデータDCgを用いたVI画像LVI(「第2のレンダリング画像」の一例に相当)とを生成する。
The synthesis processing unit 12c generates a PY image L PY (corresponding to an example of a "first rendered image") using only the real shot information measured from the above input, and a VI using the above real shot information and CG data D Cg . An image LVI (corresponding to an example of a "second rendered image") is generated.
また、合成処理部12cは、PY画像LPYおよびVI画像LVIの陰影の差に基づくADR法(「第1の手法」の一例に相当)によるADR画像LADR(「第1の合成画像」の一例に相当)と、上記陰影の比率に基づくQI法(「第2の手法」の一例に相当)によるQI画像LQI(「第2の合成画像」の一例に相当)とを生成する。
In addition, the synthesis processing unit 12c generates an ADR image L ADR (“first composite image”) based on the ADR method (corresponding to an example of the “first method”) based on the difference in shading between the PY image L PY and the VI image L VI . ) and a QI image L QI (corresponding to an example of a "second composite image") by the QI method (corresponding to an example of a "second method") based on the shading ratio.
また、合成処理部12cは、上記実写情報のみを用いた場合と、上記実写情報およびCGデータDCgを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像SIを生成し、シャドウインパクト画像SIを用いてADR画像LADRとQI画像LQIとの線形結合を行う。
In addition, the composition processing unit 12c generates a shadow impact image S I representing a change in optical radiation energy between the case where only the above-mentioned live-action information is used and the case where the above-mentioned live-action information and CG data DCg are used, A linear combination of the ADR image L ADR and the QI image L QI is performed using the shadow impact image S I.
これにより、実空間を表す3次元空間に対するCGデータDCgの合成品質をより向上させることができる。
This makes it possible to further improve the quality of compositing the CG data DCg with respect to the three-dimensional space representing the real space.
以上、本開示の各実施形態について説明したが、本開示の技術的範囲は、上述の各実施形態そのままに限定されるものではなく、本開示の要旨を逸脱しない範囲において種々の変更が可能である。また、異なる実施形態及び変形例にわたる構成要素を適宜組み合わせてもよい。
Although each embodiment of the present disclosure has been described above, the technical scope of the present disclosure is not limited to each of the above-mentioned embodiments as is, and various changes can be made without departing from the gist of the present disclosure. be. Furthermore, components of different embodiments and modifications may be combined as appropriate.
また、本明細書に記載された各実施形態における効果はあくまで例示であって限定されるものでは無く、他の効果があってもよい。
Further, the effects in each embodiment described in this specification are merely examples and are not limited, and other effects may also be provided.
なお、本技術は以下のような構成も取ることができる。
(1)
RGB画像、デプス画像および全天球画像を入力として前記RGB画像に対応する実空間を表す3次元空間にCGデータを配置する合成処理を行う合成処理部
を備え、
前記合成処理部は、
前記入力から測距された実写情報のみを用いた第1のレンダリング画像と、前記実写情報および前記CGデータを用いた第2のレンダリング画像とを生成し、
前記第1のレンダリング画像および前記第2のレンダリング画像の陰影の差に基づく第1の手法による第1の合成画像と、前記陰影の比率に基づく第2の手法による第2の合成画像とを生成し、
前記実写情報のみを用いた場合と、前記実写情報および前記CGデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成し、
前記シャドウインパクト画像を用いて前記第1の合成画像と前記第2の合成画像との線形結合を行う、
情報処理装置。
(2)
前記合成処理部は、
前記合成処理により光放射エネルギーが増加した領域では低い画素値を有し、光放射エネルギーが減少した領域では高い画素値を有するように前記シャドウインパクト画像を生成し、前記低い画素値の領域には前記第1の合成画像を、前記高い画素値の領域には前記第2の合成画像をそれぞれ割り当てる、
前記(1)に記載の情報処理装置。
(3)
前記合成処理部は、
前記低い画素値の領域は前記合成処理により明るくなる前記RGB画像の領域に対応し、前記高い画素値の領域は前記合成処理により暗くなる前記RGB画像の領域に対応するように前記シャドウインパクト画像を生成する、
前記(2)に記載の情報処理装置。
(4)
前記合成処理部は少なくとも、
測距された前記全天球画像から算出される照明に関する情報、および、前記CGデータの可視性に関する情報に基づいて、前記シャドウインパクト画像を生成する、
前記(1)、(2)または(3)に記載の情報処理装置。
(5)
前記合成処理部は、
前記照明に関する情報および前記可視性に関する情報のみに基づいて、前記シャドウインパクト画像を生成する、
前記(4)に記載の情報処理装置。
(6)
前記合成処理部は、
前記シャドウインパクト画像を用いて、前記合成処理において生じるアーチファクトを除去する、
前記(1)~(5)のいずれか一つに記載の情報処理装置。
(7)
前記合成処理部は、
前記第1のレンダリング画像が前記RGB画像よりも極めて暗くなるライトギャップ領域を示すライトギャップ画像を生成し、前記ライトギャップ領域に対し前記第2のレンダリング画像を線形結合によって優先的に割り当てることで前記アーチファクトを除去する、
前記(6)に記載の情報処理装置。
(8)
前記合成処理部は、
前記第1のレンダリング画像と前記RGB画像の差分により表現される測距誤差が極めて大きい領域を前記ライトギャップ領域として検出し、さらに前記シャドウインパクト画像を掛けることによって、前記アーチファクトの除去が及ぶ領域を限定する、
前記(7)に記載の情報処理装置。
(9)
RGB画像、デプス画像および全天球画像を入力として前記RGB画像に対応する実空間を表す3次元空間にCGデータを配置する合成処理を行うこと、
を含み、
前記合成処理を行うことは、
前記入力から測距された実写情報のみを用いた第1のレンダリング画像と、前記実写情報および前記CGデータを用いた第2のレンダリング画像とを生成することと、
前記第1のレンダリング画像および前記第2のレンダリング画像の陰影の差に基づく第1の手法による第1の合成画像と、前記陰影の比率に基づく第2の手法による第2の合成画像とを生成することと、
前記実写情報のみを用いた場合と、前記実写情報および前記CGデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成することと、
前記シャドウインパクト画像を用いて前記第1の合成画像と前記第2の合成画像との線形結合を行うことと、
をさらに含む、情報処理方法。
(10)
RGB画像、デプス画像および全天球画像を入力として前記RGB画像に対応する実空間を表す3次元空間にCGデータを配置する合成処理を行うこと、
をコンピュータに実行させ、
前記合成処理を行うことは、
前記入力から測距された実写情報のみを用いた第1のレンダリング画像と、前記実写情報および前記CGデータを用いた第2のレンダリング画像とを生成すること、
前記第1のレンダリング画像および前記第2のレンダリング画像の陰影の差に基づく第1の手法による第1の合成画像と、前記陰影の比率に基づく第2の手法による第2の合成画像とを生成すること、
前記実写情報のみを用いた場合と、前記実写情報および前記CGデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成すること、
前記シャドウインパクト画像を用いて前記第1の合成画像と前記第2の合成画像との線形結合を行うこと、
をさらに前記コンピュータに実行させる情報処理プログラムが記録されたコンピュータ読み取り可能な記録媒体。 Note that the present technology can also have the following configuration.
(1)
a composition processing unit that receives an RGB image, a depth image, and a spherical image as input and performs a composition process of arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
The synthesis processing section is
Generating a first rendered image using only the live-action information measured from the input, and a second rendering image using the live-action information and the CG data,
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. death,
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
Information processing device.
(2)
The synthesis processing section is
The shadow impact image is generated so that areas where light radiant energy has increased through the synthesis process have low pixel values, and areas where light radiant energy has decreased have high pixel values, and the areas with low pixel values have low pixel values. assigning the first composite image and the second composite image to the high pixel value area, respectively;
The information processing device according to (1) above.
(3)
The synthesis processing section is
The shadow impact image is created such that the low pixel value area corresponds to the area of the RGB image that becomes brighter due to the compositing process, and the high pixel value area corresponds to an area of the RGB image that becomes dark due to the compositing process. generate,
The information processing device according to (2) above.
(4)
The synthesis processing section at least includes:
generating the shadow impact image based on information regarding illumination calculated from the distance-measured spherical image and information regarding visibility of the CG data;
The information processing device according to (1), (2) or (3) above.
(5)
The synthesis processing section is
generating the shadow impact image based only on the illumination information and the visibility information;
The information processing device according to (4) above.
(6)
The synthesis processing section is
using the shadow impact image to remove artifacts occurring in the compositing process;
The information processing device according to any one of (1) to (5) above.
(7)
The synthesis processing section is
By generating a light gap image indicating a light gap region in which the first rendered image is much darker than the RGB image, and preferentially allocating the second rendered image to the light gap region by linear combination. remove artifacts,
The information processing device according to (6) above.
(8)
The synthesis processing section is
An area where the distance measurement error expressed by the difference between the first rendered image and the RGB image is extremely large is detected as the light gap area, and the area where the artifacts are removed is determined by multiplying the area by the shadow impact image. limit,
The information processing device according to (7) above.
(9)
performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
including;
Performing the above-mentioned compositing process includes:
Generating a first rendered image using only the live-action information measured from the input, and a second rendered image using the live-action information and the CG data;
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. to do and
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
Information processing methods, further including:
(10)
performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
make the computer run
Performing the above-mentioned compositing process includes:
generating a first rendered image using only live-action information measured from the input, and a second rendered image using the live-action information and the CG data;
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. to do,
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
A computer-readable recording medium having recorded thereon an information processing program that causes the computer to further execute the following.
(1)
RGB画像、デプス画像および全天球画像を入力として前記RGB画像に対応する実空間を表す3次元空間にCGデータを配置する合成処理を行う合成処理部
を備え、
前記合成処理部は、
前記入力から測距された実写情報のみを用いた第1のレンダリング画像と、前記実写情報および前記CGデータを用いた第2のレンダリング画像とを生成し、
前記第1のレンダリング画像および前記第2のレンダリング画像の陰影の差に基づく第1の手法による第1の合成画像と、前記陰影の比率に基づく第2の手法による第2の合成画像とを生成し、
前記実写情報のみを用いた場合と、前記実写情報および前記CGデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成し、
前記シャドウインパクト画像を用いて前記第1の合成画像と前記第2の合成画像との線形結合を行う、
情報処理装置。
(2)
前記合成処理部は、
前記合成処理により光放射エネルギーが増加した領域では低い画素値を有し、光放射エネルギーが減少した領域では高い画素値を有するように前記シャドウインパクト画像を生成し、前記低い画素値の領域には前記第1の合成画像を、前記高い画素値の領域には前記第2の合成画像をそれぞれ割り当てる、
前記(1)に記載の情報処理装置。
(3)
前記合成処理部は、
前記低い画素値の領域は前記合成処理により明るくなる前記RGB画像の領域に対応し、前記高い画素値の領域は前記合成処理により暗くなる前記RGB画像の領域に対応するように前記シャドウインパクト画像を生成する、
前記(2)に記載の情報処理装置。
(4)
前記合成処理部は少なくとも、
測距された前記全天球画像から算出される照明に関する情報、および、前記CGデータの可視性に関する情報に基づいて、前記シャドウインパクト画像を生成する、
前記(1)、(2)または(3)に記載の情報処理装置。
(5)
前記合成処理部は、
前記照明に関する情報および前記可視性に関する情報のみに基づいて、前記シャドウインパクト画像を生成する、
前記(4)に記載の情報処理装置。
(6)
前記合成処理部は、
前記シャドウインパクト画像を用いて、前記合成処理において生じるアーチファクトを除去する、
前記(1)~(5)のいずれか一つに記載の情報処理装置。
(7)
前記合成処理部は、
前記第1のレンダリング画像が前記RGB画像よりも極めて暗くなるライトギャップ領域を示すライトギャップ画像を生成し、前記ライトギャップ領域に対し前記第2のレンダリング画像を線形結合によって優先的に割り当てることで前記アーチファクトを除去する、
前記(6)に記載の情報処理装置。
(8)
前記合成処理部は、
前記第1のレンダリング画像と前記RGB画像の差分により表現される測距誤差が極めて大きい領域を前記ライトギャップ領域として検出し、さらに前記シャドウインパクト画像を掛けることによって、前記アーチファクトの除去が及ぶ領域を限定する、
前記(7)に記載の情報処理装置。
(9)
RGB画像、デプス画像および全天球画像を入力として前記RGB画像に対応する実空間を表す3次元空間にCGデータを配置する合成処理を行うこと、
を含み、
前記合成処理を行うことは、
前記入力から測距された実写情報のみを用いた第1のレンダリング画像と、前記実写情報および前記CGデータを用いた第2のレンダリング画像とを生成することと、
前記第1のレンダリング画像および前記第2のレンダリング画像の陰影の差に基づく第1の手法による第1の合成画像と、前記陰影の比率に基づく第2の手法による第2の合成画像とを生成することと、
前記実写情報のみを用いた場合と、前記実写情報および前記CGデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成することと、
前記シャドウインパクト画像を用いて前記第1の合成画像と前記第2の合成画像との線形結合を行うことと、
をさらに含む、情報処理方法。
(10)
RGB画像、デプス画像および全天球画像を入力として前記RGB画像に対応する実空間を表す3次元空間にCGデータを配置する合成処理を行うこと、
をコンピュータに実行させ、
前記合成処理を行うことは、
前記入力から測距された実写情報のみを用いた第1のレンダリング画像と、前記実写情報および前記CGデータを用いた第2のレンダリング画像とを生成すること、
前記第1のレンダリング画像および前記第2のレンダリング画像の陰影の差に基づく第1の手法による第1の合成画像と、前記陰影の比率に基づく第2の手法による第2の合成画像とを生成すること、
前記実写情報のみを用いた場合と、前記実写情報および前記CGデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成すること、
前記シャドウインパクト画像を用いて前記第1の合成画像と前記第2の合成画像との線形結合を行うこと、
をさらに前記コンピュータに実行させる情報処理プログラムが記録されたコンピュータ読み取り可能な記録媒体。 Note that the present technology can also have the following configuration.
(1)
a composition processing unit that receives an RGB image, a depth image, and a spherical image as input and performs a composition process of arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
The synthesis processing section is
Generating a first rendered image using only the live-action information measured from the input, and a second rendering image using the live-action information and the CG data,
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. death,
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
Information processing device.
(2)
The synthesis processing section is
The shadow impact image is generated so that areas where light radiant energy has increased through the synthesis process have low pixel values, and areas where light radiant energy has decreased have high pixel values, and the areas with low pixel values have low pixel values. assigning the first composite image and the second composite image to the high pixel value area, respectively;
The information processing device according to (1) above.
(3)
The synthesis processing section is
The shadow impact image is created such that the low pixel value area corresponds to the area of the RGB image that becomes brighter due to the compositing process, and the high pixel value area corresponds to an area of the RGB image that becomes dark due to the compositing process. generate,
The information processing device according to (2) above.
(4)
The synthesis processing section at least includes:
generating the shadow impact image based on information regarding illumination calculated from the distance-measured spherical image and information regarding visibility of the CG data;
The information processing device according to (1), (2) or (3) above.
(5)
The synthesis processing section is
generating the shadow impact image based only on the illumination information and the visibility information;
The information processing device according to (4) above.
(6)
The synthesis processing section is
using the shadow impact image to remove artifacts occurring in the compositing process;
The information processing device according to any one of (1) to (5) above.
(7)
The synthesis processing section is
By generating a light gap image indicating a light gap region in which the first rendered image is much darker than the RGB image, and preferentially allocating the second rendered image to the light gap region by linear combination. remove artifacts,
The information processing device according to (6) above.
(8)
The synthesis processing section is
An area where the distance measurement error expressed by the difference between the first rendered image and the RGB image is extremely large is detected as the light gap area, and the area where the artifacts are removed is determined by multiplying the area by the shadow impact image. limit,
The information processing device according to (7) above.
(9)
performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
including;
Performing the above-mentioned compositing process includes:
Generating a first rendered image using only the live-action information measured from the input, and a second rendered image using the live-action information and the CG data;
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. to do and
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
Information processing methods, further including:
(10)
performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
make the computer run
Performing the above-mentioned compositing process includes:
generating a first rendered image using only live-action information measured from the input, and a second rendered image using the live-action information and the CG data;
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. to do,
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
A computer-readable recording medium having recorded thereon an information processing program that causes the computer to further execute the following.
10 画像合成装置
11 記憶部
11a ジオメトリ情報
11b 反射率情報
11c 照明マップ情報
11d DCCツールプログラム
12 制御部
12a 取得部
12b 変換部
12c 合成処理部
12ca 第1生成部
12cb 第2生成部
12cc 第3生成部
12cd 第4生成部
12ce 第5生成部
12cf 出力画像生成部
12d 出力部 10Image synthesis device 11 Storage unit 11a Geometry information 11b Reflectance information 11c Illumination map information 11d DCC tool program 12 Control unit 12a Acquisition unit 12b Conversion unit 12c Synthesis processing unit 12ca First generation unit 12cb Second generation unit 12cc Third generation unit 12cd Fourth generation section 12ce Fifth generation section 12cf Output image generation section 12d Output section
11 記憶部
11a ジオメトリ情報
11b 反射率情報
11c 照明マップ情報
11d DCCツールプログラム
12 制御部
12a 取得部
12b 変換部
12c 合成処理部
12ca 第1生成部
12cb 第2生成部
12cc 第3生成部
12cd 第4生成部
12ce 第5生成部
12cf 出力画像生成部
12d 出力部 10
Claims (10)
- RGB画像、デプス画像および全天球画像を入力として前記RGB画像に対応する実空間を表す3次元空間にCGデータを配置する合成処理を行う合成処理部
を備え、
前記合成処理部は、
前記入力から測距された実写情報のみを用いた第1のレンダリング画像と、前記実写情報および前記CGデータを用いた第2のレンダリング画像とを生成し、
前記第1のレンダリング画像および前記第2のレンダリング画像の陰影の差に基づく第1の手法による第1の合成画像と、前記陰影の比率に基づく第2の手法による第2の合成画像とを生成し、
前記実写情報のみを用いた場合と、前記実写情報および前記CGデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成し、
前記シャドウインパクト画像を用いて前記第1の合成画像と前記第2の合成画像との線形結合を行う、
情報処理装置。 a composition processing unit that receives an RGB image, a depth image, and a spherical image as input and performs a composition process of arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
The synthesis processing section is
Generating a first rendered image using only the live-action information measured from the input, and a second rendering image using the live-action information and the CG data,
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. death,
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
Information processing device. - 前記合成処理部は、
前記合成処理により光放射エネルギーが増加した領域では低い画素値を有し、光放射エネルギーが減少した領域では高い画素値を有するように前記シャドウインパクト画像を生成し、前記低い画素値の領域には前記第1の合成画像を、前記高い画素値の領域には前記第2の合成画像をそれぞれ割り当てる、
請求項1に記載の情報処理装置。 The synthesis processing section is
The shadow impact image is generated so that areas where light radiant energy has increased through the synthesis process have low pixel values, and areas where light radiant energy has decreased have high pixel values, and the areas with low pixel values have low pixel values. assigning the first composite image and the second composite image to the high pixel value area, respectively;
The information processing device according to claim 1. - 前記合成処理部は、
前記低い画素値の領域は前記合成処理により明るくなる前記RGB画像の領域に対応し、前記高い画素値の領域は前記合成処理により暗くなる前記RGB画像の領域に対応するように前記シャドウインパクト画像を生成する、
請求項2に記載の情報処理装置。 The synthesis processing section is
The shadow impact image is created such that the low pixel value area corresponds to the area of the RGB image that becomes brighter due to the compositing process, and the high pixel value area corresponds to an area of the RGB image that becomes dark due to the compositing process. generate,
The information processing device according to claim 2. - 前記合成処理部は少なくとも、
測距された前記全天球画像から算出される照明に関する情報、および、前記CGデータの可視性に関する情報に基づいて、前記シャドウインパクト画像を生成する、
請求項1に記載の情報処理装置。 The synthesis processing section at least includes:
generating the shadow impact image based on information regarding illumination calculated from the distance-measured spherical image and information regarding visibility of the CG data;
The information processing device according to claim 1. - 前記合成処理部は、
前記照明に関する情報および前記可視性に関する情報のみに基づいて、前記シャドウインパクト画像を生成する、
請求項4に記載の情報処理装置。 The synthesis processing section is
generating the shadow impact image based only on the illumination information and the visibility information;
The information processing device according to claim 4. - 前記合成処理部は、
前記シャドウインパクト画像を用いて、前記合成処理において生じるアーチファクトを除去する、
請求項1に記載の情報処理装置。 The synthesis processing section is
using the shadow impact image to remove artifacts occurring in the compositing process;
The information processing device according to claim 1. - 前記合成処理部は、
前記第1のレンダリング画像が前記RGB画像よりも極めて暗くなるライトギャップ領域を示すライトギャップ画像を生成し、前記ライトギャップ領域に対し前記第2のレンダリング画像を線形結合によって優先的に割り当てることで前記アーチファクトを除去する、
請求項6に記載の情報処理装置。 The synthesis processing section is
By generating a light gap image indicating a light gap region in which the first rendered image is much darker than the RGB image, and preferentially allocating the second rendered image to the light gap region by linear combination. remove artifacts,
The information processing device according to claim 6. - 前記合成処理部は、
前記第1のレンダリング画像と前記RGB画像の差分により表現される測距誤差が極めて大きい領域を前記ライトギャップ領域として検出し、さらに前記シャドウインパクト画像を掛けることによって、前記アーチファクトの除去が及ぶ領域を限定する、
請求項7に記載の情報処理装置。 The synthesis processing section is
An area where the distance measurement error expressed by the difference between the first rendered image and the RGB image is extremely large is detected as the light gap area, and the area where the artifacts are removed is determined by multiplying the area by the shadow impact image. limit,
The information processing device according to claim 7. - RGB画像、デプス画像および全天球画像を入力として前記RGB画像に対応する実空間を表す3次元空間にCGデータを配置する合成処理を行うこと、
を含み、
前記合成処理を行うことは、
前記入力から測距された実写情報のみを用いた第1のレンダリング画像と、前記実写情報および前記CGデータを用いた第2のレンダリング画像とを生成することと、
前記第1のレンダリング画像および前記第2のレンダリング画像の陰影の差に基づく第1の手法による第1の合成画像と、前記陰影の比率に基づく第2の手法による第2の合成画像とを生成することと、
前記実写情報のみを用いた場合と、前記実写情報および前記CGデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成することと、
前記シャドウインパクト画像を用いて前記第1の合成画像と前記第2の合成画像との線形結合を行うことと、
を含む、情報処理方法。 performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
including;
Performing the above-mentioned compositing process includes:
Generating a first rendered image using only the live-action information measured from the input, and a second rendered image using the live-action information and the CG data;
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. to do and
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
information processing methods, including - RGB画像、デプス画像および全天球画像を入力として前記RGB画像に対応する実空間を表す3次元空間にCGデータを配置する合成処理を行うこと、
をコンピュータに実行させ、
前記合成処理を行うことは、
前記入力から測距された実写情報のみを用いた第1のレンダリング画像と、前記実写情報および前記CGデータを用いた第2のレンダリング画像とを生成すること、
前記第1のレンダリング画像および前記第2のレンダリング画像の陰影の差に基づく第1の手法による第1の合成画像と、前記陰影の比率に基づく第2の手法による第2の合成画像とを生成すること、
前記実写情報のみを用いた場合と、前記実写情報および前記CGデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成すること、
前記シャドウインパクト画像を用いて前記第1の合成画像と前記第2の合成画像との線形結合を行うこと、
をさらに前記コンピュータに実行させる情報処理プログラムが記録されたコンピュータ読み取り可能な記録媒体。 performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
make the computer run
Performing the above-mentioned compositing process includes:
generating a first rendered image using only live-action information measured from the input, and a second rendered image using the live-action information and the CG data;
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. to do,
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
A computer-readable recording medium having recorded thereon an information processing program that causes the computer to further execute the following.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2022048330 | 2022-03-24 | ||
JP2022-048330 | 2022-03-24 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023181904A1 true WO2023181904A1 (en) | 2023-09-28 |
Family
ID=88100750
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2023/008482 WO2023181904A1 (en) | 2022-03-24 | 2023-03-07 | Information processing device, information processing method, and recording medium |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2023181904A1 (en) |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009163610A (en) * | 2008-01-09 | 2009-07-23 | Canon Inc | Image processing apparatus and image processing method |
JP2013127774A (en) * | 2011-11-16 | 2013-06-27 | Canon Inc | Image processing device, image processing method, and program |
-
2023
- 2023-03-07 WO PCT/JP2023/008482 patent/WO2023181904A1/en unknown
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009163610A (en) * | 2008-01-09 | 2009-07-23 | Canon Inc | Image processing apparatus and image processing method |
JP2013127774A (en) * | 2011-11-16 | 2013-06-27 | Canon Inc | Image processing device, image processing method, and program |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CA2810865C (en) | Streaming light propagation | |
AU2006318379B2 (en) | Methods and apparatus for determining high quality sampling data from low quality sampling data | |
JP6639877B2 (en) | Image processing apparatus and method | |
US10572984B2 (en) | Method for inverse tone mapping of an image with visual effects | |
US10388063B2 (en) | Variable rate shading based on temporal reprojection | |
JP2010510571A (en) | Method and system for color correction of 3D image | |
JP5731566B2 (en) | Information processing apparatus, control method, and recording medium | |
US10529125B2 (en) | Image processing device and method therefor | |
EP3355279A1 (en) | Method and computer programs for identifying differences between 3-dimensional scenes | |
CN113920036A (en) | Interactive relighting editing method based on RGB-D image | |
AU2023233092A1 (en) | Artificial intelligence techniques for extrapolating hdr panoramas from ldr low fov images | |
WO2023181904A1 (en) | Information processing device, information processing method, and recording medium | |
US8942476B1 (en) | Saturation varying and lighting independent color color control for computer graphics | |
KR102448551B1 (en) | Method and apparatus processing images by using algorithm configured by node based graph | |
Gierlinger et al. | Rendering techniques for mixed reality | |
Dai | Stylized rendering for virtual furniture layout | |
Colbert et al. | Painting in high dynamic range | |
JP7116767B2 (en) | Image processing device and method | |
Ludwig et al. | Environment map based lighting for reflectance transformation images | |
US11380048B2 (en) | Method and system for determining a spectral representation of a color | |
Koch et al. | Semantic Screen-Space Occlusion for Multiscale Molecular Visualization. | |
Qi et al. | Denoising 1SPP Monte carlo renderings based on human visual perception | |
Sharif et al. | Halo reduction multi-exposure image fusion technique | |
GB2624103A (en) | Artificial intelligence techniques for extrapolating HDR panoramas from LDR low FOV images | |
FRASER et al. | PROJECT PROPOSAL: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23774503 Country of ref document: EP Kind code of ref document: A1 |