WO2023181904A1 - Information processing device, information processing method, and recording medium - Google Patents

Information processing device, information processing method, and recording medium Download PDF

Info

Publication number
WO2023181904A1
WO2023181904A1 PCT/JP2023/008482 JP2023008482W WO2023181904A1 WO 2023181904 A1 WO2023181904 A1 WO 2023181904A1 JP 2023008482 W JP2023008482 W JP 2023008482W WO 2023181904 A1 WO2023181904 A1 WO 2023181904A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
rgb
information
data
synthesis
Prior art date
Application number
PCT/JP2023/008482
Other languages
French (fr)
Japanese (ja)
Inventor
健太郎 深水
Original Assignee
ソニーグループ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーグループ株式会社 filed Critical ソニーグループ株式会社
Publication of WO2023181904A1 publication Critical patent/WO2023181904A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/50—Lighting effects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules

Definitions

  • the present disclosure relates to an information processing device, an information processing method, and a recording medium.
  • the synthesis quality of the ADR method and the QI method has advantages and disadvantages depending on the situation of the actual background, so in many cases, the user needs to manually select which method to use depending on the content of the background.
  • the algorithms of these methods assume that the shadow area cast by CG on the real image is only a planar shape, so it is difficult to apply shadows to areas where many three-dimensional shapes intersect in a complex manner and cause occlusion. When applied, there is also the problem that it is strongly influenced by noise and is reflected as an artifact.
  • the present disclosure proposes an information processing device, an information processing method, and a recording medium that can further improve the quality of synthesizing CG data into a three-dimensional space representing a real space.
  • an information processing device inputs an RGB image, a depth image, and a spherical image and converts CG data into a three-dimensional space representing a real space corresponding to the RGB image.
  • a compositing processing unit is provided that performs compositing processing for arranging.
  • the synthesis processing unit generates a first rendered image using only the real-life information measured from the input, and a second rendered image using the real-life information and the CG data, and A first composite image by a first method based on the difference in shading between the rendered image and the second rendered image and a second composite image by a second method based on the ratio of the shading are generated, A shadow impact image representing a change in optical radiation energy between the case of using only the information and the case of using the live-action information and the CG data is generated, and the first composite image is generated using the shadow impact image. and the second composite image are linearly combined.
  • FIG. 1 is a schematic explanatory diagram of an image synthesis method according to an embodiment of the present disclosure.
  • FIG. 1 is a block diagram illustrating a configuration example of an image synthesis device according to an embodiment of the present disclosure.
  • FIG. 2 is a block diagram showing an example of the configuration of a composition processing section.
  • FIG. 3 is a diagram showing an example of an output image.
  • FIG. 3 is a diagram showing an example of an RGB image. It is a figure showing an example of CG data. It is a figure showing an example of a depth image. It is a figure showing an example of a PY image.
  • FIG. 3 is a diagram showing an example of a VI image. It is a figure showing an example of a mask image.
  • FIG. 1 is a block diagram illustrating a configuration example of an image synthesis device according to an embodiment of the present disclosure.
  • FIG. 2 is a block diagram showing an example of the configuration of a composition processing section.
  • FIG. 3 is a diagram showing an example
  • FIG. 3 is a diagram showing an example of an RPY image. It is a figure showing an example of an RVI image.
  • FIG. 3 is a diagram showing an example of a shadow impact image. It is a figure showing an example of an ADR image.
  • FIG. 2 is a diagram (part 1) showing an example of a QI image.
  • FIG. 2 is a diagram (part 2) showing an example of a QI image.
  • FIG. 2 is a diagram (part 1) showing an example of a light gap image.
  • FIG. 2 is a diagram (part 2) showing an example of a light gap image.
  • FIG. 7 is a diagram showing the effect of correction using a light gap image.
  • 3 is a flowchart illustrating a processing procedure executed by the image compositing device.
  • FIG. 2 is a hardware configuration diagram showing an example of a computer that implements the functions of an image synthesis device.
  • the information processing device is the image composition device 10.
  • the information processing method according to the embodiment of the present disclosure is an image synthesis method.
  • This information is obtained by processing the RGB image L RGB captured in real space, the depth image I Dep , and the spherical image I Sep using general DCC (Digital Content Creation) tools such as Maya and Blender. be able to.
  • DCC Digital Content Creation
  • Both the ADR method and the QI method generate three images, a PY image L PY , a VI image L VI , and a mask image M, using the aforementioned geometry, reflectance, illumination map, and CG data to be synthesized.
  • PY image L PY is an image rendered using only actual photographic information measured from a depth image.
  • VI image L VI is an image rendered using both real-shot information and CG data to be synthesized.
  • the mask image M is an image indicating an area where CG data to be synthesized exists.
  • the RGB image L RGB , the PY image L PY , the VI image L VI and the mask image M are input to the synthesis algorithm in the ADR method and the QI method, and each of these algorithms has advantages and disadvantages in terms of their structure.
  • an ADR image L ADR is calculated based on the following equation (1).
  • a QI image L QI is calculated based on the following equation (3).
  • the QI method can add shadows to the real background by multiplying the input RGB image L RGB by the ratio of shadows L VI /L PY produced by the compositing process of the CG data D Cg . can.
  • the second term of equation (3) is actually operated by the following equation (4) in order to prevent the synthesis result from becoming unstable due to division by zero.
  • ⁇ is a constant to prevent division by zero.
  • L QI is limited to a maximum of 1.01 times L PY .
  • the QI method has a problem in that there is an upper limit to the range in which a live photograph can be brightened by, for example, combining processing of emitting CG data DCg .
  • equation (3) due to the nature of equation (3), if the input depth image IDep is of low quality and has a large error with respect to the true value, an operation extremely similar to division by zero will occur in the second term of equation (4). There is a high possibility that this will occur, and in that case, there is a problem that it will be reflected in the QI image LQI as an artifact.
  • such a region is referred to as a light-gap region.
  • the light gap region occurs when the distance measurement quality of the depth image IDep is low quality and the three-dimensional object shielding environment in real space cannot be accurately acquired.
  • the following equation (7) automatically holds true unless a virtual object that emits light is included in the CG data D Cg .
  • Equation (8) Equation (8) below holds true, and the desired output cannot be expected in the region where shading is to be added to the RGB image L RGB by the compositing process.
  • Equation (9) Equation (9) below holds true, causing a phenomenon similar to that of the ADR method.
  • the technical issues of both the ADR method and the QI method are summarized as follows.
  • the ADR method there are restrictions on the expression of darkening the RGB image L RGB by the compositing process of the CG data D Cg .
  • the ADR method tends to cause artifacts due to the low quality ranging quality of reflectance.
  • artifacts in the light gap region become noticeable.
  • the QI method there are restrictions on the expression of brightening the RGB image L RGB by the synthesis process of the CG data D Cg .
  • the QI method tends to cause artifacts due to the low ranging quality of the depth image IDep .
  • artifacts in the light gap region become noticeable.
  • FIG. 1 is a schematic explanatory diagram of an image synthesis method according to an embodiment of the present disclosure.
  • the image synthesis device 10 inputs the RGB image L RGB , the depth image I Dep , and the spherical image I Sep and converts the RGB image L RGB into an RGB image L RGB .
  • a synthesis process is performed to arrange the CG data D Cg in a three-dimensional space representing the corresponding real space.
  • the image compositing device 10 combines the PY image LPY , which is a rendered image using only the actual photographic information measured from the input, and the aforementioned photographic information and CG data DCG .
  • a VI image LVI which is the rendered image used, is generated (steps S1-1, S1-2).
  • PY image L PY corresponds to an example of a "first rendered image.”
  • VI image L VI corresponds to an example of a "second rendered image.”
  • the image synthesis device 10 also generates an ADR image L ADR, which is a synthesized image based on the ADR method based on the difference in shading between the PY image L PY and a VI image L VI , and a QI image, which is a synthesized image based on the QI method based on the ratio of the shadings.
  • Images L and QI are generated (steps S2-1 and S2-2).
  • ADR image L ADR corresponds to an example of a "first composite image.”
  • QI image L QI corresponds to an example of a "second composite image.”
  • the image synthesis device 10 also generates a shadow impact image SI representing a change in light radiation energy between the case where only the above-mentioned real-time photograph information is used and the case where the above-mentioned live-action information and CG data DCg are used. Step S3).
  • the image synthesis device 10 also performs a linear combination of the ADR image LADR and the QI image LQI using the shadow impact image S I to generate an LC image L LC (step S4).
  • the RGB image L RGB , the depth image I Dep , and the spherical image I Sep are input, and CG is created in a three-dimensional space representing the real space corresponding to the RGB image L RGB .
  • the ADR image L ADR generated by the ADR method and the QI image L QI generated by the QI method are linearly combined using the shadow impact image S I generated by the new algorithm of the present disclosure, and The advantages of the and QI methods are adaptively mixed for each image pixel.
  • the image synthesis method according to the embodiment of the present disclosure it is possible to further improve the synthesis quality of CG data DCg with respect to a three-dimensional space representing a real space.
  • a configuration example of the image synthesis apparatus 10 to which the image synthesis method according to the embodiment of the present disclosure is applied will be described in more detail.
  • FIG. 2 is a block diagram illustrating a configuration example of the image synthesis device 10 according to the embodiment of the present disclosure. Note that FIG. 2 and FIG. 3 shown later show only the constituent elements necessary for explaining the features of the embodiment of the present disclosure, and descriptions of general constituent elements are omitted.
  • each component illustrated in FIGS. 2 and 3 is functionally conceptual, and does not necessarily need to be physically configured as illustrated.
  • the specific form of distributing/integrating each block is not limited to what is shown in the diagram, and all or part of the blocks can be functionally or physically distributed/integrated in arbitrary units depending on various loads and usage conditions. It is possible to configure them in an integrated manner.
  • the image synthesis device 10 includes a storage section 11 and a control section 12.
  • the storage unit 11 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a ROM (Read Only Memory), or a flash memory, or a storage device such as a hard disk or an optical disk.
  • the storage unit 11 stores geometry information 11a, reflectance information 11b, illumination map information 11c, and DCC tool program 11d.
  • the geometry information 11a is information corresponding to the above-mentioned geometry.
  • the reflectance information 11b is information corresponding to the above-mentioned reflectance.
  • the illumination map information 11c is information corresponding to the aforementioned illumination map.
  • the DCC tool program 11d is program data of the DCC tool.
  • the control unit 12 is a controller, and includes, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. This is realized by executing the information processing program according to the embodiment using the RAM as a work area. Further, the control unit 12 can be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
  • ASIC Application Specific Integrated Circuit
  • FPGA Field Programmable Gate Array
  • the control unit 12 includes an acquisition unit 12a, a conversion unit 12b, a composition processing unit 12c, and an output unit 12d, and realizes or executes information processing functions and operations described below.
  • the acquisition unit 12a acquires the RGB image L RGB , the depth image I Dep , the spherical image I Sep , and the CG data D Cg .
  • the converter 12b converts the RGB image LRGB , the depth image IDep , and the spherical image ISep into geometry information 11a, reflectance information 11b, and illumination map information 11c that can be read by the DCC tool.
  • the synthesis processing unit 12c receives the converted geometry information 11a, reflectance information 11b, illumination map information 11c, CG data D , and RGB image L as input, and generates a three-dimensional space representing a real space corresponding to the RGB image L. A compositing process is performed to arrange CG data D Cg .
  • FIG. 3 is a block diagram showing a configuration example of the composition processing section 12c.
  • the composition processing section 12c includes a first generation section 12ca, a second generation section 12cb, a third generation section 12cc, a fourth generation section 12cd, a fifth generation section 12ce, and an output image. It has a generation unit 12cf.
  • the first generation unit 12ca reads the converted geometry information 11a, reflectance information 11b, illumination map information 11c, and CG data D Cg using the DCC tool, and generates a PY image L PY , a VI image L VI , and a mask image within the DCC tool. M, generate an RPY image R PY and an RVI image R VI .
  • the RPY image R PY and the RVI image R VI are images that are input when generating the shadow impact image S I.
  • the second generation unit 12cb generates an ADR image L ADR and a QI image L QI using the generated PY image L PY , VI image L VI , mask image M, and RGB image L RGB .
  • the third generation unit 12cc receives the RPY image RPY and the RVI image RVI as input and generates a shadow impact image SI in parallel with the second generation unit 12cb.
  • the fourth generation unit 12cd receives the ADR image L ADR , the QI image L QI , and the shadow impact image S I and generates the LC image L LC .
  • the fifth generation unit 12ce receives the RGB image L RGB , the PY image L PY , and the shadow impact image S I in parallel with the fourth generation unit 12 cd and generates a light gap image w g .
  • the output image generation unit 12cf receives the VI image L VI , LC image L LC , and light gap image w g as input, generates an output image L end , and outputs it.
  • the output unit 12d outputs the output image L end generated by the composition processing unit 12c to an external device such as a display device.
  • FIG. 4 is a diagram showing an example of the output image L end .
  • FIG. 5 is a diagram showing an example of an RGB image L RGB .
  • FIG. 6 is a diagram showing an example of CG data DCg .
  • FIG. 7 is a diagram showing an example of the depth image IDep .
  • the output image L end shown in FIG. 4 is finally output from the composition processing unit 12c after undergoing composition processing.
  • FIG. 4 shows an example in which the mannequin in the rear is a mannequin that exists in real space, and the person in the foreground is CG. Note that, below, the figures are appropriately simplified in order to make the explanation easier to understand. Therefore, the examples shown below do not limit the synthesis quality of the synthesis processing according to the embodiments of the present disclosure.
  • the M1 portion in the diagram of FIG. 4 will be described later.
  • the RGB image L RGB obtained by the obtaining unit 12a is as shown in FIG. 5.
  • the M2 portion in the diagram of FIG. 5 will be described later.
  • the CG data D Cg similarly obtained by the obtaining unit 12a is as shown in FIG.
  • the depth image IDep acquired by the acquisition unit 12a is as shown in FIG. Note that the spherical image I Sep acquired by the acquisition unit 12a is omitted here.
  • the synthesis processing unit 12c generates a PY image L PY , a VI image L VI , a mask image M, and an RPY image R PY based on these RGB image L RGB , CG data D Cg , depth image I Dep , and spherical image I Sep , generates an RVI image RVI .
  • FIG. 8 is a diagram showing an example of the PY image LPY .
  • FIG. 9 is a diagram showing an example of the VI image LVI.
  • FIG. 10 is a diagram showing an example of the mask image M.
  • FIG. 11 is a diagram showing an example of the RPY image RPY .
  • FIG. 12 is a diagram showing an example of the RVI image RVI .
  • FIG. 13 is a diagram showing an example of a shadow impact image SI .
  • the PY image L PY becomes as shown in FIG. 8.
  • the VI image LVI becomes as shown in FIG.
  • the mask image M becomes as shown in FIG.
  • a new algorithm generates a shadow impact image SI , which can be called a weighting function image that detects areas in which the ADR method and the QI method are respectively good.
  • the above equation (11) is the intensity of illumination incident on the object plane from the direction vector ⁇ i to the three-dimensional position x in real space corresponding to the pixel in the image, and is the intensity of the illumination that enters the object plane from the direction vector ⁇ i , and Calculated from I Sep.
  • Equation ( 12) shows that when looking at the direction vector ⁇ i from the three-dimensional position Visibility) function.
  • the approximation R has no color component and is gray scaled.
  • the RPY image R PY obtained as a result of calculating irradiance using only the distance-measured real-photo information, and the result of calculating irradiance using both the distance-measured real-photo information and the CG data D Cg to be synthesized.
  • the resulting RVI images RVI are respectively generated.
  • the RPY image RPY becomes as shown in FIG. 11.
  • the RVI image RVI becomes as shown in FIG.
  • composition processing unit 12c generates a shadow impact image S1 using the RVI image RVI and the RPY image RPY according to the following equation (13).
  • dilate() is a general dilation process applied to the entire image in order to reduce noise that cannot be canceled out due to the light gap.
  • the synthesis processing unit 12c performs a linear combination of the ADR image L ADR and the QI image L QI using the shadow impact image S I according to the following equation (14) to generate an LC image L LC .
  • FIG. 14 is a diagram showing an example of the ADR image L ADR .
  • FIG. 15 is a diagram (part 1) showing an example of QI image L QI .
  • FIG. 16 is a diagram (part 2) showing an example of QI image L QI .
  • FIG. 14 is an ADR image LADR in which the person is a CG image.
  • the ADR image L ADR color disharmony or the like may appear in the vicinity of the step of the pillar, as shown, for example, in the M41 part of the M4 part in the real space.
  • color blurring or the like may appear at the boundary of the object, as shown in the M42 section of the M4 section, for example.
  • FIG. 15 is a QI image LQI when the person is a CG image.
  • inappropriate noise may appear in the QI image LQI in a portion where geometry noise is large (lower end of the image), as shown, for example, in the M51 portion of the M5 portion in real space.
  • FIG. 16 is a QI image LQI in which the person is a real object and a CG of a character holding a light-emitting sword is synthesized next to the person.
  • a situation may occur in which an area that should be brightened does not become bright due to reflection from the CG light-emitting sword, as shown in the M61 part of the M6 part in the real space, for example.
  • the synthesis processing unit 12c combines the ADR image L ADR and the QI image using the shadow impact image S I so that the defects of the ADR image L ADR and the QI image L QI are eliminated.
  • a linear combination of LQI is performed to generate an LC image LLC . Therefore, the LC image LLC is generated as an image in which such defects are eliminated.
  • the compositing processing unit 12c uses the shadow impact image SI to remove artifacts in the light gap region.
  • a light gap image w g indicating a light gap region is calculated using the following equation (15).
  • saturate() is a function that limits the input range to 0 to 1
  • opening() is a morphological operation for denoising.
  • the synthesis processing unit 12c preferentially allocates the VI image LVI to the light gap area, that is, the area where wg is high, by linear combination, and outputs the final synthesis result from which artifacts have been removed.
  • Image L end is obtained as shown in equation (16) below.
  • FIG. 17 is a diagram (part 1) showing an example of the light gap image wg .
  • FIG. 18 is a diagram (part 2) showing an example of the light gap image wg .
  • FIG. 17 corresponds to the case where the synthesis process of the output image L end shown in FIG. 4 is performed. In such a case, the light gap image wg will be as shown in FIG. 17.
  • the M7 section shown in FIG. 17 corresponds to the M2 section of the RGB image LRGB shown in FIG. Furthermore, the M7 section corresponds to the M3 section of the PY image LPY shown in FIG.
  • FIG. 18 shows these M2 portion, M3 portion, and M7 portion arranged side by side for comparison. As shown in FIG. 18, it can be seen that the rendering result of the PY image LPY is significantly different from the RGB image L RGB in the space between the right arm and torso of the mannequin in the image.
  • the composition processing unit 12c detects this portion as a light gap image wg , as shown in FIG.
  • FIG. 19 is a diagram showing the effect of correction using the light gap image wg .
  • FIG. 19 shows the M1 portion of the output image L end shown in FIG. 4, the M1 ADR portion of the ADR image L ADR corresponding to the M1 portion, and the M1 QI portion of the QI image L QI corresponding to the M1 portion. They are arranged side by side for comparison.
  • the shadow impact image S I has a low pixel value in an area where the light radiant energy has increased due to the synthesis process of the CG data D Cg , and has a high pixel value in an area where the light radiant energy has decreased.
  • RVI image RVI and the RPY image RPY which are irradiance images.
  • general CG methods also consider the direction of normal lines in space, but normal lines are susceptible to noise when the depth image IDep is of low quality. Therefore, by not using normal information to generate the RVI image RVI and the RPY image RPY , it is possible to generate a filter that is not affected by noise originating from the low-quality depth image IDep .
  • the light gap region is defined using the above equation (15), and calculation and detection are performed.
  • the range measurement error expressed by the difference between the input RGB image L RGB and the PY image L PY is raised to the power of ⁇ p to identify areas where the range measurement error is significantly large. can be detected.
  • the light gap image w g is not a binary image, and the light gap changes smoothly from a low region to a high region, so the light gap image w g does not have sharp edges.
  • the synthesis process can be performed so that the joint of the blended images is not noticeable.
  • FIG. 20 is a flowchart showing the processing procedure executed by the image synthesis device 10.
  • the acquisition unit 12a first acquires an RGB image L RGB , a depth image I Dep , a spherical image I Sep , and CG data D Cg (step S101). Then, the conversion unit 12b converts the RGB image L RGB , the depth image I Dep , and the spherical image I Sep so that they can be read into the DCC tool (step S102). Then, the composition processing unit 12c reads the converted data and the CG data DCg using the DCC tool (step S103).
  • the synthesis processing unit 12c generates a PY image L PY , a VI image L VI , a mask image M, an RPY image R PY , and an RVI image R VI within the DCC tool (step S104).
  • the synthesis processing unit 12c generates the ADR image L ADR and the QI image L QI using the PY image L PY , the VI image L VI , the mask image M, and the RGB image L RGB (step S105).
  • the synthesis processing unit 12c generates a shadow impact image S I using the RPY image R PY and the RVI image R VI (step S106).
  • the synthesis processing unit 12c When steps S105 and S106 are completed, the synthesis processing unit 12c generates an LC image LLC using the ADR image L ADR , the QI image L QI , and the shadow impact image S I (step S107).
  • the synthesis processing unit 12c generates a light gap image w g using the shadow impact image S I , the RGB image L RGB , and the PY image L PY (step S108).
  • the synthesis processing unit 12c When steps S107 and S108 are completed, the synthesis processing unit 12c generates an output image L end using the LC image L LC and the light gap image w g (step S109). Then, the output unit 12d outputs the output image L end (step S110), and the process ends.
  • FIG. 21 is a flowchart showing the processing procedure when outputting a still image.
  • a user photographs an RGB image L RGB with an RGB camera (step S201). Further, the user photographs a depth image I Dep with a depth camera (step S202). Further, the user photographs a spherical image I Sep with a spherical camera (step S203). Further, the user creates CG data DCg using the DCC tool (step S204).
  • the image synthesis device 10 inputs each data acquired in steps S201 to S204 and executes the image synthesis process shown in FIG. 20 (step S205).
  • the display device displays the output image L end output from the image synthesis device 10 as a still image (step S206), and the process ends.
  • FIG. 22 is a flowchart showing the processing procedure when outputting a moving image. Note that steps S301 to S304 in FIG. 22 are the same as steps S201 to S204 shown in FIG. 21, so the description thereof will be omitted here.
  • the user updates the CG data DCg according to the frame number (step S305).
  • the image synthesis device 10 inputs each data acquired in steps S301 to S305, and executes the image synthesis process shown in FIG. 20 (step S306).
  • the image synthesis device 10 determines whether the output image L end outputted by the image synthesis process has reached a predetermined number of frames (step S307). Here, if the predetermined number of frames has not been reached (step S307, No), the frame number is updated (step S308), and the processing from step S305 is repeated.
  • step S307 if the predetermined number of frames has been reached (step S307, Yes), for example, the display device combines the output images L end of each frame in time series and displays it as a moving image (step S309). Then, the process ends.
  • each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings.
  • the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.
  • FIG. 23 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the image synthesis apparatus 10.
  • Computer 1000 has CPU 1100, RAM 1200, ROM 1300, HDD (Hard Disk Drive) 1400, communication interface 1500, and input/output interface 1600. Each part of computer 1000 is connected by bus 1050.
  • the CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each part. For example, the CPU 1100 loads programs stored in the ROM 1300 or HDD 1400 into the RAM 1200, and executes processes corresponding to various programs.
  • the ROM 1300 stores boot programs such as BIOS (Basic Input Output System) that are executed by the CPU 1100 when the computer 1000 is started, programs that depend on the hardware of the computer 1000, and the like.
  • BIOS Basic Input Output System
  • the HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by the programs.
  • HDD 1400 is a recording medium that records an information processing program according to an embodiment of the present disclosure, which is an example of program data 1450.
  • Communication interface 1500 is an interface for connecting computer 1000 to external network 1550 (for example, network N).
  • CPU 1100 receives data from other devices or transmits data generated by CPU 1100 to other devices via communication interface 1500.
  • the input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000.
  • the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input/output interface 1600. Further, the CPU 1100 transmits data to an output device such as a display, speaker, or printer via an input/output interface 1600.
  • the input/output interface 1600 may function as a media interface that reads programs and the like recorded on a predetermined recording medium.
  • Media includes, for example, optical recording media such as DVD (Digital Versatile Disc) and PD (Phase change rewritable disk), magneto-optical recording media such as MO (Magneto-Optical disk), tape media, magnetic recording media, semiconductor memory, etc. It is.
  • the CPU 1100 of the computer 1000 realizes the functions of the control unit 12 by executing a program loaded onto the RAM 1200.
  • the HDD 1400 stores an information processing program according to the present disclosure and data in the storage unit 11. Note that although the CPU 1100 reads and executes the program data 1450 from the HDD 1400, as another example, these programs may be obtained from another device via the external network 1550.
  • the image synthesis device 10 (corresponding to an example of an "information processing device") generates an RGB image L RGB , a depth image I Dep , and a spherical image I Sep.
  • a composition processing unit 12c is provided which performs composition processing of arranging CG data DCg in a three-dimensional space representing a real space corresponding to the RGB image L RGB as an input.
  • the synthesis processing unit 12c generates a PY image L PY (corresponding to an example of a "first rendered image") using only the real shot information measured from the above input, and a VI using the above real shot information and CG data D Cg .
  • An image LVI (corresponding to an example of a "second rendered image”) is generated.
  • the synthesis processing unit 12c generates an ADR image L ADR (“first composite image”) based on the ADR method (corresponding to an example of the “first method”) based on the difference in shading between the PY image L PY and the VI image L VI . ) and a QI image L QI (corresponding to an example of a "second composite image”) by the QI method (corresponding to an example of a "second method”) based on the shading ratio.
  • first composite image (“first composite image”) based on the ADR method (corresponding to an example of the “first method”) based on the difference in shading between the PY image L PY and the VI image L VI . ) and a QI image L QI (corresponding to an example of a "second composite image”) by the QI method (corresponding to an example of a "second method”) based on the shading ratio.
  • composition processing unit 12c generates a shadow impact image S I representing a change in optical radiation energy between the case where only the above-mentioned live-action information is used and the case where the above-mentioned live-action information and CG data DCg are used, A linear combination of the ADR image L ADR and the QI image L QI is performed using the shadow impact image S I.
  • the present technology can also have the following configuration.
  • a composition processing unit that receives an RGB image, a depth image, and a spherical image as input and performs a composition process of arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
  • the synthesis processing section is Generating a first rendered image using only the live-action information measured from the input, and a second rendering image using the live-action information and the CG data, Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings.
  • the synthesis processing section is The shadow impact image is generated so that areas where light radiant energy has increased through the synthesis process have low pixel values, and areas where light radiant energy has decreased have high pixel values, and the areas with low pixel values have low pixel values. assigning the first composite image and the second composite image to the high pixel value area, respectively;
  • the information processing device according to (1) above.
  • the synthesis processing section is The shadow impact image is created such that the low pixel value area corresponds to the area of the RGB image that becomes brighter due to the compositing process, and the high pixel value area corresponds to an area of the RGB image that becomes dark due to the compositing process. generate, The information processing device according to (2) above.
  • the synthesis processing section at least includes: generating the shadow impact image based on information regarding illumination calculated from the distance-measured spherical image and information regarding visibility of the CG data; The information processing device according to (1), (2) or (3) above.
  • the synthesis processing section is generating the shadow impact image based only on the illumination information and the visibility information; The information processing device according to (4) above.
  • the synthesis processing section is using the shadow impact image to remove artifacts occurring in the compositing process; The information processing device according to any one of (1) to (5) above.
  • the synthesis processing section is By generating a light gap image indicating a light gap region in which the first rendered image is much darker than the RGB image, and preferentially allocating the second rendered image to the light gap region by linear combination. remove artifacts, The information processing device according to (6) above.
  • the synthesis processing section is An area where the distance measurement error expressed by the difference between the first rendered image and the RGB image is extremely large is detected as the light gap area, and the area where the artifacts are removed is determined by multiplying the area by the shadow impact image. limit, The information processing device according to (7) above.
  • (9) performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image; including;
  • Performing the above-mentioned compositing process includes: Generating a first rendered image using only the live-action information measured from the input, and a second rendered image using the live-action information and the CG data; Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings.
  • Information processing methods further including: (10) performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image; make the computer run
  • Performing the above-mentioned compositing process includes: generating a first rendered image using only live-action information measured from the input, and a second rendered image using the live-action information and the CG data; Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings.
  • a computer-readable recording medium having recorded thereon an information processing program that causes the computer to further execute the following.
  • Image synthesis device 11 Storage unit 11a Geometry information 11b Reflectance information 11c Illumination map information 11d DCC tool program 12 Control unit 12a Acquisition unit 12b Conversion unit 12c Synthesis processing unit 12ca First generation unit 12cb Second generation unit 12cc Third generation unit 12cd Fourth generation section 12ce Fifth generation section 12cf Output image generation section 12d Output section

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)

Abstract

An image synthesis device (10), which corresponds to an example of an information processing device, is provided with a synthesis processing unit (12c) that uses an RGB image (LRGB), a depth image (IDep), and an omnidirectional image (ISep) as inputs to perform a synthesis process for disposing CG data (DCg) in a three-dimensional space representing a real space corresponding to the RGB image (LRGB). The synthesis processing unit (12c): generates a PY image (LPY) using only captured image information measured from the inputs, and a VI image (LVI) using the captured image information and the CG data (DCg); generates an ADR image (LADR) using an ADR method based on the difference between the shades of the PY image (LPY) and the VI image (LVI), and a QI image (LQI) using a QI method based on the ratio between the shades; generates a shadow impact image (SI) representing the change in light radiant energy between when only the captured image information is used and when the captured image information and the CG data (DCg) are used; and linearly combines the ADR image (L)ADR) and the QI image (LQI) using the shadow impact image (SI).

Description

情報処理装置、情報処理方法および記録媒体Information processing device, information processing method, and recording medium
　本開示は、情報処理装置、情報処理方法および記録媒体に関する。 The present disclosure relates to an information processing device, an information processing method, and a recording medium.
　従来、ＶＦＸ（visual　effects）をはじめとする、ＣＧ（computer　graphics）または合成処理によって実写映像を加工・編集する技術は、映画やテレビドラマなどの映像分野において、現実にはない画面効果を実現する作品の普及に貢献している。 Traditionally, technology that processes and edits live-action footage using CG (computer graphics) or compositing processing, including VFX (visual effects), has been used to create screen effects that do not exist in reality in the video field such as movies and TV dramas. Contributing to the spread of the work.
　このようなＶＦＸの現場では、合成するＣＧの仮想オブジェクトが実写映像上であたかも存在すると視聴者に感じさせるために、仮想オブジェクトの見た目とその合成処理によって生じる陰影の変化までも忠実に再現するような合成処理が求められる。 At such VFX sites, in order to make the viewer feel that the CG virtual object to be synthesized exists in the live-action video, it is necessary to faithfully reproduce the appearance of the virtual object and even the changes in shadows caused by the compositing process. A unique synthesis process is required.
　従来、このような陰影の変化の計算を高品位に行ううえで、ＡＤＲ（Additive　differential　rendering）法や、ＱＩ（Quotient　image）法といった技術が知られている（例えば、非特許文献１，２参照）。 Conventionally, techniques such as the ADR (Additive Differential Rendering) method and the QI (Quotient Image) method have been known to perform high-quality calculations of such shadow changes (for example, see Non-Patent Documents 1 and 2). ).
　しかしながら、上述した従来技術には、実空間を表す３次元空間に対するＣＧデータの合成品質をより向上させるうえで、さらなる改善の余地がある。 However, the above-mentioned conventional technology has room for further improvement in improving the quality of synthesizing CG data with respect to a three-dimensional space representing a real space.
　例えば、ＡＤＲ法やＱＩ法は、実写背景の状況によって合成品質が一長一短であるため、多くの場合、背景の内容に応じてどちらを採用するかユーザが手動で選択する必要がある。加えて、これらの手法のアルゴリズムは、ＣＧが実写に対して落とす陰影の領域は平面形状のみであることを想定しているため、多くの３次元形状が複雑に交差して遮蔽が生じる領域に適用すると、ノイズの影響を強く受け、アーチファクトとして反映されるという問題点もある。 For example, the synthesis quality of the ADR method and the QI method has advantages and disadvantages depending on the situation of the actual background, so in many cases, the user needs to manually select which method to use depending on the content of the background. In addition, the algorithms of these methods assume that the shadow area cast by CG on the real image is only a planar shape, so it is difficult to apply shadows to areas where many three-dimensional shapes intersect in a complex manner and cause occlusion. When applied, there is also the problem that it is strongly influenced by noise and is reflected as an artifact.
　そこで、本開示では、実空間を表す３次元空間に対するＣＧデータの合成品質をより向上させることができる情報処理装置、情報処理方法および記録媒体を提案する。 Therefore, the present disclosure proposes an information processing device, an information processing method, and a recording medium that can further improve the quality of synthesizing CG data into a three-dimensional space representing a real space.
　上記の課題を解決するために、本開示に係る一形態の情報処理装置は、ＲＧＢ画像、デプス画像および全天球画像を入力として前記ＲＧＢ画像に対応する実空間を表す３次元空間にＣＧデータを配置する合成処理を行う合成処理部を備える。前記合成処理部は、前記入力から測距された実写情報のみを用いた第１のレンダリング画像と、前記実写情報および前記ＣＧデータを用いた第２のレンダリング画像とを生成し、前記第１のレンダリング画像および前記第２のレンダリング画像の陰影の差に基づく第１の手法による第１の合成画像と、前記陰影の比率に基づく第２の手法による第２の合成画像とを生成し、前記実写情報のみを用いた場合と、前記実写情報および前記ＣＧデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成し、前記シャドウインパクト画像を用いて前記第１の合成画像と前記第２の合成画像との線形結合を行う。 In order to solve the above problems, an information processing device according to an embodiment of the present disclosure inputs an RGB image, a depth image, and a spherical image and converts CG data into a three-dimensional space representing a real space corresponding to the RGB image. A compositing processing unit is provided that performs compositing processing for arranging. The synthesis processing unit generates a first rendered image using only the real-life information measured from the input, and a second rendered image using the real-life information and the CG data, and A first composite image by a first method based on the difference in shading between the rendered image and the second rendered image and a second composite image by a second method based on the ratio of the shading are generated, A shadow impact image representing a change in optical radiation energy between the case of using only the information and the case of using the live-action information and the CG data is generated, and the first composite image is generated using the shadow impact image. and the second composite image are linearly combined.
本開示の実施形態に係る画像合成方法の概要説明図である。FIG. 1 is a schematic explanatory diagram of an image synthesis method according to an embodiment of the present disclosure. 本開示の実施形態に係る画像合成装置の構成例を示すブロック図である。FIG. 1 is a block diagram illustrating a configuration example of an image synthesis device according to an embodiment of the present disclosure. 合成処理部の構成例を示すブロック図である。FIG. 2 is a block diagram showing an example of the configuration of a composition processing section. 出力画像の一例を示す図である。FIG. 3 is a diagram showing an example of an output image. ＲＧＢ画像の一例を示す図である。FIG. 3 is a diagram showing an example of an RGB image. ＣＧデータの一例を示す図である。It is a figure showing an example of CG data. デプス画像の一例を示す図である。It is a figure showing an example of a depth image. ＰＹ画像の一例を示す図である。It is a figure showing an example of a PY image. ＶＩ画像の一例を示す図である。FIG. 3 is a diagram showing an example of a VI image. マスク画像の一例を示す図である。It is a figure showing an example of a mask image. ＲＰＹ画像の一例を示す図である。FIG. 3 is a diagram showing an example of an RPY image. ＲＶＩ画像の一例を示す図である。It is a figure showing an example of an RVI image. シャドウインパクト画像の一例を示す図である。FIG. 3 is a diagram showing an example of a shadow impact image. ＡＤＲ画像の一例を示す図である。It is a figure showing an example of an ADR image. ＱＩ画像の一例を示す図（その１）である。FIG. 2 is a diagram (part 1) showing an example of a QI image. ＱＩ画像の一例を示す図（その２）である。FIG. 2 is a diagram (part 2) showing an example of a QI image. ライトギャップ画像の一例を示す図（その１）である。FIG. 2 is a diagram (part 1) showing an example of a light gap image. ライトギャップ画像の一例を示す図（その２）である。FIG. 2 is a diagram (part 2) showing an example of a light gap image. ライトギャップ画像を用いた補正の効果を示す図である。FIG. 7 is a diagram showing the effect of correction using a light gap image. 画像合成装置が実行する処理手順を示すフローチャートである。3 is a flowchart illustrating a processing procedure executed by the image compositing device. 静止画を出力する場合の処理手順を示すフローチャートである。It is a flowchart which shows the processing procedure when outputting a still image. 動画を出力する場合の処理手順を示すフローチャートである。It is a flowchart showing a processing procedure when outputting a moving image. 画像合成装置の機能を実現するコンピュータの一例を示すハードウェア構成図である。FIG. 2 is a hardware configuration diagram showing an example of a computer that implements the functions of an image synthesis device.
　以下に、本開示の実施形態について図面に基づいて詳細に説明する。なお、以下の各実施形態において、同一の部位には同一の符号を付することにより重複する説明を省略する。 Below, embodiments of the present disclosure will be described in detail based on the drawings. In addition, in each of the following embodiments, the same portions are given the same reference numerals and redundant explanations will be omitted.
　また、以下では、本開示の実施形態に係る情報処理装置が、画像合成装置１０であるものとする。また、以下では、本開示の実施形態に係る情報処理方法が、画像合成方法であるものとする。 Furthermore, in the following description, it is assumed that the information processing device according to the embodiment of the present disclosure is the image composition device 10. Further, in the following, it is assumed that the information processing method according to the embodiment of the present disclosure is an image synthesis method.
　また、以下に示す項目順序に従って本開示を説明する。
　　１．概要
　　　１－１．ＡＤＲ法とＱＩ法の技術的課題
　　　１－２．本開示の実施形態に係る画像合成方法の概要
　　２．画像合成装置の構成
　　３．合成処理部が実行する合成処理の詳細
　　４．変形例
　　５．ハードウェア構成
　　６．むすび Further, the present disclosure will be described according to the order of items shown below.
1. Overview 1-1. Technical issues of ADR method and QI method 1-2. Outline of image synthesis method according to embodiment of the present disclosure 2. Configuration of image synthesis device 3. Details of the compositing process executed by the compositing processing unit 4. Modification example 5. Hardware configuration 6. Conclusion
＜＜１．概要＞＞
＜１－１．ＡＤＲ法とＱＩ法の技術的課題＞
　まず、本開示の実施形態に係る画像合成方法の説明に先立って、ＡＤＲ法とＱＩ法の技術的課題について詳細に述べておく。 <<1. Overview >>
<1-1. Technical issues of ADR method and QI method>
First, prior to explaining the image synthesis method according to the embodiment of the present disclosure, technical issues of the ADR method and the QI method will be described in detail.
　本開示の実施形態の先行技術であるＡＤＲ法やＱＩ法で合成処理を行うためには、合成対象のＣＧデータＤＣｇに加え、実空間の３次元的形状であるジオメトリ、ジオメトリに対応する反射率、実空間の照明マップが入力として必要である。 In order to perform synthesis processing using the ADR method or QI method, which is the prior art of the embodiment of the present disclosure, in addition to the CG data D Cg to be synthesized, geometry that is a three-dimensional shape in real space, and reflections corresponding to the geometry are required. A real-space illumination map is required as input.
　これらの情報は、ＭａｙａやＢｌｅｎｄｅｒ等の一般的なＤＣＣ（Digital　Content　Creation）ツールを用い、実空間を撮像したＲＧＢ画像ＬＲＧＢ、デプス画像ＩＤｅｐ、全天球画像ＩＳｅｐを加工することで得ることができる。 This information is obtained by processing the RGB image L RGB captured in real space, the depth image I Dep , and the spherical image I Sep using general DCC (Digital Content Creation) tools such as Maya and Blender. be able to.
　なお、こうして得られるジオメトリ、および反射率の各品質は、低品位であるという前提の下、以下説明する。ＡＤＲ法、ＱＩ法はともに、前述のジオメトリ、反射率、照明マップおよび合成対象のＣＧデータを用いて、ＰＹ画像ＬＰＹ、ＶＩ画像ＬＶＩおよびマスク画像Ｍの３枚の画像を生成する。 Note that the following description will be made on the premise that the quality of the geometry and reflectance obtained in this way are of low quality. Both the ADR method and the QI method generate three images, a PY image L PY , a VI image L VI , and a mask image M, using the aforementioned geometry, reflectance, illumination map, and CG data to be synthesized.
　ＰＹ画像ＬＰＹは、デプス画像から測距した実写情報のみでレンダリングを行った画像である。ＶＩ画像ＬＶＩは、実写情報と合成処理するＣＧデータの両方を用いてレンダリングを行った画像である。マスク画像Ｍは、合成処理するＣＧデータが存在する領域を示す画像である。これらの画像は、前述のＤＣＣツールでレンダリングすることで生成可能である。 PY image L PY is an image rendered using only actual photographic information measured from a depth image. VI image L VI is an image rendered using both real-shot information and CG data to be synthesized. The mask image M is an image indicating an area where CG data to be synthesized exists. These images can be generated by rendering with the DCC tool described above.
　ＲＧＢ画像ＬＲＧＢ、ＰＹ画像ＬＰＹ、ＶＩ画像ＬＶＩおよびマスク画像Ｍは、ＡＤＲ法およびＱＩ法における合成アルゴリズムの入力であり、それらのアルゴリズムの構造上、それぞれ利点と欠点を有する。 The RGB image L RGB , the PY image L PY , the VI image L VI and the mask image M are input to the synthesis algorithm in the ADR method and the QI method, and each of these algorithms has advantages and disadvantages in terms of their structure.
　まず、ＡＤＲ法では、以下の式（１）に基づいてＡＤＲ画像ＬＡＤＲを計算する。 First, in the ADR method, an ADR image L ADR is calculated based on the following equation (1).
Figure JPOXMLDOC01-appb-M000001
Figure JPOXMLDOC01-appb-M000001
　式（１）の第一項は、ＣＧデータＤＣｇが存在する領域、すなわちＭ＝１である場合は、ＶＩ画像ＬＶＩをそのまま表示することを表している。また、式（１）の第二項は、入力されるＲＧＢ画像ＬＲＧＢに対し、ＣＧデータＤＣｇの合成処理により生じる陰影の変化ＬＶＩ－ＬＰＹを加えることを意味する。 The first term of equation (1) indicates that in the area where the CG data D Cg exists, that is, when M=1, the VI image L VI is displayed as is. Furthermore, the second term in equation (1) means adding a change in shading L VI −L PY caused by the compositing process of the CG data D Cg to the input RGB image L RGB .
　これにより、測距品質が十分に高くない場合でも、陰影の差分の情報だけをＶＩ画像ＬＶＩから抽出し、実写背景に対し効果的に高品位な陰影を付加することができる。 As a result, even if the ranging quality is not sufficiently high, only the information on the difference in shadows can be extracted from the VI image LVI , and high-quality shadows can be effectively added to the real background.
　しかし、ＶＩ画像ＬＶＩの画素値は非負である。すなわちＬＶＩ≧０であるため、ＡＤＲ法の第二項においては、以下の式（２）が成り立つ。 However, the pixel values of the VI image LVI are non-negative. That is, since L VI ≧0, the following formula (2) holds true in the second term of the ADR method.
Figure JPOXMLDOC01-appb-M000002
Figure JPOXMLDOC01-appb-M000002
　つまり、ＡＤＲ法には、ＬＲＧＢ－ＬＰＹが正である場合には、ＬＡＤＲのピクセル値をＬＲＧＢ－ＬＰＹ未満にすることができず、陰影により実写を暗くできる範囲に下限が存在するという問題がある。加えて、上記の式（１）の性質上、入力の反射率が低品位であり、真値に対して誤差が大きい場合、ＲＧＢ画像ＬＲＧＢとＰＹ画像ＬＰＹの色分布が乖離し、アーチファクトとしてＡＤＲ画像ＬＡＤＲに反映されるという問題もある。 In other words, in the ADR method, if L RGB - L PY is positive, the L ADR pixel value cannot be made less than L RGB - L PY , and there is a lower limit to the range in which the actual photograph can be darkened by shading. There is a problem with doing so. In addition, due to the nature of equation (1) above, if the input reflectance is of low quality and has a large error with respect to the true value, the color distributions of the RGB image L RGB and the PY image L PY will diverge, causing artifacts. There is also the problem that this is reflected in the ADR image LADR .
　次に、ＱＩ法では、以下の式（３）に基づいてＱＩ画像ＬＱＩを計算する。 Next, in the QI method, a QI image L QI is calculated based on the following equation (3).
Figure JPOXMLDOC01-appb-M000003
Figure JPOXMLDOC01-appb-M000003
　ＱＩ法もＡＤＲ法と同様に、入力されるＲＧＢ画像ＬＲＧＢに対し、ＣＧデータＤＣｇの合成処理により生じる陰影の比率ＬＶＩ／ＬＰＹを掛けることで、実写背景に陰影を付加することができる。 Similar to the ADR method, the QI method can add shadows to the real background by multiplying the input RGB image L RGB by the ratio of shadows L VI /L PY produced by the compositing process of the CG data D Cg . can.
　また、ＱＩ法において、式（３）の第二項は、ゼロ除算による合成結果の不安定化を防止するために、実際は以下の式（４）で運用される。 Furthermore, in the QI method, the second term of equation (3) is actually operated by the following equation (4) in order to prevent the synthesis result from becoming unstable due to division by zero.
Figure JPOXMLDOC01-appb-M000004
Figure JPOXMLDOC01-appb-M000004
　ここで、εはゼロ除算防止のための定数である。このとき、ＬＱＩはＬＰＹの最大１．０１倍に制限される。このため、ＱＩ法には、例えば発光するＣＧデータＤＣｇの合成処理により実写を明るくできる範囲には上限が存在するという問題がある。加えて、式（３）の性質上、入力のデプス画像ＩＤｅｐが低品位であり、真値に対して誤差が大きい場合、式（４）の第二項においてゼロ除算にきわめて近い演算が発生する可能性が高く、その場合アーチファクトとしてＱＩ画像ＬＱＩに反映されるという問題もある。 Here, ε is a constant to prevent division by zero. At this time, L QI is limited to a maximum of 1.01 times L PY . For this reason, the QI method has a problem in that there is an upper limit to the range in which a live photograph can be brightened by, for example, combining processing of emitting CG data DCg . In addition, due to the nature of equation (3), if the input depth image IDep is of low quality and has a large error with respect to the true value, an operation extremely similar to division by zero will occur in the second term of equation (4). There is a high possibility that this will occur, and in that case, there is a problem that it will be reflected in the QI image LQI as an artifact.
　最後に、ＡＤＲ法とＱＩ法の両者で共通した技術的課題を示す。上記の式（１），式（３）の合成処理では、実空間のジオメトリ、反射率、照明マップの測距品質が高く、画像全体にわたって下記の式（５）が成り立つことを前提として、高品位な陰影の合成を保証している。 Finally, we will show technical issues common to both the ADR method and the QI method. In the synthesis process of equations (1) and (3) above, we assume that the real space geometry, reflectance, and illumination map have high distance measurement quality, and that equation (5) below holds true over the entire image. It guarantees high-quality shading composition.
Figure JPOXMLDOC01-appb-M000005
Figure JPOXMLDOC01-appb-M000005
　しかし、前提として、ジオメトリおよび反射率の各測距品質が低い場合、式（５）が成り立たない領域が画像中に存在する場合がある。このような領域中で、特に合成品質に影響を及ぼすのは、ＰＹ画像ＬＰＹがＲＧＢ画像ＬＲＧＢよりも極めて暗くなる、すなわち下記の式（６）となる領域である。 However, as a premise, if the ranging qualities of geometry and reflectance are low, there may be an area in the image where equation (5) does not hold. Among these regions, the region where the PY image L PY is much darker than the RGB image L RGB , that is, the region where the following equation (6) is satisfied, particularly affects the synthesis quality.
Figure JPOXMLDOC01-appb-M000006
Figure JPOXMLDOC01-appb-M000006
　本開示の実施形態では、このような領域をライトギャップ（Light-gap）領域と呼ぶ。ライトギャップ領域は、上記前提のように、デプス画像ＩＤｅｐの測距品質が低品位であり、実空間の３次元的な物体の遮蔽環境が正確に取得できない場合に生じる。ライトギャップ領域に対してＣＧデータＤＣｇの合成処理を行う場合、ＣＧデータＤＣｇ中に発光する仮想オブジェクトが含まれていなければ、自動的に下記の式（７）が成り立つ。 In embodiments of the present disclosure, such a region is referred to as a light-gap region. As stated above, the light gap region occurs when the distance measurement quality of the depth image IDep is low quality and the three-dimensional object shielding environment in real space cannot be accurately acquired. When performing compositing processing of CG data D Cg on the light gap region, the following equation (7) automatically holds true unless a virtual object that emits light is included in the CG data D Cg .
Figure JPOXMLDOC01-appb-M000007
Figure JPOXMLDOC01-appb-M000007
　ゆえに、ＡＤＲ法の式（１）の第二項においては、下記の式（８）が成り立ち、合成処理によりＲＧＢ画像ＬＲＧＢに陰影を付加すべき領域において、所望の出力を期待することができない。 Therefore, in the second term of Equation (1) of the ADR method, Equation (8) below holds true, and the desired output cannot be expected in the region where shading is to be added to the RGB image L RGB by the compositing process. .
Figure JPOXMLDOC01-appb-M000008
Figure JPOXMLDOC01-appb-M000008
　同様に、ＱＩ法の式（４）の第二項においても、下記の式（９）が成り立ち、ＡＤＲ法と同様の現象が引き起こされる。 Similarly, in the second term of Equation (4) of the QI method, Equation (9) below holds true, causing a phenomenon similar to that of the ADR method.
Figure JPOXMLDOC01-appb-M000009
Figure JPOXMLDOC01-appb-M000009
　このように、ライトギャップ領域では、合成処理によりＲＧＢ画像ＬＲＧＢに陰影を付加することができず、その結果アーチファクトとしてＡＤＲ画像ＬＡＤＲやＱＩ画像ＬＱＩに反映されるという技術的課題がある。 As described above, in the light gap region, there is a technical problem that shading cannot be added to the RGB image L RGB by the synthesis process, and as a result, it is reflected in the ADR image L ADR and the QI image L QI as an artifact.
　以上を総括し、ＡＤＲ法とＱＩ法の両者の技術的課題をまとめると、次の通りである。まずＡＤＲ法は、ＣＧデータＤＣｇの合成処理によってＲＧＢ画像ＬＲＧＢを暗くする表現に制約が存在する。また、ＡＤＲ法は、反射率の低品位な測距品質に対し、アーチファクトが生じやすい。また、ＡＤＲ法は、ライトギャップ領域におけるアーチファクトが顕著となる。 Summarizing the above, the technical issues of both the ADR method and the QI method are summarized as follows. First, in the ADR method, there are restrictions on the expression of darkening the RGB image L RGB by the compositing process of the CG data D Cg . In addition, the ADR method tends to cause artifacts due to the low quality ranging quality of reflectance. Furthermore, in the ADR method, artifacts in the light gap region become noticeable.
　一方、ＱＩ法は、ＣＧデータＤＣｇの合成処理によってＲＧＢ画像ＬＲＧＢを明るくする表現に制約が存在する。また、ＱＩ法は、デプス画像ＩＤｅｐの低品位な測距品質に対し、アーチファクトが生じやすい。また、ＱＩ法は、ライトギャップ領域におけるアーチファクトが顕著となる。 On the other hand, in the QI method, there are restrictions on the expression of brightening the RGB image L RGB by the synthesis process of the CG data D Cg . In addition, the QI method tends to cause artifacts due to the low ranging quality of the depth image IDep . Furthermore, in the QI method, artifacts in the light gap region become noticeable.
＜１－２．本開示の実施形態に係る画像合成方法の概要＞
　図１は、本開示の実施形態に係る画像合成方法の概要説明図である。上記の技術的課題に対し、本開示の実施形態に係る画像合成方法では、画像合成装置１０が、ＲＧＢ画像ＬＲＧＢ、デプス画像ＩＤｅｐおよび全天球画像ＩＳｅｐを入力としてＲＧＢ画像ＬＲＧＢに対応する実空間を表す３次元空間にＣＧデータＤＣｇを配置する合成処理を行う。 <1-2. Overview of image synthesis method according to embodiment of the present disclosure>
FIG. 1 is a schematic explanatory diagram of an image synthesis method according to an embodiment of the present disclosure. In order to solve the above technical problem, in the image synthesis method according to the embodiment of the present disclosure, the image synthesis device 10 inputs the RGB image L RGB , the depth image I Dep , and the spherical image I Sep and converts the RGB image L RGB into an RGB image L RGB . A synthesis process is performed to arrange the CG data D Cg in a three-dimensional space representing the corresponding real space.
　図１に示すように、かかる合成処理において、画像合成装置１０は、上記入力から測距された実写情報のみを用いたレンダリング画像であるＰＹ画像ＬＰＹと、上記実写情報およびＣＧデータＤＣｇを用いたレンダリング画像であるＶＩ画像ＬＶＩとを生成する（ステップＳ１－１，Ｓ１－２）。ＰＹ画像ＬＰＹは、「第１のレンダリング画像」の一例に相当する。ＶＩ画像ＬＶＩは、「第２のレンダリング画像」の一例に相当する。 As shown in FIG. 1, in this compositing process, the image compositing device 10 combines the PY image LPY , which is a rendered image using only the actual photographic information measured from the input, and the aforementioned photographic information and CG data DCG . A VI image LVI , which is the rendered image used, is generated (steps S1-1, S1-2). PY image L PY corresponds to an example of a "first rendered image." VI image L VI corresponds to an example of a "second rendered image."
　また、画像合成装置１０は、ＰＹ画像ＬＰＹおよびＶＩ画像ＬＶＩの陰影の差に基づくＡＤＲ法による合成画像であるＡＤＲ画像ＬＡＤＲと、上記陰影の比率に基づくＱＩ法による合成画像であるＱＩ画像ＬＱＩとを生成する（ステップＳ２－１，Ｓ２－２）。ＡＤＲ画像ＬＡＤＲは、「第１の合成画像」の一例に相当する。ＱＩ画像ＬＱＩは、「第２の合成画像」の一例に相当する。 The image synthesis device 10 also generates an ADR image L ADR, which is a synthesized image based on the ADR method based on the difference in shading between the PY image L PY and a VI image L VI , and a QI image, which is a synthesized image based on the QI method based on the ratio of the shadings. Images L and QI are generated (steps S2-1 and S2-2). ADR image L ADR corresponds to an example of a "first composite image." QI image L QI corresponds to an example of a "second composite image."
　また、画像合成装置１０は、上記実写情報のみを用いた場合と、上記実写情報およびＣＧデータＤＣｇを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像ＳＩを生成する（ステップＳ３）。また、画像合成装置１０は、シャドウインパクト画像ＳＩを用いてＡＤＲ画像ＬＡＤＲとＱＩ画像ＬＱＩとの線形結合を行い、ＬＣ画像ＬＬＣを生成する（ステップＳ４）。 The image synthesis device 10 also generates a shadow impact image SI representing a change in light radiation energy between the case where only the above-mentioned real-time photograph information is used and the case where the above-mentioned live-action information and CG data DCg are used. Step S3). The image synthesis device 10 also performs a linear combination of the ADR image LADR and the QI image LQI using the shadow impact image S I to generate an LC image L LC (step S4).
　すなわち、本開示の実施形態に係る画像合成方法では、ＲＧＢ画像ＬＲＧＢ、デプス画像ＩＤｅｐ、全天球画像ＩＳｅｐを入力として、ＲＧＢ画像ＬＲＧＢに対応する実空間を表す３次元空間にＣＧデータＤＣｇを配置する合成処理を行う。その際、ＡＤＲ法により生成されるＡＤＲ画像ＬＡＤＲと、ＱＩ法により生成されるＱＩ画像ＬＱＩを、本開示の新規アルゴリズムにより生成するシャドウインパクト画像ＳＩを用いて線形結合を行い、ＡＤＲ法とＱＩ法の長所を画像ピクセルごとに適応的に混合する。 That is, in the image synthesis method according to the embodiment of the present disclosure, the RGB image L RGB , the depth image I Dep , and the spherical image I Sep are input, and CG is created in a three-dimensional space representing the real space corresponding to the RGB image L RGB . Performs synthesis processing to arrange data D Cg . At that time, the ADR image L ADR generated by the ADR method and the QI image L QI generated by the QI method are linearly combined using the shadow impact image S I generated by the new algorithm of the present disclosure, and The advantages of the and QI methods are adaptively mixed for each image pixel.
　これにより、ユーザは、ＡＤＲ画像ＬＡＤＲとＱＩ画像ＬＱＩの手動的選択を行う必要がなくなる。さらに、シャドウインパクト画像ＳＩを用いて遮蔽が生じる領域のアーチファクトを解消し、多くの３次元形状が複雑に交差して遮蔽が生じる実空間での自動的な合成処理を実現することができる。 This eliminates the need for the user to manually select the ADR image L ADR and the QI image L QI . Furthermore, it is possible to eliminate artifacts in areas where occlusion occurs using the shadow impact image SI , and to realize automatic synthesis processing in real space where many three-dimensional shapes intersect in a complex manner and occlusion occurs.
　すなわち、本開示の実施形態に係る画像合成方法によれば、実空間を表す３次元空間に対するＣＧデータＤＣｇの合成品質をより向上させることができる。以下、本開示の実施形態に係る画像合成方法を適用した画像合成装置１０の構成例について、より具体的に説明する。 That is, according to the image synthesis method according to the embodiment of the present disclosure, it is possible to further improve the synthesis quality of CG data DCg with respect to a three-dimensional space representing a real space. Hereinafter, a configuration example of the image synthesis apparatus 10 to which the image synthesis method according to the embodiment of the present disclosure is applied will be described in more detail.
＜＜２．画像合成装置の構成＞＞
　図２は、本開示の実施形態に係る画像合成装置１０の構成例を示すブロック図である。なお、図２および後に示す図３では、本開示の実施形態の特徴を説明するために必要な構成要素のみを表しており、一般的な構成要素についての記載を省略している。 <<2. Configuration of image synthesis device >>
FIG. 2 is a block diagram illustrating a configuration example of the image synthesis device 10 according to the embodiment of the present disclosure. Note that FIG. 2 and FIG. 3 shown later show only the constituent elements necessary for explaining the features of the embodiment of the present disclosure, and descriptions of general constituent elements are omitted.
　換言すれば、図２および図３に図示される各構成要素は機能概念的なものであり、必ずしも物理的に図示の如く構成されていることを要しない。例えば、各ブロックの分散・統合の具体的形態は図示のものに限られず、その全部または一部を、各種の負荷や使用状況などに応じて、任意の単位で機能的または物理的に分散・統合して構成することが可能である。 In other words, each component illustrated in FIGS. 2 and 3 is functionally conceptual, and does not necessarily need to be physically configured as illustrated. For example, the specific form of distributing/integrating each block is not limited to what is shown in the diagram, and all or part of the blocks can be functionally or physically distributed/integrated in arbitrary units depending on various loads and usage conditions. It is possible to configure them in an integrated manner.
　また、図２および図３を用いた説明では、既に説明済みの構成要素については、説明を簡略するか、省略する場合がある。 Furthermore, in the explanation using FIGS. 2 and 3, the explanation of components that have already been explained may be simplified or omitted.
　図２に示すように、画像合成装置１０は、記憶部１１と、制御部１２とを有する。記憶部１１は、例えば、ＲＡＭ（Random　Access　Memory）、ＲＯＭ（Read　Only　Memory）、フラッシュメモリ（Flash　Memory）等の半導体メモリ素子、または、ハードディスク、光ディスク等の記憶装置によって実現される。 As shown in FIG. 2, the image synthesis device 10 includes a storage section 11 and a control section 12. The storage unit 11 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a ROM (Read Only Memory), or a flash memory, or a storage device such as a hard disk or an optical disk.
　図２に示す例では、記憶部１１は、ジオメトリ情報１１ａと、反射率情報１１ｂと、照明マップ情報１１ｃと、ＤＣＣツールプログラム１１ｄを記憶する。ジオメトリ情報１１ａは、前述のジオメトリに対応する情報である。反射率情報１１ｂは、前述の反射率に対応する情報である。照明マップ情報１１ｃは、前述の照明マップに対応する情報である。ＤＣＣツールプログラム１１ｄは、ＤＣＣツールのプログラムデータである。 In the example shown in FIG. 2, the storage unit 11 stores geometry information 11a, reflectance information 11b, illumination map information 11c, and DCC tool program 11d. The geometry information 11a is information corresponding to the above-mentioned geometry. The reflectance information 11b is information corresponding to the above-mentioned reflectance. The illumination map information 11c is information corresponding to the aforementioned illumination map. The DCC tool program 11d is program data of the DCC tool.
　制御部１２は、コントローラ（controller）であり、例えば、ＣＰＵ（Central　Processing　Unit）やＭＰＵ（Micro　Processing　Unit）、ＧＰＵ（Graphics　Processing　Unit）等によって、記憶部１１に記憶された図示略の本開示の実施形態に係る情報処理プログラムがＲＡＭを作業領域として実行されることにより実現される。また、制御部１２は、例えば、ＡＳＩＣ（Application　Specific　Integrated　Circuit）やＦＰＧＡ（Field　Programmable　Gate　Array）等の集積回路により実現することができる。 The control unit 12 is a controller, and includes, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. This is realized by executing the information processing program according to the embodiment using the RAM as a work area. Further, the control unit 12 can be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
　制御部１２は、取得部１２ａと、変換部１２ｂと、合成処理部１２ｃと、出力部１２ｄとを有し、以下に説明する情報処理の機能や作用を実現または実行する。 The control unit 12 includes an acquisition unit 12a, a conversion unit 12b, a composition processing unit 12c, and an output unit 12d, and realizes or executes information processing functions and operations described below.
　取得部１２ａは、ＲＧＢ画像ＬＲＧＢ、デプス画像ＩＤｅｐ、全天球画像ＩＳｅｐおよびＣＧデータＤＣｇを取得する。 The acquisition unit 12a acquires the RGB image L RGB , the depth image I Dep , the spherical image I Sep , and the CG data D Cg .
　変換部１２ｂは、ＲＧＢ画像ＬＲＧＢ、デプス画像ＩＤｅｐ、全天球画像ＩＳｅｐをＤＣＣツールが読み込み可能なジオメトリ情報１１ａ、反射率情報１１ｂおよび照明マップ情報１１ｃに変換する。 The converter 12b converts the RGB image LRGB , the depth image IDep , and the spherical image ISep into geometry information 11a, reflectance information 11b, and illumination map information 11c that can be read by the DCC tool.
　合成処理部１２ｃは、変換されたジオメトリ情報１１ａ、反射率情報１１ｂ、照明マップ情報１１ｃ、ＣＧデータＤＣｇおよびＲＧＢ画像ＬＲＧＢを入力として、ＲＧＢ画像ＬＲＧＢに対応する実空間を表す３次元空間にＣＧデータＤＣｇを配置する合成処理を行う。 The synthesis processing unit 12c receives the converted geometry information 11a, reflectance information 11b, illumination map information 11c, CG data D , and RGB image L as input, and generates a three-dimensional space representing a real space corresponding to the RGB image L. A compositing process is performed to arrange CG data D Cg .
　図３は、合成処理部１２ｃの構成例を示すブロック図である。図３に示すように、合成処理部１２ｃは、第１生成部１２ｃａと、第２生成部１２ｃｂと、第３生成部１２ｃｃと、第４生成部１２ｃｄと、第５生成部１２ｃｅと、出力画像生成部１２ｃｆとを有する。 FIG. 3 is a block diagram showing a configuration example of the composition processing section 12c. As shown in FIG. 3, the composition processing section 12c includes a first generation section 12ca, a second generation section 12cb, a third generation section 12cc, a fourth generation section 12cd, a fifth generation section 12ce, and an output image. It has a generation unit 12cf.
　第１生成部１２ｃａは、変換されたジオメトリ情報１１ａ、反射率情報１１ｂ、照明マップ情報１１ｃおよびＣＧデータＤＣｇをＤＣＣツールで読み込み、ＤＣＣツール内でＰＹ画像ＬＰＹ、ＶＩ画像ＬＶＩ、マスク画像Ｍ、ＲＰＹ画像ＲＰＹ、ＲＶＩ画像ＲＶＩを生成する。ＲＰＹ画像ＲＰＹ、ＲＶＩ画像ＲＶＩは、シャドウインパクト画像ＳＩを生成する際の入力となる画像である。 The first generation unit 12ca reads the converted geometry information 11a, reflectance information 11b, illumination map information 11c, and CG data D Cg using the DCC tool, and generates a PY image L PY , a VI image L VI , and a mask image within the DCC tool. M, generate an RPY image R PY and an RVI image R VI . The RPY image R PY and the RVI image R VI are images that are input when generating the shadow impact image S I.
　第２生成部１２ｃｂは、生成されたＰＹ画像ＬＰＹ、ＶＩ画像ＬＶＩ、マスク画像Ｍ、ＲＧＢ画像ＬＲＧＢを用いて、ＡＤＲ画像ＬＡＤＲおよびＱＩ画像ＬＱＩを生成する。第３生成部１２ｃｃは、第２生成部１２ｃｂに並行して、ＲＰＹ画像ＲＰＹとＲＶＩ画像ＲＶＩを入力として、シャドウインパクト画像ＳＩを生成する。 The second generation unit 12cb generates an ADR image L ADR and a QI image L QI using the generated PY image L PY , VI image L VI , mask image M, and RGB image L RGB . The third generation unit 12cc receives the RPY image RPY and the RVI image RVI as input and generates a shadow impact image SI in parallel with the second generation unit 12cb.
　そして、第４生成部１２ｃｄは、ＡＤＲ画像ＬＡＤＲ、ＱＩ画像ＬＱＩ、シャドウインパクト画像ＳＩを入力として、ＬＣ画像ＬＬＣを生成する。第５生成部１２ｃｅは、第４生成部１２ｃｄに並行して、ＲＧＢ画像ＬＲＧＢ、ＰＹ画像ＬＰＹ、シャドウインパクト画像ＳＩを入力として、ライトギャップ画像ｗｇを生成する。最後に、出力画像生成部１２ｃｆは、ＶＩ画像ＬＶＩ、ＬＣ画像ＬＬＣ、ライトギャップ画像ｗｇを入力として、出力画像Ｌｅｎｄを生成し、出力する。 Then, the fourth generation unit 12cd receives the ADR image L ADR , the QI image L QI , and the shadow impact image S I and generates the LC image L LC . The fifth generation unit 12ce receives the RGB image L RGB , the PY image L PY , and the shadow impact image S I in parallel with the fourth generation unit 12 cd and generates a light gap image w g . Finally, the output image generation unit 12cf receives the VI image L VI , LC image L LC , and light gap image w g as input, generates an output image L end , and outputs it.
　かかる合成処理部１２ｃが実行する合成処理のより詳細な内容については、図４以降を用いた説明で後述する。 More detailed contents of the compositing process executed by the compositing processing unit 12c will be described later in the explanation using FIG. 4 and subsequent figures.
　図３の説明に戻る。出力部１２ｄは、合成処理部１２ｃによって生成された出力画像Ｌｅｎｄを表示装置等の外部装置へ出力する。 Returning to the explanation of FIG. 3. The output unit 12d outputs the output image L end generated by the composition processing unit 12c to an external device such as a display device.
＜＜３．合成処理部が実行する合成処理の詳細＞＞
　次に、合成処理部１２ｃが実行する合成処理の詳細について、図４～図１９を用いて各画像の例を挙げつつ説明する。 <<3. Details of the compositing process executed by the compositing processing unit >>
Next, the details of the compositing process executed by the compositing processing unit 12c will be explained while giving examples of each image using FIGS. 4 to 19.
　図４は、出力画像Ｌｅｎｄの一例を示す図である。また、図５は、ＲＧＢ画像ＬＲＧＢの一例を示す図である。また、図６は、ＣＧデータＤＣｇの一例を示す図である。また、図７は、デプス画像ＩＤｅｐの一例を示す図である。以下では、図４に示す出力画像Ｌｅｎｄが、合成処理を経て合成処理部１２ｃから最終的に出力される例を挙げる。 FIG. 4 is a diagram showing an example of the output image L end . Further, FIG. 5 is a diagram showing an example of an RGB image L RGB . Further, FIG. 6 is a diagram showing an example of CG data DCg . Further, FIG. 7 is a diagram showing an example of the depth image IDep . In the following, an example will be given in which the output image L end shown in FIG. 4 is finally output from the composition processing unit 12c after undergoing composition processing.
　図４は、後方に存在するマネキンが実空間に存在するマネキンであり、手前の人物がＣＧである例を示している。なお、以下では、説明を分かりやすくするために図を適宜模式化している。したがって、以下に示す例は、本開示の実施形態に係る合成処理の合成品質を限定するものではない。 FIG. 4 shows an example in which the mannequin in the rear is a mannequin that exists in real space, and the person in the foreground is CG. Note that, below, the figures are appropriately simplified in order to make the explanation easier to understand. Therefore, the examples shown below do not limit the synthesis quality of the synthesis processing according to the embodiments of the present disclosure.
　また、図４の図中のＭ１部については後述する。図４に示す出力画像Ｌｅｎｄの合成処理を行う場合、取得部１２ａが取得するＲＧＢ画像ＬＲＧＢは、図５に示すようなものとなる。なお、図５の図中のＭ２部については後述する。 Further, the M1 portion in the diagram of FIG. 4 will be described later. When performing the synthesis process of the output image L end shown in FIG. 4, the RGB image L RGB obtained by the obtaining unit 12a is as shown in FIG. 5. Note that the M2 portion in the diagram of FIG. 5 will be described later.
　また、同じく取得部１２ａが取得するＣＧデータＤＣｇは、図６に示すようなものとなる。同様に、取得部１２ａが取得するデプス画像ＩＤｅｐは、図７に示すようなものとなる。なお、取得部１２ａが取得する全天球画像ＩＳｅｐについては、ここでは省略した。 Further, the CG data D Cg similarly obtained by the obtaining unit 12a is as shown in FIG. Similarly, the depth image IDep acquired by the acquisition unit 12a is as shown in FIG. Note that the spherical image I Sep acquired by the acquisition unit 12a is omitted here.
　合成処理部１２ｃは、これらＲＧＢ画像ＬＲＧＢ、ＣＧデータＤＣｇ、デプス画像ＩＤｅｐおよび全天球画像ＩＳｅｐに基づいて、ＰＹ画像ＬＰＹ、ＶＩ画像ＬＶＩ、マスク画像Ｍ、ＲＰＹ画像ＲＰＹ、ＲＶＩ画像ＲＶＩを生成する。 The synthesis processing unit 12c generates a PY image L PY , a VI image L VI , a mask image M, and an RPY image R PY based on these RGB image L RGB , CG data D Cg , depth image I Dep , and spherical image I Sep , generates an RVI image RVI .
　図８は、ＰＹ画像ＬＰＹの一例を示す図である。また、図９は、ＶＩ画像ＬＶＩの一例を示す図である。また、図１０は、マスク画像Ｍの一例を示す図である。また、図１１は、ＲＰＹ画像ＲＰＹの一例を示す図である。また、図１２は、ＲＶＩ画像ＲＶＩの一例を示す図である。また、図１３は、シャドウインパクト画像ＳＩの一例を示す図である。 FIG. 8 is a diagram showing an example of the PY image LPY . Further, FIG. 9 is a diagram showing an example of the VI image LVI. Moreover, FIG. 10 is a diagram showing an example of the mask image M. Further, FIG. 11 is a diagram showing an example of the RPY image RPY . Further, FIG. 12 is a diagram showing an example of the RVI image RVI . Further, FIG. 13 is a diagram showing an example of a shadow impact image SI .
　図４に示す出力画像Ｌｅｎｄの合成処理を行う場合、ＰＹ画像ＬＰＹは、図８に示すようなものとなる。同様に、ＶＩ画像ＬＶＩは、図９に示すようなものとなる。同様に、マスク画像Ｍは、図１０に示すようなものとなる。 When performing the synthesis process of the output image L end shown in FIG. 4, the PY image L PY becomes as shown in FIG. 8. Similarly, the VI image LVI becomes as shown in FIG. Similarly, the mask image M becomes as shown in FIG.
　ところで、本開示の実施形態では、２段階の合成処理によってＡＤＲ法とＱＩ法の技術的課題を克服する。まず１段階目の合成処理では、ＡＤＲ法とＱＩ法がそれぞれ得意とする領域を検出する重み関数画像とも言えるシャドウインパクト画像ＳＩを新規のアルゴリズムによって生成する。 By the way, in the embodiment of the present disclosure, the technical problems of the ADR method and the QI method are overcome by a two-step synthesis process. First, in the first stage of synthesis processing, a new algorithm generates a shadow impact image SI , which can be called a weighting function image that detects areas in which the ADR method and the QI method are respectively good.
　ここで、シャドウインパクト画像ＳＩの生成アルゴリズムについて説明する。本アルゴリズムでは、以下の式（１０）により定義される、単位面積当たりの光放射エネルギーである放射照度の近似値Ｒを計算する。 Here, the generation algorithm of the shadow impact image SI will be explained. In this algorithm, an approximate value R of irradiance, which is the optical radiation energy per unit area, is calculated by the following equation (10).
Figure JPOXMLDOC01-appb-M000010
Figure JPOXMLDOC01-appb-M000010
Figure JPOXMLDOC01-appb-M000011
Figure JPOXMLDOC01-appb-M000011
Figure JPOXMLDOC01-appb-M000012
Figure JPOXMLDOC01-appb-M000012
　ここで、上記の式（１１）は、画像中のピクセルに対応する実空間の３次元位置ｘに方向ベクトルωｉから物体面に入射する照明の強さであり、測距した全天球画像ＩＳｅｐから計算される。 Here, the above equation (11) is the intensity of illumination incident on the object plane from the direction vector ω i to the three-dimensional position x in real space corresponding to the pixel in the image, and is the intensity of the illumination that enters the object plane from the direction vector ω i , and Calculated from I Sep.
　また、上記の式（１２）は、３次元位置ｘから方向ベクトルωｉを見たときに、合成処理を行うＣＧデータＤＣｇが存在すれば０を、存在しなければ１を返す可視性（Visibility）関数である。加えて、近似値Ｒはカラー要素を持たず、グレースケール化されている。 In addition, the above equation ( 12) shows that when looking at the direction vector ω i from the three-dimensional position Visibility) function. In addition, the approximation R has no color component and is gray scaled.
　本アルゴリズムでは、測距した実写情報のみを用いて放射照度を計算した結果得られるＲＰＹ画像ＲＰＹ、測距した実写情報と合成処理するＣＧデータＤＣｇの両方を用いて放射照度を計算した結果得られるＲＶＩ画像ＲＶＩをそれぞれ生成する。 In this algorithm, the RPY image R PY obtained as a result of calculating irradiance using only the distance-measured real-photo information, and the result of calculating irradiance using both the distance-measured real-photo information and the CG data D Cg to be synthesized. The resulting RVI images RVI are respectively generated.
　図４に示す出力画像Ｌｅｎｄの合成処理を行う場合、ＲＰＹ画像ＲＰＹは、図１１に示すようなものとなる。同様に、ＲＶＩ画像ＲＶＩは、図１２に示すようなものとなる。 When performing the synthesis process of the output image L end shown in FIG. 4, the RPY image RPY becomes as shown in FIG. 11. Similarly, the RVI image RVI becomes as shown in FIG.
　そして、合成処理部１２ｃは、ＲＶＩ画像ＲＶＩとＲＰＹ画像ＲＰＹを用いてシャドウインパクト画像ＳＩを以下の式（１３）によって生成する。 Then, the composition processing unit 12c generates a shadow impact image S1 using the RVI image RVI and the RPY image RPY according to the following equation (13).
Figure JPOXMLDOC01-appb-M000013
Figure JPOXMLDOC01-appb-M000013
　ここで、ｄｉｌａｔｅ（）はライトギャップにより相殺しきれないノイズを軽減するために、画像全体に適用される一般的な膨張処理である。そして、合成処理部１２ｃは、シャドウインパクト画像ＳＩを用いて、以下の式（１４）によってＡＤＲ画像ＬＡＤＲとＱＩ画像ＬＱＩの線形結合を行い、ＬＣ画像ＬＬＣを生成する。 Here, dilate() is a general dilation process applied to the entire image in order to reduce noise that cannot be canceled out due to the light gap. Then, the synthesis processing unit 12c performs a linear combination of the ADR image L ADR and the QI image L QI using the shadow impact image S I according to the following equation (14) to generate an LC image L LC .
Figure JPOXMLDOC01-appb-M000014
Figure JPOXMLDOC01-appb-M000014
　図４に示す出力画像Ｌｅｎｄの合成処理を行う場合、シャドウインパクト画像ＳＩは、図１３に示すようなものとなる。 When performing the synthesis process of the output image L end shown in FIG. 4, the shadow impact image S I becomes as shown in FIG. 13.
　なお、説明が前後するが、ここで、ＡＤＲ画像ＬＡＤＲに現れるＡＤＲ法の欠点、および、ＱＩ画像ＬＱＩに現れるＱＩ法の欠点のそれぞれの具体例を示しておく。図１４は、ＡＤＲ画像ＬＡＤＲの一例を示す図である。また、図１５は、ＱＩ画像ＬＱＩの一例を示す図（その１）である。また、図１６は、ＱＩ画像ＬＱＩの一例を示す図（その２）である。 Although the explanation is complicated, specific examples of the drawbacks of the ADR method that appear in the ADR image L ADR and the drawbacks of the QI method that appear in the QI image L QI will be shown here. FIG. 14 is a diagram showing an example of the ADR image L ADR . Further, FIG. 15 is a diagram (part 1) showing an example of QI image L QI . Further, FIG. 16 is a diagram (part 2) showing an example of QI image L QI .
　図１４は、図４と同様に、人物がＣＧである場合のＡＤＲ画像ＬＡＤＲであるものとする。かかる場合、ＡＤＲ画像ＬＡＤＲには、例えば実空間中のＭ４部の中のＭ４１部に示すように、柱の段差付近での色の不調和等が現れることがある。 As with FIG. 4, FIG. 14 is an ADR image LADR in which the person is a CG image. In such a case, in the ADR image L ADR , color disharmony or the like may appear in the vicinity of the step of the pillar, as shown, for example, in the M41 part of the M4 part in the real space.
　また、ＡＤＲ画像ＬＡＤＲには、例えば同じくＭ４部の中のＭ４２部に示すように、物体の境界での色のにじみ等が現れることがある。 Further, in the ADR image LADR , color blurring or the like may appear at the boundary of the object, as shown in the M42 section of the M4 section, for example.
　また、図１５は、図１４と同様に、人物がＣＧである場合のＱＩ画像ＬＱＩであるものとする。かかる場合、ＱＩ画像ＬＱＩには、例えば実空間中のＭ５部の中のＭ５１部に示すように、ジオメトリノイズが大きい部分（画像の下端部）で、不適当なノイズが走ることがある。 Furthermore, similarly to FIG. 14, it is assumed that FIG. 15 is a QI image LQI when the person is a CG image. In such a case, inappropriate noise may appear in the QI image LQI in a portion where geometry noise is large (lower end of the image), as shown, for example, in the M51 portion of the M5 portion in real space.
　また、図１６は、図１４および図１５とは逆に人物が実物体であり、人物の横に発光する剣を持ったキャラクタのＣＧを合成する場合のＱＩ画像ＬＱＩであるものとする。かかる場合、ＱＩ画像ＬＱＩには、例えば実空間中のＭ６部の中のＭ６１部に示すように、ＣＧの発光する剣による照り返しで明るくなるべき領域が明るくならないといった事態が起こることがある。 Furthermore, in contrast to FIGS. 14 and 15, FIG. 16 is a QI image LQI in which the person is a real object and a CG of a character holding a light-emitting sword is synthesized next to the person. In such a case, in the QI image LQI , a situation may occur in which an area that should be brightened does not become bright due to reflection from the CG light-emitting sword, as shown in the M61 part of the M6 part in the real space, for example.
　前述の１段階目の合成処理では、合成処理部１２ｃは、こうしたＡＤＲ画像ＬＡＤＲおよびＱＩ画像ＬＱＩの欠点が解消されるように、シャドウインパクト画像ＳＩを用いてＡＤＲ画像ＬＡＤＲとＱＩ画像ＬＱＩの線形結合を行い、ＬＣ画像ＬＬＣを生成する。したがって、ＬＣ画像ＬＬＣは、こうした欠点が解消された画像として生成されることとなる。 In the above-mentioned first stage synthesis processing, the synthesis processing unit 12c combines the ADR image L ADR and the QI image using the shadow impact image S I so that the defects of the ADR image L ADR and the QI image L QI are eliminated. A linear combination of LQI is performed to generate an LC image LLC . Therefore, the LC image LLC is generated as an image in which such defects are eliminated.
　次に、２段階目の合成処理では、合成処理部１２ｃは、シャドウインパクト画像ＳＩを用いてライトギャップ領域におけるアーチファクトを除去する。本アルゴリズムでは、ライトギャップ領域を示すライトギャップ画像ｗｇを以下の式（１５）により計算する。 Next, in the second stage of compositing processing, the compositing processing unit 12c uses the shadow impact image SI to remove artifacts in the light gap region. In this algorithm, a light gap image w g indicating a light gap region is calculated using the following equation (15).
Figure JPOXMLDOC01-appb-M000015
Figure JPOXMLDOC01-appb-M000015
　ここで、ｓａｔｕｒａｔｅ（）は入力を０から１に範囲制限する関数であり、ｏｐｅｎｉｎｇ（）はデノイズのためのモルフォロジー演算である。また、λｇ、λｐはハイパーパラメータであり、本開示の実施形態では全ての合成実行において経験的にλｇ＝２．０、λｐ＝２．０に設定する。 Here, saturate() is a function that limits the input range to 0 to 1, and opening() is a morphological operation for denoising. Furthermore, λ g and λ p are hyperparameters, and in the embodiment of the present disclosure, they are empirically set to λ g =2.0 and λ p =2.0 in all synthesis executions.
　そして、最後に、合成処理部１２ｃは、ライトギャップ領域、すなわちｗｇが高い領域にＶＩ画像ＬＶＩを線形結合によって優先的に割り当てることで、アーチファクトが除去された最終的な合成結果である出力画像Ｌｅｎｄを以下の式（１６）のように得る。 Finally, the synthesis processing unit 12c preferentially allocates the VI image LVI to the light gap area, that is, the area where wg is high, by linear combination, and outputs the final synthesis result from which artifacts have been removed. Image L end is obtained as shown in equation (16) below.
Figure JPOXMLDOC01-appb-M000016
Figure JPOXMLDOC01-appb-M000016
　図１７は、ライトギャップ画像ｗｇの一例を示す図（その１）である。また、図１８は、ライトギャップ画像ｗｇの一例を示す図（その２）である。 FIG. 17 is a diagram (part 1) showing an example of the light gap image wg . Further, FIG. 18 is a diagram (part 2) showing an example of the light gap image wg .
　図１７は、図４に示す出力画像Ｌｅｎｄの合成処理を行う場合に対応している。かかる場合、ライトギャップ画像ｗｇは、図１７に示すようなものとなる。 FIG. 17 corresponds to the case where the synthesis process of the output image L end shown in FIG. 4 is performed. In such a case, the light gap image wg will be as shown in FIG. 17.
　図１７に示すＭ７部は、図５に示したＲＧＢ画像ＬＲＧＢのＭ２部に対応している。また、Ｍ７部は、図８に示したＰＹ画像ＬＰＹのＭ３部に対応している。 The M7 section shown in FIG. 17 corresponds to the M2 section of the RGB image LRGB shown in FIG. Furthermore, the M7 section corresponds to the M3 section of the PY image LPY shown in FIG.
　これらＭ２部，Ｍ３部，Ｍ７部を比較のために並べたものが図１８である。図１８に示すように、画像中のマネキンの右腕と胴体の間の空間において、ＰＹ画像ＬＰＹのレンダリング結果は、ＲＧＢ画像ＬＲＧＢと大きく異なることが分かる。合成処理部１２ｃは、図１８に示すように、この部分をライトギャップ画像ｗｇとして検出する。 FIG. 18 shows these M2 portion, M3 portion, and M7 portion arranged side by side for comparison. As shown in FIG. 18, it can be seen that the rendering result of the PY image LPY is significantly different from the RGB image L RGB in the space between the right arm and torso of the mannequin in the image. The composition processing unit 12c detects this portion as a light gap image wg , as shown in FIG.
　また、図１９は、ライトギャップ画像ｗｇを用いた補正の効果を示す図である。図１９は、図４に示した出力画像ＬｅｎｄのＭ１部、かかるＭ１部に対応するＡＤＲ画像ＬＡＤＲのＭ１ＡＤＲ部、および、同様にＭ１部に対応するＱＩ画像ＬＱＩのＭ１ＱＩ部を比較のために並べたものとなっている。 Further, FIG. 19 is a diagram showing the effect of correction using the light gap image wg . FIG. 19 shows the M1 portion of the output image L end shown in FIG. 4, the M1 ADR portion of the ADR image L ADR corresponding to the M1 portion, and the M1 QI portion of the QI image L QI corresponding to the M1 portion. They are arranged side by side for comparison.
　図１８と同様に、図１９に示すように、ライトギャップが大きいマネキンの右腕と胴体の間の空間では、ＡＤＲ画像ＬＡＤＲもＱＩ画像ＬＱＩもアーチファクトが大きいことが分かる。例えば、ＡＤＲ画像ＬＡＤＲは影となるべき部分が明る過ぎることが分かる。また、ＱＩ画像ＬＱＩは色のにじみや不調和があることが分かる。 Similar to FIG. 18, as shown in FIG. 19, it can be seen that in the space between the mannequin's right arm and torso where the light gap is large, artifacts are large in both the ADR image L ADR and the QI image L QI . For example, it can be seen that in the ADR image L ADR , the portion that should be a shadow is too bright. It can also be seen that the QI image LQI has color bleeding and disharmony.
　これに対し、出力画像Ｌｅｎｄは、ライトギャップ画像ｗｇを用いた補正の効果により、ＡＤＲ画像ＬＡＤＲやＱＩ画像ＬＱＩに見られたアーチファクトが解消されていることが分かる。 On the other hand, it can be seen that in the output image L end , the artifacts observed in the ADR image LADR and the QI image L QI are eliminated due to the effect of the correction using the light gap image w g .
　以上のような本開示の実施形態に係る合成処理のアルゴリズムによる効果をまとめる。まず、シャドウインパクト画像ＳＩは、ＣＧデータＤＣｇの合成処理により光放射エネルギーが増加した領域では低い画素値を、光放射エネルギーが減少した領域では高い画素値を持つ。 The effects of the synthesis processing algorithm according to the embodiment of the present disclosure as described above will be summarized. First, the shadow impact image S I has a low pixel value in an area where the light radiant energy has increased due to the synthesis process of the CG data D Cg , and has a high pixel value in an area where the light radiant energy has decreased.
　これは、ＣＧデータＤＣｇの合成処理によりＲＧＢ画像ＬＲＧＢが明るくなる場合と、暗くなる場合とにそれぞれ対応する。ゆえに、シャドウインパクトが低い領域にはＡＤＲ画像ＬＡＤＲを、シャドウインパクトが高い領域にはＱＩ画像ＬＱＩを割り当てることで、両手法の長所を自動的に折衷した合成処理が可能となる。 This corresponds to a case where the RGB image L RGB becomes brighter and a case where it becomes darker due to the composition processing of the CG data DCg . Therefore, by assigning the ADR image L ADR to an area with a low shadow impact and the QI image L QI to an area with a high shadow impact, it is possible to automatically combine the advantages of both methods.
　また、放射照度画像であるＲＶＩ画像ＲＶＩとＲＰＹ画像ＲＰＹの生成には、照明と可視性のみの情報を用いている。一般的なＣＧの手法は、これに加えて空間中の法線の向きも考慮するが、法線はデプス画像ＩＤｅｐが低品位である場合にノイズの影響を受けやすい。そのため、ＲＶＩ画像ＲＶＩとＲＰＹ画像ＲＰＹの生成に法線の情報を用いないことで、低品位なデプス画像ＩＤｅｐに由来するノイズの影響を受けないフィルタを生成することができる。 Further, information only about illumination and visibility is used to generate the RVI image RVI and the RPY image RPY , which are irradiance images. In addition to this, general CG methods also consider the direction of normal lines in space, but normal lines are susceptible to noise when the depth image IDep is of low quality. Therefore, by not using normal information to generate the RVI image RVI and the RPY image RPY , it is possible to generate a filter that is not affected by noise originating from the low-quality depth image IDep .
　次に、シャドウインパクト画像ＳＩを用いたアーチファクトの除去効果を述べる。本アルゴリズムでは、上記の式（１５）によってライトギャップ領域の定義を行い、計算および検出を行う。式（１５）中のｏｐｅｎｉｎｇ関数の引数では、入力ＲＧＢ画像ＬＲＧＢとＰＹ画像ＬＰＹとの差分により表現される測距誤差をλｐによって累乗することで、測距誤差が顕著に大きい領域を検出することができる。 Next, the effect of removing artifacts using the shadow impact image SI will be described. In this algorithm, the light gap region is defined using the above equation (15), and calculation and detection are performed. In the argument of the opening function in equation (15), the range measurement error expressed by the difference between the input RGB image L RGB and the PY image L PY is raised to the power of λ p to identify areas where the range measurement error is significantly large. can be detected.
　さらに、式（１５）中でシャドウインパクト画像ＳＩを掛けることで、ＲＧＢ画像ＬＲＧＢにおいて合成処理により画素値の変化が起きる可能性がある領域にのみアーチファクトの除去処理を限定することができる。 Furthermore, by multiplying the shadow impact image S I in Equation (15), it is possible to limit the artifact removal process to only areas where there is a possibility that a change in pixel value will occur due to the composition process in the RGB image L RGB .
　また、ライトギャップ画像ｗｇは２値画像ではなく、ライトギャップが低い領域から高い領域になめらかに変化するため、ライトギャップ画像ｗｇは急峻なエッジを持たない。その結果、上記の式（１６）中でＶＩ画像ＬＶＩとＬＣ画像ＬＬＣを線形結合させてブレンドする際に、ブレンドした画像の接合部が目立たないように合成処理を行うことができる。 Further, the light gap image w g is not a binary image, and the light gap changes smoothly from a low region to a high region, so the light gap image w g does not have sharp edges. As a result, when the VI image L VI and the LC image L LC are linearly combined and blended in the above equation (16), the synthesis process can be performed so that the joint of the blended images is not noticeable.
　次に、画像合成装置１０が実行する処理手順について、図２０を用いて説明する。図２０は、画像合成装置１０が実行する処理手順を示すフローチャートである。 Next, the processing procedure executed by the image synthesis device 10 will be explained using FIG. 20. FIG. 20 is a flowchart showing the processing procedure executed by the image synthesis device 10.
　図２０に示すように、まず取得部１２ａが、ＲＧＢ画像ＬＲＧＢ、デプス画像ＩＤｅｐ、全天球画像ＩＳｅｐ、ＣＧデータＤＣｇを取得する（ステップＳ１０１）。そして、変換部１２ｂが、ＲＧＢ画像ＬＲＧＢ、デプス画像ＩＤｅｐ、全天球画像ＩＳｅｐをＤＣＣツールに読み込み可能に変換する（ステップＳ１０２）。そして、合成処理部１２ｃが、変換後データおよびＣＧデータＤＣｇをＤＣＣツールで読み込む（ステップＳ１０３）。 As shown in FIG. 20, the acquisition unit 12a first acquires an RGB image L RGB , a depth image I Dep , a spherical image I Sep , and CG data D Cg (step S101). Then, the conversion unit 12b converts the RGB image L RGB , the depth image I Dep , and the spherical image I Sep so that they can be read into the DCC tool (step S102). Then, the composition processing unit 12c reads the converted data and the CG data DCg using the DCC tool (step S103).
　つづいて、合成処理部１２ｃは、ＤＣＣツール内でＰＹ画像ＬＰＹ、ＶＩ画像ＬＶＩ、マスク画像Ｍ、ＲＰＹ画像ＲＰＹ、ＲＶＩ画像ＲＶＩを生成する（ステップＳ１０４）。 Subsequently, the synthesis processing unit 12c generates a PY image L PY , a VI image L VI , a mask image M, an RPY image R PY , and an RVI image R VI within the DCC tool (step S104).
　そして、合成処理部１２ｃは、ＰＹ画像ＬＰＹ、ＶＩ画像ＬＶＩ、マスク画像Ｍ、ＲＧＢ画像ＬＲＧＢを用いてＡＤＲ画像ＬＡＤＲ、ＱＩ画像ＬＱＩを生成する（ステップＳ１０５）。 Then, the synthesis processing unit 12c generates the ADR image L ADR and the QI image L QI using the PY image L PY , the VI image L VI , the mask image M, and the RGB image L RGB ( step S105).
　また、これと並列に、合成処理部１２ｃは、ＲＰＹ画像ＲＰＹ、ＲＶＩ画像ＲＶＩを用いてシャドウインパクト画像ＳＩを生成する（ステップＳ１０６）。 Further, in parallel with this, the synthesis processing unit 12c generates a shadow impact image S I using the RPY image R PY and the RVI image R VI (step S106).
　ステップＳ１０５，Ｓ１０６が終了すると、合成処理部１２ｃは、ＡＤＲ画像ＬＡＤＲ、ＱＩ画像ＬＱＩ、シャドウインパクト画像ＳＩを用いてＬＣ画像ＬＬＣを生成する（ステップＳ１０７）。 When steps S105 and S106 are completed, the synthesis processing unit 12c generates an LC image LLC using the ADR image L ADR , the QI image L QI , and the shadow impact image S I (step S107).
　また、これと並列に、合成処理部１２ｃは、シャドウインパクト画像ＳＩ、ＲＧＢ画像ＬＲＧＢ、ＰＹ画像ＬＰＹを用いてライトギャップ画像ｗｇを生成する（ステップＳ１０８）。 Further, in parallel with this, the synthesis processing unit 12c generates a light gap image w g using the shadow impact image S I , the RGB image L RGB , and the PY image L PY (step S108).
　ステップＳ１０７，Ｓ１０８が終了すると、合成処理部１２ｃは、ＬＣ画像ＬＬＣ、ライトギャップ画像ｗｇを用いて出力画像Ｌｅｎｄを生成する（ステップＳ１０９）。そして、出力部１２ｄが出力画像Ｌｅｎｄを出力し（ステップＳ１１０）、処理を終了する。 When steps S107 and S108 are completed, the synthesis processing unit 12c generates an output image L end using the LC image L LC and the light gap image w g (step S109). Then, the output unit 12d outputs the output image L end (step S110), and the process ends.
　次に、図２１は、静止画を出力する場合の処理手順を示すフローチャートである。まず、ユーザがＲＧＢカメラでＲＧＢ画像ＬＲＧＢを撮影する（ステップＳ２０１）。また、ユーザは、デプスカメラでデプス画像ＩＤｅｐを撮影する（ステップＳ２０２）。また、ユーザは、全天球カメラで全天球画像ＩＳｅｐを撮影する（ステップＳ２０３）。また、ユーザは、ＤＣＣツールでＣＧデータＤＣｇを作成する（ステップＳ２０４）。 Next, FIG. 21 is a flowchart showing the processing procedure when outputting a still image. First, a user photographs an RGB image L RGB with an RGB camera (step S201). Further, the user photographs a depth image I Dep with a depth camera (step S202). Further, the user photographs a spherical image I Sep with a spherical camera (step S203). Further, the user creates CG data DCg using the DCC tool (step S204).
　そして、画像合成装置１０は、ステップＳ２０１～Ｓ２０４で取得された各データを入力し、図２０に示した画像合成処理を実行する（ステップＳ２０５）。 Then, the image synthesis device 10 inputs each data acquired in steps S201 to S204 and executes the image synthesis process shown in FIG. 20 (step S205).
　そして、例えば表示装置が、画像合成装置１０から出力された出力画像Ｌｅｎｄを静止画として表示し（ステップＳ２０６）、処理を終了する。 Then, for example, the display device displays the output image L end output from the image synthesis device 10 as a still image (step S206), and the process ends.
　次に、図２２は、動画を出力する場合の処理手順を示すフローチャートである。なお、図２２のステップＳ３０１～Ｓ３０４は、図２１に示したステップＳ２０１～Ｓ２０４と同様であるため、ここでの説明を省略する。 Next, FIG. 22 is a flowchart showing the processing procedure when outputting a moving image. Note that steps S301 to S304 in FIG. 22 are the same as steps S201 to S204 shown in FIG. 21, so the description thereof will be omitted here.
　つづいて、ユーザはフレーム番号に応じてＣＧデータＤＣｇを更新する（ステップＳ３０５）。そして、画像合成装置１０は、ステップＳ３０１～Ｓ３０５で取得された各データを入力し、図２０に示した画像合成処理を実行する（ステップＳ３０６）。 Subsequently, the user updates the CG data DCg according to the frame number (step S305). Then, the image synthesis device 10 inputs each data acquired in steps S301 to S305, and executes the image synthesis process shown in FIG. 20 (step S306).
　そして、画像合成装置１０は、画像合成処理によって出力された出力画像Ｌｅｎｄが所定のフレーム数に到達したか否かを判定する（ステップＳ３０７）。ここで、所定のフレーム数に到達していなければ（ステップＳ３０７，Ｎｏ）、フレーム番号を更新し（ステップＳ３０８）、ステップＳ３０５からの処理を繰り返す。 Then, the image synthesis device 10 determines whether the output image L end outputted by the image synthesis process has reached a predetermined number of frames (step S307). Here, if the predetermined number of frames has not been reached (step S307, No), the frame number is updated (step S308), and the processing from step S305 is repeated.
　一方、所定のフレーム数に到達していれば（ステップＳ３０７，Ｙｅｓ）、例えば表示装置が、各フレームの出力画像Ｌｅｎｄを時系列に結合し、動画として表示する（ステップＳ３０９）。そして、処理を終了する。 On the other hand, if the predetermined number of frames has been reached (step S307, Yes), for example, the display device combines the output images L end of each frame in time series and displays it as a moving image (step S309). Then, the process ends.
＜＜４．変形例＞＞
　上述した本開示の実施形態において説明した各処理のうち、自動的に行われるものとして説明した処理の全部又は一部を手動的に行うこともでき、あるいは、手動的に行われるものとして説明した処理の全部又は一部を公知の方法で自動的に行うこともできる。この他、上記文書中や図面中で示した処理手順、具体的名称、各種のデータやパラメータを含む情報については、特記する場合を除いて任意に変更することができる。例えば、各図に示した各種情報は、図示した情報に限られない。 <<4. Modified example >>
Among the processes described in the embodiments of the present disclosure described above, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed manually. All or part of the processing can also be performed automatically using known methods. In addition, information including the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.
　また、図示した各装置の各構成要素は機能概念的なものであり、必ずしも物理的に図示の如く構成されていることを要しない。すなわち、各装置の分散・統合の具体的形態は図示のものに限られず、その全部又は一部を、各種の負荷や使用状況などに応じて、任意の単位で機能的又は物理的に分散・統合して構成することができる。 Furthermore, each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. In other words, the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.
　また、上述した本開示の実施形態は、処理内容を矛盾させない領域で適宜組み合わせることが可能である。また、本実施形態のシーケンス図或いはフローチャートに示された各ステップは、適宜順序を変更することが可能である。 Furthermore, the embodiments of the present disclosure described above can be combined as appropriate in areas where the processing contents do not conflict. Further, the order of each step shown in the sequence diagram or flowchart of this embodiment can be changed as appropriate.
＜＜５．ハードウェア構成＞＞
　また、上述してきた本開示の実施形態に係る画像合成装置１０は、例えば図２３に示すような構成のコンピュータ１０００によって実現される。図２３は、画像合成装置１０の機能を実現するコンピュータ１０００の一例を示すハードウェア構成図である。コンピュータ１０００は、ＣＰＵ１１００、ＲＡＭ１２００、ＲＯＭ１３００、ＨＤＤ（Hard　Disk　Drive）１４００、通信インターフェイス１５００、及び入出力インターフェイス１６００を有する。コンピュータ１０００の各部は、バス１０５０によって接続される。 <<5. Hardware configuration >>
Further, the image synthesis apparatus 10 according to the embodiment of the present disclosure described above is realized by, for example, a computer 1000 having a configuration as shown in FIG. 23. FIG. 23 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the image synthesis apparatus 10. Computer 1000 has CPU 1100, RAM 1200, ROM 1300, HDD (Hard Disk Drive) 1400, communication interface 1500, and input/output interface 1600. Each part of computer 1000 is connected by bus 1050.
　ＣＰＵ１１００は、ＲＯＭ１３００又はＨＤＤ１４００に格納されたプログラムに基づいて動作し、各部の制御を行う。例えば、ＣＰＵ１１００は、ＲＯＭ１３００又はＨＤＤ１４００に格納されたプログラムをＲＡＭ１２００に展開し、各種プログラムに対応した処理を実行する。 The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each part. For example, the CPU 1100 loads programs stored in the ROM 1300 or HDD 1400 into the RAM 1200, and executes processes corresponding to various programs.
　ＲＯＭ１３００は、コンピュータ１０００の起動時にＣＰＵ１１００によって実行されるＢＩＯＳ（Basic　Input　Output　System）等のブートプログラムや、コンピュータ１０００のハードウェアに依存するプログラム等を格納する。 The ROM 1300 stores boot programs such as BIOS (Basic Input Output System) that are executed by the CPU 1100 when the computer 1000 is started, programs that depend on the hardware of the computer 1000, and the like.
　ＨＤＤ１４００は、ＣＰＵ１１００によって実行されるプログラム、及び、かかるプログラムによって使用されるデータ等を非一時的に記録する、コンピュータが読み取り可能な記録媒体である。具体的には、ＨＤＤ１４００は、プログラムデータ１４５０の一例である本開示の実施形態に係る情報処理プログラムを記録する記録媒体である。 The HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by the programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to an embodiment of the present disclosure, which is an example of program data 1450.
　通信インターフェイス１５００は、コンピュータ１０００が外部ネットワーク１５５０（例えばネットワークＮ）と接続するためのインターフェイスである。例えば、ＣＰＵ１１００は、通信インターフェイス１５００を介して、他の機器からデータを受信したり、ＣＰＵ１１００が生成したデータを他の機器へ送信したりする。 Communication interface 1500 is an interface for connecting computer 1000 to external network 1550 (for example, network N). For example, CPU 1100 receives data from other devices or transmits data generated by CPU 1100 to other devices via communication interface 1500.
　入出力インターフェイス１６００は、入出力デバイス１６５０とコンピュータ１０００とを接続するためのインターフェイスである。例えば、ＣＰＵ１１００は、入出力インターフェイス１６００を介して、キーボードやマウス等の入力デバイスからデータを受信する。また、ＣＰＵ１１００は、入出力インターフェイス１６００を介して、ディスプレイやスピーカーやプリンタ等の出力デバイスにデータを送信する。また、入出力インターフェイス１６００は、所定の記録媒体（メディア）に記録されたプログラム等を読み取るメディアインターフェイスとして機能してもよい。メディアとは、例えばＤＶＤ（Digital　Versatile　Disc）、ＰＤ（Phase　change　rewritable　Disk）等の光学記録媒体、ＭＯ（Magneto-Optical　disk）等の光磁気記録媒体、テープ媒体、磁気記録媒体、または半導体メモリ等である。 The input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input/output interface 1600. Further, the CPU 1100 transmits data to an output device such as a display, speaker, or printer via an input/output interface 1600. Furthermore, the input/output interface 1600 may function as a media interface that reads programs and the like recorded on a predetermined recording medium. Media includes, for example, optical recording media such as DVD (Digital Versatile Disc) and PD (Phase change rewritable disk), magneto-optical recording media such as MO (Magneto-Optical disk), tape media, magnetic recording media, semiconductor memory, etc. It is.
　例えば、コンピュータ１０００が本開示の実施形態に係る画像合成装置１０として機能する場合、コンピュータ１０００のＣＰＵ１１００は、ＲＡＭ１２００上にロードされたプログラムを実行することにより、制御部１２の機能を実現する。また、ＨＤＤ１４００には、本開示に係る情報処理プログラムや、記憶部１１内のデータが格納される。なお、ＣＰＵ１１００は、プログラムデータ１４５０をＨＤＤ１４００から読み取って実行するが、他の例として、外部ネットワーク１５５０を介して、他の装置からこれらのプログラムを取得してもよい。 For example, when the computer 1000 functions as the image composition device 10 according to the embodiment of the present disclosure, the CPU 1100 of the computer 1000 realizes the functions of the control unit 12 by executing a program loaded onto the RAM 1200. Further, the HDD 1400 stores an information processing program according to the present disclosure and data in the storage unit 11. Note that although the CPU 1100 reads and executes the program data 1450 from the HDD 1400, as another example, these programs may be obtained from another device via the external network 1550.
＜＜６．むすび＞＞
　以上説明したように、本開示の一実施形態によれば、画像合成装置１０（「情報処理装置」の一例に相当）は、ＲＧＢ画像ＬＲＧＢ、デプス画像ＩＤｅｐおよび全天球画像ＩＳｅｐを入力としてＲＧＢ画像ＬＲＧＢに対応する実空間を表す３次元空間にＣＧデータＤＣｇを配置する合成処理を行う合成処理部１２ｃを備える。 <<6. Conclusion >>
As described above, according to an embodiment of the present disclosure, the image synthesis device 10 (corresponding to an example of an "information processing device") generates an RGB image L RGB , a depth image I Dep , and a spherical image I Sep. A composition processing unit 12c is provided which performs composition processing of arranging CG data DCg in a three-dimensional space representing a real space corresponding to the RGB image L RGB as an input.
　合成処理部１２ｃは、上記入力から測距された実写情報のみを用いたＰＹ画像ＬＰＹ（「第１のレンダリング画像」の一例に相当）と、上記実写情報およびＣＧデータＤＣｇを用いたＶＩ画像ＬＶＩ（「第２のレンダリング画像」の一例に相当）とを生成する。 The synthesis processing unit 12c generates a PY image L PY (corresponding to an example of a "first rendered image") using only the real shot information measured from the above input, and a VI using the above real shot information and CG data D Cg . An image LVI (corresponding to an example of a "second rendered image") is generated.
　また、合成処理部１２ｃは、ＰＹ画像ＬＰＹおよびＶＩ画像ＬＶＩの陰影の差に基づくＡＤＲ法（「第１の手法」の一例に相当）によるＡＤＲ画像ＬＡＤＲ（「第１の合成画像」の一例に相当）と、上記陰影の比率に基づくＱＩ法（「第２の手法」の一例に相当）によるＱＩ画像ＬＱＩ（「第２の合成画像」の一例に相当）とを生成する。 In addition, the synthesis processing unit 12c generates an ADR image L ADR (“first composite image”) based on the ADR method (corresponding to an example of the “first method”) based on the difference in shading between the PY image L PY and the VI image L VI . ) and a QI image L QI (corresponding to an example of a "second composite image") by the QI method (corresponding to an example of a "second method") based on the shading ratio.
　また、合成処理部１２ｃは、上記実写情報のみを用いた場合と、上記実写情報およびＣＧデータＤＣｇを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像ＳＩを生成し、シャドウインパクト画像ＳＩを用いてＡＤＲ画像ＬＡＤＲとＱＩ画像ＬＱＩとの線形結合を行う。 In addition, the composition processing unit 12c generates a shadow impact image S I representing a change in optical radiation energy between the case where only the above-mentioned live-action information is used and the case where the above-mentioned live-action information and CG data DCg are used, A linear combination of the ADR image L ADR and the QI image L QI is performed using the shadow impact image S I.
　これにより、実空間を表す３次元空間に対するＣＧデータＤＣｇの合成品質をより向上させることができる。 This makes it possible to further improve the quality of compositing the CG data DCg with respect to the three-dimensional space representing the real space.
　以上、本開示の各実施形態について説明したが、本開示の技術的範囲は、上述の各実施形態そのままに限定されるものではなく、本開示の要旨を逸脱しない範囲において種々の変更が可能である。また、異なる実施形態及び変形例にわたる構成要素を適宜組み合わせてもよい。 Although each embodiment of the present disclosure has been described above, the technical scope of the present disclosure is not limited to each of the above-mentioned embodiments as is, and various changes can be made without departing from the gist of the present disclosure. be. Furthermore, components of different embodiments and modifications may be combined as appropriate.
　また、本明細書に記載された各実施形態における効果はあくまで例示であって限定されるものでは無く、他の効果があってもよい。 Further, the effects in each embodiment described in this specification are merely examples and are not limited, and other effects may also be provided.
　なお、本技術は以下のような構成も取ることができる。
（１）
　ＲＧＢ画像、デプス画像および全天球画像を入力として前記ＲＧＢ画像に対応する実空間を表す３次元空間にＣＧデータを配置する合成処理を行う合成処理部
　を備え、
　前記合成処理部は、
　前記入力から測距された実写情報のみを用いた第１のレンダリング画像と、前記実写情報および前記ＣＧデータを用いた第２のレンダリング画像とを生成し、
　前記第１のレンダリング画像および前記第２のレンダリング画像の陰影の差に基づく第１の手法による第１の合成画像と、前記陰影の比率に基づく第２の手法による第２の合成画像とを生成し、
　前記実写情報のみを用いた場合と、前記実写情報および前記ＣＧデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成し、
　前記シャドウインパクト画像を用いて前記第１の合成画像と前記第２の合成画像との線形結合を行う、
　情報処理装置。
（２）
　前記合成処理部は、
　前記合成処理により光放射エネルギーが増加した領域では低い画素値を有し、光放射エネルギーが減少した領域では高い画素値を有するように前記シャドウインパクト画像を生成し、前記低い画素値の領域には前記第１の合成画像を、前記高い画素値の領域には前記第２の合成画像をそれぞれ割り当てる、
　前記（１）に記載の情報処理装置。
（３）
　前記合成処理部は、
　前記低い画素値の領域は前記合成処理により明るくなる前記ＲＧＢ画像の領域に対応し、前記高い画素値の領域は前記合成処理により暗くなる前記ＲＧＢ画像の領域に対応するように前記シャドウインパクト画像を生成する、
　前記（２）に記載の情報処理装置。
（４）
　前記合成処理部は少なくとも、
　測距された前記全天球画像から算出される照明に関する情報、および、前記ＣＧデータの可視性に関する情報に基づいて、前記シャドウインパクト画像を生成する、
　前記（１）、（２）または（３）に記載の情報処理装置。
（５）
　前記合成処理部は、
　前記照明に関する情報および前記可視性に関する情報のみに基づいて、前記シャドウインパクト画像を生成する、
　前記（４）に記載の情報処理装置。
（６）
　前記合成処理部は、
　前記シャドウインパクト画像を用いて、前記合成処理において生じるアーチファクトを除去する、
　前記（１）～（５）のいずれか一つに記載の情報処理装置。
（７）
　前記合成処理部は、
　前記第１のレンダリング画像が前記ＲＧＢ画像よりも極めて暗くなるライトギャップ領域を示すライトギャップ画像を生成し、前記ライトギャップ領域に対し前記第２のレンダリング画像を線形結合によって優先的に割り当てることで前記アーチファクトを除去する、
　前記（６）に記載の情報処理装置。
（８）
　前記合成処理部は、
　前記第１のレンダリング画像と前記ＲＧＢ画像の差分により表現される測距誤差が極めて大きい領域を前記ライトギャップ領域として検出し、さらに前記シャドウインパクト画像を掛けることによって、前記アーチファクトの除去が及ぶ領域を限定する、
　前記（７）に記載の情報処理装置。
（９）
　ＲＧＢ画像、デプス画像および全天球画像を入力として前記ＲＧＢ画像に対応する実空間を表す３次元空間にＣＧデータを配置する合成処理を行うこと、
　を含み、
　前記合成処理を行うことは、
　前記入力から測距された実写情報のみを用いた第１のレンダリング画像と、前記実写情報および前記ＣＧデータを用いた第２のレンダリング画像とを生成することと、
　前記第１のレンダリング画像および前記第２のレンダリング画像の陰影の差に基づく第１の手法による第１の合成画像と、前記陰影の比率に基づく第２の手法による第２の合成画像とを生成することと、
　前記実写情報のみを用いた場合と、前記実写情報および前記ＣＧデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成することと、
　前記シャドウインパクト画像を用いて前記第１の合成画像と前記第２の合成画像との線形結合を行うことと、
　をさらに含む、情報処理方法。
（１０）
　ＲＧＢ画像、デプス画像および全天球画像を入力として前記ＲＧＢ画像に対応する実空間を表す３次元空間にＣＧデータを配置する合成処理を行うこと、
　をコンピュータに実行させ、
　前記合成処理を行うことは、
　前記入力から測距された実写情報のみを用いた第１のレンダリング画像と、前記実写情報および前記ＣＧデータを用いた第２のレンダリング画像とを生成すること、
　前記第１のレンダリング画像および前記第２のレンダリング画像の陰影の差に基づく第１の手法による第１の合成画像と、前記陰影の比率に基づく第２の手法による第２の合成画像とを生成すること、
　前記実写情報のみを用いた場合と、前記実写情報および前記ＣＧデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成すること、
　前記シャドウインパクト画像を用いて前記第１の合成画像と前記第２の合成画像との線形結合を行うこと、
　をさらに前記コンピュータに実行させる情報処理プログラムが記録されたコンピュータ読み取り可能な記録媒体。 Note that the present technology can also have the following configuration.
(1)
a composition processing unit that receives an RGB image, a depth image, and a spherical image as input and performs a composition process of arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
The synthesis processing section is
Generating a first rendered image using only the live-action information measured from the input, and a second rendering image using the live-action information and the CG data,
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. death,
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
Information processing device.
(2)
The synthesis processing section is
The shadow impact image is generated so that areas where light radiant energy has increased through the synthesis process have low pixel values, and areas where light radiant energy has decreased have high pixel values, and the areas with low pixel values have low pixel values. assigning the first composite image and the second composite image to the high pixel value area, respectively;
The information processing device according to (1) above.
(3)
The synthesis processing section is
The shadow impact image is created such that the low pixel value area corresponds to the area of the RGB image that becomes brighter due to the compositing process, and the high pixel value area corresponds to an area of the RGB image that becomes dark due to the compositing process. generate,
The information processing device according to (2) above.
(4)
The synthesis processing section at least includes:
generating the shadow impact image based on information regarding illumination calculated from the distance-measured spherical image and information regarding visibility of the CG data;
The information processing device according to (1), (2) or (3) above.
(5)
The synthesis processing section is
generating the shadow impact image based only on the illumination information and the visibility information;
The information processing device according to (4) above.
(6)
The synthesis processing section is
using the shadow impact image to remove artifacts occurring in the compositing process;
The information processing device according to any one of (1) to (5) above.
(7)
The synthesis processing section is
By generating a light gap image indicating a light gap region in which the first rendered image is much darker than the RGB image, and preferentially allocating the second rendered image to the light gap region by linear combination. remove artifacts,
The information processing device according to (6) above.
(8)
The synthesis processing section is
An area where the distance measurement error expressed by the difference between the first rendered image and the RGB image is extremely large is detected as the light gap area, and the area where the artifacts are removed is determined by multiplying the area by the shadow impact image. limit,
The information processing device according to (7) above.
(9)
performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
including;
Performing the above-mentioned compositing process includes:
Generating a first rendered image using only the live-action information measured from the input, and a second rendered image using the live-action information and the CG data;
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. to do and
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
Information processing methods, further including:
(10)
performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
make the computer run
Performing the above-mentioned compositing process includes:
generating a first rendered image using only live-action information measured from the input, and a second rendered image using the live-action information and the CG data;
Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. to do,
generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
performing a linear combination of the first composite image and the second composite image using the shadow impact image;
A computer-readable recording medium having recorded thereon an information processing program that causes the computer to further execute the following.
　１０　画像合成装置
　１１　記憶部
　１１ａ　ジオメトリ情報
　１１ｂ　反射率情報
　１１ｃ　照明マップ情報
　１１ｄ　ＤＣＣツールプログラム
　１２　制御部
　１２ａ　取得部
　１２ｂ　変換部
　１２ｃ　合成処理部
　１２ｃａ　第１生成部
　１２ｃｂ　第２生成部
　１２ｃｃ　第３生成部
　１２ｃｄ　第４生成部
　１２ｃｅ　第５生成部
　１２ｃｆ　出力画像生成部
　１２ｄ　出力部 10 Image synthesis device 11 Storage unit 11a Geometry information 11b Reflectance information 11c Illumination map information 11d DCC tool program 12 Control unit 12a Acquisition unit 12b Conversion unit 12c Synthesis processing unit 12ca First generation unit 12cb Second generation unit 12cc Third generation unit 12cd Fourth generation section 12ce Fifth generation section 12cf Output image generation section 12d Output section

Claims (10)

  1. 　ＲＧＢ画像、デプス画像および全天球画像を入力として前記ＲＧＢ画像に対応する実空間を表す３次元空間にＣＧデータを配置する合成処理を行う合成処理部
    　を備え、
    　前記合成処理部は、
    　前記入力から測距された実写情報のみを用いた第１のレンダリング画像と、前記実写情報および前記ＣＧデータを用いた第２のレンダリング画像とを生成し、
    　前記第１のレンダリング画像および前記第２のレンダリング画像の陰影の差に基づく第１の手法による第１の合成画像と、前記陰影の比率に基づく第２の手法による第２の合成画像とを生成し、
    　前記実写情報のみを用いた場合と、前記実写情報および前記ＣＧデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成し、
    　前記シャドウインパクト画像を用いて前記第１の合成画像と前記第２の合成画像との線形結合を行う、
    　情報処理装置。     a composition processing unit that receives an RGB image, a depth image, and a spherical image as input and performs a composition process of arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
    The synthesis processing section is
    Generating a first rendered image using only the live-action information measured from the input, and a second rendering image using the live-action information and the CG data,
    Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. death,
    generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
    performing a linear combination of the first composite image and the second composite image using the shadow impact image;
    Information processing device.
  2. 　前記合成処理部は、
    　前記合成処理により光放射エネルギーが増加した領域では低い画素値を有し、光放射エネルギーが減少した領域では高い画素値を有するように前記シャドウインパクト画像を生成し、前記低い画素値の領域には前記第１の合成画像を、前記高い画素値の領域には前記第２の合成画像をそれぞれ割り当てる、
    　請求項１に記載の情報処理装置。     The synthesis processing section is
    The shadow impact image is generated so that areas where light radiant energy has increased through the synthesis process have low pixel values, and areas where light radiant energy has decreased have high pixel values, and the areas with low pixel values have low pixel values. assigning the first composite image and the second composite image to the high pixel value area, respectively;
    The information processing device according to claim 1.
  3. 　前記合成処理部は、
    　前記低い画素値の領域は前記合成処理により明るくなる前記ＲＧＢ画像の領域に対応し、前記高い画素値の領域は前記合成処理により暗くなる前記ＲＧＢ画像の領域に対応するように前記シャドウインパクト画像を生成する、
    　請求項２に記載の情報処理装置。     The synthesis processing section is
    The shadow impact image is created such that the low pixel value area corresponds to the area of the RGB image that becomes brighter due to the compositing process, and the high pixel value area corresponds to an area of the RGB image that becomes dark due to the compositing process. generate,
    The information processing device according to claim 2.
  4. 　前記合成処理部は少なくとも、
    　測距された前記全天球画像から算出される照明に関する情報、および、前記ＣＧデータの可視性に関する情報に基づいて、前記シャドウインパクト画像を生成する、
    　請求項１に記載の情報処理装置。     The synthesis processing section at least includes:
    generating the shadow impact image based on information regarding illumination calculated from the distance-measured spherical image and information regarding visibility of the CG data;
    The information processing device according to claim 1.
  5. 　前記合成処理部は、
    　前記照明に関する情報および前記可視性に関する情報のみに基づいて、前記シャドウインパクト画像を生成する、
    　請求項４に記載の情報処理装置。     The synthesis processing section is
    generating the shadow impact image based only on the illumination information and the visibility information;
    The information processing device according to claim 4.
  6. 　前記合成処理部は、
    　前記シャドウインパクト画像を用いて、前記合成処理において生じるアーチファクトを除去する、
    　請求項１に記載の情報処理装置。     The synthesis processing section is
    using the shadow impact image to remove artifacts occurring in the compositing process;
    The information processing device according to claim 1.
  7. 　前記合成処理部は、
    　前記第１のレンダリング画像が前記ＲＧＢ画像よりも極めて暗くなるライトギャップ領域を示すライトギャップ画像を生成し、前記ライトギャップ領域に対し前記第２のレンダリング画像を線形結合によって優先的に割り当てることで前記アーチファクトを除去する、
    　請求項６に記載の情報処理装置。     The synthesis processing section is
    By generating a light gap image indicating a light gap region in which the first rendered image is much darker than the RGB image, and preferentially allocating the second rendered image to the light gap region by linear combination. remove artifacts,
    The information processing device according to claim 6.
  8. 　前記合成処理部は、
    　前記第１のレンダリング画像と前記ＲＧＢ画像の差分により表現される測距誤差が極めて大きい領域を前記ライトギャップ領域として検出し、さらに前記シャドウインパクト画像を掛けることによって、前記アーチファクトの除去が及ぶ領域を限定する、
    　請求項７に記載の情報処理装置。     The synthesis processing section is
    An area where the distance measurement error expressed by the difference between the first rendered image and the RGB image is extremely large is detected as the light gap area, and the area where the artifacts are removed is determined by multiplying the area by the shadow impact image. limit,
    The information processing device according to claim 7.
  9. 　ＲＧＢ画像、デプス画像および全天球画像を入力として前記ＲＧＢ画像に対応する実空間を表す３次元空間にＣＧデータを配置する合成処理を行うこと、
    　を含み、
    　前記合成処理を行うことは、
    　前記入力から測距された実写情報のみを用いた第１のレンダリング画像と、前記実写情報および前記ＣＧデータを用いた第２のレンダリング画像とを生成することと、
    　前記第１のレンダリング画像および前記第２のレンダリング画像の陰影の差に基づく第１の手法による第１の合成画像と、前記陰影の比率に基づく第２の手法による第２の合成画像とを生成することと、
    　前記実写情報のみを用いた場合と、前記実写情報および前記ＣＧデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成することと、
    　前記シャドウインパクト画像を用いて前記第１の合成画像と前記第２の合成画像との線形結合を行うことと、
    　を含む、情報処理方法。     performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
    including;
    Performing the above-mentioned compositing process includes:
    Generating a first rendered image using only the live-action information measured from the input, and a second rendered image using the live-action information and the CG data;
    Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. to do and
    generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
    performing a linear combination of the first composite image and the second composite image using the shadow impact image;
    information processing methods, including
  10. 　ＲＧＢ画像、デプス画像および全天球画像を入力として前記ＲＧＢ画像に対応する実空間を表す３次元空間にＣＧデータを配置する合成処理を行うこと、
    　をコンピュータに実行させ、
    　前記合成処理を行うことは、
    　前記入力から測距された実写情報のみを用いた第１のレンダリング画像と、前記実写情報および前記ＣＧデータを用いた第２のレンダリング画像とを生成すること、
    　前記第１のレンダリング画像および前記第２のレンダリング画像の陰影の差に基づく第１の手法による第１の合成画像と、前記陰影の比率に基づく第２の手法による第２の合成画像とを生成すること、
    　前記実写情報のみを用いた場合と、前記実写情報および前記ＣＧデータを用いた場合との間の光放射エネルギーの変化を表すシャドウインパクト画像を生成すること、
    　前記シャドウインパクト画像を用いて前記第１の合成画像と前記第２の合成画像との線形結合を行うこと、
    　をさらに前記コンピュータに実行させる情報処理プログラムが記録されたコンピュータ読み取り可能な記録媒体。     performing a composition process of inputting an RGB image, a depth image, and a spherical image and arranging CG data in a three-dimensional space representing a real space corresponding to the RGB image;
    make the computer run
    Performing the above-mentioned compositing process includes:
    generating a first rendered image using only live-action information measured from the input, and a second rendered image using the live-action information and the CG data;
    Generating a first composite image using a first method based on a difference in shading between the first rendered image and the second rendered image, and a second composite image using a second method based on the ratio of the shadings. to do,
    generating a shadow impact image representing a change in optical radiation energy between a case where only the live-action information is used and a case where the live-action information and the CG data are used;
    performing a linear combination of the first composite image and the second composite image using the shadow impact image;
    A computer-readable recording medium having recorded thereon an information processing program that causes the computer to further execute the following.
PCT/JP2023/008482 2022-03-24 2023-03-07 Information processing device, information processing method, and recording medium WO2023181904A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022048330 2022-03-24
JP2022-048330 2022-03-24

Publications (1)

Publication Number Publication Date
WO2023181904A1 true WO2023181904A1 (en) 2023-09-28

Family

ID=88100750

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/008482 WO2023181904A1 (en) 2022-03-24 2023-03-07 Information processing device, information processing method, and recording medium

Country Status (1)

Country Link
WO (1) WO2023181904A1 (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009163610A (en) * 2008-01-09 2009-07-23 Canon Inc Image processing apparatus and image processing method
JP2013127774A (en) * 2011-11-16 2013-06-27 Canon Inc Image processing device, image processing method, and program

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009163610A (en) * 2008-01-09 2009-07-23 Canon Inc Image processing apparatus and image processing method
JP2013127774A (en) * 2011-11-16 2013-06-27 Canon Inc Image processing device, image processing method, and program

Similar Documents

Publication Publication Date Title
CA2810865C (en) Streaming light propagation
AU2006318379B2 (en) Methods and apparatus for determining high quality sampling data from low quality sampling data
JP6639877B2 (en) Image processing apparatus and method
US10572984B2 (en) Method for inverse tone mapping of an image with visual effects
US10388063B2 (en) Variable rate shading based on temporal reprojection
JP2010510571A (en) Method and system for color correction of 3D image
JP5731566B2 (en) Information processing apparatus, control method, and recording medium
US10529125B2 (en) Image processing device and method therefor
EP3355279A1 (en) Method and computer programs for identifying differences between 3-dimensional scenes
CN113920036A (en) Interactive relighting editing method based on RGB-D image
AU2023233092A1 (en) Artificial intelligence techniques for extrapolating hdr panoramas from ldr low fov images
WO2023181904A1 (en) Information processing device, information processing method, and recording medium
US8942476B1 (en) Saturation varying and lighting independent color color control for computer graphics
KR102448551B1 (en) Method and apparatus processing images by using algorithm configured by node based graph
Gierlinger et al. Rendering techniques for mixed reality
Dai Stylized rendering for virtual furniture layout
Colbert et al. Painting in high dynamic range
JP7116767B2 (en) Image processing device and method
Ludwig et al. Environment map based lighting for reflectance transformation images
US11380048B2 (en) Method and system for determining a spectral representation of a color
Koch et al. Semantic Screen-Space Occlusion for Multiscale Molecular Visualization.
Qi et al. Denoising 1SPP Monte carlo renderings based on human visual perception
Sharif et al. Halo reduction multi-exposure image fusion technique
GB2624103A (en) Artificial intelligence techniques for extrapolating HDR panoramas from LDR low FOV images
FRASER et al. PROJECT PROPOSAL: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23774503

Country of ref document: EP

Kind code of ref document: A1