JP2021068272A - Image processing system, image processing method and program - Google Patents

Abstract

To provide an image processing system which estimates information about a light source in a space of interest from a photographed image of a real space of interest and a three-dimensional shape model of the space of interest without preparing an object having a known shape and preparing a reference image by capturing an image of the object under a known light source and which correctly reproduces the incidence of light at any point of the three-dimensional shape model.SOLUTION: An image processing system comprises: a shape information estimation unit which estimates space shape information including at least a three-dimensional shape of a space of interest from a single or a plurality of photographed images obtained by photographing the space of interest; and a light source information estimation unit which estimates light source information including at least the position and intensity of a light source in the space of interest from the photographed images and the space shape information.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing system, an image processing method and a program.

When remodeling a building or adding equipment, it may be desirable to check the appearance after completion in advance before actually remodeling or arranging equipment.
For example, when adding furniture, walls, and equipment as equipment, or considering the position of the added furniture, what kind of impression will be given when the equipment is placed, that is, the equipment that is placed is the room. There is a request to visually examine whether or not it is in harmony with the environment of the furniture by using the image of the simulation result.
At this time, if the light source of the room is not accurately reflected in the virtual space, the impression obtained from the appearance in the simulation and the impression obtained from the appearance in the actual layout change may be significantly different.

For this reason, the three-dimensional shape of this target space is modeled as a virtual space from the captured image of the target space, which is the space to be reformed, and the equipment is arranged in this virtual space to display the equipment in the room or the like in the virtual space. The appearance is simulated using virtual space.
At this time, in order to make the appearance of the article in the virtual space closer to the target space, it is possible to accurately estimate the light source information of the light source in the room as the target space and generate a virtual light source in the virtual space based on this light source information. It has been done (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2018-036884

However, in Cited Document 1, when an image captured in the target space is captured, an object having a known shape and for which a reference image captured under a known light source is prepared in advance is arranged with respect to the target space, and the captured image is captured. The light source information is estimated by comparing the image of the object captured in the image with the reference image.
Therefore, it is necessary to prepare an object having a known shape, and to prepare an image of this object under a known light source to prepare a reference image, which takes time and effort to image the target space.

Further, in this method, only the information of the light emitted from the light source can be estimated, and the three-dimensional shape of the space in which the radiated light is emitted is unknown.
Therefore, it is unclear whether the light emitted from the light source is shielded or reflected, and it is not possible to simulate the appearance corresponding to the target space in the virtual space with high accuracy.

The present invention has been made in view of such a situation, and it is not necessary to prepare an object having a known shape and to take an image of this object under a known light source to prepare a reference image, which is an actual object. Provided are an image processing system, an image processing method, and a program for estimating at least information on the position and intensity of a light source in the target space from an captured image of the space and a three-dimensional shape model of the target space.

The image processing system of the present invention includes a shape information estimation unit that estimates spatial shape information including at least a three-dimensional shape of the target space from a single or a plurality of captured images of the target space, the captured image, and the above-mentioned image processing system. It is characterized by including a light source information estimation unit that estimates light source information including at least the position and intensity of the light source in the target space from the space shape information.

In the image processing system of the present invention, the light source information estimation unit estimates the reflectance information of the three-dimensional shape in the target space from the captured image and the space shape information, and uses the reflectance information to estimate the light source information. Is characterized by estimating.

In the image processing system of the present invention, the light source information estimation unit estimates the pixel value of the captured image from a predetermined formula showing the relationship between the reflectance information, the light source information, and the spatial shape information. It is characterized in that the light source information is estimated using the reflectance information obtained.

The image processing system of the present invention generates an observation image of a virtual space in which a shape model of a predetermined object is arranged at a predetermined position of the captured image from the captured image, the spatial shape information, and the light source information. It is characterized by further including an image compositing unit.

In the image processing system of the present invention, the shape model of the predetermined object is arranged at a predetermined position of the captured image from the captured image, the spatial shape information, and the light source information, and the shape model is arranged. It is characterized by further including an image compositing unit that generates an observation image of a virtual space in which the influence is reflected.

The image processing system of the present invention is characterized in that the light source information estimation unit generates IBL (image based lighting) information from the reflectance information of the three-dimensional shape, the spatial shape information, and the light source information.

In the image processing system of the present invention, the shape information estimation unit separately estimates each of the three-dimensional shapes of the structural part and the object part in the target space, and obtains the three-dimensional shape of each of the structural part and the object part. It is characterized in that a three-dimensional shape model of the entire target space is generated by synthesizing.

The image processing system of the present invention is characterized in that the shape information estimation unit estimates the three-dimensional shape of the target space and the shape information of the light source including the object recognition information in the target space.

The image processing system of the present invention is further provided with an imaging condition acquisition unit that acquires imaging conditions when the captured image is captured.

The image processing method of the present invention includes a shape information estimation process in which the shape information estimation unit estimates spatial shape information including at least a three-dimensional shape of the target space from a single or a plurality of captured images in which the target space is captured. The light source information estimation unit includes a light source information estimation process for estimating light source information including at least the position and intensity of the light source in the target space from the captured image and the space shape information.

The program of the present invention is a shape information estimation means for estimating spatial shape information including at least a three-dimensional shape of the target space from a single or a plurality of captured images in which the target space is captured, the captured image, and the above. This is a program that functions as a light source information estimation means for estimating light source information including at least the position and intensity of a light source in the target space from space shape information.

As described above, according to the present invention, it is not necessary to prepare an object having a known shape and to capture an image of this object under a known light source to prepare a reference image, and an image captured in an actual target space. An image processing system, an image processing method, and a program that estimate the information of the light source in the target space from the three-dimensional shape model of the target space and accurately reproduce the incident of light at an arbitrary point of the three-dimensional shape model. It will be possible to provide.

It is a block diagram which shows the structural example of the image processing system by one Embodiment of this invention. It is a figure explaining the process of extracting the vanishing point portion and the boundary portion of the ceiling, the wall and the floor in a room from the captured image, and obtaining a depth map. It is a figure which shows the area division of the object of the object space by the machine learning model which performs semantic segmentation. It is a figure explaining the composition of the depth map generated by the machine learning model and the depth map generated by the Manhattan world hypothesis. It is a figure which shows the three-dimensional shape of the room structure part (ceiling, wall and floor) obtained from the depth map generated by the Manhattan world hypothesis. It is a figure which shows the 3D shape obtained from the depth map of the object part obtained by the area divided by the semantic segmentation from the depth map obtained by the machine learning model in the 3D shape estimation method (1). It is a figure which the light source information estimation unit 14 shows in the grayscale image the area (light source area) of the light source extracted by the light source information estimation method (1). It is a reflectance component image which shows the reflectance information in the spatial shape information obtained by a light source information estimation unit 14. An example is shown in which the light source information estimated by the light source information estimation unit 14 is arranged in a three-dimensional shape in the spatial shape information. It is a figure which shows the image which looked at the 3D virtual space generated by the image synthesis part from the image-taking direction of the captured image. It is a figure explaining the control which performs the additional arrangement of a CG object with respect to a virtual space. It is a figure explaining the control which performs the additional arrangement of a CG object with respect to a virtual space. It is a flowchart which shows the operation example of the process which synthesizes the virtual space generated from the selected captured image of the target space, and a CG object in the image processing system by this embodiment.

Hereinafter, a configuration example of the image processing system in FIG. 1 will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration example of an image processing system according to an embodiment of the present invention. In FIG. 1, the image processing system 10 includes a data input / output unit 11, an imaging condition acquisition unit 12, a shape information estimation unit 13, a light source information estimation unit 14, an image synthesis unit 15, a display control unit 16, a display unit 17, and an captured image. Each of a storage unit 18, a spatial information storage unit 19, and a composite image storage unit 20 is provided.
Here, the image processing system 10 is configured by installing an application that performs image processing from each functional unit described below on a personal computer, a tablet terminal, a smartphone, or the like.

The data input / output unit 11 inputs the captured image in which the target space to be simulated is captured from an external device, adds the captured image identification information, and writes and stores the captured image in the captured image storage unit 18.
The imaging condition acquisition unit 12 acquires the imaging conditions when the captured image of the target space is captured, corresponds to the captured image identification information of each captured image, and writes and stores the captured image storage unit 18. The imaging conditions include the camera parameters of the imaging device, the sensor size, the actual size distance from the imaging device to the three-dimensional shape of the target space, the number of pixels of the captured image, the positional relationship of the captured imaging position, and the frame if it is a moving image. Rate etc. Further, it is not necessary for the user to input the information that can be acquired from the Exif (exchangeable image file format) information of the captured image under the imaging conditions. In the case of a single image, the actual size distance is calculated by placing a marker whose size is known in the target space to be imaged and based on the image of the marker.

The shape information estimation unit 13 estimates the three-dimensional shape of the captured target space from the captured image stored in the captured image storage unit 18, and generates spatial shape information including the three-dimensional shape model (virtual space). ..
Then, the shape information estimation unit 13 writes and stores the generated spatial shape information in the spatial information storage unit 19. This spatial shape information includes at least the data of the three-dimensional shape model of the target space, the normal vector, and the like.
In the present embodiment, the shape information estimation unit 13 generates a three-dimensional shape model by the following three three-dimensional shape estimation methods.

Three-dimensional shape estimation method (1):
The shape information estimation unit 13 uses a machine learning model to estimate the depth map directly from a single captured image (RGB image) in which the target space is captured (for example, Ibraheem Alhashim, Peter Wonka "High Quality Monocular Depth". Estimation via Transfer Learning "CoRR, Vol. Abs / 1901.03861, 2019. Use the method described in the literature).
The shape information estimation unit 13 can calculate a three-dimensional point cloud and a normal in each pixel of the captured image based on the obtained depth map, and mesh the three-dimensional point cloud.
Then, the three-dimensional point cloud writes and stores the information of the virtual space, which is a meshed three-dimensional shape model, as the spatial shape information in the captured image storage unit 18 in correspondence with the captured image.

Further, when there are a plurality of captured images captured from different imaging positions in the same target space, the shape information estimation unit 13 utilizes the fact that the same target space is imaged to obtain SfM (structure from motion). ), The positional relationship between the respective imaging positions of the captured image is estimated.
Then, the shape information estimation unit 13 integrates each of the depth maps obtained from each of the captured images to generate a more accurate depth map of the target space.

Three-dimensional shape estimation method (2):
Further, when there are a plurality of captured images captured from different imaging positions in the same target space, the shape information estimation unit 13 uses the plurality of captured images by using the algorithm of MVS (Multi-View Stereo). The three-dimensional shape of the target space may be estimated, and a virtual space, which is a three-dimensional shape model, may be generated and used as space shape information.

Three-dimensional shape estimation method (3):
Further, the shape information estimation unit 13 may use an algorithm for separately obtaining a shape model for each of the room structure unit in the target space and the object unit (room equipment, etc.) arranged in other rooms. good.
As shown in FIG. 2 below, the shape information estimation unit 13 extracts the boundary between the ceiling and the wall and the boundary between the wall and the floor from the captured image.

FIG. 2 is a diagram illustrating a process of extracting a boundary portion between a ceiling, a wall, and a floor in a room from a captured image and obtaining a depth map. FIG. 2A shows the captured image 100. In the captured image 100, the ceiling 200C, the wall 200W, the floor 200F, the pillar 200P, the table 200T, the light source 200L, and the like in the room 200 of the target space are imaged.
FIG. 2B shows a boundary portion 200CW between the ceiling 200C and the wall 200W and a boundary portion 200WF between the wall 200W and the floor 200F. Further, in FIG. 2B, information on the pillar 200P, the table 200T, and the light source 200L is not extracted.
FIG. 2C shows a depth map obtained from the captured image of FIG. 2A and the boundary portion of FIG. 2B.

Then, the shape information estimation unit 13 uses the Manhattan world hypothesis to obtain the three axes (x-axis, y-axis and the y-axis) of the three-dimensional space from the captured image of FIG. 2 (a) and the boundary portion of FIG. 2 (b). Corresponds to the z-axis), it detects ceilings, walls and floors and generates a depth map for each. The detection of ceilings, walls and floors is performed directly from RGB images by a machine learning model (for example, CNN (convolutional neural network)), for example, "Chuhang Zou, Alex Colburn, Qi Shan, Derek Hoiem" LayoutNet: Reconstructing. There is the 3D Room Layout from a Single RGB Image "Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 2051-2059."
Further, the shape information estimation unit 13 extracts each region of the object in the captured image by a machine learning model (for example, CNN) that performs semantic segmentation.
FIG. 3 is a diagram showing the region division of an object in the target space by a machine learning model that performs semantic segmentation. In FIG. 3, the regions of each object in the target space captured in the captured image are separated as segments.
The shape information estimation unit 13 uses a machine learning model that performs semantic segmentation, and in the image of the captured image 100, the structural parts of the ceiling 200C, the wall 200W, the floor 200F, the pillar 200P, the light sources 200L_1, 200L_2, the table 200T, and the like. It is separated as each image area of the object part.

As described above, the shape information estimation unit 13 captures the captured image of the room in the target space, such as the room structure unit (ceiling, wall, floor) and the object unit (pillars, desks (tables), chairs, shelves, etc. as equipment). By dividing into areas such as furniture), each area is recognized.
Further, the shape information estimation unit 13 estimates the depth map directly from one captured image (RGB image) in which the target space is captured by using the machine learning model described in the three-dimensional shape estimation method (1). ..
Then, the shape information estimation unit 13 extracts the area of the pillar 200P and the table 200T as the area of the object part excluding the area of the room structure part from the image area where the target space is separated in the semantic segmentation result of FIG.

FIG. 4 is a diagram for explaining the composition of the depth map generated by the machine learning model and the depth map generated by the Manhattan World hypothesis.
FIG. 4A is a column 200P extracted from the depth map generated by the machine learning model described in the three-dimensional shape estimation method (1) using the column 200P of the object part and the area of the table 200T in the semantic segmentation result. , Table 200T shows the depth map of each.
FIG. 4B shows a target space in which the depth map of the room structure generated by the Manhattan world hypothesis shown in FIG. 2C and the depth map of the object portion shown in FIG. 4A are scaled and integrated. An example of a depth map is shown.

As shown in FIG. 4B, the shape information estimation unit 13 synthesizes the depth map generated by the machine learning model and the depth map generated by the Manhattan world hypothesis.
As a result, the Manhattan World hypothesis is based on the assumption that many of the artifacts are constrained in parallel to the Cartesian coordinate system, so that depth maps in room structures such as ceilings, walls, and floors can be accurately mapped. Obtainable.
However, the depth map is not generated for the object part such as the table or the chair in the target space because it does not correspond to the above constraint.

On the other hand, in the depth map generated by the machine learning model described in the three-dimensional shape estimation method (1), the depth map is also generated for the object part in the target space.
However, in the depth map obtained by machine learning, it is not possible to recognize which is the area where the depth map of the object part exists.
Therefore, the shape information estimation unit 13 uses the region of the object portion (200P, 200T) in the depth map obtained by machine learning to extract from the region of the object portion in the semantic segmentation shown in FIG. Acquire the depth map of the area of the object part shown in a). As a result, the depth map of the room structure part and the object part can be obtained separately.
At this time, the shape information estimation unit 13 scales the depth map of the object unit extracted from the depth map obtained by machine learning based on the values of the room structure unit of FIG.

By this three-dimensional shape estimation method (3), the depth map showing the three-dimensional shape of the room structure part shielded by the object part is more accurate than the depth map obtained by using the three-dimensional shape estimation method (1). Obtainable.
Then, the shape information estimation unit 13 obtains a three-dimensional point cloud and a normal at each three-dimensional point for each depth map of the room structure unit and the object unit, and meshes these three-dimensional point cloud points. Is performed, and it is written and stored in the spatial information storage unit 19 as spatial shape information.
By this method, it becomes possible to memorize the three-dimensional shape of the room structure portion shielded by the object portion.

Further, when there are a plurality of captured images captured from different imaging positions in the same target space, the shape information estimation unit 13 utilizes the fact that the same target space is imaged to obtain SfM (structure from motion). ), The positional relationship between the respective imaging positions of the captured image is estimated.
Then, the shape information estimation unit 13 integrates the depth maps of the room structure unit and the object unit obtained from each of the captured images in each of the room structure unit and the object unit, and the room in the target space with higher accuracy. Generates the depth map of the structure part and the object part.

FIG. 5 is a diagram showing the three-dimensional shape of the room structure (ceiling, wall and floor) obtained from the depth map generated by the Manhattan World hypothesis.
FIG. 5 shows the three-dimensional shape of the room structure portion including the ceiling 200C, the wall 200W, and the floor 200F of the room in the target space.

FIG. 6 is a diagram showing a three-dimensional shape obtained from the depth map of the object portion obtained by the region divided by the semantic segmentation from the depth map obtained by the machine learning model in the three-dimensional shape estimation method (1).
FIG. 6 shows the three-dimensional shapes of the pillar 200P and the table 200T obtained from the depth map of the object portion in the virtual space 250 of the target space.

As described above, the shape information estimation unit 13 estimates the three-dimensional shape of the target space by, for example, the three-dimensional shape estimation method (1) to the three-dimensional shape estimation method (3).
However, the above-mentioned methods from the three-dimensional shape estimation method (1) to the three-dimensional shape estimation method (3) are shown as an example of the three-dimensional restoration technique from the captured image. The three-dimensional restoration technique used in the shape information estimation unit 13 is not necessarily limited to a specific method, and any generally used method may be used.

Further, since the depth map obtained by the three-dimensional shape estimation method (1) to the three-dimensional shape estimation method (3) described above is obtained as a relative distance, the distance from the actual imaging position is unknown. The actual distance in the depth map is estimated by the following method, and the size of each object in the restored target space is calculated as the actual size.
For example, if the actual size distance between a specific pixel in the imaging device and the three-dimensional shape of the target space is stored as an imaging condition when the captured image stored in the captured image storage unit 18 is captured, the distance is stored. Using this actual size distance, the distance of each pixel in the depth map can be obtained.

Alternatively, an object of known size may be arranged and imaged, and each of the objects in the target space may be scaled based on the image of the captured object of known size. At this time, the pixel area of the known object is extracted by image analysis such as pattern recognition, semantic segmentation, and object recognition of the known object, and the depth map of the other area is obtained by using the pixel depth value in the extracted area. Scale to actual size.
Further, a configuration may be used in which the correspondence between the amount of blurring of the image and the focal length of the imaging device is acquired in advance by a measurement experiment, and the distance is estimated from the amount of blurring of the captured image.

The light source information estimation unit 14 estimates the light source information from the captured image of the target space and the spatial shape information (three-dimensional shape of the virtual space) of the target space, and supplies the estimated light source information to the spatial information storage unit 19. Write and memorize. This light source information is at least the position of the light source in the target space, the intensity of the light emitted by the light source, and the tint of the light (spectral information).
In the present embodiment, the light source information estimation unit 14 generates light source information by the following three light source information estimation methods.

Light source information estimation method (1):
The light source information estimation unit 14 converts the captured image (RGB image) into a grayscale image, and extracts the region of the pixel having the highest degree of gradation.
Then, the light source information estimation unit 14 is, for example, a region of the pixel having the highest gradation degree and a pixel having a gradation degree of a predetermined ratio (for example, 90% or more) with respect to the gradation degree of the pixel in the region. The area is extracted as the area of the light source.

FIG. 7 is a diagram showing a region of the light source (light source region) extracted by the light source information estimation method (1) in a grayscale image by the light source information estimation unit 14. In FIG. 7, the grayscale image 300 is a grayscale image of the captured image 100 (FIG. 2A) which is an RGB image.
Among the pixels of the grayscale image 300, the pixels of the region 301 have the highest gradation. The pixels in the area 302 have a gradation degree of 90% or more of the pixels in the area 301. As a result, the light source information estimation unit 14 extracts each of the area 301 and the area 302 as the light source area.

Light source information estimation method (2):
The light source information estimation unit 14 extracts a light source region from the captured image by the light source information estimation method (1) described above.
Then, the light source information estimation unit 14 arranges a point light source as a three-dimensional shape of the light source region in the space shape information which is a virtual space based on the shape information of the light source region extracted by the shape information estimation unit 13.
Further, the light source information estimation unit 14 estimates the reflectance information (reflectance information for each wavelength) of the target space from the three-dimensional shape in the captured image and the spatial shape information.

Here, the light source information estimation unit 14 in estimating the reflectance information of the target space, for example, is a method of the algorithm of the Intrinsic Image Problem based on the intrinsic image decomposition (for example, Qifeng Chen, Vladlen Koltun "A Simple". Model for Intrinsic Image Decomposition with Depth Cues "The IEEE International Conference on Computer Vision (ICCV), 2013, pp. 241-248) is used.
The light source information estimation unit 14 separates the reflectance and the shadow by the algorithm of the Intricic Image Problem.
That is, when the light source information estimation unit 14 separates the captured image into each of the reflectance component image (reflectance information) and the shadow component image, the light source information estimation unit 14 "images (captured image) I is the reflectance" based on the algorithm of the Intrinsic Image Problem. Based on the assumption that it can be represented by the product of the component R and the shadow component S, the captured image I is separated into the reflectance component image R and the shadow component image S, respectively.

FIG. 8 is a reflectance component image showing the reflectance information of the three-dimensional shape in the spatial shape information obtained by the light source information estimation unit 14.
In FIG. 8, in the reflectance component image, the reflectance component (reflectance information) of each pixel indicated by the two-dimensional coordinate points indicated by the two-dimensional coordinate values in the captured image is a pixel value of RGB (Red Green Blue). It is shown.
On the other hand, in the shadow component image, the shadow component of each two-dimensional coordinate point indicated by the two-dimensional coordinate value in the captured image, that is, the data depending on the light source when the captured image is captured is shown as the pixel value.

Next, the light source information estimation unit 14 estimates the intensity of the light source based on the gradation of each pixel of the captured image and the reflectance information of the three-dimensional shape in the spatial shape information.
Then, the light source information estimation unit 14 arbitrarily sets a region A for obtaining the intensity of the light source in the captured image, and extracts the reflectance information of this region A.
Here, it is desirable that the region A to be set is a region in which only the light of the target light source for which the intensity and the tint are to be obtained (or most of the incident light is only the light of the target light source) is irradiated.
That is, when there are a plurality of light sources, the amount of calculation for obtaining the intensity and color of the target light source is reduced so that the calculation for obtaining the influence from each light source is not complicated.

Next, the light source information estimation unit 14 extracts the region A set in the captured image and the region B at the position corresponding to the region A in the reflectance component image.
Then, the light source information estimation unit 14 obtains the intensity and color of the light source from the RGB gradation of the region A and the reflectance information of the region B.
Further, when determining the intensity of the light source, the light source information estimation unit 14 considers that the intensity of light is inversely proportional to the square of the distance, and in the spatial shape information with respect to the intensity of light estimated to be incident on the region B. The intensity of the light source is obtained by multiplying the square of the distance between the light source and the region B obtained from the three-dimensional shape.

By this light source information estimation method (2), even if the intensity of the light source saturates the RGB values of the captured image, the intensity of the light source is indirectly obtained from the region where the light from the light source is incident, so that accurate light source information can be obtained. Can be obtained.
FIG. 9 shows an example in which the light source information estimated by the light source information estimation unit 14 is arranged in a three-dimensional shape in the spatial shape information. In FIG. 9, the light sources 200L_1 and 200L_2 are arranged in the virtual space 250 of the target space corresponding to the estimated positions in the three-dimensional shape of the target space.

Light source information estimation method (3):
The light source information estimation unit 14 extracts a light source region from the captured image by the light source information estimation method (1). Further, the light source information estimation unit 14 is the third dimension of the light source region in the virtual space in the spatial shape information based on the shape information of the light source region extracted by the shape information estimation unit 13 as in the light source information estimation method (2) described above. Place a point light source as the original shape.
At this time, the intensity and color of the point light source are set to temporary arbitrary numerical values (temporary light source information), and for example, the RGB value of the region estimated to be the point light source of the captured image is set as a temporary numerical value.

Next, the light source information estimation unit 14 extracts the coordinate points of the three-dimensional shape in the spatial shape information corresponding to each pixel of the captured image, and under the assumption that the light is completely diffused and reflected at these coordinate points, each of the point light sources The shadow information when light is incident from is calculated.
Then, the light source information estimation unit 14 obtains the reflectance information in each pixel from the pixel value of the captured image and the shadow information of the corresponding pixel of the shadow component image. That is, the light source information estimation unit 14 divides the pixel value of the captured image by the shadow information of the corresponding pixel of the shadow component image, and uses the division result as the reflectance information.

Then, the light source information estimation unit 14 scales the reflectance information using the assumption that the average value of the reflectance of the natural object is 20%.
That is, the light source information estimation unit 14 obtains the reflectance A (x, y) of each pixel of the captured image by using the following equation (1). Here, the light source information estimation unit 14 obtains a scaling coefficient α at which the average value of the reflectances A (x, y) of all the pixels in the captured image is 20%. That is, the light source information estimation unit 14 is the result of dividing the pixel value I (x, y) of the captured image by the shadow information (pixel value S (x, y) of the shadow component image) of the corresponding pixel of the shadow component image. In the numerical value, the scaling coefficient α is obtained so that the average value of the reflectance A (x, y) is 20%. The light source information estimation unit 14 sets the reflectance A (x, y) when the scaling coefficient α is obtained as the reflectance A (x, y) of each pixel of the captured image.

Next, the light source information estimation unit 14 uses the reflectances A (x, y) of each pixel in the captured image obtained by the equation (1), and the color of the light source and the color of the light source according to the equation (2) shown below. Obtain the intensity light source information. At this time, the light source information estimation unit 14 adjusts the intensity and color of the point light source so that the pixel value I (x, y) obtained by Eq. (2) is the same as the pixel value at the corresponding position of the captured image. The intensity and color of the point light source are finally obtained by changing from a tentative arbitrary numerical value.
In the following equation (2), A (x, y) indicates the reflectance of the pixel (x, y), k (l) indicates the color of the light source l, and k _α (l) indicates the light of the light source l. R (x, y, l) indicates the distance from the pixel (x, y) to the light source l, and n (x, y, l) is the vector from the pixel (x, y) to the light source l. , And n _I (x, y) indicates the normal vector of the pixel (x, y).
_{The light source information estimation unit 14 sets the color k (l) and the intensity k α} (l), which are the light source information of each light source, to (2) so as to be the pixel value I (x, y) of each pixel of the captured image. ) Calculated by numerical calculation.

Further, as the algorithm for obtaining the light source information, a method other than the above-mentioned light source information estimation method (1) to the light source information estimation method (3) may be used.
In each of the above-mentioned light source information estimation method (1) to light source information estimation method (3), the light source has been described as a point light source, but for example, a line light source or a surface light source other than the point light source may be used.
For example, the light source information estimation unit 14 obtains the object recognition result (shape of the light source in the captured image) of each light source by using the semantic segmentation already described. Then, the light source information estimation unit 14 uses a point light source if the object recognition result is a light bulb shape, a directional light (parallel light source) if the object recognition result is a downlight shape, and a line light source if the object recognition result is a fluorescent lamp shape. If it has a window shape, it will be a surface light source.

Further, as the light source information, IBL (image based lighting) information at an arbitrary position in the virtual space where the CG object generated by CG is synthesized is generated by the captured image, the spatial shape information of the target space, and the light source information. be able to.
As a result, the image synthesizing unit 15 can easily obtain the light source information at the position where the CG object generated by CG is arranged from the IBL information based on the captured image and the spatial shape information of the target space. Here, the data format of the IBL information in which the pixel values are not saturated can be used for the light source information obtained by the light source information estimation method (2) and the light source information estimation method (3).

The image synthesizing unit 15 generates a three-dimensional virtual space based on the captured image, the spatial shape information of the target space, and the light source information based on the light source information.
Here, the image synthesizing unit 15 uses the three-dimensional shape in the spatial shape information as a shape model, the captured image as texture data, and the light source information as a rendering parameter to generate a virtual space in the target space.

FIG. 10 is a diagram showing an image of a three-dimensional virtual space generated by the image synthesizing unit as viewed from the imaging direction of the captured image. FIG. 10 is a diagram in which the display control unit 16 displays the virtual space on the display screen of the display unit 17 as a screen observed from a predetermined viewpoint direction (for example, an imaging direction).
In FIG. 10, when the three-dimensional virtual space generated by the captured image is displayed as an image observed from a predetermined viewpoint, the captured image is set as texture data for a plane or a celestial sphere according to the angle of view of the imaging device. (Paste) to create a virtual space. Alternatively, in the virtual space, in order to display the observed image without discomfort even if the viewpoint position to be observed is changed (without discomfort within a predetermined range with respect to the imaging position), the captured image should be set as texture data in the shape model. (Paste) just do.

In FIG. 10, in the shape model of the pillar 200P, a bright bright region 200PL irradiated with light from each of the light sources 200L_1 and 200L_2 and a shadow region 200PS (shadow) not irradiated with light are used as texture data in the shape model. It is formed by using the captured image, such as a process of pasting (setting) the captured image.
Similarly, at the position where the shape model of the table 200T is arranged, the floor 200F below the table 200T has a shadow area 200TS generated by shielding the light from each of the light sources 200L_1 and 200L_2 by the table 200T. It is formed in the same way.
Further, on the floor 200F, a gradation region 501 indicating the degree of brightness intensity corresponding to the distances from the light sources 200L_1 and 200L_2 is also formed by using the captured image as described above.

Further, when the image synthesizing unit 15 is controlled to add and arrange the CG object generated by CG to the generated three-dimensional virtual space, the image synthesizing unit 15 additionally arranges the CG object to the virtual space. I do.
In the present embodiment, each of the CG objects generated by CG is, for example, generated in advance and is written and stored in the composite image storage unit 20.

FIG. 11 is a diagram illustrating control for additionally arranging CG objects in the virtual space. FIG. 11 is a diagram in which the display control unit 16 displays the virtual space on the display screen of the display unit 17 as a screen observed from a predetermined viewpoint direction (for example, the imaging direction) as in FIG. 10.
In FIG. 11, the CG objects 503_1, 503_2, etc. displayed in the CG object presentation area 503 are displayed by the image synthesizing unit 15 on the display screen 502 of the display unit 17 via the display control unit 16.

For example, when the user drags the CG object 503_1 in the CG object presentation area 503 and drops it at the arrangement position 504, the image synthesizing unit 15 arranges the CG object 503_1 at the arrangement position 504 in the virtual space.

FIG. 12 is a diagram illustrating control for additionally arranging CG objects in the virtual space. FIG. 12 is a diagram in which the display control unit 16 displays the virtual space on the display screen of the display unit 17 as a screen observed from a predetermined viewpoint direction (for example, the imaging direction) as in FIG. 10.
In FIG. 12, the virtual space is reconstructed by reflecting the light source information of each of the light sources 200L_1 and 200L_1 with respect to the shape information of the CG object 503_1 corresponding to the CG object 503_1 arranged in the virtual space.

The image synthesizing unit 15 corresponds to the shape information of the arranged CG object 503_1 and the light source information (arrangement position, color and intensity of light) of the light sources 200L_1 and 200L_2, and determines the brightness due to the irradiation light from the light sources 200L_1 and 200L_2. The reflected rendering is performed on the CG object 503_1.
As a result, the image synthesizing unit 15 corresponds to the shape information of the arranged CG object 503_1 by a technique such as ray tracing, and the brightness of the upper part irradiated with the light from the light sources 200L_1 and 200L_2. The intensity of the CG object 503_1 is increased, the area shielded by the three-dimensional shape of the CG object 503_1 is obtained as a shadow, and the brightness intensity of the shadow area 503_1S to be a shadow is decreased.

Further, the image synthesizing unit 15 reflects the irradiation light by using the reflectance information and the three-dimensional shape of the ceiling 200C, the wall 200W, the floor 200F, the pillar 200P, the table 200T, etc. in the virtual space obtained by the light source information estimation unit 14. The brightness of the components may be reflected in the rendering.
That is, the image synthesizing unit 15 corresponds to the three-dimensional shape in the virtual space and the light source information (arrangement position, light tint and intensity) of the light sources 200L_1 and 200L_2, and the reflected light of the irradiation light from the light sources 200L_1 and 200L_2. The effect of the brightness on the CG object arranged in the virtual space may be reflected on the rendering of the virtual space.

FIG. 13 is a flowchart showing an operation example of the process of synthesizing the virtual space generated from the selected captured image of the target space and the CG object in the image processing system according to the present embodiment. In the following description, the data input / output unit 11 transmits the captured image identification information to the captured image storage unit 18 for a single or a plurality of captured images in which each of the target spaces is captured in advance, which is supplied from an external device. It is added, and each of the captured images is written and stored together with the imaging conditions at the time of imaging.

Step S1:
The data input / output unit 11 displays the captured image stored in the captured image storage unit 18 on the display unit 17 by the display control unit 16.
The user selects from the captured images displayed on the display unit 17 by clicking the captured image of the target space to be observed with the mouse or the like.
As a result, the data input / output unit 11 outputs the captured image identification information of the captured image selected by the user to each of the imaging condition acquisition unit 12, the shape information estimation unit 13, the light source information estimation unit 14, and the image synthesis unit 15. Output.

Step S2:
The imaging condition acquisition unit 12 reads out the imaging conditions of the captured image selected by the user from the captured image storage unit 18 based on the captured image identification information of the captured image.
Then, the imaging condition acquisition unit 12 outputs the read imaging condition to each of the shape information estimation unit 13 and the light source information estimation unit 14.

Step S3:
The shape information estimation unit 13 is an object of the captured image selected by a method such as the three-dimensional shape estimation method (1) to the three-dimensional shape estimation method (3) already described using each of the captured image and the imaging condition. Estimate the three-dimensional shape of space.
Then, the shape information estimation unit 13 adds the captured image identification information of the corresponding captured image to the spatial information storage unit 19 by using the generated three-dimensional shape data of the target space as the spatial shape information and stores it. Let me.

Step S4:
The light source information estimation unit 14 reads out the spatial shape information corresponding to the captured image identification information from the spatial information storage unit 19.
Then, the light source information estimation unit 14 is selected by any method from the light source information estimation method (1) to the light source information estimation method (3), which has already been described, using each of the captured image and the spatial shape information. Estimate the light source information in the virtual space corresponding to the captured image. The light source information estimation unit 14 writes and stores the estimated light source information in the spatial information storage unit 19 in correspondence with the spatial shape information.

Step S5:
The image synthesizing unit 15 reads out each of the spatial shape information and the light source information corresponding to the captured image identification information from the spatial information storage unit 19, and reads out the captured image corresponding to the captured image identification information from the captured image storage unit 18.
The image synthesizing unit 15 uses the light source information and the captured image for the three-dimensional shape in the spatial shape information to generate a virtual space corresponding to the target space captured in the captured image.
Then, the display control unit 16 generates an observation image observed from the position of the viewpoint in the virtual space and displays it on the display screen of the display unit 17 as shown in FIG.

Step S6:
For example, the user controls the image processing system 10 to add (or change) a CG object to the virtual space corresponding to the observation image displayed on the display screen of the display unit 17.
As a result, the image synthesizing unit 15 reads the image data of the CG object stored in advance in the synthesizing image storage unit 20.

Then, as shown in FIG. 11, the image synthesizing unit 15 displays the read image of the CG object on the CG object presentation area 503 on the display screen of the display unit 17.
The user selects and drags the CG object 503_1 in the CG object presentation area 503, drops it at the arrangement position 504 on the floor 200F in the virtual space, and adds the CG object 503_1 to the virtual space.

Step S7:
The image composition unit 15 performs a composition process for newly arranging the CG object 503_1 with respect to the arrangement position 504 in the virtual space.
Then, the image synthesizing unit 15 reconstructs the virtual space by newly rendering based on the three-dimensional shape of the virtual space, the three-dimensional shape of the CG object 503_1, and the light source information of the light sources 200L_1 and 200L_1. ..

Here, the image synthesizing unit 15 writes and stores the three-dimensional shape data of the reconstructed virtual space in the synthesizing image storage unit 20 in correspondence with the captured image identification information of the corresponding captured image.
The display control unit 16 generates an observation image observed from the position of the viewpoint in the reconstructed virtual space, and displays it on the display screen of the display unit 17 as shown in FIG.

Step S8:
The data input / output unit 11 displays an input display area (not shown) for inputting whether or not to continuously arrange (including additions and changes) CG objects on the display screen of the display unit 17.
Then, when the user inputs the information for continuing the arrangement of the CG object to the input display area, the data input / output unit 11 advances the process to step S6.
On the other hand, when the user inputs the information for terminating the arrangement of the CG object into the input display area, the data input / output unit 11 ends the process.

As described above, according to the present embodiment, it is possible to acquire the light source information based on the captured image and the three-dimensional shape of the virtual space by the light source information estimation method (3) from the light source information estimation method (1) described above. Since it is possible, it is not necessary to prepare an object having a known shape as in the past and to prepare a reference image by capturing an image of this object under a known light source, and simply obtain light source information of the light source in the target space. Obtainable.

Further, according to the present embodiment, the image synthesizing unit 15 renders a virtual space in which a CG object is newly arranged by the three-dimensional shape of the target space, the light source information in the target space, and the three-dimensional shape of the CG object. The effect of the light emitted from the light source on the CG object, the area shielded and hidden by the three-dimensional shape of the CG object, and the shadow area where the radiation from the light source is shielded by the three-dimensional shape of the CG object are reflected. Then, by rendering a virtual space (reconstructed virtual space) in which the virtual space and the CG object additionally arranged in the virtual space are combined, the target is in a light source environment similar to the light source environment in the actual target space. It is possible to virtually generate a state in which an object (an object corresponding to a CG object) is arranged in space, and easily observe the appearance of the target space in which the object is arranged as an observation image.

Further, according to the present embodiment, the reflectance information of each of the three-dimensional shapes of the structural part and the object part of the virtual space and the reflectance information of each of the CG objects are used to form the three-dimensional shape of the light emitted from the light source. By reflecting the reflection (secondary reflection) in the rendering, the appearance in the virtual space can be approximated to the actual target space.

Further, according to the present embodiment, by scaling the three-dimensional information of each of the virtual space and the CG object to the actual size, the appearance of the virtual space in which the CG object is arranged is accurate when the object is arranged in the target space. It can be visually recognized as an observation image having a texture with a sense of size.

Further, according to the present embodiment, since the virtual space is generated by estimating the three-dimensional shape of the target space using a single captured image or a plurality of captured images, the target space in a predetermined range is observed. The viewpoint can be changed arbitrarily, and the appearance of the area where the CG object is arranged in the virtual space can be observed as an observation image in which the viewpoint is changed.

Further, in the above-described embodiment, image processing for generating a virtual space and synthesizing a CG object with respect to the virtual space using the captured image captured in advance and written in the captured image storage unit 18 has been described.
However, the data input / output unit 11 may be configured to write the captured images (single or plurality of captured images) that sequentially capture the target space supplied from the imaging device to the captured image storage unit 18 in real time. In this case, a virtual space is generated from a single or a plurality of captured images that are sequentially supplied, a process of synthesizing a CG object is performed on the virtual space, and an observation image of the combined virtual space is displayed. It can be displayed on the display unit 17 in real time so that the user can visually recognize it.

Although the embodiments of the present invention have been described above with reference to the drawings, the specific configuration is not limited to this mode, and includes designs within a range that does not deviate from the gist of the present invention.

A program for realizing the function of the image processing system 10 of FIG. 1 in the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system and executed. A process of synthesizing a CG object with the virtual space generated from the captured image may be performed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices.

Further, the "computer system" shall also include a WWW system provided with a homepage providing environment (or display environment). Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Furthermore, a "computer-readable recording medium" is a volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, it shall include those that hold the program for a certain period of time.

Further, the program may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the above program may be for realizing a part of the above-mentioned functions. Further, a so-called difference file (difference program) may be used, which can realize the above-mentioned functions in combination with a program already recorded in the computer system.

10 ... Image processing system 11 ... Data input / output unit 12 ... Imaging condition acquisition unit 13 ... Shape information estimation unit 14 ... Light source information estimation unit 15 ... Image synthesis unit 16 ... Display control unit 17 ... Display unit 18 ... Captured image storage unit 19 … Spatial information storage unit 20… Composite image storage unit

Claims (11)

A shape information estimation unit that estimates spatial shape information including at least a three-dimensional shape of the target space from a single or a plurality of captured images in which the target space is captured.
An image processing system including a light source information estimation unit that estimates light source information including at least the position and intensity of a light source in the target space from the captured image and the space shape information. The light source information estimation unit
The image according to claim 1, wherein the reflectance information of the three-dimensional shape in the target space is estimated from the captured image and the spatial shape information, and the light source information is estimated using the reflectance information. Processing system. The light source information estimation unit uses the reflectance information estimated from a predetermined formula showing the relationship between the pixel value of the captured image and each of the reflectance information, the light source information, and the spatial shape information. The image processing system according to claim 2, wherein the light source information is estimated. An image compositing unit that generates a composite image of a virtual space in which a shape model of a predetermined object is arranged at a predetermined position of the captured image from the captured image, the spatial shape information, and the light source information is further provided. The image processing system according to any one of claims 1 to 3, wherein the image processing system is characterized. From the captured image, the spatial shape information, and the light source information, a shape model of the predetermined object is arranged at a predetermined position of the captured image, and the influence of arranging the shape model is reflected in the virtual space. The image processing system according to any one of claims 1 to 4, further comprising an image compositing unit for generating an observation image. The light source information estimation unit
The image according to any one of claims 1 to 5, wherein IBL (image based lighting) information is generated from the reflectance information of the three-dimensional shape, the spatial shape information, and the light source information. Processing system. The shape information estimation unit
Each of the three-dimensional shapes of the structural part and the object part in the target space is estimated separately, and the three-dimensional shapes of the structural part and the object part are combined to generate a three-dimensional shape model of the entire target space. The image processing system according to any one of claims 1 to 6, wherein the image processing system is characterized by the above. The shape information estimation unit
The image processing system according to any one of claims 1 to 7, wherein the three-dimensional shape of the target space and the shape information of the light source including the object recognition information in the target space are estimated. The image processing system according to any one of claims 1 to 8, further comprising an imaging condition acquisition unit that acquires an imaging condition when the captured image is captured. A shape information estimation process in which the shape information estimation unit estimates spatial shape information including at least a three-dimensional shape of the target space from a single or a plurality of captured images in which the target space is captured.
An image processing method comprising a light source information estimation process in which a light source information estimation unit estimates light source information including at least the position and intensity of a light source in the target space from the captured image and the space shape information. Computer,
A shape information estimation means that estimates spatial shape information including at least a three-dimensional shape of the target space from a single or a plurality of captured images in which the target space is captured.
A program that functions as a light source information estimation means that estimates light source information including at least the position and intensity of a light source in the target space from the captured image and the space shape information.

Images