WO2019088273A1

WO2019088273A1 - Image processing device, image processing method and image processing program

Info

Publication number: WO2019088273A1
Application number: PCT/JP2018/040919
Authority: WO
Inventors: 英起多田; 亮介稲森
Original assignee: ナーブ株式会社
Priority date: 2017-11-04
Filing date: 2018-11-02
Publication date: 2019-05-09
Also published as: JPWO2019088273A1; JP6570161B1

Abstract

Provided is an image processing device which is capable of synthesizing freely without discomfort a 3D model of an object such as furniture with a two-dimensional background image. The image processing device is provided with: an arrangement setting unit which sets an arrangement site of an object on a floor plan of a real estate item; a 3D location estimation unit which estimates the location of the object, which corresponds to the arrangement site of the object on the floor plan, in a 3D space surrounded by a celestial sphere image acquired as a background image; a shadow generation unit which generates a shadow generated by arranging a 3D model at the object location; an object layer generation unit which generates a first layer obtained by rendering an image obtained by projecting the 3D model onto the surface of the celestial sphere, which is the same as the background image, with the center of the 3D space employed as the projection center; a shadow layer generation unit which generates a second layer by rendering an image obtained by projecting the shadow on the surface of the celestial sphere with the center of the 3D space employed as the projection center; and an image synthesizing unit which synthesizes the first and second layers with the background image.

Description

IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM

The present invention relates to an image processing apparatus, an image processing method, and an image processing program for combining an image of an object with a background image obtained by photographing an object.

BACKGROUND ART In recent years, in the real estate industry, a service called home staging, which is performed at the time of trading, has become known. Home staging is a service that performs interior coordination on real estate properties in order to promote the sale of real estate properties such as single-family homes and condominiums. Specifically, the furniture, furniture, accessories, etc. selected in consideration of the target layer of the purchaser, fashion, characteristics of the property, etc. are placed. Purchasers (including those under consideration) are said to be able to facilitate sales smoothly because it becomes easy to imagine the state of living in the property by previewing the property to which home staging is applied.

On the other hand, recently, there is also a service of previewing real estate properties using virtual reality (hereinafter, also referred to as VR) technology. Image display of a VR is usually performed on a glasses or goggle type display device called a head mounted display (hereinafter also referred to as "HMD"). The HMD incorporates a screen on which two images provided with parallax are displayed and two lenses respectively installed in the direction of the eyes of the user's eyes, and the user can By looking at the screen, it is possible to three-dimensionally recognize the image. In addition, a gyro sensor and an acceleration sensor are built in the HMD, and an image displayed on the screen changes in accordance with the movement of the head of the user detected by these sensors. As a result, the user can experience as if it were embedded in a three-dimensional image.

As a technology relating to the simulation of the interior of a real estate property, Patent Document 1 generates a 3D model of a real estate property from property information including a floor plan image, an indoor shot photograph, etc., and uses virtual reality based on the generated 3D model. A technique for browsing indoors, simulating interiors, etc. is disclosed (see paragraph 0015 of Patent Document 1). Specifically, the scale of the 3D furniture model prepared in advance is adjusted to be placed in the 3D model of the property (see paragraph 0070 of Patent Document 1).

Further, with regard to a technique for generating composite image data of a background image of a real image and a three-dimensional CG object, Patent Document 2 discloses the periphery for generating three-dimensional shape data of the surrounding environment from environment map data consisting of two or more viewpoints. An image processing apparatus is disclosed which comprises: environmental three-dimensional shape data generation means; and virtual subject combination means for combining a virtual subject with background image data by using ambient three-dimensional shape data as a light source.

Further, in Patent Document 3, a foreground image, a background image serving as the background of the foreground image, storage means for storing depth information indicating the depth of the background image, and the background image on the foreground image are stored. Composite position acquisition means for acquiring the composite position, projection means for projecting the coordinates of the space generated from the depth information of the background image on the coordinates of the background image, projection information by the projection means and spatial coordinates of the composite position from the composite position The distance between the light source and the space coordinates of the synthesis position from the light source parameters and space coordinates of the synthesis position holding the information of the position and illuminance of the light source viewed from the position where the background image was taken Alternatively, calculation means for calculating the illuminance of the light source viewed from the space coordinates of the combined position, light source parameter changing means for changing the light source parameter according to the space coordinates of the combined position and the calculated value, light Disclosed is an image processing apparatus comprising: lighting means for applying a lighting effect to a foreground image using parameters; and combining means for combining the foreground image subjected to the lighting effect by the lighting means to a combined position of the background image It is done.

In addition, as a service for real estate agents, non-patent property 1 has a 360 ° image (sky image) by photographing a real property using a digital camera (product name: THETA) manufactured by Ricoh Co., Ltd., for VR The service which is processed into the contents of and provided to the real estate agent is disclosed.

Patent No. 6116746 gazette JP, 2013-152683, A JP, 2013-149219, A

When using a method of arranging a three-dimensional model of an object such as furniture as a three-dimensional background model of a real estate property as a method of providing home staging content usable by VR, in the construction (rendering) of 3D data It is expensive, time consuming, and expensive when displaying data. Therefore, high-spec equipment is required, making it difficult to easily use home staging content.

It is also conceivable to combine a three-dimensional model (furniture model) with a two-dimensional background image of an existing subject as a method of more easily creating home staging content usable by VR. However, the two-dimensional background image and the three-dimensional model have different positions in the computer graphics space, so the furniture may appear to float in the background image or the depth may appear different, etc. It is easy for the user who uses the content to feel uncomfortable, and it is difficult to freely arrange the three-dimensional model on the two-dimensional background image.

The present invention has been made in view of the above, and is an image processing apparatus and an image processing method capable of freely synthesizing a three-dimensional model of an object such as furniture with a two-dimensional background image freely. And providing an image processing program.

In order to solve the above problems, an image processing apparatus according to an aspect of the present invention is an image processing apparatus that combines an image of an object virtually arranged in the real estate property with a background image on which the real estate property is captured. In a three-dimensional graphics space surrounded by a celestial sphere image acquired as a background image by capturing an arrangement position of the object on the floor plan of the real estate property and photographing the real estate property. A 3D position estimation unit for estimating an object position which is a position corresponding to an arrangement position of the object on the floor plan, and a shadow for generating a shadow caused by arranging a three-dimensional model of the object at the object position. A generation unit, and the object position with the center of the three-dimensional graphics space as a projection center An object layer generation unit that generates a first layer by rendering an image obtained by projecting the arranged three-dimensional model onto the same celestial sphere as the background image; and a projection center of the three-dimensional graphics space A shadow layer generation unit that generates a second layer by rendering an image obtained by projecting the shadow onto the sky sphere, and combining the second layer and the first layer with the background image And an image combining unit.

The image processing apparatus further includes a display unit that displays the image of the floor plan, and an operation input unit that receives an operation performed by the user, and the arrangement setting unit receives the operation input unit. The arrangement location may be set based on an operation on the image of the floor plan displayed on the display unit.

In the image processing apparatus, the shadow generation unit may set a light source based on the background image, and may generate a shadow of the three-dimensional model based on the light source.

In the image processing apparatus, the 3D position estimation unit may estimate the object position based on the dimensions of each unit in the floor plan.

In the image processing apparatus, the 3D position estimation unit may estimate the object position based on a distance and a dimension of a subject located in the vicinity of the arrangement location.

In the image processing apparatus, the 3D position estimation unit may acquire a shooting point based on a distance and a dimension of a subject measured from the background image, and estimate the object position based on the shooting point.

In the image processing apparatus, the 3D position estimation unit may acquire a shooting point based on the measured distance and dimension of the subject, and estimate the object position based on the shooting point.

An image processing method according to another aspect of the present invention is an image processing method for synthesizing an image of an object virtually arranged in the real estate property with respect to a background image showing the real estate property. In the three-dimensional graphics space surrounded by the celestial sphere image acquired as a background image by setting an arrangement location of the object on the floor plan and photographing the real estate property, the object on the floor plan Estimating an object position which is a position corresponding to an arrangement position of the object, generating a shadow generated by arranging the three-dimensional model of the object at the object position, and projecting the center of the three-dimensional graphics space In front of the three-dimensional model placed at the object position as a center Generating the first layer by rendering an image projected on the same celestial sphere as the background image, and projecting the shadow on the celestial sphere with the center of the three-dimensional graphics space as the projection center The steps of generating a second layer by rendering an image, and combining the second layer and the first layer with the background image.

An image processing program according to still another aspect of the present invention is an image processing program for synthesizing an image of an object virtually placed on the real estate property against a background image showing the real estate property, and the real estate property In the three-dimensional graphics space surrounded by the celestial sphere image acquired as a background image by setting the arrangement location of the object on the floor plan, and capturing the real estate property, the three-dimensional graphics space on the floor plan Estimating an object position which is a position corresponding to an arrangement position of the object, generating a shadow generated by arranging the three-dimensional model of the object at the object position, and determining a center of the three-dimensional graphics space. The front located at the object position as the projection center Generating a first layer by rendering an image in which a three-dimensional model is projected onto the same celestial sphere as the background image, and setting the shadow as the projection center with the center of the three-dimensional graphics space as the projection center Causing a computer to execute the steps of generating a second layer by rendering an image projected on a spherical surface, and combining the second layer and the first layer with the background image. is there.

According to the present invention, an object position in the three-dimensional graphics space corresponding to the arrangement position of the object on the floor plan is estimated, and an image of the three-dimensional model when the three-dimensional model of the object is arranged at the object position Since the layer which rendered the image of the shadow of the three-dimensional model respectively is generated and these layers are combined with the background image, the three-dimensional model of the object such as furniture can be freely and uncomplicatedly with the two-dimensional background image. It becomes possible to synthesize.

FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to a first embodiment of the present invention. It is a schematic diagram for demonstrating the image compositing process which the image processing apparatus shown in FIG. 1 performs. It is a schematic diagram for demonstrating the image compositing process which the image processing apparatus shown in FIG. 1 performs. It is a flowchart which shows the image processing method which concerns on the 1st Embodiment of this invention. It is a floor plan of real estate that is an example of the subject of the background image It is a flowchart which shows the position estimation process shown in FIG. It is a schematic diagram for demonstrating the position estimation process of a 3D model. It is a schematic diagram for demonstrating the position estimation process of a 3D model. It is a schematic diagram for demonstrating the position estimation process of a 3D model. It is a schematic diagram for demonstrating the image processing shown in FIG. It is a flowchart which shows the position estimation process in the 2nd Embodiment of this invention. It is a floor plan of a real estate property which is an example of a subject of a background image. It is a schematic diagram for demonstrating the position estimation process shown in FIG. It is a schematic diagram for demonstrating the position estimation process shown in FIG. It is a schematic diagram for demonstrating the position estimation process shown in FIG. It is a schematic diagram for demonstrating the position estimation process shown in FIG. It is a flowchart which shows the image processing method which concerns on the 3rd Embodiment of this invention. It is a floor plan of a real estate property which is an example of a subject of a background image. It is a schematic diagram for demonstrating the image processing method in the 3rd Embodiment of this invention. FIG. 1 is a network diagram showing a configuration example of a system to which an image processing apparatus according to first to third embodiments of the present invention is applied. FIG. 15 is a schematic view showing another configuration example of a system to which the image processing apparatus according to the first to third embodiments of the present invention is applied.

Hereinafter, an image processing apparatus, an image processing method, and an image processing program according to an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited by these embodiments. Further, in the descriptions of the respective drawings, the same parts are denoted by the same reference numerals.

The image processing apparatus according to the embodiment of the present invention uses a celestial sphere image with real estate property as a background image, and synthesizes an image of a three-dimensional model (3D model) of an object such as furniture or furniture with this background image. By performing the image processing to be performed, it is possible to generate an image as if the object was placed on the real estate property and shooting was performed.

The celestial sphere image used as a background image is also referred to as a 360 ° image, a full celestial sphere image, an omnidirectional image, etc., and is an image in which an object around 360 ° around the shooting location is captured. The background image (sky image) may be a still image or a moving image.

In the present image processing apparatus, a virtually transparent floor is disposed on a background image, and the shadow of the 3D model is projected on the floor to enhance the realism of the 3D model. Also, by lighting the 3D model with the background image as the light source, the discomfort is reduced.

Here, in order to construct the entire image including the background of the real estate property with three-dimensional CG, rendering is necessary for each drawing, and it takes a lot of computational cost to execute this processing, so high-spec equipment Was necessary. However, in the image processing apparatus according to the present embodiment, since the 3D model of the virtually arranged object is combined with the two-dimensional background image, the calculation cost is not so high.

Further, in the present image processing apparatus, since the data of the 3D model of the object is rendered in accordance with the projection of the two-dimensional background image, there is no gap in the depth, and it is possible to display without discomfort.

In this way, by presenting the content of the 3D model combined with the background image, the customer who is considering purchasing or renting a real estate property tries to coordinate furniture etc. on the real estate property until he or she understands on the screen Can. Therefore, it is possible to promote the purchase and rental transactions of real estate properties without increasing the cost.
Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described in detail.

First Embodiment
FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to a first embodiment of the present invention. FIGS. 2A and 2B are schematic views for explaining the image combining process performed by the image processing apparatus shown in FIG. Among these, FIG. 2A shows a floor plan of real estate. Also, FIG. 2B is an image obtained by combining a 3D model of a sofa, a coffee table, a closet, and a picture with a background image (heaven ball image) acquired by shooting in the living room (see FIG. 2A) of the real estate property. Show a part. As shown in FIG. 2B, since the sense of depth according to the arrangement has been reproduced for each object of the sofa, the low table, the chest, and the picture, it was as if each object was actually arranged and photographed in the living room Thus, the background image is displayed without a sense of incongruity.

As shown in FIG. 1, the image processing apparatus 10 includes a communication interface 11, a display unit 12, an operation input unit 13, a storage unit 14, and a processor 15. As such an image processing apparatus 10, a general purpose computer can be used.

The communication interface 11 connects the image processing apparatus 10 to a communication network, and transmits and receives information to and from other devices connected to the communication network. The communication interface 11 is configured using, for example, a soft modem, a cable modem, a wireless modem, an ADSL modem, an ONU (Optical Network Unit), or the like. The communication interface 11 also functions as a data acquisition unit that loads image data generated by an omnidirectional camera, data of a 3D model, and the like into the image processing apparatus 10.

The display unit 12 is a display including a display panel formed by liquid crystal or organic EL (electroluminescence) and a drive unit.

The operation input unit 13 is an operation button, a keyboard, a pointing device such as a mouse, and an input device such as a touch sensor provided on the display unit 12. The operation input unit 13 receives an operation performed by the user and processes a signal corresponding to the operation. Enter 15

The storage unit 14 is a computer-readable storage medium such as a semiconductor memory such as a ROM or a RAM. The storage unit 14 includes a program storage unit 141, a background image data storage unit 142, a floor plan data storage unit 143, a 3D data storage unit 144, and a composite image data storage unit 145. The operating system program and driver A program, an application program for executing various functions, various parameters used during execution of these programs, image data, etc. are stored.

In detail, the program storage unit 141 stores various programs such as an image processing program.
The background image data storage unit 142 stores image data (hereinafter also referred to as background image data) of a celestial sphere image acquired as a background image by photographing a real estate property.
The floor plan data storage unit 143 stores image data of a floor plan of a real estate property (hereinafter also referred to as floor plan data).

The 3D data storage unit 144 includes data (hereinafter, also referred to as 3D data) representing a 3D model of an object such as furniture virtually disposed on a real estate, and related parameters (for example, parameters representing a texture, etc.) Remember. The 3D data may be generated by the image processing apparatus 10 or may be acquired by another apparatus through the communication network N.

The composite image data storage unit 145 stores image data (hereinafter also referred to as composite image data) of a composite image obtained by combining an image of a 3D model with a background image.

The processor 15 is configured using, for example, a central processing unit (CPU) or a graphics processing unit (GPU), and reads various programs stored in the program storage unit 141 to centrally control the respective units of the image processing apparatus 10. At the same time, various processes are performed to combine an image of an object such as furniture with a background image showing a real estate property. In detail, the functions of the image processing unit 150 realized by executing the image processing program by the processor 15 include the arrangement setting unit 151, the 3D position estimation unit 152, the object layer generation unit 153, and the shadow generation unit A shadow layer generation unit 155 and an image combining unit 156 are included.

The arrangement setting unit 151 sets an arrangement place in the floor plan of the object virtually arranged on the real estate according to the signal input from the operation input unit 13.

The 3D position estimation unit 152 is an arrangement location of an object in a floor plan in a three-dimensional graphics space (hereinafter, also referred to as a 3D space) in which a shooting point is surrounded by a sky spherical background image and is associated with a shooting point. The position corresponding to (hereinafter, also referred to as an object position) is estimated.

When the 3D model of an object is placed at the above object position in 3D space, the object layer generation unit 153 creates an image of the 3D model viewed from the center of the 3D space on a layer provided on the same celestial sphere as the background image. Render. That is, an image in which the 3D model is projected onto the celestial sphere is generated with the center of the 3D space as the projection center. Hereinafter, the layer on which the 3D model image is rendered is referred to as an object layer (first layer).

The shadow generation unit 154 generates a shadow generated by the 3D model placed at the object position in 3D space. In detail, the shadow generation unit 154 executes an image based lighting process of setting a light source based on a background image, and generates a shadow of a 3D model based on the set light source. Image-based lighting is a method of rendering a scene using an image obtained by photographing an existing subject as color information of lighting.

The shadow layer generation unit 155 renders an image obtained by viewing the shadow generated by the shadow generation unit 154 from the center of the 3D space on a layer provided on the same celestial sphere as the background image. That is, an image is generated in which the shadow of the 3D model is projected onto the sky sphere with the center of the 3D space as the projection center. Hereinafter, the layer in which the shadow image is rendered is referred to as a shadow layer (second layer).

The image combining unit 156 generates a combined image in which the 3D model image of the object is combined with the background image by superimposing the shadow layer and the object layer on the background image in this order.

Next, an image processing method according to the present embodiment will be described.
The outline of the image processing in this embodiment is as follows. That is, a background image (sky image) to be composited, a layer of transparent sky image for shadow (shadow layer), and a layer of transparent sky image for 3D model of object such as furniture (object layer) Prepare a total of three layers of celestial sphere images. The background image is acquired by photographing at the site of a real estate property using an omnidirectional camera or the like. In addition to the above two layers, layers to be superimposed on the background image may be appropriately increased.

Next, the 3D model of the object is placed in a 3D graphics space (3D space) surrounded by these celestial sphere images. Then, in order to cast a shadow on the 3D model, a light source based on a background image is set by image-based rating.

Next, a camera for rendering the celestial sphere image is placed at the origin (center of the celestial sphere) of the 3D space. In other words, the viewpoint (projection center) to see the 3D model and its shadow from the center of the 3D space is set.

Next, when a 3D model is placed in 3D space, an image of a shadow generated by a light source set based on a background image is rendered and pasted on a shadow layer. In addition, an image of the 3D model when the 3D model is arranged in the 3D space is rendered and attached to an object layer. By superimposing such a background image, a shadow layer, and an object layer, it is possible to obtain a composite image in which an object is composited seamlessly with the background image.

Hereinafter, such image processing will be described in detail with specific examples. FIG. 3 is a flowchart showing an image processing method according to the present embodiment. FIG. 4 is a floor plan of real estate, which is an example of the subject of the background image. In the present embodiment, an omnidirectional camera is installed at a position corresponding to the photographing point P ₀ on the floor plan shown in FIG. It is assumed that this celestial sphere image is used as a background image to synthesize a 3D model.

First, in step S10, the image processing unit 150 acquires a background image for combining a 3D model. In the present embodiment, the image processing unit 150 acquires a background image by reading out the background image data stored in the background image data storage unit 142.

In the subsequent step S11, the image processing unit 150 acquires data (3D data) representing a 3D model to be combined with the background image. Specifically, the image processing unit 150 causes the display unit 12 to display a name, an icon, or the like representing a 3D model in which 3D data is stored in the 3D data storage unit 144, and allows the user to select a desired 3D model. Then, the image processing unit 150 reads 3D data representing the selected 3D model from the 3D data storage unit 144 in accordance with the signal input from the operation input unit 13 according to the operation by the user.

In the subsequent step S12, the image processing unit 150 acquires parameters regarding the 3D model to be combined with the background image. Specifically, parameters for displaying each coordinate of the 3D model as a function such as a curve or a curved surface, parameters used in processing such as texture mapping are read out from the 3D data storage unit 144.

In the subsequent step S13, the image processing unit 150 estimates the position of the 3D model in the 3D space corresponding to the arrangement position of the object in the floor plan shown in FIG.

FIG. 5 is a flowchart showing the position estimation process in step S13. 6 to 8 are schematic diagrams for explaining the position estimation process of the 3D model. Among these, FIG. 6 shows a state where the omnidirectional camera 20 mounted on a tripod is installed on the floor surface. FIG. 7 shows a state in which the background image L1 obtained by photographing the real estate with the omnidirectional camera 20 is cut along a horizontal plane passing through the central point P ₀ ′ of the celestial sphere. In the background image L1 shown in FIG. 7, a sink, a stove, a wall, a Western-style door, a Japanese-style rattan, a bathroom door, etc., which enter the field of view of the omnidirectional camera 20 installed at the shooting point P ₀ (see FIG. 4). Images of the toilet door and the front door are visible. FIG. 8 shows the background image L1 cut in the vertical plane passing through the center point P ₀ ′. In the background image L1 shown in FIG. 8, images of a floor surface, a wall, a ceiling, an entrance door and the like which are included in the field of view of the omnidirectional camera 20 are captured.

In the present embodiment, the respective parts in real estate dimensions and coordinates (height of the plane coordinates and photographed point) of the shooting point P ₀ of the (each room and equipment such as shown in Mato view) is known, these dimensions and Based on the coordinates, estimate the position of the 3D model in 3D space. Incidentally, as shown in FIG. 6, the height of the imaging point P ₀ can be obtained as the height h of the omnidirectional camera 20, including the tripod.

First, in step S131, the placement setting unit 151 determines the placement of objects on the floor plan. In detail, the arrangement setting unit 151 causes the display unit 12 to display a floor plan of a real estate property, and allows the user to specify a place where an object such as furniture is to be arranged on the floor plan. Then, the arrangement setting unit 151 acquires the position information (coordinates) of the object in the floor plan according to the signal input from the operation input unit 13 according to the operation by the user. In the following, a point P ₁ on the chart summarizes shown in FIG. 4, and that determined as the arrangement position of the user-desired objects.

In subsequent step S132, 3D position estimation unit 152 obtains the position of the imaging point P ₀ at the time of photographing the background image L1.

In the subsequent step S133, the 3D position estimation unit 152 derives the position of the 3D model in the 3D space surrounded by the background image L1 based on the floor plan. Specifically, the 3D position estimation unit 152 creates a 3D space in which the shooting point P _{0 is associated} with the center point P ₀ ′ of the celestial sphere, and associates the position and size of each part in the floor plan with the coordinates in 3D space. . Thereby, the coordinates of the point P ₁ ′ in the 3D space corresponding to the point P ₁ in the floor plan can be obtained.
When the position of the 3D model is thus estimated, the process returns to the main routine.

Referring again to FIG. 3, in step S <b> 14 following step S <b> 13, the image processing unit 150 arranges the 3D model at the position of the 3D space estimated in step S <b> 13. FIG. 9 is a schematic diagram for explaining the image processing shown in FIG. 3, and shows a state in which the 3D model a11 of the sofa is disposed in the 3D space surrounded by the background image L1.

In the subsequent step S15, the shadow generation unit 154 sets a light source and a material. Specifically, the shadow generation unit 154 generates a light probe image in image-based writing based on the background image L1, and sets a material (such as surface reflection characteristics) of the 3D model. Here, the light probe image is a high dynamic range image recording incident illumination conditions in all directions.

In the subsequent step S16, the shadow generation unit 154 generates a shadow of the 3D model arranged in the 3D space based on the light source and the material set in step S15. That is, rendering is performed using the generated light probe image. As a result, a shadow is generated in 3D space with the 3D model as a shield, and lighting based on the background image L1 is also made on the surface of the 3D model. FIG. 9 shows a shadow a13 of the 3D model a11 generated by the light source a12.

In the subsequent step S17, the shadow layer generation unit 155 generates a shadow layer in which the shadow image of the 3D model is rendered. Specifically, the shadow layer generation unit 155 sets the shadow of the 3D model generated in step S16 to a shadow layer L2 located on the same celestial sphere as the background image with the central point P ₀ ′ of the 3D space as the projection center. Project FIG. 9 shows an area a14 of an image obtained by projecting the shadow a13 on the shadow layer L2 in the upper layer of the background image L1. In the shadow layer L2, the area other than the area a14 of the image is transparent.

In the subsequent step S18, the object layer generation unit 153 generates an object layer in which the image of the 3D model is rendered. Specifically, the object layer generation unit 153 places the 3D model placed in the 3D space in step S14 and illuminated in step S16 on the same sky as the background image with the center point P ₀ ′ of the 3D space as the projection center. It projects onto the object layer L3 located on the spherical surface. In FIG. 9, an area a15 of an image obtained by projecting the 3D model a11 is shown on the object layer L3 in the upper layer of the shadow layer L2. In the object layer L3, the area other than the area a15 of the image is transparent.

In the subsequent step S19, the image combining unit 156 generates a combined image in which the background image L1, the shadow layer L2, and the object layer L3 are superimposed in this order. Thereby, an image is obtained in which the image of the 3D model is synthesized at the user's desired position in the background image.

The composite image generated in this manner may be displayed as a panoramic image obtained by expanding the celestial sphere, or may be displayed as content for VR.

As described above, according to the first embodiment of the present invention, in the 3D space surrounded by the celestial sphere image acquired as the background image, the location of the user-desired object designated on the floor plan is dealt with Position the 3D model of the object at this position and render the image of the 3D model on the sky sphere, causing the object to be freely against the background image and causing a position or size mismatch. It can be arranged without

Further, according to the first embodiment of the present invention, a shadow of a 3D model is generated by a light source set based on a background image, and an image of the shadow is rendered on the sky sphere, so that a composite image without discomfort Can be generated.

Furthermore, according to the first embodiment of the present invention, since the 3D model is illuminated by the light source set based on the background image, a more realistic composite image can be generated.

Second Embodiment
Next, a second embodiment of the present invention will be described.
The configuration of the image processing apparatus according to the second embodiment of the present invention is the same as that of the first embodiment (see FIG. 1) as a whole, and 3D model position estimation processing (FIG. 3) performed by the 3D position estimation unit 152. Step S13) is different from the first embodiment.

FIG. 10 is a flow chart showing position estimation processing in the second embodiment of the present invention. FIG. 11 is a floor plan of real estate, which is an example of a subject of a background image. 12A to 13B are schematic diagrams for explaining the position estimation process in the present embodiment. Among these, FIG. 12A is a side view showing a state in which a real estate camera is photographed by the omnidirectional camera 20, and FIG. 12B passes through a central point P ₀ ′ the background image L1 obtained by the photographing. It shows the state of cutting in the vertical plane. FIG. 13A is a top view showing that the image is taken by the omnidirectional camera 20, and FIG. 13B shows a state in which the background image L1 is cut along a horizontal plane passing through the center point P ₀ ′.

In the present embodiment, illustrating the position estimation processing in the case the plane coordinate dimensions and shooting point P ₀ of each part of the real estate is unknown. In this case, as described below, the object position is estimated based on the dimensions of the subject with reference to the subject such as furniture or equipment located near the arrangement location of the object desired by the user. Hereinafter, the subject used for reference is referred to as a reference object.

First, in step S231, the placement setting unit 151 determines the placement of objects on the floor plan. The process of determining the arrangement of objects is the same as that of the first embodiment (see step S131 in FIG. 5). In the following, the point P ₂ on the chart Mato shown in FIG. 11, and that determined as the arrangement position of the object.

In the following step S232, the 3D position estimation unit 152 selects a subject near the arrangement location of the object in the floor plan as a reference object. In the following, it is assumed that the sink shown in FIG. 11 is selected as the reference object 21.

In the subsequent step S233, the 3D position estimation unit 152 acquires the vertical angle of view and the horizontal angle of view of the visual field area in which the reference object 21 captured in the background image L1 is fully contained. Specifically, as shown in FIGS. 12B and 13B, the area a21 where the reference object 21 is captured is extracted from the background image L1, and the vertical angle of view α and the horizontal angle of view β are measured. For the vertical angle of view α, an area corresponding to the height h from the floor surface to the omnidirectional camera 20 is measured.

In the subsequent step S234, the 3D position estimation unit 152 generates a 3D space surrounded by the background image L1 and associating the shooting point P ₀ in the floor plan with the center point P ₀ ′ of the sky.

In the subsequent step S235, the 3D position estimation unit 152 estimates the position of the reference object 21 in 3D space. The position of the reference object 21 may be, for example, from the center point and the shooting point P ₀ of the reference object 21 represented by a representative point such points the shortest distance. As a specific example, as shown in FIGS. 12A and 12B, from the vertical view angle α and the height h of the shooting point P _{0 according} to the following equation (1), from the shooting point P ₀ to the representative point of the reference object 21 The distance d ₁ can be calculated.
d ₁ = h / tan α (1)

Further, as shown in FIGS. 13A and 13B, the horizontal angle β and the distance d ₁ Tokyo, it is possible to calculate the position of the reference object 21 in the horizontal direction. For example, the distance x ₄ from a certain reference position (e.g., in front of the omnidirectional camera 20 lines) to the representative point of the reference object 21 can be calculated by the following equation (2).
x ₄ = d ₁ · sin (β / 2) (2)

In the following step S236, the 3D position estimation unit 152 derives the position of the 3D model in 3D space. For example, as shown in FIG. 11, when the align objects to the reference object 21 (sink), also in the 3D space, near the position of the reference object 21 estimated in step S235 (e.g., shifted by a distance x ₄ The 3D model should be placed at the position).
Thereafter, the process returns to the main routine.

As described above, according to the second embodiment of the present invention, even in the case where the coordinates of the shooting point P ₀ are unknown, if the height h of the shooting point P ₀ is known, then in the 3D space It is possible to estimate the position of the 3D model.

Here, when the 3D model is superimposed on the image being captured in real time, as in AR, the position of the 3D model in 3D space can be estimated relatively easily because the position of the shooting point is known. However, according to the present embodiment, it is possible to appropriately set the position of the 3D model in 3D space even when using a background image whose position of the shooting point is unknown.

In the above embodiment, although furniture or equipment provided in the real estate property is used as a reference object, instead, a marker whose dimensions are known in advance is disposed as a reference object in the real estate property and photographed. You may In this case, the position of the reference object in the 3D space can be more easily estimated.

Third Embodiment
Next, a third embodiment will be described.
The configuration of the image processing apparatus according to the third embodiment of the present invention is the same as that of the first embodiment (see FIG. 1) as a whole, and the 3D position estimation process performed by the 3D position estimation unit 152 It differs from the first embodiment.

FIG. 14 is a flowchart showing an image processing method according to the present embodiment. FIG. 15 is a floor plan of real estate, which is an example of the subject of the background image. FIG. 16 is a schematic view showing an example of the background image, and shows a state in which the background image L1 is cut along a horizontal plane passing through the center point P ₀ ′. In the present embodiment, based on the measurement value of the subject distance (depth) and dimensions of the furniture and interior disposed real estate, to identify the imaging point P ₀ in the real estate, to estimate the object position.

First, in step S30, the image processing unit 150 acquires a background image for combining a 3D model.

In subsequent step S31, the image processing unit 150, based on the background image L1, measures the size of the distance and the subject from the photographic point P ₀ to the object at multiple locations, and stores the measurement data. The method of measuring the distance and size of the subject is the same as that described in the second embodiment (see FIGS. 12A to 13B). The number of subjects to measure the distance and size is arbitrary, as an example, by measuring the distances to the three subjects, it is possible to identify the coordinates of the photographing point P ₀ in the horizontal plane.

In the subsequent step S32, the image processing unit 150 associates measurement data of the subject with the background image. FIG. 16 shows a state in which the distances d ₃₁ to d ₃₈ measured in step S ₃₁ and the dimensions x ₃₁ , x ₃₂ , y ₃₁ and y ₃₂ in the horizontal plane are associated with the background image L 1. The dimensions in the vertical plane are likewise associated with the background image.

In the subsequent step S33, the image processing unit 150 acquires 3D data representing a 3D model to be combined with the background image L1. Further, in step S34, the image processing unit 150 acquires parameters related to the 3D model to be combined with the background image L1. The processes of steps S33 and S34 are the same as steps S11 and S12 shown in FIG. 3, respectively.

In the subsequent step S35, the image processing unit 150 determines the arrangement of the object on the floor plan. The process of determining the arrangement of objects is the same as that of the first embodiment (see step S131 in FIG. 5). In the following, the point P ₃ of the drawing Mato shown in FIG. 15, and that determined as the arrangement position of the object.

In subsequent step S36, the image processing unit 150 acquires the position of the imaging point P ₀ at the time of photographing the background image L1. Specifically, the 3D position estimation unit 152 calculates the coordinates of the shooting point P ₀ in the floor plan from the distance measured in step S 31 (see FIG. 15).

In the following step S37, the image processing unit 150 is surrounded by the background image L1 based on the measurement data associated with the background image L1, and associates the shooting point P ₀ in the floor plan with the center point P ₀ 'of the sky. Create a 3D space.

In the following step S38, the image processing unit 150 arranges the 3D model in the 3D space based on the arrangement of the objects determined in step S35. Specifically, the 3D position estimation unit 152 calculates the coordinates of the placement location (point P ₃ ) of the object with respect to the shooting point P ₀ in the floor plan, and the location in 3D space corresponding to the placement location (FIG. Estimate a point P ₃ ′) and place a 3D model at this position.

The processes in the following steps S39 to S43 are the same as the steps S15 to S19 shown in FIG. 3, respectively.

As described above, according to the third embodiment of the present invention, since measuring the distance and size of the object at multiple locations, the imaging point P ₀ in the diagram taken between can be accurately obtained. Therefore, the position in the 3D space corresponding to the placement location of the object determined on the floor plan can be estimated more accurately.

Next, a modification of the third embodiment of the present invention will be described.
In the third embodiment, although the distance and the size of the subject are measured from the background image (see step S31), the distance and the size may be measured at the time of photographing. In detail, by attaching an auxiliary tool for distance measurement to the omnidirectional camera or smartphone, or by executing a dedicated application and shooting a real estate property, the distance to the subject and the dimensions of the subject are measured, Measured data is stored in association with image data. In this case, it is possible to more accurately reproduce the 3D space corresponding to the real estate property.

In the first to third embodiments described above, the light source is set based on the background image by image-based lighting, but the method of setting the light source is not limited to this, and various known methods may be applied. Can. For example, a light source (a window, a lighting device, etc.) shown in a background image is placed as a light source object in 3D space, and shadows of the 3D model are generated by performing global illumination calculation (radiosity method, photon mapping method, etc.) You may.

FIG. 17 is a network diagram showing a configuration example of a system to which the image processing apparatus according to the first to third embodiments of the present invention is applied. The system 1 shown in FIG. 17 includes an image processing apparatus (server) 30, a real estate management terminal 31, an object management terminal 32, and an image display terminal 33, and these devices are connected via a communication network N. There is. As the communication network N, for example, an Internet line, a telephone line, a LAN, a dedicated line, a mobile communication network, a communication line such as WiFi (Wireless Fidelity), Bluetooth (registered trademark), or a combination thereof is used. The communication network N may be wired, wireless, or a combination of these.

The image processing apparatus 30 is constituted by a host computer with high arithmetic processing capability, functions as a server that manages the system 1 in a centralized manner, and executes the image processing described in the first to third embodiments. The computer constituting the image processing apparatus 30 does not necessarily have to be one, and may be composed of a plurality of computers distributed on the communication network N.

The real estate management terminal 31 manages information on a real estate rental property and a real estate sale property. In detail, the real estate management terminal 31 stores image data (room arrangement data and background image data) regarding the real estate property as well as information regarding the transaction such as the location of the real estate property, the owner, and rental or trading conditions. The floor plan data and the background image data are uploaded from the real estate management terminal 31 to the image processing apparatus 30, and used for image processing.

The object management terminal 32 creates and stores information related to an object such as furniture to be combined with a background image, that is, data and parameters of a 3D model representing the object, and a setting file. These pieces of information are uploaded from the object management terminal 32 to the image processing apparatus 30 and used for image processing.

The image display terminal 33 is a terminal device for displaying an image combined by the image processing device 30 and causing the user to view it. The image display terminal 33 causes the user to recognize a three-dimensional virtual space (VR) by displaying a two-dimensional still image or a moving image. As the image display terminal 33, a goggle type dedicated device (so-called head mounted display (HMD)) used by being directly attached to the head of the user may be used, or a dedicated application installed on a general-purpose smartphone May be attached to a goggle type holder and used.

Inside the image display terminal 33, two lenses are respectively attached at positions corresponding to the left and right eyes of the user. At the time of reproducing the VR, two images provided with parallax are respectively displayed in the two left and right areas provided on the screen. The user can recognize an image three-dimensionally (stereoscopically) by viewing the two images with the left and right eyes respectively through the two lenses.

Of course, the image display terminal 33 is not limited to the HMD, and a tablet terminal or a stationary display may be used. In this case, an image obtained by combining a three-dimensional model of an object such as furniture with a background image may be displayed as a panoramic image.

In such a system 1, the image processing device 30 uses the image data of the background image uploaded from the real estate management terminal 31, the data of the 3D model uploaded from the object management terminal 32, etc. A composite image in which the model is composited is generated, and image data of the composite image is transmitted to the image display terminal 33. Thus, the composite image can be displayed on the image display terminal 33.

In the system 1, various terminal devices may be further connected to the communication network N. For example, a terminal device may be separately provided which is used to cause the user to select a background image or an object to be combined with the background image, or to specify an arrangement position of the object in the floor plan. Further, a plurality of image display terminals 33 may be provided so that the same content can be simultaneously viewed on these image display terminals 33.

FIG. 18 is a schematic view showing another configuration example of a system to which the image processing apparatus according to the first to third embodiments of the present invention is applied. In FIG. 18, bold solid arrows represent the flow of data to be downloaded, and dashed arrows represent the flow of data to be uploaded.

The system 2 shown in FIG. 18 includes an image processing apparatus (editor) 40, an image management server 41, an object management apparatus 42, a converter 43, a service management apparatus 44, and an image display terminal 45. These respective devices are connected to one another via a communication network.

The image processing apparatus (editor) 40 functions as an editor for processing or editing a background image by executing the image processing described in the first to third embodiments.
The image management server 41 manages images used in the system 2.

The object management device 42 is a setting file in which data representing a 3D model of an object such as furniture or furniture and related information such as a texture (hereinafter collectively referred to as a 3D file) and location information such as furniture are described Create or store from an external source. Note that “FBX” illustrated in FIG. 18 is an example of a file format of data representing a 3D model. However, the file format usable in this system is not limited to this.

The converter 43 converts data representing the 3D model into a file that can be read into the image processing device 40 and is associated with related information such as a texture.

The service management device 44 includes a storage unit for storing floor plan data and background image data, and also includes a screen capable of displaying a floor plan and a background image, and displays the floor plan on the screen to display an object such as furniture. It is used to allow the user to specify the placement position of the image, or to display on the screen the composite image 44b in which the background image 44a and the 3D model of the object are composited and to present it to the user.

The configuration of the image display terminal 45 is the same as that of the image display terminal 33 shown in FIG. 17, and is used when a user views a composite image in which a 3D model is combined with a background image as a VR.

In such a system 2, first, the object management device 42 uploads data of a 3D model of an object such as furniture to the image management server 41. The converter 43 downloads 3D model data from the image management server 41, converts the 3D model, a texture, and the like into a file, and uploads the file to the image management server 41. The image processing apparatus 40 downloads the file converted in this way from the image management server 41, downloads the setting file from the object management apparatus 42, and further downloads the background image data from the service management apparatus 44. Then, the image processing device 40 performs each process (setting of illumination, adjustment of material, etc.) for combining the 3D model with the background image, and uploads the file of the created combined image data to the service management device 44. The image display terminal 45 downloads a file of composite image data from the service management apparatus 44, and reproduces a composite image, that is, home staging content based on the file.

The present invention is not limited to the above first to third embodiments and modifications, and various combinations of components disclosed in the above first to third embodiments can be made as appropriate. The invention can be formed. For example, it may be formed by excluding some components from all the components shown in the first to third embodiments and the modification, or in the first to third embodiments and the modification. The components shown may be combined appropriately.

1, 2 system 10 image processing apparatus 11 communication interface 12 display section 13 operation input section 14 storage section 15 processor 20 all-sky camera 21 reference object 30 image processing apparatus (server)
31 Real Estate Management Terminal 32 Object Management Terminal 33 Image Display Terminal 40 Image Processing Device (Editor)
41 image management server 42 object management device 43 converter 44 service management device 45 image display terminal 141 program storage unit 142 background image data storage unit 143 floor plan data storage unit 144 3D data storage unit 145 composite image data storage unit 150 image processing unit 151 placement setting unit 152 3D position estimation unit 153 object layer generation unit 154 shadow generation unit 155 shadow layer generation unit 156 image combining unit

Claims

An image processing apparatus that synthesizes an image of an object virtually placed on the real estate property against a background image of the real estate property,
A placement setting unit configured to set the placement location of the object on the floor plan of the real estate property;
A 3D position for estimating an object position which is a position corresponding to an arrangement position of the object on the floor plan in a three-dimensional graphics space surrounded by an astronomical image acquired as a background image by photographing the real estate property An estimation unit,
A shadow generation unit that generates a shadow generated by arranging a three-dimensional model of the object at the object position;
A first layer is generated by rendering an image in which the three-dimensional model disposed at the object position is projected onto the same celestial sphere as the background image with the center of the three-dimensional graphics space as the projection center. An object layer generation unit,
A shadow layer generation unit that generates a second layer by rendering an image in which the shadow is projected onto the celestial sphere with the center of the three-dimensional graphics space as the projection center;
An image combining unit that combines the second layer and the first layer with the background image;
An image processing apparatus comprising:
A display unit for displaying an image of the floor plan;
An operation input unit that receives an operation performed by a user;
And further
The image processing apparatus according to claim 1, wherein the arrangement setting unit sets the arrangement position based on an operation on the image of the floor plan displayed on the display unit received by the operation input unit.
The image processing apparatus according to claim 1, wherein the shadow generation unit sets a light source based on the background image, and generates a shadow of the three-dimensional model based on the light source.
The image processing apparatus according to any one of claims 1 to 3, wherein the 3D position estimation unit estimates the object position based on the dimensions of each part in the floor plan.
The image processing apparatus according to any one of claims 1 to 3, wherein the 3D position estimation unit estimates the object position based on a distance and a dimension of a subject located near the arrangement location.
The said 3D position estimation part acquires an imaging | photography point based on the distance and the dimension of the to-be-measured measured from the said background image, and estimates the said object position based on this imaging | photography point. An image processing apparatus according to claim 1.
The said 3D position estimation part acquires an imaging | photography point based on the distance and dimension of the to-be-measured object, and estimates the said object position based on this imaging | photography point. Image processing device.
An image processing method for synthesizing an image of an object virtually placed on the real estate property against a background image on which the real estate property is captured,
Setting the location of the object on the floor plan of the real estate property;
Estimating an object position which is a position corresponding to an arrangement position of the object on the floor plan in a three-dimensional graphics space surrounded by an astronomical image acquired as a background image by photographing the real estate property; ,
Generating a shadow resulting from placing a three-dimensional model of the object at the object position;
A first layer is generated by rendering an image in which the three-dimensional model disposed at the object position is projected onto the same celestial sphere as the background image with the center of the three-dimensional graphics space as the projection center. Step and
Generating a second layer by rendering an image in which the shadow is projected onto the celestial sphere with the center of the three-dimensional graphics space as the projection center;
Combining the second layer and the first layer with the background image;
Image processing method including:
An image processing program for synthesizing an image of an object virtually placed on the real estate property with respect to a background image showing the real estate property,
Setting the location of the object on the floor plan of the real estate property;
Estimating an object position which is a position corresponding to an arrangement position of the object on the floor plan in a three-dimensional graphics space surrounded by an astronomical image acquired as a background image by photographing the real estate property; ,
Generating a shadow resulting from placing a three-dimensional model of the object at the object position;
A first layer is generated by rendering an image in which the three-dimensional model disposed at the object position is projected onto the same celestial sphere as the background image with the center of the three-dimensional graphics space as the projection center. Step and
Generating a second layer by rendering an image in which the shadow is projected onto the celestial sphere with the center of the three-dimensional graphics space as the projection center;
Combining the second layer and the first layer with the background image;
An image processing program that causes a computer to execute.