JPWO2019088273A1 - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

Provided is an image processing apparatus and the like that can freely and smoothly synthesize a 3D model of an object such as furniture with a two-dimensional background image. In the 3D space surrounded by the arrangement setting unit that sets the arrangement location of the object on the floor plan of the real estate property and the celestial sphere image acquired as the background image, the image processing apparatus is arranged at the location of the object on the floor plan. A 3D position estimation unit that estimates a corresponding object position, a shadow generation unit that generates a shadow generated by placing the 3D model at the object position, and a 3D model that is the same as the background image with the center of the 3D space as the projection center An object layer generation unit that generates a first layer that renders an image projected above, and a shadow that generates a second layer that renders an image obtained by projecting the shadow onto the celestial sphere with the center of the 3D space as the projection center A layer generating unit; and an image combining unit that combines the second and first layers with the background image.

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program that synthesize an object image with a background image obtained by photographing a subject.

  In recent years, in the real estate industry, a service called home staging performed at the time of sales transactions has been known. Home staging is a service that provides interior coordination for real estate properties in order to promote the sale of real estate properties such as detached houses and condominiums. Specifically, furniture, furniture, accessories, etc. selected in consideration of the target audience, fashion, and property of the purchaser are arranged. It is said that purchase applicants (including those under consideration) can easily imagine the state of living in the property by viewing the property to which the home staging is applied, so that the sale can proceed smoothly.

  On the other hand, recently, a service for inspecting real estate properties using virtual reality (virtual reality, hereinafter also referred to as VR) technology has also been provided. VR image display is usually performed in a glasses-type or goggles-type display device called a head-mounted display (hereinafter also referred to as “HMD”). The HMD incorporates a screen on which two images with parallax are displayed and two lenses respectively installed in the line-of-sight direction of the user's eyes, and the user passes through these lenses. By viewing the screen, it is possible to recognize the image three-dimensionally. The HMD includes a gyro sensor and an acceleration sensor, and an image displayed on the screen changes according to the movement of the user's head detected by these sensors. As a result, the user can experience as if entering a stereoscopic image.

  As a technology related to the simulation of the interior of a real estate property, Patent Document 1 describes that a 3D model of a real estate property is generated from property information including floor plan images and indoor shot photos, and the virtual reality is based on the generated 3D model. Techniques for performing indoor browsing and interior simulation are disclosed (see paragraph 0015 of Patent Document 1). Specifically, the scale of the 3D furniture model prepared in advance is adjusted and placed in the 3D model of the property (see paragraph 0070 of Patent Document 1).

  Further, regarding a technique for generating composite image data of a live-action background image and a three-dimensional CG object, Patent Document 2 discloses a peripheral that generates three-dimensional shape data of the surrounding environment from environment map data composed of two or more viewpoints. An image processing apparatus is disclosed that includes environmental three-dimensional shape data generation means and virtual subject synthesis means for synthesizing a virtual subject with background image data using the surrounding environment three-dimensional shape data as a light source.

  Patent Document 3 discloses a foreground image, a background image as a background of the foreground image, storage means for storing depth information indicating the depth of the background image, and a background image of the foreground image. Composite position acquisition means for acquiring the composite position, projection means for projecting the coordinates of the space generated from the depth information of the background image to the coordinates of the background image, and the spatial coordinates of the composite position from the projection information and the composite position by the projection means A spatial coordinate acquisition means for acquiring the position of the light source viewed from the position where the background image was captured and a light source parameter that holds information on the illuminance and a distance between the light source and the spatial coordinate of the composite position from the spatial coordinates of the composite position, Alternatively, calculation means for calculating the illuminance of the light source viewed from the spatial coordinates of the combined position, light source parameter changing means for changing the light source parameter according to the spatial coordinates of the combined position and the calculated value, and the light source Disclosed is a video processing apparatus having a lighting unit that applies a lighting effect to a foreground image using a parameter, and a combining unit that combines a foreground image that has been subjected to the lighting effect by the writing unit into a background image combining position. Has been.

  In addition, as a service for real estate agents, non-patent property 1 uses a Ricoh Co., Ltd. digital camera (product name: THETA) to photograph 360 ° images (celestial sphere images) for VR use. A service that is processed into content and provided to a real estate agent is disclosed.

Japanese Patent No. 6116746 JP2013-152683A JP 2013-149219 A

Ricoh Co., Ltd., “Shooting / Authoring Service THETA360.biz”, [online], [October 17, 2017 Search], Internet <URL: https://theta360.biz/en/services/authoring/>

  As a method of providing home staging content that can be used by VR, when using a method of arranging a three-dimensional model of an object such as furniture in a three-dimensional background model of a real estate property, the construction (rendering) of 3D data It requires a lot of calculation costs and takes time, and also requires a lot of calculation costs when displaying data. Therefore, high-spec equipment is required, and it becomes difficult to easily use the contents of home staging.

  As a method for more easily creating home staging content that can be used by VR, a three-dimensional model (furniture model) may be combined with a two-dimensional background image that shows a real subject. However, since the position in the computer graphics space is different between the two-dimensional background image and the three-dimensional model, the furniture appears to be floating relative to the background image, the depth looks different, etc. It is easy for users who use content to feel uncomfortable, and it is difficult to freely arrange a three-dimensional model on a two-dimensional background image.

  The present invention has been made in view of the above, and an image processing apparatus, an image processing method, and an image processing apparatus that can freely and comfortably synthesize a three-dimensional model of an object such as furniture with a two-dimensional background image, An object of the present invention is to provide an image processing program.

  In order to solve the above problems, an image processing apparatus according to an aspect of the present invention is an image processing apparatus that synthesizes an image of an object that is virtually arranged in a real estate property with a background image in which the real estate property is reflected. In a three-dimensional graphics space surrounded by an arrangement setting unit that sets an arrangement location of the object on the floor plan of the real estate property, and a celestial sphere image acquired as a background image by photographing the real estate property A 3D position estimation unit that estimates an object position that is a position corresponding to the location of the object on the floor plan, and a shadow that generates a shadow generated by placing a three-dimensional model of the object at the object position A generation unit, and the center of the three-dimensional graphics space as a projection center; An object layer generation unit that generates a first layer by rendering an image obtained by projecting the placed 3D model onto the same celestial sphere as the background image, and a center of projection of the center of the 3D graphics space A shadow layer generation unit that generates a second layer by rendering an image obtained by projecting the shadow onto the celestial sphere, and the second layer and the first layer are combined with the background image And an image composition unit.

  The image processing apparatus further includes a display unit that displays an image of the floor plan, and an operation input unit that receives an operation performed by a user, and the placement setting unit is received by the operation input unit, The arrangement location may be set based on an operation on the floor plan image displayed on the display unit.

  In the image processing apparatus, the shadow generation unit may set a light source based on the background image and generate a shadow of the three-dimensional model based on the light source.

  In the image processing apparatus, the 3D position estimation unit may estimate the object position based on a size of each part in the floor plan.

  In the image processing apparatus, the 3D position estimation unit may estimate the object position based on a distance and a size of a subject located in the vicinity of the arrangement location.

  In the image processing apparatus, the 3D position estimation unit may acquire a shooting point based on a distance and a size of a subject measured from the background image, and estimate the object position based on the shooting point.

  In the image processing apparatus, the 3D position estimation unit may acquire a shooting point based on the actually measured distance and size of the subject, and may estimate the object position based on the shooting point.

  An image processing method according to another aspect of the present invention is an image processing method for synthesizing an image of an object virtually arranged in a real estate property with a background image showing the real estate property, The object on the floor plan is set in a three-dimensional graphics space surrounded by a celestial sphere image acquired as a background image by photographing the real estate property, and setting the location of the object on the floor plan. Estimating an object position that is a position corresponding to the arrangement position of the object, generating a shadow generated by arranging a three-dimensional model of the object at the object position, and projecting a center of the three-dimensional graphics space The three-dimensional model arranged at the object position is used as the center. Rendering an image projected on the same celestial sphere as a scene image, and generating a first layer, and projecting the shadow onto the celestial sphere with the center of the three-dimensional graphics space as the projection center Rendering an image to generate a second layer and synthesizing the second layer and the first layer to the background image.

  An image processing program according to another aspect of the present invention is an image processing program for synthesizing an image of an object virtually arranged in the real estate property with a background image showing the real estate property, wherein the real estate property In a three-dimensional graphics space surrounded by a celestial sphere image acquired as a background image by photographing the real estate property, and setting the location of the object on the floor plan. Estimating an object position which is a position corresponding to an object placement location, generating a shadow generated by placing a three-dimensional model of the object at the object position, and centering the three-dimensional graphics space The projection center is arranged at the object position. Rendering an image obtained by projecting a three-dimensional model onto the same celestial sphere as the background image, generating a first layer; and using the center of the three-dimensional graphics space as the projection center, the shadow as the celestial sphere Rendering the image projected above causes the computer to execute a step of generating a second layer and a step of combining the second layer and the first layer with the background image .

  According to the present invention, the object position in the three-dimensional graphics space corresponding to the arrangement position of the object on the floor plan is estimated, and the image of the three-dimensional model when the three-dimensional model of the object is arranged at the object position, and Layers are created by rendering the shadow images of the 3D model, and these layers are combined with the background image, so the 3D model of furniture and other objects can be freely and comfortably added to the 2D background image. It is possible to synthesize.

1 is a block diagram illustrating a schematic configuration of an image processing apparatus according to a first embodiment of the present invention. It is a schematic diagram for demonstrating the image composition process which the image processing apparatus shown in FIG. 1 performs. It is a schematic diagram for demonstrating the image composition process which the image processing apparatus shown in FIG. 1 performs. It is a flowchart which shows the image processing method which concerns on the 1st Embodiment of this invention. It is a floor plan of a real estate property that is an example of a subject of a background image It is a flowchart which shows the position estimation process shown in FIG. It is a schematic diagram for demonstrating the position estimation process of 3D model. It is a schematic diagram for demonstrating the position estimation process of 3D model. It is a schematic diagram for demonstrating the position estimation process of 3D model. It is a schematic diagram for demonstrating the image processing shown in FIG. It is a flowchart which shows the position estimation process in the 2nd Embodiment of this invention. It is a floor plan of a real estate article which is an example of a subject of a background image. It is a schematic diagram for demonstrating the position estimation process shown in FIG. It is a schematic diagram for demonstrating the position estimation process shown in FIG. It is a schematic diagram for demonstrating the position estimation process shown in FIG. It is a schematic diagram for demonstrating the position estimation process shown in FIG. It is a flowchart which shows the image processing method which concerns on the 3rd Embodiment of this invention. It is a floor plan of a real estate article which is an example of a subject of a background image. It is a schematic diagram for demonstrating the image processing method in the 3rd Embodiment of this invention. It is a network diagram which shows the structural example of the system by which the image processing apparatus which concerns on the 1st-3rd embodiment of this invention is applied. It is a schematic diagram which shows another structural example of the system with which the image processing apparatus which concerns on the 1st-3rd embodiment of this invention is applied.

  Hereinafter, an image processing apparatus, an image processing method, and an image processing program according to embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not limited by these embodiments. Moreover, in description of each drawing, the same code | symbol is attached | subjected and shown to the same part.

  An image processing apparatus according to an embodiment of the present invention uses a celestial sphere image in which a real estate is reflected as a background image, and synthesizes an image of a three-dimensional model (3D model) of an object such as furniture or furniture with the background image. By executing the image processing to be performed, an image is generated as if the object was placed and photographed on the real estate property.

  The celestial sphere image used as the background image is also called a 360 ° image, an omnidirectional image, an omnidirectional image, or the like, and is an image in which a subject of approximately 360 ° around the shooting location is captured. The background image (celestial sphere image) may be a still image or a moving image.

  In this image processing apparatus, a virtual transparent floor is arranged on the background image, and the shadow of the 3D model is projected onto the floor, thereby enhancing the sense of reality of the 3D model. Also, lighting the 3D model using the background image as a light source reduces the sense of discomfort.

  Here, in order to compose the entire image including the background of the real estate property with 3D CG, rendering is required for each drawing, and it takes a lot of calculation costs to execute this processing. Was necessary. However, since the image processing apparatus according to the present embodiment synthesizes a 3D model of an object that is virtually arranged on a two-dimensional background image, the calculation cost is not so high.

  Further, in the present image processing apparatus, since the 3D model data of the object is rendered in accordance with the projection of the two-dimensional background image, there is no shift in the depth, and the image can be displayed without a sense of incongruity.

In this way, by presenting content that combines the 3D model with the background image, customers who are considering buying or renting real estate properties can try to coordinate the furniture on the real estate property until they are satisfied on the screen. Can do. Accordingly, it is possible to promote the purchase of real estate properties and rental transactions without incurring much cost.
Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described in detail.

(First embodiment)
FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to the first embodiment of the present invention. 2A and 2B are schematic diagrams for explaining an image composition process executed by the image processing apparatus shown in FIG. Of these, FIG. 2A shows a floor plan of a real estate property. FIG. 2B shows an image obtained by synthesizing a 3D model of a sofa, a low table, a chiffon, and a picture with a background image (celestial sphere image) obtained by photographing in the living room of the real estate property (see FIG. 2A). Some are shown. As shown in FIG. 2B, the sofa, the low table, the chiffon, and the picture objects have a sense of depth according to the arrangement, so it is as if each object was actually placed in the living room and shot. As shown, the background image is displayed without a sense of incongruity.

  As illustrated in FIG. 1, the image processing apparatus 10 includes a communication interface 11, a display unit 12, an operation input unit 13, a storage unit 14, and a processor 15. As such an image processing apparatus 10, a general-purpose computer can be used.

  The communication interface 11 connects the image processing apparatus 10 to a communication network, and transmits / receives information to / from other devices connected to the communication network. The communication interface 11 is configured using, for example, a soft modem, a cable modem, a wireless modem, an ADSL modem, an ONU (Optical Network Unit), or the like. The communication interface 11 also functions as a data acquisition unit that captures image data generated by the omnidirectional camera, 3D model data, and the like into the image processing apparatus 10.

  The display unit 12 is a display including a display panel formed by liquid crystal or organic EL (electroluminescence) and a driving unit.

  The operation input unit 13 is an input device such as an operation button, a pointing device such as a keyboard and a mouse, and a touch sensor provided on the display unit 12, and receives an operation performed by the user and outputs a signal corresponding to the operation to the processor. 15 is input.

  The storage unit 14 is a computer-readable storage medium such as a semiconductor memory such as a ROM or a RAM. The storage unit 14 includes a program storage unit 141, a background image data storage unit 142, a floor plan data storage unit 143, a 3D data storage unit 144, and a composite image data storage unit 145, and an operating system program and driver Stores programs, application programs for executing various functions, various parameters used during the execution of these programs, image data, and the like.

Specifically, the program storage unit 141 stores various programs such as an image processing program.
The background image data storage unit 142 stores image data (hereinafter also referred to as background image data) of a celestial sphere image acquired as a background image by photographing a real estate property.
The floor plan data storage unit 143 stores image data of floor plans of real estate properties (hereinafter also referred to as floor plan data).

  The 3D data storage unit 144 stores data representing a 3D model of an object such as furniture virtually arranged in a real estate property (hereinafter also referred to as 3D data) and related parameters (for example, a parameter representing texture). Remember. The 3D data may be generated by the image processing apparatus 10 or may be acquired via the communication network N that is generated by another device.

  The composite image data storage unit 145 stores image data of a composite image obtained by combining a 3D model image with a background image (hereinafter also referred to as composite image data).

  The processor 15 is configured using, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and comprehensively controls each unit of the image processing apparatus 10 by reading various programs stored in the program storage unit 141. At the same time, various processes for synthesizing an image of an object such as furniture with a background image showing a real estate property are executed. Specifically, the functions of the image processing unit 150 realized by the processor 15 executing the image processing program include an arrangement setting unit 151, a 3D position estimation unit 152, an object layer generation unit 153, and a shadow generation unit. 154, a shadow layer generation unit 155, and an image composition unit 156 are included.

  The arrangement setting unit 151 sets the arrangement location in the floor plan of the object that is virtually arranged in the real estate property according to the signal input from the operation input unit 13.

  The 3D position estimation unit 152 is located in a floor plan in a three-dimensional graphics space (hereinafter also referred to as a 3D space) surrounded by a celestial background image and associated with a shooting point and a center point of the celestial sphere. Is estimated (hereinafter also referred to as object position).

  When the 3D model of the object is placed at the object position in the 3D space, the object layer generation unit 153 creates an image obtained by viewing the 3D model from the center of the 3D space as a layer provided on the same spherical surface as the background image. Render. That is, an image is generated by projecting the 3D model onto the celestial sphere with the center of the 3D space as the projection center. Hereinafter, the layer on which the image of the 3D model is rendered is referred to as an object layer (first layer).

  The shadow generation unit 154 generates a shadow generated by the 3D model arranged at the object position in the 3D space. Specifically, the shadow generation unit 154 executes an image based lighting process that sets a light source based on a background image, and generates a shadow of the 3D model based on the set light source. Image-based lighting is a technique for rendering a scene using an image of a real subject as lighting color information.

  The shadow layer generation unit 155 renders an image obtained by viewing the shadow generated by the shadow generation unit 154 from the center of the 3D space on a layer provided on the same celestial sphere as the background image. That is, an image is generated by projecting the shadow of the 3D model onto the celestial sphere with the center of the 3D space as the projection center. Hereinafter, the layer on which the shadow image is rendered is referred to as a shadow layer (second layer).

  The image composition unit 156 generates a composite image in which the image of the 3D model of the object is combined with the background image by superimposing the shadow layer and the object layer in this order on the background image.

Next, an image processing method according to the present embodiment will be described.
The outline of the image processing in this embodiment is as follows. That is, a background image (celestial sphere image) to be synthesized, a transparent celestial sphere image layer (shadow layer) for shadows, and a transparent celestial sphere image layer (object layer) for 3D models of objects such as furniture, A total of three layers of celestial sphere images are prepared. The background image is acquired by taking a picture using a spherical camera or the like at the real estate property site. Note that the number of layers to be superimposed on the background image may be increased as appropriate in addition to the above two layers.

  Next, a 3D model of the object is arranged in a three-dimensional graphics space (3D space) surrounded by these celestial sphere images. In order to add a shadow to the 3D model, a light source based on the background image is set by image-based triting.

  Next, a camera for rendering the celestial sphere image is arranged at the origin (the center of the celestial sphere) of the 3D space. In other words, the viewpoint (projection center) for viewing the 3D model and its shadow is set from the center of the 3D space.

  Next, when the 3D model is arranged in the 3D space, a shadow image generated by the light source set based on the background image is rendered and pasted on the shadow layer. Also, an image of the 3D model when the 3D model is arranged in the 3D space is rendered and pasted on the object layer. By superimposing such a background image, a shadow layer, and an object layer, it is possible to obtain a composite image in which the object is combined with the background image without a sense of incongruity.

Hereinafter, such image processing will be described in detail with specific examples. FIG. 3 is a flowchart showing an image processing method according to the present embodiment. FIG. 4 is a floor plan of a real estate property that is an example of the subject of the background image. In the present embodiment, a celestial image of a still image or a moving image is generated by setting the omnidirectional camera at a position corresponding to the shooting point P ₀ on the floor plan shown in FIG. In the following description, it is assumed that the 3D model is synthesized using this celestial sphere image as a background image.

  First, in step S10, the image processing unit 150 acquires a background image for synthesizing the 3D model. In the present embodiment, the image processing unit 150 acquires a background image by reading the background image data stored in the background image data storage unit 142.

  In subsequent step S11, the image processing unit 150 acquires data (3D data) representing a 3D model to be combined with the background image. Specifically, the image processing unit 150 causes the display unit 12 to display a name, an icon, and the like representing a 3D model in which 3D data is stored in the 3D data storage unit 144, and allows the user to select a desired 3D model. Then, the image processing unit 150 reads 3D data representing the selected 3D model from the 3D data storage unit 144 in accordance with a signal input from the operation input unit 13 in accordance with an operation by the user.

  In subsequent step S12, the image processing unit 150 acquires parameters relating to the 3D model to be combined with the background image. Specifically, parameters for displaying each coordinate of the 3D model as a function such as a curve or a curved surface, parameters used in processing such as texture mapping, and the like are read from the 3D data storage unit 144.

  In subsequent step S <b> 13, the image processing unit 150 estimates the position of the 3D model in the 3D space corresponding to the arrangement position of the object in the floor plan illustrated in FIG. 4.

FIG. 5 is a flowchart showing the position estimation process in step S13. 6 to 8 are schematic diagrams for explaining the position estimation process of the 3D model. Among these, FIG. 6 has shown the state which installed the omnidirectional camera 20 attached to the tripod on the floor surface. FIG. 7 shows a state in which a background image L1 obtained by photographing a real estate property with the omnidirectional camera 20 is cut by a horizontal plane passing through the center point P ₀ ′ of the celestial sphere. The background image L1 shown in FIG. 7, sink entering the field of view of the imaging point P ₀ (see FIG. 4) omnidirectional camera 20 installed in, stove, walls, Western-style door, Japanese-style sliding doors, bathroom doors, There are images of toilet doors and entrance doors. FIG. 8 shows a state in which the background image L1 is cut by a vertical plane passing through the center point P ₀ ′. The background image L1 illustrated in FIG. 8 includes images of a floor surface, a wall, a ceiling, an entrance door, and the like that fall within the field of view of the omnidirectional camera 20.

In this embodiment, the dimensions of each part (such as each room and equipment shown in the floor plan) and the coordinates of the shooting point P ₀ (planar coordinates and the height of the shooting point) in the real estate property are known. Based on the coordinates, the position of the 3D model in the 3D space is estimated. As shown in FIG. 6, the height of the shooting point P ₀ can be acquired as the height h of the omnidirectional camera 20 including the tripod.

First, in step S131, the arrangement setting unit 151 determines the arrangement of objects on the floor plan. Specifically, the arrangement setting unit 151 displays a floor plan of the real estate property on the display unit 12 and allows the user to specify a place where an object such as furniture is to be arranged on the floor plan. And the arrangement | positioning setting part 151 acquires the positional information (coordinate) of the object in a floor plan according to the signal input from the operation input part 13 according to operation by a user. In the following, it is assumed that the point P ₁ on the floor plan shown in FIG. 4 is determined as the arrangement location of the object desired by the user.

In subsequent step S132, the 3D position estimation unit 152 acquires the position of the shooting point P ₀ when the background image L1 is shot.

In subsequent step S133, the 3D position estimation unit 152 derives the position of the 3D model in the 3D space surrounded by the background image L1 based on the floor plan. Specifically, the 3D position estimation unit 152 creates a 3D space in which the shooting point P _{0 is associated} with the center point P ₀ ′ of the celestial sphere, and associates the position and size of each part in the floor plan with the coordinates in the 3D space. . Thereby, the coordinates of the point P ₁ ′ in the 3D space corresponding to the point P ₁ in the floor plan can be acquired.
When the position of the 3D model is estimated in this way, the process returns to the main routine.

  Referring to FIG. 3 again, in step S14 following step S13, the image processing unit 150 places the 3D model at the position in the 3D space estimated in step S13. FIG. 9 is a schematic diagram for explaining the image processing shown in FIG. 3, and shows a state in which the 3D model a11 of the sofa is arranged in the 3D space surrounded by the background image L1.

  In subsequent step S15, the shadow generation unit 154 sets a light source and a material. Specifically, the shadow generation unit 154 generates a light probe image in image-based lighting based on the background image L1, and sets a 3D model material (surface reflection characteristics and the like). Here, the light probe image is a high dynamic range image that records incident illumination conditions in all directions.

  In subsequent step S16, the shadow generation unit 154 generates a shadow of the 3D model arranged in the 3D space based on the light source and the material set in step S15. That is, rendering is performed using the generated light probe image. Thereby, a shadow using the 3D model as a shielding object is generated in the 3D space, and lighting based on the background image L1 is also performed on the surface of the 3D model. FIG. 9 shows a shadow a13 of the 3D model a11 generated by the light source a12.

In subsequent step S17, the shadow layer generation unit 155 generates a shadow layer in which the shadow image of the 3D model is rendered. Specifically, the shadow layer generation unit 155 applies the shadow of the 3D model generated in step S16 to the shadow layer L2 located on the same celestial sphere as the background image with the center point P ₀ ′ of the 3D space as the projection center. Project. FIG. 9 shows a region a14 of an image obtained by projecting a shadow a13 on the shadow layer L2 in the upper layer of the background image L1. In the shadow layer L2, the area other than the image area a14 is transparent.

In subsequent step S18, the object layer generation unit 153 generates an object layer in which an image of the 3D model is rendered. Specifically, the object layer generation unit 153 arranges the 3D model placed in the 3D space in step S14 and further lit in the step S16 with the center point P ₀ ′ in the 3D space as the projection center and the same sky as the background image. Projecting onto the object layer L3 located on the spherical surface. FIG. 9 shows a region a15 of an image obtained by projecting the 3D model a11 on the object layer L3, which is an upper layer of the shadow layer L2. In the object layer L3, the area other than the image area a15 is transparent.

  In subsequent step S19, the image composition unit 156 generates a composite image in which the background image L1, the shadow layer L2, and the object layer L3 are superimposed in this order. As a result, an image in which the image of the 3D model is synthesized at the position desired by the user in the background image is obtained.

  The composite image generated in this way may be displayed as a panoramic image in which the celestial sphere is developed, or may be displayed as VR content.

  As described above, according to the first embodiment of the present invention, in the 3D space surrounded by the celestial sphere image acquired as the background image, it corresponds to the arrangement location of the user-desired object specified on the floor plan. Since the 3D model of the object is placed at this position and the image of the 3D model is rendered on the celestial sphere, the object can be freely adjusted with respect to the background image, and the position and size are inconsistent. It can arrange without making it.

  According to the first embodiment of the present invention, a shadow of the 3D model is generated by a light source set based on the background image, and the shadow image is rendered on the celestial sphere. Can be generated.

  Furthermore, according to the first embodiment of the present invention, since the 3D model is illuminated with the light source set based on the background image, a more realistic composite image can be generated.

(Second Embodiment)
Next, a second embodiment of the present invention will be described.
The configuration of the image processing apparatus according to the second embodiment of the present invention is generally the same as that of the first embodiment (see FIG. 1), and the 3D model position estimation process (FIG. 3) executed by the 3D position estimation unit 152. Step S13) is different from the first embodiment.

FIG. 10 is a flowchart showing position estimation processing in the second embodiment of the present invention. FIG. 11 is a floor plan of a real estate property that is an example of the subject of the background image. 12A to 13B are schematic diagrams for explaining the position estimation processing in the present embodiment. Among these, FIG. 12A is a side view showing a state in which a real estate property is photographed by the omnidirectional camera 20, and FIG. 12B passes through the center point P ₀ ′ through the background image L1 obtained by the photographing. The state cut | disconnected by the vertical surface is shown. FIG. 13A is a top view showing a state where photographing is performed by the omnidirectional camera 20, and FIG. 13B shows a state in which the background image L1 is cut by a horizontal plane passing through the center point P ₀ ′.

In the present embodiment, the position estimation process when the dimensions of each part of the real estate property and the plane coordinates of the shooting point P ₀ are unknown will be described. In this case, as described below, an object position is estimated based on the size of the subject by referring to a subject such as furniture or equipment located in the vicinity of the location where the user-desired object is placed. Hereinafter, a subject used for reference is referred to as a reference object.

First, in step S231, the arrangement setting unit 151 determines the arrangement of objects on the floor plan. The object placement determination process is the same as in the first embodiment (see step S131 in FIG. 5). In the following, it is assumed that a point P ₂ on the floor plan shown in FIG. 11 is determined as an object placement location.

  In subsequent step S232, the 3D position estimation unit 152 selects a subject in the vicinity of the location of the object in the floor plan as a reference object. In the following, it is assumed that the sink shown in FIG. 11 is selected as the reference object 21.

  In subsequent step S233, the 3D position estimating unit 152 obtains the vertical field angle and the horizontal field angle of the visual field area in which the reference object 21 captured in the background image L1 is fully contained. Specifically, as shown in FIGS. 12B and 13B, a region a21 in which the reference object 21 is captured is extracted from the background image L1, and the vertical field angle α and the horizontal field angle β are measured. For the vertical angle of view α, an area corresponding to the height h from the floor surface to the omnidirectional camera 20 is measured.

In subsequent step S234, the 3D position estimating unit 152 generates a 3D space surrounded by the background image L1 and associating the shooting point P ₀ in the floor plan with the center point P ₀ ′ of the celestial sphere.

In subsequent step S235, the 3D position estimation unit 152 estimates the position of the reference object 21 in the 3D space. The position of the reference object 21 can be represented by a representative point such as a center point of the reference object 21 or a point at the shortest distance from the shooting point P ₀ . As a specific example, as shown in FIGS. 12A and 12B, and a vertical angle α and the height h of the imaging point P _0, the following equation (1), from the photographing point P ₀ to the representative point of the reference object 21 The distance d ₁ can be calculated.
d ₁ = h / tan α (1)

Further, as shown in FIGS. 13A and 13B, the position of the reference object 21 in the horizontal direction can be calculated from the horizontal angle of view β and the distance d ₁ . For example, the distance x ₄ from a certain reference position (for example, the front line of the omnidirectional camera 20) to the representative point of the reference object 21 can be obtained by the following equation (2).
x ₄ = d ₁ · sin (β / 2) (2)

In subsequent step S236, the 3D position estimation unit 152 derives the position of the 3D model in the 3D space. For example, as shown in FIG. 11, when the objects are arranged on the reference object 21 (a sink), the vicinity of the position of the reference object 21 estimated in step S235 (for example, the distance x _{4 is} shifted) also in the 3D space. The 3D model may be arranged at (position).
Thereafter, the process returns to the main routine.

As described above, according to the second embodiment of the present invention, even if the coordinates of the shooting point P ₀ are unknown, if the height h of the shooting point P ₀ is known, the 3D space can be used. It is possible to estimate the position of the 3D model.

  Here, when the 3D model is superimposed on an image being captured in real time as in AR, since the position of the capturing point is known, the position of the 3D model in the 3D space can be estimated relatively easily. However, according to the present embodiment, it is possible to appropriately set the position of the 3D model in the 3D space even when a background image whose shooting point position is unknown is used.

  In the above embodiment, the furniture and equipment provided in the real estate property are used as the reference object. Instead, a marker whose size is known in advance is placed in the real estate property as the reference object and photographed. May be performed. In this case, the position of the reference object in the 3D space can be estimated more easily.

(Third embodiment)
Next, a third embodiment will be described.
The configuration of the image processing apparatus according to the third embodiment of the present invention is generally the same as that of the first embodiment (see FIG. 1), and the 3D model position estimation process executed by the 3D position estimation unit 152 is the first. Different from the first embodiment.

FIG. 14 is a flowchart showing an image processing method according to the present embodiment. FIG. 15 is a floor plan of a real estate property that is an example of the subject of the background image. FIG. 16 is a schematic diagram showing an example of the background image, and shows a state in which the background image L1 is cut along a horizontal plane passing through the center point P ₀ ′. In the present embodiment, the shooting point P ₀ in the real estate is specified and the object position is estimated based on the measured value of the distance (depth) and dimensions of subjects such as furniture and interior arranged in the real estate.

  First, in step S30, the image processing unit 150 acquires a background image for synthesizing the 3D model.

In subsequent step S31, the image processing unit 150, based on the background image L1, measures the size of the distance and the subject from the photographic point P ₀ to the object at multiple locations, and stores the measurement data. The method for measuring the distance and dimensions of the subject is the same as that described in the second embodiment (see FIGS. 12A to 13B). The number of subjects whose distances and dimensions are measured is arbitrary, but as an example, the coordinates of the photographing point P _{0 on} the horizontal plane can be specified by measuring the distances to three subjects.

In subsequent step S32, the image processing unit 150 associates the measurement data of the subject with the background image. FIG. 16 shows a state in which the distances d _{31 to} d ₃₈ measured in step S31 and the dimensions x ₃₁ , x ₃₂ , y ₃₁ , y _{32 on} the horizontal plane are associated with the background image L1. Similarly, the dimension in the vertical plane is associated with the background image.

  In subsequent step S33, the image processing unit 150 acquires 3D data representing a 3D model to be combined with the background image L1. In step S34, the image processing unit 150 acquires parameters related to the 3D model to be combined with the background image L1. Steps S33 and S34 are the same as steps S11 and S12 shown in FIG.

In subsequent step S35, the image processing unit 150 determines the arrangement of the objects on the floor plan. The object placement determination process is the same as in the first embodiment (see step S131 in FIG. 5). In the following, the point P ₃ of the drawing Mato shown in FIG. 15, and that determined as the arrangement position of the object.

In subsequent step S36, the image processing unit 150 acquires the position of the imaging point P ₀ at the time of photographing the background image L1. Specifically, the 3D position estimation unit 152 calculates the coordinates of the shooting point P ₀ in the floor plan from the distance measured in step S31 (see FIG. 15).

In subsequent step S37, the image processing unit 150 is surrounded by the background image L1 based on the measurement data associated with the background image L1, and associates the shooting point P ₀ in the floor plan with the center point P ₀ ′ of the celestial sphere. 3D space is generated.

In subsequent step S38, the image processing unit 150 arranges the 3D model in the 3D space based on the arrangement of the objects determined in step S35. Specifically, the 3D position estimation unit 152 calculates the coordinates of the object placement location (point P ₃ ) with respect to the shooting point P ₀ in the floor plan, and the position in the 3D space corresponding to this placement location (in FIG. 16). A point P ₃ ′) is estimated and a 3D model is placed at this position.

  The subsequent processes in steps S39 to S43 are the same as steps S15 to S19 shown in FIG.

As described above, according to the third embodiment of the present invention, since the distances and dimensions of a plurality of subjects are measured, the photographing point P ₀ in the floor plan can be obtained with high accuracy. Therefore, it is possible to estimate the position in the 3D space corresponding to the arrangement location of the object determined on the floor plan with higher accuracy.

Next, a modification of the third embodiment of the present invention will be described.
In the third embodiment, the distance and dimensions of the subject are measured from the background image (see step S31), but the distance and dimensions may be measured at the time of shooting. Specifically, by attaching an auxiliary tool for distance measurement to an omnidirectional camera or smartphone, or by shooting a real estate property by executing a dedicated application, the distance to the subject and the dimensions of the subject are measured, The actual measurement data is stored in association with the image data. In this case, the 3D space corresponding to the real estate property can be reproduced more accurately.

  In the first to third embodiments described above, the light source is set based on the background image by image-based lighting, but the method of setting the light source is not limited to this, and various known methods are applied. Can do. For example, a 3D model shadow is generated by placing a light source (window, lighting device, etc.) reflected in a background image as a light source object in 3D space and performing global illumination calculation (radiosity method, photon mapping method, etc.) You may do it.

  FIG. 17 is a network diagram illustrating a configuration example of a system to which the image processing apparatus according to the first to third embodiments of the present invention is applied. The system 1 shown in FIG. 17 includes an image processing device (server) 30, a real estate management terminal 31, an object management terminal 32, and an image display terminal 33, and these devices are connected via a communication network N. Yes. As the communication network N, for example, an Internet line, a telephone line, a LAN, a dedicated line, a mobile communication network, a communication line such as WiFi (Wireless Fidelity), Bluetooth (registered trademark), or a combination thereof is used. The communication network N may be wired, wireless, or a combination thereof.

  The image processing apparatus 30 is configured by a host computer having high arithmetic processing capability, functions as a server that manages the system 1 in an integrated manner, and executes the image processing described in the first to third embodiments. The number of computers constituting the image processing apparatus 30 is not necessarily one, and may be composed of a plurality of computers distributed on the communication network N.

  The real estate management terminal 31 manages information related to real estate rental properties and real estate sales properties. Specifically, the real estate management terminal 31 stores image data (floor plan data and background image data) related to real estate properties, in addition to information related to transactions such as the location, owner, lease, or sales conditions of the real estate properties. Floor plan data and background image data are uploaded from the real estate management terminal 31 to the image processing apparatus 30 and used for image processing.

  The object management terminal 32 creates and stores information related to objects such as furniture to be synthesized with the background image, that is, 3D model data and parameters representing the object, and a setting file. These pieces of information are uploaded from the object management terminal 32 to the image processing apparatus 30 and used for image processing.

  The image display terminal 33 is a terminal device for displaying an image synthesized by the image processing device 30 and allowing the user to browse. The image display terminal 33 displays a two-dimensional still image or moving image to allow the user to recognize a three-dimensional virtual space (VR). As the image display terminal 33, a goggle-type dedicated device (so-called head mounted display (HMD)) that is directly mounted on the user's head may be used, or a dedicated application installed on a general-purpose smartphone. May be attached to a goggle-type holder.

  Inside the image display terminal 33, two lenses are respectively attached at positions corresponding to the left and right eyes of the user. When playing back VR, two images with parallax are displayed in two left and right areas provided on the screen, respectively. The user can recognize (stereoscopically) the image three-dimensionally by viewing the two images with the left and right eyes through the two lenses.

  Of course, the image display terminal 33 is not limited to the HMD, and a tablet terminal or a stationary display may be used. In this case, an image obtained by combining a three-dimensional model of an object such as furniture with a background image may be displayed as a panoramic image.

  In such a system 1, the image processing apparatus 30 uses a background image image data uploaded from the real estate management terminal 31, a 3D model data uploaded from the object management terminal 32, and the like as a 3D background image. A composite image obtained by combining the models is generated, and the image data of the composite image is transmitted to the image display terminal 33. Thereby, a composite image can be displayed on the image display terminal 33.

  In the system 1, various terminal devices may be further connected to the communication network N. For example, a terminal device that is used for allowing the user to select a background image or an object to be combined with the background image or to specify an arrangement position of the object in the floor plan may be separately provided. Further, a plurality of image display terminals 33 may be provided so that the same content can be simultaneously viewed on these image display terminals 33.

  FIG. 18 is a schematic diagram illustrating another configuration example of a system to which the image processing apparatus according to the first to third embodiments of the present invention is applied. In FIG. 18, a bold solid line arrow represents a flow of data to be downloaded, and a broken arrow represents a flow of data to be uploaded.

  The system 2 illustrated in FIG. 18 includes an image processing device (editor) 40, an image management server 41, an object management device 42, a converter 43, a service management device 44, and an image display terminal 45. These devices are connected to each other via a communication network.

The image processing apparatus (editor) 40 functions as an editor for processing and editing a background image by executing the image processing described in the first to third embodiments.
The image management server 41 manages images used in the system 2.

  The object management device 42 is a setting file in which data representing a 3D model of objects such as furniture and furniture, related information such as textures (hereinafter collectively referred to as a 3D file), and arrangement information of furniture and the like are described. Is created or stored externally. Note that “FBX” illustrated in FIG. 18 is an example of a file format of data representing the 3D model. However, the file format that can be used in this system is not limited to this.

  The converter 43 converts the data representing the 3D model into a file that can be read into the image processing apparatus 40 and is collected together with related information such as texture.

  The service management device 44 includes a storage unit that stores floor plan data and background image data, and a screen that can display the floor plan and background image, and displays the floor plan on the screen to display objects such as furniture. Is used to allow the user to designate the arrangement location of the image, or to display the composite image 44b obtained by combining the background image 44a and the 3D model of the object on the screen and present it to the user.

  The configuration of the image display terminal 45 is the same as that of the image display terminal 33 shown in FIG. 17, and is used when a user browses a synthesized image obtained by synthesizing a 3D model with a background image as a VR.

  In such a system 2, first, the object management device 42 uploads 3D model data of an object such as furniture to the image management server 41. The converter 43 downloads the data of the 3D model from the image management server 41, converts the 3D model, texture, and the like into a file that is collected, and uploads the file to the image management server 41. The image processing apparatus 40 downloads the file thus converted from the image management server 41, downloads the setting file from the object management apparatus 42, and further downloads the background image data from the service management apparatus 44. Then, the image processing apparatus 40 performs each process (such as lighting setting and material adjustment) for synthesizing the 3D model with the background image, and uploads the created synthesized image data file to the service management apparatus 44. The image display terminal 45 downloads a composite image data file from the service management apparatus 44 and reproduces a composite image, that is, home staging content based on the file.

  The present invention is not limited to the first to third embodiments and modified examples described above, but various combinations of a plurality of components disclosed in the first to third embodiments can be used. An invention can be formed. For example, some constituent elements may be excluded from all the constituent elements shown in the first to third embodiments and modifications, and the first to third embodiments and modifications may be used. You may form combining the shown component suitably.

DESCRIPTION OF SYMBOLS 1, 2 System 10 Image processing apparatus 11 Communication interface 12 Display part 13 Operation input part 14 Storage part 15 Processor 20 Global camera 21 Reference object 30 Image processing apparatus (server)
31 Real Estate Management Terminal 32 Object Management Terminal 33 Image Display Terminal 40 Image Processing Device (Editor)
41 Image management server 42 Object management device 43 Converter 44 Service management device 45 Image display terminal 141 Program storage unit 142 Background image data storage unit 143 Floor plan data storage unit 144 3D data storage unit 145 Composite image data storage unit 150 Image processing unit 151 Arrangement Setting Unit 152 3D Position Estimation Unit 153 Object Layer Generation Unit 154 Shadow Generation Unit 155 Shadow Layer Generation Unit 156 Image Composition Unit

Claims (9)

An image processing apparatus that synthesizes an image of an object virtually placed in the real estate property against a background image showing the real estate property,
An arrangement setting unit for setting an arrangement location of the object on the floor plan of the real estate property;
A 3D position for estimating an object position that is a position corresponding to an arrangement position of the object on the floor plan in a three-dimensional graphics space surrounded by a celestial sphere image acquired as a background image by photographing the real estate property An estimation unit;
A shadow generator that generates a shadow generated by placing a three-dimensional model of the object at the object position;
A first layer is generated by rendering an image obtained by projecting the three-dimensional model arranged at the object position on the same celestial sphere as the background image with the center of the three-dimensional graphics space as the projection center. An object layer generator,
A shadow layer generation unit that generates a second layer by rendering an image obtained by projecting the shadow on the celestial sphere with the center of the three-dimensional graphics space as a projection center;
An image composition unit for compositing the second layer and the first layer to the background image;
An image processing apparatus comprising: A display unit for displaying an image of the floor plan;
An operation input unit for receiving an operation performed by the user;
Further comprising
The image processing apparatus according to claim 1, wherein the arrangement setting unit sets the arrangement location based on an operation on the image of the floor plan displayed on the display unit received by the operation input unit.   The image processing apparatus according to claim 1, wherein the shadow generation unit sets a light source based on the background image, and generates a shadow of the three-dimensional model based on the light source.   The image processing apparatus according to claim 1, wherein the 3D position estimation unit estimates the object position based on a size of each part in the floor plan.   The image processing apparatus according to claim 1, wherein the 3D position estimation unit estimates the object position based on a distance and a size of a subject located in the vicinity of the arrangement location.   The said 3D position estimation part acquires an imaging | photography point based on the distance and dimension of the to-be-photographed object measured from the said background image, and estimates the said object position based on this imaging | photography point. The image processing apparatus according to item.   The said 3D position estimation part acquires an imaging | photography point based on the distance and dimension of the to-be-measured subject, and estimates the said object position based on this imaging | photography point. Image processing device. An image processing method for synthesizing an image of an object virtually placed in a real estate property against a background image showing the real estate property,
Setting an arrangement location of the object on the floor plan of the real estate property;
Estimating an object position that is a position corresponding to the location of the object on the floor plan in a three-dimensional graphics space surrounded by a celestial sphere image acquired as a background image by photographing the real estate property; ,
Generating a shadow caused by placing a three-dimensional model of the object at the object position;
A first layer is generated by rendering an image obtained by projecting the three-dimensional model arranged at the object position on the same celestial sphere as the background image with the center of the three-dimensional graphics space as the projection center. Steps,
Generating a second layer by rendering an image obtained by projecting the shadow onto the celestial sphere with the center of the three-dimensional graphics space as a projection center;
Combining the second layer and the first layer with the background image;
An image processing method including: An image processing program that synthesizes an image of an object virtually placed in the real estate property against a background image showing the real estate property,
Setting an arrangement location of the object on the floor plan of the real estate property;
Estimating an object position that is a position corresponding to the location of the object on the floor plan in a three-dimensional graphics space surrounded by a celestial sphere image acquired as a background image by photographing the real estate property; ,
Generating a shadow caused by placing a three-dimensional model of the object at the object position;
A first layer is generated by rendering an image obtained by projecting the three-dimensional model arranged at the object position on the same celestial sphere as the background image with the center of the three-dimensional graphics space as the projection center. Steps,
Generating a second layer by rendering an image obtained by projecting the shadow onto the celestial sphere with the center of the three-dimensional graphics space as a projection center;
Combining the second layer and the first layer with the background image;
An image processing program for causing a computer to execute.