JP2006040053A - Image processing method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method which easily constructs an indoor space model of high reality in a virtual space and provides a person, who wants to buy a house, with more realistic VR pictures, and also provide a program therefor. <P>SOLUTION: Actual pictures of an indoor space photographed with a plurality of photographing devices are obtained, and the obtained actual pictures are compared with virtual pictures obtained by viewing the virtual model of the same indoor model from virtual viewpoints to compute a correlation between the actual pictures and the virtual model. Then, the surface area of the virtual model for mapping the textures of the actual pictures is designated, and the partial areas of the actual pictures after conversion of their forms are mapped to the surface area. Then, the VR images obtained by viewing the virtual model after the texture mapping from arbitrary viewpoints are generated and displayed on a display device. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing method and a program for displaying a VR image obtained by reproducing an indoor space of a building displayed in a housing exhibition hall or the like in a virtual space to a VR experience person.

  In recent years, with the development of computer hardware technology, more realistic CG (Computer Graphics) images are being actively expressed. Therefore, virtual reality (virtual reality) that constructs a virtual world using highly developed CG technology and perceives the world like reality is being used in various fields. For example, a VR image (video) in which a newly constructed house is virtually constructed in a computer using a three-dimensional model, and a three-dimensional indoor space model of the house to be purchased for a person who wants to buy a house is projected from an arbitrary viewpoint. ) Can be presented. Furthermore, by exchanging and moving the parts of the three-dimensional model, the color of the indoor wall and the type of decoration such as furniture can be changed or the arrangement can be changed. As a result, housing manufacturers, etc. should provide not only two-dimensional information such as design drawings and external views to purchase applicants, but also provide three-dimensional images of houses, etc., in order to contribute to considerations when purchasing. Is possible.

  However, even though CG technology has been developed, the space realized by the CG video is still poor in reality compared to the space of the live-action video. Therefore, methods such as 360 ° panoramic video and QuickTime VR are also known as methods for expressing a three-dimensional space using live-action video.

In addition, in response to the viewer view in the virtual space, a plurality of subject video cuts (small fragmented live-action video images) that have been previously cut and accumulated from the live-action video are played back to the viewer. There is also known a technique for presenting an image in accordance with movement (for example, see Patent Document 1).
Japanese Patent Laid-Open No. 11-175745

  However, an expression in a three-dimensional space expressed by using a live-action video or an expression for reproducing a small fragmented live-action video image described in Patent Document 1 in accordance with the movement of the viewer in the virtual space is arbitrary. In terms of presenting the video from the viewpoint to the purchase applicant, the video is limited as compared with the VR video as described above.

  The present invention has been made in consideration of such circumstances, and can easily construct a highly realistic indoor space model in a virtual space, which is more realistic for those who want to purchase a house. An object of the present invention is to provide an image processing method and program capable of presenting a VR video provided with video.

In order to solve the above problems, an image processing method according to the present invention includes:
An acquisition step of acquiring an actual image of the indoor space imaged using the imaging device;
A two-dimensional coordinate in the real image is obtained by comparing a virtual image obtained by viewing a virtual model corresponding to the indoor space constructed in a computer from a virtual viewpoint corresponding to the viewpoint of the imaging device and the real image. And a calculation step of calculating a correspondence relationship between the virtual model and the three-dimensional coordinates of the virtual model;
A designating step of designating a surface area of the virtual model for mapping a texture in the real video;
A shape conversion step of converting a partial region in the real image corresponding to the surface region into the same shape as the surface region based on the correspondence calculated by the calculation step;
A mapping step of mapping the partial area in the real image after shape conversion to the surface area;
An image generation step of generating a VR image when the virtual model after mapping the texture in the real video is viewed from an arbitrary viewpoint;
A display step of displaying the VR image on a display device.

The image processing method according to the present invention includes:
The acquisition step includes the partial region mapped to the surface region designated by the designation step among real images photographed by a plurality of photographing devices capable of photographing the indoor space from a plurality of different directions. The image is obtained from a photographing device determined based on a positional relationship between the orientation of the surface area and the photographing direction of each photographing device.

Furthermore, an image processing method according to the present invention includes:
The acquisition step acquires a real image that is captured and distributed by the imaging device via a network in a predetermined time unit,
The display step sequentially displays the VR image generated by the image generation step in the predetermined time unit on the display device.

Furthermore, the program according to the present invention is:
On the computer,
An acquisition procedure for acquiring a real image of the indoor space photographed using the photographing device;
By comparing a virtual image obtained by viewing a virtual model corresponding to the indoor space constructed in the computer from a virtual viewpoint corresponding to the viewpoint of the imaging device with the actual image, a two-dimensional image in the actual image is obtained. A calculation procedure for calculating the correspondence between the coordinates and the three-dimensional coordinates of the virtual model;
A designation procedure for designating a surface area of the virtual model for mapping a texture in the real video;
A shape conversion procedure for converting a partial region in the actual video corresponding to the surface region into the same shape as the surface region based on the correspondence calculated by the calculation procedure;
A mapping procedure for mapping the partial area in the real image after shape conversion to the surface area;
An image generation procedure for generating a VR image when the virtual model after mapping the texture in the real video is viewed from an arbitrary viewpoint;
And a display procedure for displaying the VR image on a display device.

  According to the present invention, it is possible to easily construct a highly realistic indoor space model in a virtual space, and it is possible to present a VR video with a more realistic feeling to those who want to purchase a house. .

  Hereinafter, an image processing method according to an embodiment of the present invention will be described with reference to the drawings. In the present embodiment, the same virtual model (indoor space model) as the indoor space of the model house displayed in a certain house exhibition hall is constructed in the virtual space of the computer, and an arbitrary space included in the indoor space model is included. A description will be given of a method of mapping the texture of the object by cutting it out from the video image of the actual model house.

  Note that an apparatus (device) for performing image processing according to the present embodiment can be realized using a general-purpose computer such as a three-dimensional CAD, a personal computer, or a workstation.

  FIG. 1 is a schematic diagram showing a configuration of an image processing system that takes a picture of an actual model house displayed in a house exhibition hall and performs processing for pasting the picture on a house model built in a computer. It is.

  In FIG. 1, reference numeral 11 denotes a model house actually constructed in a residential exhibition hall. Reference numerals 12a and 12b denote video cameras for photographing the indoor space of the model house 11 from various viewpoints. The number of video cameras can be arbitrarily determined. For example, a minimum number of video cameras may be installed based on the number of objects to be cut out of textures from captured images, and may be one or two or more. Reference numeral 13 denotes a texture of an arbitrary object (for example, a wall, a floor, a table, etc.) in the video captured by the video cameras 12a and 12b, which is mapped and stored in a three-dimensional indoor space model. Is a computer that can view a video from an arbitrary viewpoint with a mouse or the like. In addition, the computer 13 can distribute the actual video to other computers 14, 15 and the like via a network 16 such as a LAN.

  14 and 15, the same three-dimensional indoor space model as the model house 11 is constructed in the virtual space, and the texture of the object cut out is mapped and stored in the three-dimensional indoor space model in the same manner as the computer 13 to purchase a house. This is a computer that allows a user or the like to view a VR video from an arbitrary viewpoint with a mouse or the like. The actual video used by the computers 14 and 15 may be acquired directly from the imaging devices 12a and 12b, or only the texture cut out by the computer 13 may be acquired.

  That is, the computers 14 and 15 do not need to be installed in the housing exhibition hall, and can be installed at any position remote from the housing exhibition hall. In addition, any number of computers can be connected to the LAN 16 nationwide. Therefore, a person who wants to purchase a house does not go to the house exhibition hall where the model house is actually exhibited, but simply goes to a nearby office or the like where a computer connected to the LAN 16 is installed. The VR image by the highly realistic three-dimensional virtual model constructed in the above can be viewed. In addition, the appearance changes according to the external environment that changes every moment by cutting out textures from video captured by the video cameras 12a and 12b at regular intervals or in real time and mapping them to a three-dimensional indoor space model. It is also possible to provide a VR image having an atmosphere corresponding to the time of the indoor space.

  FIG. 2 is a block diagram illustrating a configuration of a computer that executes image processing for providing VR video based on the three-dimensional indoor space model according to the present embodiment. In FIG. 2, the operation unit 120 includes, for example, a mouse, a keyboard, buttons, and the like, and an operator can input operation data and give various instructions to the image processing unit 110. The temporary storage unit 130 is configured by hardware such as a RAM, and an application program such as an image processing program, which will be described later, is developed and becomes a work area of the image processing unit 110. The storage unit 140 is composed of a fixed or removable hard disk or ROM, or a portable storage medium (recording medium) such as a DVD or MO, and maps programs and textures related to various information processing according to the present embodiment. Various data including a three-dimensional indoor space model which is a virtual space is stored.

  The image processing unit 110 remaps the texture of the specified object of the 3D indoor space model to the texture of the actual video acquired from the communication unit 160 based on the image processing program stored in the storage unit 140 and the like. A VR image viewed from a virtual viewpoint is generated. The generated VR images are stored in the storage unit 140 in chronological order, and can be reproduced as a VR video on the display unit 150 in real time. The display unit 150 is realized by, for example, a CRT or a liquid crystal monitor, and outputs (reproduces) a VR image (video) generated by the processing by the image processing unit 110.

  The communication unit 160 is connected to other computers, CAD systems, etc. directly or via a network such as the Internet or a LAN, and performs various data communications with these external devices. In the present embodiment, the three-dimensional indoor space model that is a virtual space is stored in the storage unit 140 as described above, but the three-dimensional indoor space model is stored by the image processing unit 110 of the computer. It is also possible to download the one constructed by an external device via the communication unit 160 or use it while performing data communication.

  The image processing unit 110 includes a region cutout unit 111 that cuts out a region of an object that is a target for mapping a texture of a live-action video from video images captured by the video cameras 12a and 12b, and three textures of the cut out regions. A shape conversion unit 112 that converts a texture surface to map to a corresponding surface of a three-dimensional indoor space model, a mapping unit 113 that maps a texture surface whose shape has been converted, and a viewpoint recognition unit that recognizes a viewpoint in a virtual space 114, a rendering unit 115 that generates a two-dimensional VR image by rendering a 3D indoor space model in which a texture based on a live-action image is mapped from a recognized viewpoint on a projection plane, and is generated and stored in the storage unit 140 A playback unit 116 for playing back the VR image thus recorded is included.

  FIG. 3 is a block diagram showing a hardware configuration of a computer that executes image processing for mapping a texture of a live-action image according to the present embodiment to a three-dimensional indoor space model. In FIG. 3, a CPU 201 implements the image processing unit 101 of FIG. 2, and performs control for executing various applications based on the OS. The RAM 202 expands application programs executed by the CPU 201 and temporarily stores work data, and implements the temporary storage unit 130 of FIG. 1.

  The ROM 203 stores a basic input / output system (BIOS) that is a program for controlling various devices connected to the computer. The stored BIOS or the like can be electrically rewritten and can be upgraded. The network interface 204 is for accessing the computer with an external device connected to the network 205 such as a LAN or the Internet, and implements the communication unit 160 of FIG.

  Reference numeral 206 denotes an operation unit interface for taking in information instructed by the operator using an operation device 207 such as a keyboard and a mouse. The display controller 208 is a controller for processing an image drawing command from the CPU 201, and outputs (reproduces) the generated VR image on the display 209.

  Furthermore, the storage unit interface 210 is an interface for connecting an external storage device such as a hard disk drive (HDD) 211 or a media drive 212 for a portable storage medium 213 such as a DVD or MO to a computer. The hard disk drive 211 stores an OS program executed by the CPU 201, various application programs, device drivers for various devices, and the like. The hard disk drive 211 or the storage medium 213 stores the CG image generated according to the present embodiment in time series so that it can be reproduced as a moving image. Note that the CPU 201, RAM 202, ROM 203, network interface 204, operation unit interface 206, display controller 208, and storage unit interface 210 described above are connected to each other via a bus 214.

  FIG. 4 is a flowchart for explaining an image processing procedure for mapping the texture of the live-action video according to the embodiment of the present invention to the three-dimensional indoor space model. That is, in the following, the VR image generation process performed in accordance with the image processing procedure shown in FIG. 4 will be described in detail using the computer described with reference to FIGS.

  First, a live-action picture taken from the video cameras 12a, 12b, etc. installed in the model house is taken into the computer (step S11). FIG. 5 is a diagram illustrating an example of an installation situation of a video camera that captures an image in a model house and an actual image captured by the video camera in the present embodiment. In the present embodiment, it is assumed that a video camera for photographing the directions indicated by arrows 51 to 55 is installed as shown in FIG. FIG. 5B shows a display example of the indoor space model rendered with a line of sight from the direction corresponding to the arrow 55.

  In the embodiment described below, the three-dimensional indoor space model is stored in the computer shown in FIGS. 2 and 3, but the construction may be modeled using the computer, A model obtained by acquiring and storing a three-dimensional indoor space model generated by another device such as CAD via a network or the like, or a model stored in those other devices connected via a network or the like is used while communicating. You may do it.

  The video image captured in the computer is matched with the image from the VR space (step S12). Specifically, camera information and virtual camera information in the VR space are matched, and an arbitrary partial region in the video image is associated with a portion in the VR space. Then, an object to which the texture of the live-action video image is pasted and its area (surface area) are designated (step S13).

  Thereafter, in order to paste a region obtained by cutting out the texture of the designated portion from the live-action video onto a surface in the VR space, the shape is converted by performing coordinate conversion (step S14). This shape conversion is performed because the cut-out area is usually photographed at a certain angle from the relationship between the viewing direction of the camera and the orientation of the surface, and is distorted and not the original shape. Then, the texture whose shape has been converted is mapped to a corresponding surface in the VR space (step S15). Next, the presence / absence of a surface on which a texture is mapped from a live-action video in the VR space is determined (step S16).

  As a result, if there is still another surface area for texture mapping (Yes), the process returns to step S13 and the same processing is performed. On the other hand, if there is no other texture mapping surface (No), a VR image is generated in real time by rendering the virtual space viewed from the VR user's line of sight (ie, viewed from the virtual viewpoint). Playback is performed (step S17).

  FIG. 6 is a schematic diagram for explaining details of the video image acquisition processing in step S11. In FIG. 6, in the real space where the object A and the object B exist, a plurality of video cameras are installed one by one at the four corners and other parts of the real space, and real images are acquired and acquired as virtual objects in the VR space. The pasted video shall be pasted.

  In FIG. 6, in the case of video shooting from the camera viewpoint 1, depending on the type of the lens of the camera, for example, a video image acquired by aligning the focal axis with the object B is projected on the live-action image projection plane 1. It becomes. Note that an image including the projection images of the object A and the object B is acquired as the live-action image projection plane 1. Similarly, in the case of video shooting from the camera viewpoint 2, an image including the projection images of the object A and the object B is projected and acquired on the real image projection plane 2.

  Here, it is desirable that the relationship between the surface of the object in the real space and the camera line of sight is as orthogonal as possible. This is because an image of a surface taken at an angle deviating from orthogonality is distorted, so that an image of a surface with less distortion is used. Therefore, for the object A, the image of the camera viewpoint 1 is used as the texture for the surface A1 and the image of the camera viewpoint 2 is used as the texture for the surface A2. On the other hand, for the object B, the image of the camera viewpoint 1 is used for the texture as the VR image for the surface B2 and the image of the camera viewpoint B2 for the surface B2.

  In this embodiment, since the camera position for shooting the actual video is basically fixed, the necessary number of units is placed where necessary so that no blind spots are generated according to the interior of the model house or the arrangement of furniture. as well as installation, objects in the three-dimensional interior space model (e.g., furniture, walls, etc.) in advance to determine whether a video captured video to be used as a texture to be mapped on each side beforehand in which video cameras It is better to always use it fixed.

  Next, the details of the process of obtaining a video image in step S11 and then cutting out and pasting the texture from the video image will be described. FIG. 7 is a schematic diagram for explaining the details of the process of pasting the acquired video into the VR space. FIG. 7A shows a situation in which two textures are pasted on the object A and the object B, respectively, using an image from the camera 1 that is assumed to be installed in a real space. That is, the origin at the time of image conversion is set as the camera viewpoint, and the image on the live-action image projection plane 1 is converted into a plane. Here, since the specifications of the camera lens and the three-dimensional positions of the objects in the live-action space and the VR space are known, it is possible to perform video conversion.

  FIG. 7B is a schematic diagram showing how camera viewpoints are converted. As shown in FIG. 7B, since the object B is orthogonal to the object B, only one video is pasted on the object B, but the distortion is minimized. In addition, with respect to the object A, the planes to be created are different for the camera viewpoints 1 and 2, and an image with less distortion (for example, an image of a focal axis that is more orthogonal to the plane) is adopted.

  If there are overlapping portions of real images taken by a plurality of photographing devices in the same plane area, the texture may be cut out from the real images taken by any one of the photographing devices. Then, the surface area may be further divided into a plurality of partial areas, and the actual video that is optimal as each texture may be used from the actual video from each imaging device.

  Further, the three-dimensional indoor space model is not limited to the one shown in FIG. 5, and may be another indoor space or an indoor space such as an office or a store. Moreover, the external appearance of a house, a building, etc. may be sufficient.

  In the present invention, a program for realizing the functions of the above-described embodiments by a computer is recorded on a computer-readable recording medium (storage medium), and the program recorded on the recording medium is stored in the system. A computer or the like may read and execute. The computer-readable recording medium includes a portable recording medium such as a flexible disk, CD-ROM, and DVD-ROM, a hard disk built in the system or computer, a storage device such as a volatile memory (RAM) (recording). Device). The program may be downloaded to the system, computer, or the like via a network such as the Internet or a communication line such as a telephone line. Then, the functions of the above-described embodiments are realized by executing the read program in the system or the computer. In this case, the recording medium or the like stores a program corresponding to the flowchart described above.

It is an outline figure showing composition of an image processing system which takes a picture for photographing a picture of an actual model house displayed in a house exhibition hall, and affixing the picture on a house model built in a computer. It is a block diagram which shows the structure of the computer which performs the image process for providing the VR image | video based on the three-dimensional room space model which concerns on this embodiment. It is a block diagram which shows the hardware constitutions of the computer which performs the image process which maps the texture of the captured image concerning this embodiment to a three-dimensional VR model. It is a flowchart for demonstrating the image processing procedure for mapping the texture of the photographed image which concerns on one Embodiment of this invention to a three-dimensional indoor space model. It is a figure which shows an example of the installation condition of the video camera which image | photographs the image | video in a model house in this embodiment, and an actual image image | photographed by it. It is a schematic diagram for demonstrating the detail of the video image | video acquisition process of step S11. It is a schematic diagram for demonstrating the detail of the process which pastes the acquired image | video in VR space.

Claims (4)

An acquisition step of acquiring an actual image of the indoor space imaged using the imaging device;
A two-dimensional coordinate in the real image is obtained by comparing a virtual image obtained by viewing a virtual model corresponding to the indoor space constructed in a computer from a virtual viewpoint corresponding to the viewpoint of the imaging device and the real image. And a calculation step of calculating a correspondence relationship between the virtual model and the three-dimensional coordinates of the virtual model;
A designating step of designating a surface area of the virtual model for mapping a texture in the real video;
A shape conversion step of converting a partial region in the real image corresponding to the surface region into the same shape as the surface region based on the correspondence calculated by the calculation step;
A mapping step of mapping the partial area in the real image after shape conversion to the surface area;
An image generation step of generating a VR image when the virtual model after mapping the texture in the real video is viewed from an arbitrary viewpoint;
A display step of displaying the VR image on a display device.   The acquisition step includes the partial region mapped to the surface region designated by the designation step among real images photographed by a plurality of photographing devices capable of photographing the indoor space from a plurality of different directions. The image processing method according to claim 1, wherein the image processing method is obtained from an imaging device that is determined based on a positional relationship between the orientation of the surface area and the imaging direction of each imaging device. The acquisition step acquires a real image that is captured and distributed by the imaging device via a network in a predetermined time unit,
The image processing method according to claim 1, wherein the display step sequentially displays the VR image generated by the image generation step on the display device in the predetermined time unit. On the computer,
An acquisition procedure for acquiring a real image of the indoor space photographed using the photographing device;
By comparing a virtual image obtained by viewing a virtual model corresponding to the indoor space constructed in the computer from a virtual viewpoint corresponding to the viewpoint of the imaging device with the actual image, a two-dimensional image in the actual image is obtained. A calculation procedure for calculating the correspondence between the coordinates and the three-dimensional coordinates of the virtual model;
A designation procedure for designating a surface area of the virtual model for mapping a texture in the real video;
A shape conversion procedure for converting a partial region in the actual video corresponding to the surface region into the same shape as the surface region based on the correspondence calculated by the calculation procedure;
A mapping procedure for mapping the partial area in the real image after shape conversion to the surface area;
An image generation procedure for generating a VR image when the virtual model after mapping the texture in the real video is viewed from an arbitrary viewpoint;
A display procedure for displaying the VR image on a display device.

Images