JP7241812B2 - Information visualization system, information visualization method, and program - Google Patents

Description

The present invention relates to an information visualization system, an information visualization method, and a program.

In each business such as building maintenance management and layout design, it is necessary to accurately grasp the situation of the target building. In each of the above operations, drawings of buildings and facility ledgers of facilities attached to buildings are mainly used. There are a wide variety of related drawings. In addition, the required information may be scattered, for example, if the custodian of the drawings and information is different from the owner of the building. Even in such cases, information may not be obtained without referring to a wide variety of drawings. On the other hand, information written on drawings is difficult to read and understand unless one has specialized knowledge, and is difficult to use.
In recent years, attempts have been made to use BIM (Building Information Modeling) to centrally manage various types of information related to buildings so that necessary information is not scattered and used in operations such as building management and design. It is BIM digitizes a building, treats it as a 3D model, and manages the 3D model in association with various pieces of information about the building. By using such BIM as an information visualization system, it becomes possible to easily use various types of information on buildings.

JP-A-9-244522

However, when creating a detailed 3D model using BIM or the like, the more detailed the 3D model is, the more work is required to create the 3D model. In particular, when BIM or the like is applied to an existing building, it may be judged that it is difficult to obtain an explicit effect that is worth the cost of making BIM or the like available.
The problem to be solved by the present invention is to provide an information visualization system, an information visualization method, and a program that make it possible to share the situation around a certain position by a simpler method.

(1) An information visualization system according to one aspect of the present invention includes a real image acquisition unit that acquires a surrounding image showing the situation of an object around an observation point in real space and adds it to collected information; is registered in the space model information, a desired space model is selected from the space model information, and a surrounding image selected from the collected information is applied to the selected space model. a display control unit that synthesizes and associates a virtual space image representing a virtual structure in the three-dimensional virtual space with the synthesized image and displays the situation of the object, wherein the collected information includes: Image information of the surrounding image, identification information that associates the space model with the surrounding image, and date and time information of the acquired surrounding image or information indicating whether the surrounding image is the latest information. Included is an information visualization system.
(2) In the information visualization system described above, the date and time information of the acquired surrounding image includes information on the shooting date and time of the surrounding image or information on the date and time when the surrounding image was stored.
(3) In the information visualization system described above, the display control unit estimates the position of a reference point in the three-dimensional virtual space corresponding to the observation point and a projection plane provided around the reference point. match the position.
(4) In the information visualization system described above, the display control unit aligns the position of the reference point with the position from which the projection plane is viewed, and stores the surrounding image and the Associate with the virtual space image.
(5) In the above information visualization system, the space model is formed so as to use at least one of a spherical surface of a sphere and a side surface of a cylinder arranged with reference to the reference point as a projection plane.
(6) In the above information visualization system, the display control unit superimposes and displays partial images extracted from both the surrounding image and the virtual space image based on predetermined extraction conditions.
(7) An information visualization system according to one aspect of the present invention includes a real image acquisition unit that acquires a surrounding image showing the situation of an object around an observation point in real space and adds it to collected information; is registered in the space model information, a desired space model is selected from the space model information, and a surrounding image selected from the collected information is applied to the selected space model. a display control unit for synthesizing, wherein the collected information includes: image information of the surrounding image; identification information that associates the space model with the surrounding image; date and time information of the acquired surrounding image; and information indicating whether the image is the latest information, and a virtual space image indicating a virtual structure in the three-dimensional virtual space is associated with the synthesized image to obtain the status of the object. It is an information visualization system that displays
(8) Further, in the above information visualization system, the display control unit causes the BIM device to generate a virtual space image showing a virtual structure in the three-dimensional virtual space corresponding to the synthesized image.
( 9 ) Further, the information visualization method of one aspect of the present invention includes the step of acquiring a surrounding image showing the situation of an object around an observation point in real space; a step of acquiring a surrounding image showing the situation of the space and adding it to the collected information; is selected, a surrounding image selected from the collected information is synthesized with the selected space model, and a virtual space image showing a virtual structure in the three-dimensional virtual space is associated with the synthesized image , and displaying the status of the object, wherein the collected information includes image information of the surrounding image, identification information that associates the spatial model with the surrounding image, and the date and time of the acquired surrounding image. or information indicating whether the surrounding image is the latest information .
( 10 ) In addition, a program according to one aspect of the present invention includes steps of acquiring a surrounding image showing a situation of an object around an observation point in real space and adding it to collected information ; information of a three-dimensional virtual space is registered in the space model information, a desired space model is selected from the space model information, and a surrounding image selected from the collected information is combined with the selected space model; a step of associating a virtual space image showing a virtual structure in the three-dimensional virtual space with the synthesized image and displaying the situation of the object , wherein the collecting The information includes image information of the surrounding image, identification information that associates the space model with the surrounding image, date and time information of the acquired surrounding image, or whether or not the surrounding image is the latest information. A program that contains information and

According to the present invention, it is possible to provide an information visualization system, an information visualization method, and a program that make it possible to share the situation around a certain position by a simpler method.

1 is a configuration diagram of an information visualization system according to an embodiment of the present invention; FIG. It is a figure which shows an example of the space model M which forms the projection plane S which concerns on embodiment. 1 is a configuration diagram of an imaging device 100 according to an embodiment; FIG. FIG. 10 is a diagram showing an image by the equirectangular projection method according to the embodiment; FIG. 4B illustrates an example of the image of FIG. 4A drawn inside the surface of a hollow cylinder, according to an embodiment; FIG. 4B is a diagram illustrating an example of a state in which the image of FIG. 4A is projected inside the surface of a hollow sphere according to the embodiment; 1 is a configuration diagram of a terminal device 200 according to an embodiment; FIG. It is a figure which shows an example which displays a three-dimensional virtual space by the 3D model based on the numerical data by BIM300 which concerns on embodiment. FIG. 10 is a diagram showing an example of display of a target range 10 in a three-dimensional virtual space as a comparative example; 2 is a hardware configuration diagram of an information provision support device 400 according to an embodiment; FIG. 1 is a functional configuration diagram of an information provision support device 400 according to an embodiment; FIG. It is a figure which shows an example of the content of the space model information 411 which concerns on embodiment. It is a figure which shows an example of the content of the positional information 413 to expect which concerns on embodiment. It is a figure which shows an example of the content of the collection information 412 which concerns on embodiment. FIG. 4 is a diagram showing the relationship between a reference point R, a projection plane S, and a position VP looking into the projection plane S in a three-dimensional virtual space according to the embodiment; It is a figure which shows the three-dimensional virtual space which has arrange|positioned the space model M which concerns on embodiment. 6 is a flowchart showing an example of information visualization processing according to the embodiment; FIG. 16 is a flowchart showing an example of information visualization processing according to the fourth embodiment; FIG. It is a figure which shows an example of the image which looks into a viewpoint direction from the observation point which concerns on embodiment. FIG. 11 is a diagram for explaining processing for synthesizing a plurality of images supplied from the imaging device 100 according to the second embodiment; FIG. FIG. 14 is a diagram for explaining processing for correcting distortion of an omnidirectional image according to the fourth embodiment; FIG. 14 is a diagram for explaining processing for correcting distortion of an omnidirectional image according to the fourth embodiment; FIG. 14 is a diagram for explaining processing for correcting distortion of an omnidirectional image according to the fourth embodiment;

Hereinafter, an information visualization system, an information visualization method, and a program according to embodiments of the present invention will be described with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a configuration diagram of an information visualization system according to an embodiment of the present invention.

[Information visualization system]
The information visualization system 1 includes an imaging device 100 , a terminal device 200 , a BIM 300 and an information provision support device 400 . The imaging device 100, terminal device 200, BIM 300, and information provision support device 400 are connected via a network NW.

First, an overview of the information visualization system 1 will be described. An information visualization system 1 visualizes information about a target range 10 of an existing building 2 . For example, the target range 10 includes part or all of the existing building 2 . Furthermore, the target range 10 includes facilities, fixtures, and the like provided in the building. The target range 10 is predetermined in its range. For example, positions in the target range 10 are identified using a Cartesian coordinate system (X, Y, Z). Several observation points (not shown) are provided within the target range 10 . For example, the position of the observation point P is determined at a position where the situation around the observation point P can be easily grasped. In the following description, the observation point P is simply referred to as an observation point.

The information visualization system 1 visualizes information about the surrounding situation on the basis of this observation point. The information visualized by the information visualization system 1 includes an image of the surroundings of the observation point, a 3D model converted into data by the BIM 300 (hereinafter simply referred to as a 3D model), a two-dimensional drawing (2D drawing), and the like. The surrounding image of the observation point is an image obtained by imaging the imaging device 100 at the observation point, or a set of a plurality of images obtained by imaging at the observation point. For example, the position of the observation point is determined based on the position of the optical system of imaging device 100 . Details of the surrounding image will be described later. The surrounding image of the observation point may be captured by the user U1 operating the imaging device 100, or may be captured by the imaging device 100 at a predetermined timing. The 3D model is information that indicates the structure of the building 2 including the target range 10, the equipment arranged in the building 2, fixtures, etc. as three-dimensional information.

The information visualization system 1 generates and displays an image based on one or both of the surrounding image of the observation point and the 3D model.

The surrounding image of the observation point contains accurate information indicating the situation in the target range 10 included in the imaging range. However, the above image information does not include information such as what the object (subject) shown in the image is, what it is like, and physical characteristics such as the size and surface condition of each individual subject. No information included.

On the other hand, a 3D model is a virtual structure formed in a three-dimensional virtual space, and physical features such as types, properties, feature amounts, sizes, and surface conditions of constituent elements that make up the virtual structure. Information can be included. By using the above 3D model, it is possible to use information registered correctly in advance. However, it is difficult to detect the presence or absence of unregistered information, the presence of erroneously registered information, and the like using only the 3D model. Also, it is not easy to supplement information that has not been registered.

Therefore, the information visualization system 1 associates the data of the image captured by the imaging device 100 with the 3D model of the BIM 300 to complement the respective information and enhance the convenience. The case of generating an image based on both the surrounding image of the observation point and the 3D model is illustrated.

For example, the information visualization system 1 associates a surrounding image with an image based on a 3D model (a virtual space image showing a virtual structure in a 3D virtual space). The information visualization system 1 generates a display image including an area common to the display area displaying the associated surrounding image and the display area displaying the image based on the 3D model.

In this embodiment, the above processing is roughly divided into a first processing for correcting the distortion of the surrounding image and a second processing for adjusting the position of the image based on the surrounding image and the 3D model.

The first processing includes, for example, (1-1) a method using the space model M and (1-2) a method using geometric arithmetic processing without using the space model M. A space model M is an example of a projection plane formed as a virtual structure in a three-dimensional virtual space. Details of the space model M and the first process will be described later.

The second processing includes, for example, (2-1) a method of rotating and adjusting an image along the spatial model M when using the spatial model M, and and (2-3) a method of adjusting so that the feature points in the surrounding image match the feature points in the image based on the 3D model.

Any combination of the methods of the first processing and the second processing can be selected except for the method of “(2-1)”. Hereinafter, in this embodiment, the above "(1-1) Method of using the space model M" and "(2-1) When using the space model M, rotate the image along the space model M An example of the combination with the "method for adjusting" will be described below. Note that other methods will be described later in a modified example, a second embodiment, or the like.

For example, the information visualization system 1 associates the surrounding image with the image based on the 3D model using the projection plane S common to them. The information visualization system 1 treats the projection plane S as a space model M, which is a virtual structure in a three-dimensional virtual space. A space model M in the three-dimensional virtual space is provided corresponding to an observation point in the real space. The space model M has a virtual plane on which the situation within the visible range from the observation point is projected. For example, the information visualization system 1 projects the situation within the visible range from the observation point onto the surface of the space model M so as to maintain the viewing direction from the observation point. The information visualization system 1 may use the space model M to manage information related to the situation around the observation point. The details will be described below.

For example, the information provision support device 400 collects image data captured by the imaging device 100 and supplies the BIM 300 with an all-around image based on the collected image data. The BIM 300 generates an image that combines the supplied omnidirectional image and information from the 3D model. The information provision support device 400 displays the image generated by the BIM 300 on the display unit of the terminal device 200 operated by the user U2 or the like in response to a request from the terminal device 200. FIG.

The information visualization system 1 uses the spatial model M in performing the process of generating an image obtained by synthesizing the omnidirectional image and the information of the 3D model as described above.

FIG. 2 is a diagram showing an example of the space model M forming the projection plane S according to the embodiment. For example, the space model M is formed as a hollow sphere. The information visualization system 1 projects a surrounding image (omnidirectional image) captured at the position of the observation point onto the inner surface of the sphere. By placing the viewpoint at the center C of the sphere, the direction in which the surrounding image projected onto the surface of the sphere is viewed from the center C of the sphere coincides with the direction in which the subject is viewed from the position of the observation point P (Fig. 1) in real space. can be made If the space model M is tilted with respect to the axial direction of the orthogonal coordinate system, the polar direction of the spatial model M should be adjusted to coincide with the axial direction of the orthogonal coordinate system. For example, the information provision support device 400 adjusts the polar direction of the space model M so as to match the Z-axis direction of the orthogonal coordinate system as shown in FIG. For example, after adjusting the inclination of the space model M in this way, the surrounding image may be projected.

In the above description, the surrounding image at the observation point is shown as an example, but the type of information associated with the space model M is not limited to this. The information visualization system 1 may associate various types of information with the spatial model M in addition to the surrounding image. For example, the information visualization system 1 forms the space model M as a hollow sphere, and associates various types of information such as collected information with the surface of the sphere. The above collected information is information collected to share the situation around the observation point. The collected information includes image information obtained by imaging the surroundings of the observation point, information of a type different from the image information, and the like. Information different in type from the image information may include, for example, text (character information) indicating the circumstances around the observation point.

Each device related to the information visualization system 1 will be described below.

[Imaging device]
An example of the configuration of the imaging device 100 will be described. FIG. 3 is a configuration diagram of the imaging device 100 according to the embodiment. The imaging device 100 captures an image of its surroundings and outputs the obtained surrounding image.

The imaging device 100 includes an optical system 111 , an optical system 112 , an imaging section 121 , an imaging section 122 , a signal processing section 130 , a reception section 140 , a control section 150 and an output section 160 .

The optical system 111 and the optical system 112 each form a fish-eye lens, and are capable of photographing at an angle of view of 180 degrees or more. The optical system 111 and the optical system 112 are supported by the housing of the imaging device 100 or the like so that the optical axis of the optical system 111 and the optical axis of the optical system 112 are substantially parallel. The optical axis direction of the optical system 111 and the optical axis direction of the optical system 112 are directed in opposite directions.

The imaging unit 121 is provided so as to be paired with the optical system 111 and images an object (subject) within the range of the angle of view of the optical system 111 . The imaging unit 122 is provided so as to be paired with the optical system 112 and images an object (subject) within the range of the angle of view of the optical system 112 .

The signal processing unit 130 synthesizes the image obtained by imaging by the imaging unit 121 and the image obtained by imaging by the imaging unit 122 to obtain a Generate a composite image, such as an image or an all-around image.

The accepting unit 140 accepts a user's operation of instructing imaging.

The control unit 150 causes the image capturing unit 121 and the image capturing unit 122 to capture an image based on the imaging instruction received by the receiving unit 140 . The control unit 150 controls each imaging unit such that the imaging unit 121 and the imaging unit 122 perform imaging in synchronization with each other. The control unit 150 causes the signal processing unit 130 to generate a composite image based on the results of imaging by the imaging unit 121 and the imaging unit 122 . The control unit 150 causes the output unit 160 to output the composite image generated by the signal processing unit 130 .

For example, the surrounding image output by the imaging device 100 as a composite image preferably includes an omnidirectional image such as an equirectangular projection image or an omnidirectional image. FIG. 4A is a diagram showing an image by equirectangular projection according to the embodiment. The equirectangular projection is a projection method in which the surface of a sphere as shown in FIG. 2 is developed and projected into a rectangle. The latitude and longitude of the sphere are scaled so that they intersect at right angles and at equal intervals within the rectangle. FIG. 4B is a diagram illustrating an example of the image of FIG. 4A drawn inside the surface of a hollow cylinder.

In the embodiment described below, the equirectangular projection image shown in FIG. 4 is projected inside the surface of the sphere as the space model M shown in FIG. FIG. 5 is a diagram showing an example of a state in which the image of FIG. 4A is projected inside the surface of a hollow sphere. This FIG. 5 shows the far side hemisphere in the depth direction of the sphere and omits drawing of the near side hemisphere. As shown in FIG. 5, the image projected inside the surface of the sphere becomes an omnidirectional image when the entire sphere is viewed from the inside of the sphere. That is, by projecting an image captured by the imaging device 100 onto the inside of the surface of the space model M formed as the surface of a hollow sphere, image information indicating the surrounding state of the observation point is arranged on the surface of the sphere. can be obtained. Note that the imaging apparatus 100 outputs a part or all of at least one of the equirectangular projection image and the omnidirectional image. With such an imaging device 100, it is possible to collectively photograph the entire surroundings with one operation, and to photograph the surroundings without omission.

In other words, the imaging apparatus 100 saves the user the trouble of synthesizing the images, and can improve the quality of continuity of the images in the spatial direction. In addition, the imaging device 100 can shorten the time required for imaging by imaging the entire surroundings with one operation. In addition, with the image pickup apparatus 100 as described above, there is no possibility of partial loss due to omission of photographing or the like. Due to these, the work efficiency of the user is enhanced.

In the following description of the embodiments, it is assumed that the imaging device 100 outputs an equirectangular projection image as a surrounding image of the object. The imaging device 100 transmits the surrounding image of the target to the information provision support device 400 .

[Terminal device]
FIG. 6 is a configuration diagram of the terminal device 200 according to the embodiment. The terminal device 200 includes, for example, a CPU 200A, a RAM (Random Access Memory) 200B, a nonvolatile storage device 200C, a portable storage medium drive device 200D, an input/output device 200E, and a communication interface 200F. The terminal device 200 may include any form of processor instead of the CPU 200A, or may omit some of the components shown in FIG.

The CPU 200A expands the program stored in the nonvolatile storage device 200C or the program stored in the portable storage medium attached to the portable storage medium drive device 200D to the RAM 200B and executes the program as described below. Various processing is performed. RAM 200B is used as a working area by CPU 200A. The nonvolatile storage device 200C is, for example, an HDD, flash memory, ROM (Read Only Memory), or the like. A portable storage medium such as a DVD (Digital Versatile Disc), a CD (Compact Disc), an SD card, or the like is attached to the portable storage medium drive device 200D. The input/output device 200E includes, for example, a keyboard, mouse, touch panel, display device, and the like. The communication interface 200</b>F is connected to the network NW and controls communication in the terminal device 200 .

A functional configuration of the terminal device 200 according to the embodiment will be described with reference to FIG. The terminal device 200 includes a reception processing section 211 , a data acquisition section 212 and a display control section 213 . These functional units are implemented by, for example, CPU 200A executing a program. In addition, these functional units may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), etc. It may be realized by

The reception processing unit 211 receives a user's operation detected by the input/output device 200E. For example, the reception processing unit 211 detects an operation of selecting a button or the like on an image displayed by the input/output device 200E and receives it as a user operation. The reception processing unit 211 transmits a request for sharing information to the information provision support apparatus 400 based on the detected user's operation.

The data acquisition unit 212 acquires information transmitted from the information provision support device 400 . The information acquired from the information provision support device 400 includes acquired information corresponding to a specific observation point.

The display control unit 213 generates an image to be displayed on the display unit of the input/output device 200E based on the information transmitted from the information provision support device 400. FIG. For example, the display control unit 213 generates a browsing image based on acquired information corresponding to a specific observation point, and causes the display unit to display it.

Note that the terminal device 200 may realize part or all of the reception processing unit 211, the data acquisition unit 212, and the display control unit 213 using a browser applied to a web system. The terminal device 200 may be permitted to be used only by a predetermined user.

[BIM]
The BIM 300 is a computer that quantifies and manages information about the building 2 including the target range. For example, as shown in FIG. 1, the BIM 300 includes a storage section 310 and a control section 320. FIG.

The storage unit 310 stores data obtained by digitizing the building 2 as a 3D model, various drawings related to the building 2, attribute information for identifying equipment attached to the building 2, history information of maintenance and inspection of the building 2 and attached equipment, and the like. is stored. For example, data obtained by digitizing the building 2 as a 3D model includes data such as a structural drawing of the building 2 . Various drawings related to the building 2 include part or all of various drawings such as a structural drawing of the building 2, a plan view of each floor, an equipment layout diagram, a piping diagram, and a power system diagram.

Control unit 320 receives information supplied from the outside and writes the information in storage unit 310 by executing a program. Also, the control unit 320 generates a 3D model according to an instruction from the outside, and stores the generated 3D model in the storage unit 310 . Details of the processing by the control unit 320 will be described later.

FIG. 7 is a diagram showing an example of displaying a three-dimensional virtual space with a 3D model based on numerical data by BIM300. FIG. 7 shows a bird's-eye view of the three-dimensional virtual space corresponding to the target range 10. As shown in FIG. A 3D model exists in the three-dimensional virtual space shown in FIG. It can be seen that the 3D model shows the outline of the building 2 . In this 3D model, data of a target range 10 for the building 2 are registered. As shown in FIG. 7, it is not essential to register information indicating the configuration of windows, devices, etc. provided on the wall surface as a 3D model. It should be noted that the diagram shown in FIG. 7 is displayed with the ceiling portion removed. In addition to the building 2, the figure also shows a layout plan for a new device.

FIG. 8 is a diagram showing an example of display of a virtual space image as a comparative example. The virtual space image shown in FIG. 8 is a perspective view showing the target range 10 in the three-dimensional virtual space. If the target range shown in FIG. 7 is converted into a perspective view as it is, a virtual space image as shown in FIG. 8 is obtained. As shown in FIG. 8, it is not possible to identify the current state of the target range 10 as shown in FIG. 1 from only the simple data registered in the BIM 300. FIG.

Therefore, in this embodiment, by using an information provision support device 400 described below, it is possible to share the situation around a specific position by a simple method.

[Information provision support device]
FIG. 9 is a hardware configuration diagram of the information provision support device 400 according to the embodiment. The information provision support device 400 includes, for example, a CPU 400A, a RAM 400B, a nonvolatile storage device 400C, a portable storage medium drive device 400D, an input/output device 400E, and a communication interface 400F. The information provision support device 400 may include any form of processor instead of the CPU 400A, or may omit some of the components shown in FIG.

The CPU 400A expands the program stored in the nonvolatile storage device 400C or the program stored in the portable storage medium attached to the portable storage medium drive device 400D to the RAM 400B and executes the program as described below. Various processing is performed. RAM 400B is used as a working area by CPU 400A. The nonvolatile storage device 400C is, for example, an HDD, flash memory, ROM, or the like. A portable storage medium such as a DVD, a CD, or an SD card is installed in the portable storage medium drive device 400D. The input/output device 400E includes, for example, a keyboard, mouse, touch panel, and display device. The communication interface 400</b>F is connected to the network NW and controls communication in the information provision support device 400 . Note that the communication interface 400F may have a function as a web server applied to the web system.

FIG. 10 is a functional configuration diagram of the information provision support device 400 according to the embodiment. Information provision support device 400 includes storage unit 410 and control unit 420 .

The storage unit 410 is a storage area provided in the RAM 400B or the nonvolatile storage device 400C. The storage unit 410 may be realized by an external storage device such as a NAS device that can be accessed from the information provision support device 400 .

The storage unit 410 stores information such as space model information 411 , collection information 412 , expected position information 413 , and integration information 414 .

The spatial model information 411 includes information on the spatial model M provided for each observation point. FIG. 11A is a diagram showing an example of the contents of the spatial model information 411 according to the embodiment. For example, one spatial model M is assigned in principle to each observation point. The spatial model information 411 includes information about the spatial model M. FIG. For example, the space model information 411 includes information such as space model ID, area ID, position in real space, position in virtual space, radius and direction. The spatial model information 411 includes information for each spatial model.

The space model ID is identification information for identifying the space model M assigned to the observation point. For example, the spatial model ID may be an observation point ID (observation point identification information) for identifying the observation point, or may be the coordinate value of the observation point, or a value generated based on the coordinate value. good. For example, the spatial model M is associated with one or more surrounding images. The spatial model ID makes it possible to identify one or more surrounding images associated with the spatial model M using it as a key. A surrounding image selected from them becomes an image seen from the position of the observation point. The spatial model ID can be used to manage the spatial model M, and the surrounding images associated with a specific spatial model M can be referenced using the spatial model ID. The surrounding images corresponding to the space model M may be associated 1:1, or one of a plurality of surrounding images may be selected according to a defined condition.

The area ID is identification information for identifying an area including the space model M. FIG. For example, the area ID is information such as the identification number of each divided area when the building is divided into multiple areas according to desired rules, such as the number of floors in the building, zones on the same floor, tenants, rooms, etc. including.

The position on the real space indicates the position of the observation point on the coordinate system determined for each area. For example, the coordinate system determined for each area is an orthogonal coordinate system or a polar coordinate system, and the example shown in the figure is the case of the orthogonal coordinate system. The position on the virtual space indicates the position of the observation point on the coordinate system of the virtual space. For example, the coordinate system of the virtual space is an orthogonal coordinate system or a polar coordinate system, and the example shown in the figure is the case of the orthogonal coordinate system. An observation point on the virtual space is determined for each space model M. FIG. In this embodiment, the coordinate system of the real space and the coordinate system of the virtual space corresponding to the real space are made to correspond in order to facilitate understanding of the explanation. For example, it is assumed that the origin of the coordinate system of the real space coincides with the origin of the coordinate system of the virtual space corresponding to the real space. In this case, the position of the observation point in the real space and the position of the reference point of the space model M can be matched. The above is just an example, and does not limit that the origin of the coordinate system of the real space and the origin of the coordinate system of the virtual space corresponding to the real space are different. For example, the origin of the coordinate system in the real space may be determined based on the interior surface or the like, and the origin of the coordinate system in the virtual space may be determined based on the center of the wall or the like.
The radius is information indicating the size of the space model M. FIG. For example, if the radius (R1) is 2 m and the height of the observation point from the floor surface is 1.5 m, the lower part of the space model M is arranged in a state of protruding under the floor. Alternatively, if the radius (R1) is 0.5 m and the height of the observation point from the floor surface is 1.5 m, the space model will be arranged in a floating state on the floor. When the space model M is included in the range in which the virtual space is displayed, a sphere with a radius corresponding to the space model M is displayed at that location. The radius may be set to the same value for all the spatial models M, or may be set to different values according to usage, purpose, attributes of registrants, and the like.

The direction indicates the amount of rotation of the coordinate system of the space model M with respect to the coordinate system of the virtual space. For example, the direction value indicates the polar rotation angle of the space model M around the virtual space coordinate X axis as α, and the polar rotation angle of the space model M around the virtual space coordinate Y axis as β. , the angle of rotation in the polar direction of the space model M around the coordinate Z axis of the virtual space is denoted by γ. The spatial model M shown in FIG. 2 described above is the result of adjusting the above α, β, and γ to desired values.

The projected position information 413 includes information indicating the position VP where the projection plane S of the space model M is projected. FIG. 11B is a diagram showing an example of the contents of the expected position information 413 according to the embodiment. For example, the spatial model M is assigned in principle one prospective position VP. For example, the expected position information 413 includes information such as a space model ID, a position in the virtual space, a direction, and an angle of view. The position on the virtual space indicates the position where the position VP on which the projection plane S of the space model M is viewed is indicated by the coordinate system of the virtual space coordinates. The direction indicates the direction of the viewpoint directed from the position VP on which the projection plane S of the space model M is viewed. The angle of view indicates the visible range based on the direction of the viewpoint. If the value of the angle of view is increased, the image will be taken with a wide-angle lens, and if it is decreased, the image will be taken with a telephoto lens.

The collected information 412 will be described later.

Returning to FIG. 10 described above, the control unit 420 will be described. Control unit 420 includes designation information acquisition unit 421 , surrounding image acquisition unit 422 , display control unit 423 , position derivation unit 424 , integration processing control unit 426 , and output processing unit 427 . The specified information acquisition unit 421 and the surrounding image acquisition unit 422 are examples of acquisition units.

The designated information acquisition unit 421 acquires from the terminal device 200 the detection result of the user's operation in the terminal device 200 . The specified information acquisition unit 421 adds the information acquired from the terminal device 200 to the collected information 412 in the storage unit 410 .

The ambient image acquisition unit 422 acquires the ambient image of the observation point in the real space, a drawing showing the state of the real space, and the like, and similarly adds them to the collected information 412 .

Here, an example of the contents of the collected information 412 will be described. FIG. 12 is a diagram showing an example of the content of collected information 412 according to the embodiment. The collected information 412 includes image information and the like collected for each space model.

For example, the collected information 412 includes space model ID (observation point ID), object, position information, valid flag, data type, data identification ID (file name), date and time (time), angle of view, management information, reference direction ( θ0φ0), and includes information such as a user ID.

A spatial model ID corresponds to the spatial model ID of the spatial model information 411 .
The object indicates an object arranged in the virtual space, a structure in the virtual space, or an object arranged within the target range in the real space. This object is defined as having a two-dimensional or three-dimensional shape. In the case of an object arranged in the virtual space and a structure in the virtual space, the object is projected onto the space model from the position in the virtual space and becomes visible from the observation point of the space model M. Alternatively, in the case of an object placed within the target range of the real space, the image of the target range of the real space is projected onto the space model and becomes visible from the observation point of the space model M.

The position information is identification information for identifying the position of a point on the space model indicated by a polar coordinate system with the observation point corresponding to the space model ID as the origin. The position of the above target is
For example, it may be the position of the point representing the above "object", the position of the point representing the range of the above "object", the position of the center of gravity of the "object", and the like. Further, for example, a predetermined range from the position of the target may be determined as the "target portion". A predetermined range from the target position may be defined as a circle with a predetermined radius centered on the representative point defined above.

The validity flag is information indicating validity as collected information. For example, when the collected information cannot be used, it is invalidated, and when it is available, it is validated. Situations in which the collected information is unavailable include cases such as when the data is determined to be damaged, when new information is collected and the information becomes obsolete, and the like.

The data type is information indicating the type of data indicated by an image, text, or the like. For example, the types of data include a "whole image (image A)" that shows the entire circumference in a single file, an "uncombined image (image B)" that shows the entire circumference by combining multiple images, and an image of each facility. "Individual images (image C)" indicating individual states, "drawings" indicating information such as notes posted by users and facilities, and "texts" indicating items to be handed over are included. For example, the information on equipment may include information on the location of the equipment in real space or virtual space. The type of data may be identified by using the extension of the file name.

The data identification ID is identification information for identifying data treated as collected information, such as a file name when collected information is treated as a file.

The date and time is information indicating the date and time related to the collected information. For example, when the data type is an image ("whole image" or "uncombined image"), the date and time are the date and time when the image was captured, the date and time when the image information was stored, and the like.

The angle of view is applied when the type of information is the omnidirectional image (image A), and corresponds to information for designating the range of the image. When allocating an image to the space model M, the ambient image acquisition unit 422 adjusts the size of the original image specified by the data identification ID so that the image covers the space model M according to the range of the image. For example, the omnidirectional image includes an image that can be circulated around and an image that cannot be circulated around. When the ambient image acquisition unit 42 allocates an image to the space model M, the image may include only a portion of the omnidirectional image and fill only a portion of the projection plane of the space model M. In this case, the surrounding image acquisition unit 422 adjusts the range of the original image specified by the data identification ID to the range determined based on this angle of view. By this adjustment, the image can be allocated to the space model M without the image being distorted. For example, the angle of view may be indicated by the angle in the horizontal direction (θ direction), the angle of attack (depression angle) direction (φ direction (FIG. 2)), or a combination thereof. Even in the case of the omnidirectional image, if the obtained image does not satisfy the vertical direction, the angle of view includes at least one of the upper limit and the lower limit of the above range in the φ direction (FIG. 2). There may be.

The reference direction (θ0φ0) is applied when the type of information is the omnidirectional image (image A). For example, when allocating an image to the space model M, the ambient image acquisition unit 422 adjusts the position of the starting point of the omnidirectional image (image A) based on the reference direction θ0φ0. When adjusting based on the reference direction θ0, the image direction is adjusted by rotating the image around the vertical z-axis (FIG. 2) so that the image direction matches the direction in the three-dimensional virtual space. When adjusting based on the reference direction φ0, the image direction is adjusted by moving the image in the φ (FIG. 2) direction so that the image direction matches the direction in the three-dimensional virtual space. The reference direction (θ0φ0) is a correction angle when adjustment is required, and its value is set to 0 when correction by adjustment is not performed.

The management information is information such as the status of the contents of the collected information. For example, if the type of collected information is image information, it includes a flag indicating whether or not the information is the latest information, a flag indicating whether or not update is necessary, and the like. If the type of collected information is text, it includes the progress of countermeasures according to the content. The progress of the countermeasures may indicate various stages such as during consideration of the countermeasures, after the countermeasures are decided, during preparation for countermeasures, during countermeasures, and after countermeasures are completed.
The user ID is identification information of the user who has executed the process of writing information in the collected information 412 .

Some examples of data are described. For example, the specified information acquisition unit 421 obtains the data of the omnidirectional image (image A) referenced with the space model ID of "1001" and the data identification ID of "FP001" from the imaging device 100 based on the operation of the user U1. It acquires, sets the user ID to “U1”, and adds each of the above data to the collected information 412 .

For example, the ambient image acquisition unit 422 acquires the ambient image of the observation point in the real space from the imaging device 100 or the like, and adds it to the collected information 412 . As shown in FIG. 12 , the surrounding image acquisition unit 422 acquires drawing data referred to with a spatial model ID of “1001” and a data identification ID of “FD002” from an external device such as an operation management system, Add each of the above data to the collected information 412 .

Further, for example, the designation information acquisition unit 421 acquires the data of the image C referred to with the space model ID of "1001" and the data identification ID of "FP011" from the imaging device 100 based on the operation of the user U1. , the user ID is set to “U1”, and each of the above data is added to the collected information 412 .

It should be noted that the surrounding image referred to as "FP101" is not the omnidirectional image (image A), but the "uncombined image (image B)" that becomes the omnidirectional image by combining. For example, the ambient image acquisition unit 422 combines the uncombined image (image B) identified as “FP101” to generate the omnidirectional image (image A) identified as FP102. The ambient image acquisition unit 422 adds the generated omnidirectional image (image A) of the FP 102 to the collected information 412 .

In addition to the image A, image B, and image C described above, information whose information type is different from that of the image is added to the collected information 412 in the same manner as the image described above.

As described above, the specified information acquisition unit 421 and the surrounding image acquisition unit 422 are examples of the actual image acquisition unit. The designated information acquisition unit 421 and the surrounding image acquisition unit 422 acquire surrounding images (such as omnidirectional images) that indicate the surroundings of the observation point in real space.

The display control unit 423 controls the position of the reference point R in the three-dimensional virtual space corresponding to the observation point in the real space corresponding to the target range 10, and the position VP looking into the projection plane S provided around the reference point R. The positions of the reference point R and the position VP looking into the projection plane S in the three-dimensional virtual space are adjusted so that both are within a predetermined range. FIG. 13 is a diagram showing the relationship between the reference point R, the projection plane S, and the position VP from which the projection plane S is viewed in the three-dimensional virtual space according to the embodiment. The position of reference point R is determined based on spatial model information 411 . A position VP from which projection plane S is viewed is determined based on position information 413 to be viewed. For example, the position of the reference point R and the position VP from which the projection plane S is expected are determined based on the user's operation of the terminal device 100, and the display control unit 423 updates the space model information 411 and the expected position information 413. be done.

The three-dimensional virtual space corresponding to the observation point is a virtual three-dimensional space for arranging the "space model M (FIG. 2)" corresponding to the observation point.
The projection plane S provided around the reference point R includes at least a part of the inner surface of the surface of the hollow sphere or the inner surface of the side surface of the hollow cylinder arranged with reference to the reference point R. . In the following description, it is assumed that the projection plane S provided around the reference point R forms the inner surface of the hollow sphere.

FIG. 14 is a diagram showing a three-dimensional virtual space in which the space model M according to the embodiment is arranged. A projection plane S provided around the reference point R is part or all of the surface of a sphere with the reference point R as its center. For example, the projection plane S provided around the reference point R is the inner surface of the spatial model M corresponding to the reference point R. In the following description, the inner surface of the spatial model M may be simply referred to as a projection plane S.

Further, the predetermined range is a minute range corresponding to a range that does not affect the accuracy of the generated display image. For example, the display control unit 423 may match the position of the reference point R with the position VP from which the projection plane S is viewed. In this case, the display control unit 423 may refer to the space model information 411 to determine the position VP of the reference point R and the projection plane S at positions corresponding to the positions of the observation points in the real space. .

The display control unit 423 associates the surrounding image (see FIG. 5) projected onto the projection plane S with the virtual space image (see FIG. 8) showing the virtual structure in the three-dimensional virtual space. The display control unit 423 causes the BIM 300 to display a display image including an area common to the display area for displaying the surrounding image and the display area for displaying the virtual space image in the associated surrounding image and virtual space image. Control BIM 300 to generate. Note that the above-mentioned common area is, for example, a part of the target range 10 and includes an object to be examined, verified, etc. based on the display image.

The display control unit 423 determines the position based on the position of the reference point R or the position VP from which the projection plane S is viewed, and displays the virtual structure viewed from the determined position in the virtual space image based on the virtual structure. The BIM 300 is controlled so that the display image is generated by the BIM 300 . The display control unit 423 adjusts in advance three relative positional relationships among the space model M, the surrounding image, and the virtual space image. The adjustment method includes the following. For example, as a first adjustment method, the display control unit 423 adjusts the position of the surrounding image with respect to the space model M, and then adjusts the direction of the space model M with respect to the three-dimensional virtual space. Alternatively, as a second adjustment method, the display control unit 423 adjusts the position with respect to the space model M independently of the surrounding image and the virtual space image. As a third adjustment method, the display control unit 423 first adjusts the relative positions of the surrounding image and the virtual space image, and then adjusts the position with respect to the space model M collectively. By performing any of these, the display control unit 423 adjusts the direction of the representative point of the surrounding image in at least the three-dimensional virtual space.

After adjusting the three relative positional relationships among the space model M, the surrounding image, and the virtual space image, the display control unit 423 adjusts the range extracted from the surrounding image and the range extracted from the virtual space image. The BIM 300 is controlled so that the BIM 300 generates a display image including the range to be displayed. The display control unit 423 converts at least one of the surrounding image and the virtual space image into a transparent image, and arranges the transparent image at a position overlapping the other image. Under such control, the BIM 300 generates a display image in which the other image can be viewed through the transparent image.

The output processing unit 427 outputs desired information for the request to the terminal device 200 in response to the request from the terminal device 200 .

[Regarding processing in BIM300]
The BIM 300 performs processing such as data registration, drawing, drawing, editing, and display of a 3D model in a 3D virtual space. For example, the BIM 300 builds a 3D model in which structures such as the building 2 are digitized according to user's operations. The BIM 300 converts 3D model data (hereinafter referred to as 3D model data) into data in a specific file format, and can share that data with computers that run other BIM software, CAD software, viewer software, and the like. .

The BIM 300 in this embodiment acquires image data captured by the imaging device 100 or the like in addition to numerical data registered as numerical information, and uses the data for processing such as drawing. The BIM 300 of the embodiment uses, for example, data of images such as omnidirectional images and equirectangular projections. The BIM 300 superimposes a corresponding omnidirectional image and an image based on 3D model data on the same viewpoint. For example, the omnidirectional image data includes the current wall usage status. On the other hand, it is assumed that the 3D model data does not contain detailed information about the usage status of the wall surface. The BIM 300 combines "current" information acquired as image information with information such as "numerical data" and "geometric model data" based on 3D model data, thereby supplementing information that is lacking in each data. The user can intuitively recognize the situation from the image obtained as a result, and can obtain objective data based on numerical data.

When the BIM 300 draws an image based on each of the omnidirectional image data and the 3D model data, it aligns the perspective (perspective drawing method) with either the three-point perspective method or the two-point perspective method. The BIM 300 aligns the perspectives of the omnidirectional image data and the 3D model data to synthesize an image in a three-dimensional virtual space.

(Image information displayed by BIM300)
By the way, data (3D model data) handled by the BIM 300 in a three-dimensional virtual space is in a three-dimensional format. On the other hand, the omnidirectional image generated by the imaging device 100 or the like is information in a two-dimensional format. It is easy to display an omnidirectional image on a surface, but it cannot be used as it is as three-dimensional information. As described above, the data structures of the omnidirectional image and the 3D model data are different, and it is difficult to combine them as they are.

Therefore, the BIM 300 of the present embodiment maps 2D format image data onto the surface of an object in a 3D virtual space, thereby making it possible to use data in different formats.
An object in the three-dimensional virtual space shown below as an example is a hollow sphere (space model M) and corresponds to the space model M described above. BIM 300 maps image data in two-dimensional format inside the surface of this sphere.

(Range of image displayed by BIM300)
The wider the display range of the BIM 300, the more the image is distorted. If there is a difference from an image that is normally viewed (an image with a standard angle of view), it will look unnatural. On the other hand, if the display range is standardized, the distortion produced in the image is not conspicuous, and spatial recognition is facilitated.

The BIM 300 maps both the omnidirectional image and the image based on the 3D model data inside the surface of the virtual sphere, and draws based on the respective data. BIM 300 selects a portion of the virtual spherical surface and displays the selected area as a two-dimensional image (2D view). As described above, the image mapped inside the surface of the sphere (projection plane S) is cut out and displayed as a two-dimensional image. , the occurrence of distortion due to the projection plane S being a curved surface is reduced.

Under the control of the display control unit 423, the BIM 300 detects, for example, a range common to both the range extracted from the omnidirectional image (surrounding image) and the range extracted from the image (virtual space image) based on the 3D model data. An included display image may be generated.

Note that the BIM 300 may convert at least one of the surrounding image and the virtual space image into a transparent image under the control of the display control unit 423 . For example, the BIM 300 places the transparent image at a position overlapping the other image. Thereby, the BIM 300 generates a display image that allows the other image to be visible through the transparent image.

FIG. 15A is a flowchart illustrating an example of information visualization processing according to the embodiment; An image of the site is captured by the imaging device 100 .

First, the ambient image acquisition unit 422 acquires an image from the imaging device 100 (S11) and adds it to the collected information 412. FIG. The surrounding image acquisition unit 422 determines whether or not the image acquired in S11 is an image that requires compositing of photo data, based on the data type of the collected information 412 (S12). An image that requires photographic data synthesis is an image that cannot be treated as an all-around image without being processed. If the image requires photographic data synthesis (S12: YES), the ambient image acquisition unit 422 combines the multiple acquired images to obtain an all-surrounding image based on the observation point where the image was captured. is generated (S13), and the generated image is added to the collected information 412 as a type of an all-surrounding image.

If the image does not require photographic data synthesis (S12: NO), or after the processing in S13 is finished, the designation information acquisition unit 421 acquires the 3D model data of the target range 10 based on the information from the BIM 300. (S14). If there is no 3D model data for the target range 10 (S14: NO), the designation information acquisition unit 421 controls to generate 3D model data (S15). For example, the specified information acquisition unit 421 controls the BIM 300 to generate 3D model data.

In some BIMs of the comparative example, the existing situation cannot be expressed unless various data are prepared as 3D model data. On the other hand, the BIM 300 of the present embodiment only needs to include, as 3D model data, minimum information on existing elements (existing objects) related to the purpose of study. For example, for the purpose of arranging information about the interior of a room or the like, information indicating the shape of the room may be input. For example, the shape of the room can be shown by showing the inner wall surface, the ceiling surface, and the floor surface in a two-dimensional drawing (2D drawing). Information such as windows and equipment provided in the room is not essential information. These pieces of information are complemented by a process of superimposing images, which will be described later.

If there is 3D model data of the target range 10 (S14: YES), or after the processing in S15 is finished, the specified information acquisition unit 421 acquires the position of the observation point from the imaging device 100 or the terminal device 200 (S16 ).

Next, the designation information acquisition unit 421 determines whether or not there is a desired space model M among the space models M registered in the space model information 411 (S17). When it is determined that the desired space model M does not exist in the space model information 411 (S17: NO), the designation information acquisition unit 421 adds the space model M to the observation points of the space model information 411 (S18). The specified information acquisition unit 421 controls the BIM 300 to add the space model M to the 3D model data by the BIM 300 .

When it is determined that the desired space model M is present in the space model information 411 (S17: YES), or after finishing the processing in S18, the display control unit 423 processes the peripheral image (S19). For example, the central portion of the omnidirectional image (equirectangular image) output from the camera may not always match the front of the camera. In the case of an omnidirectional image (equirectangular image) whose starting point is the front direction of the camera, the display control unit 423 corrects the position of the starting point on the data. The display control unit 423 reflects the determined correction amount in the “reference direction” of the collected information 412 .

Further, when the range and size of the image are not suitable for use as BIM 300 data, the display control unit 423 converts a part of the image according to a predetermined standard so that it can be used as BIM 300 data. is extracted (trimmed), and processing such as adjusting the size of the image is applied to the peripheral image. The display control unit 423 reflects the adjustment amount determined above in the “angle of view” of the collected information 412 .

Next, the display control unit 423 selects a desired space model M from among the space models M registered in the space model information 411, and adds the peripheral image selected from the collected information 412 to the selected space model M. (Surrounding image) are synthesized (S20). Note that when there is a relative positional deviation between the space model M, the surrounding image, and the virtual space image, the display control unit 423 determines the positional relationship of each other based on the space model. can be adjusted by

The display control unit 423 acquires from the terminal device 200 the direction of the viewpoint (viewpoint direction) that determines the direction of the image to be displayed on the terminal device 200 (S21).
The display control unit 423 causes the terminal device 200 to display an image looking in the viewpoint direction from the observation point (S22). FIG. 16 is a diagram illustrating an example of an image viewed in a viewpoint direction from an observation point according to the embodiment; For example, as shown in the figure, it becomes clear that a new device scheduled to be installed within the target range 10 interferes with an existing device.

The display control unit 423 determines whether or not to display another direction with a different viewpoint direction according to an instruction from the terminal device 200 (S23), and when it determines to display another direction (S23: YES). , S21 and subsequent steps are repeated. If it is determined not to display another direction by the determination processing in S23 (S23: NO), the illustrated series of processing ends.

Through the processing described above, the information provision support device 400 makes it possible to share the situation around the position of a certain observation point by a simpler method. The information provision support apparatus 400 may be applied to various business uses to share desired information based on the above images.

The projection plane S set as the space model M is provided so as to intersect a virtual straight line connecting the reference point R from which the image to be displayed is viewed and the virtual structure provided in the three-dimensional virtual space. For example, the position of the projection plane S is between the reference point R and the virtual structure.

(Modification 1 of the first embodiment)
Modification 1 of the first embodiment will be described. In the first embodiment, at least one of the surrounding image and the virtual space image is converted into a transmissive image, and the other image is displayed through the transmissive image. Therefore, in this modified example, display without transmission will be described.

The display control unit 423 extracts a part of at least one of the surrounding image and the virtual space image based on a predetermined extraction condition, and superimposes the extracted partial image on the other image. For example, assume that the BIM 300 registers information about indoor wiring and piping installed in areas hidden by walls, floors, ceilings, and the like. When the display control unit 423 supplies the surrounding image to the BIM 300, the BIM 300 superimposes the partial image of the indoor wiring, piping, etc. on the surrounding image based on the supplied surrounding image.
Alternatively, it is assumed that the BIM 300 is registered with image information including objects such as indoor wiring and piping, and mask information indicating a range other than the above objects. When the display control unit 423 supplies the surrounding image to the BIM 300, the BIM 300 uses the supplied surrounding image as an underlay, and out of the image information including the target object, the area other than the range specified by the mask information is displayed on the surrounding image. placed on top of each other.

According to this modification, it is possible to generate a display image in which the partial image extracted by the BIM 300 is superimposed on the surrounding image. Note that the information provision support apparatus 400 may control the BIM 300 to generate the above display image.

Note that the information provision support device 400 receives control from the terminal device 200, converts at least one of the surrounding image and the virtual space image into a transparent image, and displays the other image through the transparent image. 1 display mode and the second display mode in which the image is displayed without being transparent, as exemplified in the present modification, may be switched, or both may be combined for display.

(Modification 2 of the first embodiment)
Modification 2 of the first embodiment will be described. In the first embodiment, the imaging device 100 captures an image based on equirectangular projection or a omnidirectional image as a surrounding image, and the information provision support device 400 does not require a synthesis process for generating the surrounding image. explained. Instead of this, in the present embodiment, the information provision support device 400 synthesizes a plurality of images supplied from the imaging device 100 to perform synthesis processing to generate a surrounding image. The description will focus on differences from the first embodiment.

The imaging device 100 of the embodiment uses a super-wide-angle lens, a wide-angle lens, a standard lens, and the like. A lens such as a super-wide-angle lens, a wide-angle lens, and a standard lens has a narrower angle of view than the fish-eye lens shown in the first embodiment. When the entire surroundings are photographed using a super-wide-angle lens, a wide-angle lens, and a standard lens, the entire surroundings are divided into three or more and photographed. The information provision support device 400 acquires images captured by dividing the entire surroundings into three or more pluralities, and performs synthesis processing of combining these images.

FIG. 17 is a diagram for explaining processing for synthesizing a plurality of images supplied from the imaging device 100 according to the embodiment.

For example, the imaging apparatus 100 captures an image using an ultra-wide-angle lens having an angle of view of 90 degrees or more and less than 180 degrees. The imaging device 100 directs the optical axis in the positive and negative directions of the XC axis, the positive and negative directions of the YC axis, and the positive and negative directions of the ZC axis, and takes a total of six images. The XC axis, YC axis, and ZC axis are orthogonal coordinates with the point C as the origin. The imaging apparatus 100 may add information regarding the direction of the optical axis at the time of imaging and the orientation of the image to each of the images. The imaging device 100 supplies each of the above images to the information provision support device 400 . The information provision support device 400 adds the acquired image to the collected information 412 as a omnidirectional image.

The information provision support device 400 arranges the obtained images as shown in the figure, centering on the point C corresponding to the observation point.

For example, an image in the positive direction and an image in the negative direction of the XC axis are arranged so as to be orthogonal to the XC axis, and their positions are in the positive and negative directions of the x axis. For example, the positive and negative values of the XC axis that determine the position are made equal in absolute value. In other words, the image in the positive direction and the image in the negative direction of the XC axis are arranged at positions at equal distances from point C. FIG. The image corresponding to the YC axis and the image corresponding to the ZC axis are also the same as the image corresponding to the XC axis.

By arranging the six images as described above, a cube surrounding point C is formed. The information provision support device 400 projects the image drawn inside each face of the cube onto the inside of the surface of the sphere as the space model M. FIG. As a result, the information provision support device 400 generates an all-surrounding image projected onto the space model M. FIG.

According to the above-described embodiment, it is possible to combine a plurality of images supplied from the imaging device 100 to generate a surrounding image, in addition to achieving the same effects as the first embodiment.

It is important to manage the position of the image pickup device 100 and accurately orient the optical axis direction when photographing by the above method. For example, the imaging apparatus 100 accurately orients the direction of the optical axis in the direction of the reference coordinate axis when capturing each of the above images, so that the position at the time of capturing does not deviate from the position of the origin of the reference coordinate axis. It is better to

(Modification 3 of the first embodiment)
Modification 3 of the first embodiment will be described. In the first embodiment, "(1-1) a method of using the space model M" and "(2-1) when using the space model M, adjust by rotating the image along the space model M He explained the case of combining "method". In this modification, instead of "(2-1) a method of adjusting by rotating an image along the space model M when using the space model M", "(2-3) feature points of the surrounding image and the feature points of the image based on the 3D model”. The description will focus on differences from the first embodiment.

A point T1 in the image of the virtual space shown in FIG. 8 is a point having a feature specifying a position in the virtual space. It is assumed that a point T2 corresponding to the point T1 can be identified in the peripheral image of the real space corresponding to this, that is, the image shown in FIG. 4A. The point T2 is shown in FIG. 4A.

For example, after finishing the processing of S19 shown in FIG. 15A, the information provision support device 400 performs the following processing based on the information acquired from the imaging device 100. FIG.
The imaging device 100 detects the visual feature quantity of the point T3 (see FIG. 1) in the real space corresponding to the points T1 and T2, and supplies the feature quantity to the information provision support device 400. FIG. For example, the imaging apparatus 100 determines a feature amount (hereinafter referred to as feature amount T) that can characterize the point T3 from the color, brightness, etc. of the surroundings centering on the point T3.

Based on the feature amount T detected in the real space, the control unit 420 extracts points specified from the feature amount T as points in the image shown in FIG. 4A. The extracted point becomes point T2. It should be noted that the control unit 420 may perform the processing for the above extraction by a technique such as pattern matching of image processing. At that time, the control unit 420 appropriately selects processing such as a color filter, binarization for each brightness or color component, contour enhancement, contrast adjustment, and distortion correction as the processing for emphasizing the feature amount. It may be performed in the order in which feature points can be extracted satisfactorily.

The control unit 420 acquires the point T1 in the virtual space image (FIG. 8) designated by the user. The control unit 420 synthesizes the peripheral image with the space model M so that the position of the point T2 overlaps with the position of the point T1 (S20). Note that the control unit 420 may perform conversion processing for adjusting so that the peripheral image can be mapped onto the space model M at the time of this synthesis.

Subsequent processing is as described above.

According to the above-described embodiment, in addition to the effects common to those of the first embodiment, the surrounding image in the real space and the virtual space image in the virtual space can be associated without using the projection plane.

(Second embodiment)
A second embodiment will be described. In the first embodiment, "(1-1) a method of using the space model M" and "(2-1) when using the space model M, adjust by rotating the image along the space model M He explained the case of combining "method". Instead of this, in the second embodiment, "(1-2) a method not using the spatial model M" and "(2-2) a method for adjusting by projecting the surrounding image onto an image based on the 3D model". will be described below. The description will focus on differences from the first embodiment.

15B is a flowchart illustrating an example of information visualization processing according to the embodiment; FIG.

The processing from S11 to S16 is the same as in FIG. 15A. After completing the processing up to S16, the display control unit 423 acquires from the terminal device 200 the direction of the viewpoint (viewpoint direction) that determines the direction of the image to be displayed on the terminal device 200 (S31).

The control unit 420 corrects the distortion of the omnidirectional image based on the direction of the viewpoint specified in S31 so that the image can be viewed from a desired observation point (S32). The details of the processing for correcting the distortion of the omnidirectional image will be described later.

The control unit 420 determines whether or not position correction of the image after distortion correction is necessary (S33).
If position correction of the image after distortion correction is necessary, the control unit 420 performs position correction of the image after distortion correction (S34).

If the position correction of the image after distortion correction is not necessary, or after the process of S34 is completed, the control unit 420 performs the position correction of the image after distortion correction (S34). For example, the control unit 420 maps 2D format image data onto the surface of the object in the 3D virtual space. The control unit 420 corrects the position of the distortion-corrected image by its own processing. Alternatively, the control unit 420 may control the BIM 300 to perform position correction of the image after distortion correction by processing of the BIM 300 . In this case, the BIM 300 performs position correction of the image after distortion correction.

The display control unit 423 causes the terminal device 200 to display an image looking in the viewpoint direction from the observation point (S35). As a result, an image as shown in FIG. 16 described above is generated, and it becomes clear that the device to be newly installed within the target range 10 interferes with the existing device.

The display control unit 423 determines whether or not to display another direction with a different viewpoint direction according to an instruction from the terminal device 200 (S36), and when it determines to display another direction (S36: YES). , S31 and subsequent steps are repeated. If it is determined not to display another direction by the determination processing in S36 (S36: NO), the illustrated series of processing ends.

18 to 20 are diagrams for explaining the process of correcting the distortion of the omnidirectional image.
FIG. 18 shows the pixel configuration of the omnidirectional image. If this omnidirectional image is in the equirectangular projection, the aspect ratio of the omnidirectional image is 1:2. This omnidirectional image has pixels of vertical v (pixels) and horizontal u (pixels). If the aspect ratio of a unit pixel is 1, the number of horizontal pixels (u) is twice the number of vertical pixels (v). Define two-dimensional coordinates with the center of the image as the origin.

The position (u1, v1) of any point PP in the omnidirectional image can be converted to polar coordinates (r, θ, φ) by the following equation (1).

θ=(u1/u)×360
φ=(v1/v)×360 (1)

By making the radius r of the sphere a unit quantity, the position (θ, φ) of an arbitrary point P in the omnidirectional image can be represented by a vector equation as shown in Equation (2) below.

PP = kP
PP=[u1 v1] ^T
k = [(360/u) (360/v)]
P = [θ φ] ^T
... (2)

FIG. 19 illustrates a case where the omnidirectional image projected onto the sphere representing the space model M is projected onto the display image.
FIG. 19(a) shows a state in which a sphere representing the space model M and a plane onto which a display image is projected (hereinafter referred to as a display plane) are arranged so as to be in contact with each other. In the example shown in this figure, the sphere touches the display surface at its origin O(0,0). A point P(θ, φ) on the sphere representing the space model M is projected onto a point PP(x, y) on the display surface.
FIG. 19(b) is a plan view of the state shown in FIG. 19(a) from above the sphere along the y-axis direction of the display surface, that is, from the polar direction of the sphere. As is clear from the figure, x is a function of θ. The relationship is shown in Formula (3).

x=Atan θ (3)

A in the above equation (3) is the radius of the sphere in contact with the display surface.

FIG. 19(c) is an elevational view of the state shown in FIG. 19(a) viewed from the side of the sphere along the x-axis direction of the display surface, that is, from a direction orthogonal to the polar direction of the sphere. As is clear from the figure, y is a function of φ. The relationship is shown in Formula (4).

y=Atanφ (4)

FIG. 20 illustrates a case where the sphere representing the space model M touches other than the origin O of the display surface. FIG. 20 shows a state where the sphere is in contact at a point moved by Δx in the +x-axis direction from the state shown in FIG. 19(b). The relationship between x and θ in this case is shown in equation (5).

x=Atan(θ ₀ +θ) (5)

θ ₀ in equation (5) is the rotation angle around the axis of the sphere.
Similarly, the relationship between y and φ when the sphere touches at a point moved by Δy in the +y-axis direction from the state shown in FIG. 19C is shown in Equation (6).

y=Atan(φ ₀ +φ) (6)

φ ₀ in equation (6) is the rotation angle when the sphere is rotated around an axis parallel to the x-axis direction.

As described above, by using the relational expressions shown in this embodiment, the positions on the omnidirectional image can be projected onto the display surface. By this processing, the distortion-corrected image of the omnidirectional image can be corrected by an arithmetic method.

According to the above-described embodiment, in addition to the effects common to those of the first embodiment, the surrounding image in the real space and the virtual space image in the virtual space are associated without using the projection plane of the space model M. be able to.

According to at least one of the above embodiments, the information provision support device 400 includes a real image acquisition unit (designated information acquisition unit 421 and surrounding image acquisition unit 422), and a display control unit 423 that causes the BIM 300 to generate a display image including an area common to the display area for displaying the surrounding image and the display area for displaying the virtual space image. , allows sharing of the surroundings of a location in a simpler way.
Further, the display control unit 423 controls whether both the position of the reference point R in the three-dimensional virtual space corresponding to the observation point in the real space and the position VP on which the projection plane S provided around the reference point R is viewed are within a predetermined range. , the position of the reference point R and the position VP looking into the projection plane S in the three-dimensional virtual space are adjusted so that the surrounding image projected onto the projection plane S and the virtual structure in the three-dimensional virtual space are shown. space image, and causes the image generation unit to generate a display image including an area common to the display area displaying the associated surrounding image and the display area displaying the virtual space image. This makes it possible to share the surroundings of a location in a simpler way.

Further, according to the above-described embodiment, the information registered in the BIM 300 may be general information about the range to be used as the 3D model. As a result, the user can obtain the same effect as drawing detailed information only by drawing the minimum information required to use it as a 3D model.

Further, according to the above embodiment, the information provision support device 400 acquires the actual state of the real space as an image captured in the real space. By providing an image in which the image is superimposed on the 3D model by the information provision support device 400, the user can reduce oversight and prejudice of the actual state compared to the information only as a result of modeling as the 3D model.

Further, according to the above embodiment, the BIM 300 provides the 3D model with information such as dimensions and component type information. The information visualization system 1 makes it possible to easily compare the actual state of the real space on the BIM 300 . As a result, the efficiency of various tasks using the BIM 300 can be improved. The various types of work using the BIM 300 include, for example, design work, supervision work, management work, and work that uses dimensions, such as estimation work such as quantity calculation.

Further, according to the above embodiment, the information visualization system 1 creates a perspective view (bird's-eye view) that reflects the actual state of the real space. The BIM 300 uses the above space model M to create a perspective view based on part or all of the surrounding image of the observation point, so that the actual state of the real space can be objectively and easily determined. Become.

Furthermore, the perspective view makes it possible to provide intuitively understandable information. In particular, it is effective when there is an existing object within the target range 10 and the relationship between the new planned object and the existing object is indicated. According to the perspective view above, the interference between the plan (new construction) and the existing construction becomes clear. Also, the above perspective view provides the user with necessary information for examining the state after removal even when the existing object is removed.
For example, the designer determines the plan after making adjustments so that there is no interference between the planned object (new object) and the existing object. If integration processing is required to implement the plan, the integration processing control unit 426 may cause the BIM 300 to perform integration processing within a specified range. In that case, the display control unit 423 acquires the result and adds it to the integration information 414 .

Furthermore, the above perspective view is suitable as an explanatory material for showing the above relationship to the administrator, user, etc. of the target range 10 . It becomes easier for the participants of the briefing session who receive the explanations from managers, users, etc., to understand the explanations, and to be easily convinced by the explanations. In addition, it becomes easier for the participants of the briefing to participate in questions and discussions, and it becomes easier to improve the design based on the opinions of the participants of the briefing obtained on the spot.

Further, according to the above-described embodiment, it is possible to obtain accurate information of the target range 10 without omission by a simple operation of capturing an image of the target range 10 with the imaging device 100 .
As a comparative example for the imaging device 100, a laser scanner is known that acquires the site situation as point group information. The information visualization system 1 can easily obtain necessary information by using the imaging device 100 without using a special device such as a laser scanner when acquiring the situation of a wall surface or the like. can. The information visualization system 1 provides simple work as described above, and may be applied to maintenance work.

In addition to maintenance work, the information visualization system 1 can be applied to work and applications other than design work, supervision work, and management work for the building 2 .

Although the embodiment of the present invention has been described above, the information visualization system 1 shown in FIG. 1 has a computer system inside. A series of processes related to the above-described processes are stored in a computer-readable storage medium in the form of a program, and the above processes are performed by reading and executing this program by a computer. Here, computer-readable storage media include magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, semiconductor memories, and the like. Alternatively, the computer program may be distributed to a computer via a communication line, and the computer receiving the distribution may execute the program. In addition, the term "computer system" as used herein includes an OS and the like.

Although one embodiment of the present invention has been described above, the embodiment of the present invention is not limited to the above. For example, the methods exemplified in each embodiment and its modifications may be combined in combinations other than the exemplified combinations. Moreover, the embodiment of the present invention can be modified from the above embodiment as follows.
For example, in the above-described embodiments, the information visualization system 1 was divided into the imaging device 100, the terminal device 200, the BIM 300, and the information provision support device 400 for convenience of explanation of the configuration related to the present invention. The division of the imaging device 100, the terminal device 200, the BIM 300, and the information provision support device 400 may be changed from those illustrated above, and the respective devices may be integrated. Also, a part of the configuration included in each device may be included in the configuration of another device.

Note that the data structure shown in the figure is shown in a tabular format for ease of explanation, but the format of the data structure may be another format.

In the above-described embodiment, the method of using the image of the real space captured by the imaging device 100 is exemplified. may be used.

Note that the information visualization system 1 can be applied to tasks and uses other than design work and management work for the building 2 .

1 information visualization system, 100 imaging device, 200 terminal device, 300 BIM (image generation unit), 400 information provision support device, 410 storage unit, 411 space model information, 412 collected information, 413 expected position information, 414 integration information, 420 Control unit 421 Specified information acquisition unit 422 Surrounding image acquisition unit (actual image acquisition unit) 423 Display control unit 424 Position derivation unit 426 Integration processing control unit 427 Output processing unit.

Claims (10)

a real image acquisition unit that acquires a surrounding image showing the situation of an object around an observation point in real space and adds it to collected information ;
Information of a three-dimensional virtual space corresponding to the real space is registered in the space model information, a desired space model is selected from the space model information, and the selected space model is selected from the collected information. a display control unit that synthesizes surrounding images, associates a virtual space image showing a virtual structure in the three-dimensional virtual space with the synthesized image , and displays the situation of the object;
with
The collected information includes:
image information of the surrounding image;
identification information that associates the spatial model with the surrounding image;
information indicating the date and time of the acquired surrounding image or information indicating whether the surrounding image is the latest information;
Information visualization system. The date and time information of the acquired surrounding image includes information on the shooting date and time of the surrounding image or information on the date and time when the surrounding image is stored.
The information visualization system according to claim 1. The display control unit
3. The information visualization system according to claim 1, wherein a position of a reference point in the three-dimensional virtual space corresponding to the observation point is matched with a position on which a projection plane provided around the reference point is viewed. The display control unit
4. The information visualization system according to claim 3, wherein the position of the reference point and the position from which the projection plane is viewed are matched, and the surrounding image and the virtual space image are associated with the space model corresponding to the observation point. The space model is formed so as to use at least one of a spherical surface of a sphere and a side surface of a cylinder arranged with reference to the reference point as a projection plane.
The information visualization system according to claim 4. The display control unit
overlapping and displaying partial images extracted based on predetermined extraction conditions from both the surrounding image and the virtual space image;
The information visualization system according to any one of claims 1 to 5. a real image acquisition unit that acquires a surrounding image showing the situation of an object around an observation point in real space and adds it to collected information;
Information of a three-dimensional virtual space corresponding to the real space is registered in the space model information, a desired space model is selected from the space model information, and the selected space model is selected from the collected information. a display control unit that synthesizes a surrounding image;
with
The collected information includes:
image information of the surrounding image;
identification information that associates the spatial model with the surrounding image;
Information indicating the date and time of the acquired surrounding image or information indicating whether the surrounding image is the latest information,
displaying the situation of the object by associating the synthesized image with a virtual space image showing a virtual structure in the three-dimensional virtual space;
Information visualization system. The display control unit
causing a BIM device to generate a virtual space image showing a virtual structure in the three-dimensional virtual space corresponding to the synthesized image;
The information visualization system according to any one of claims 1 to 7. Acquiring a surrounding image showing the situation of an object around the observation point in real space and adding it to the collected information ;
Information of a three-dimensional virtual space corresponding to the real space is registered in the space model information, a desired space model is selected from the space model information, and the selected space model is selected from the collected information. a step of synthesizing a surrounding image, associating a virtual space image showing a virtual structure in the three-dimensional virtual space with the synthesized image , and displaying the situation of the object;
including
The collected information includes:
image information of the surrounding image;
identification information that associates the spatial model with the surrounding image;
information indicating the date and time of the acquired surrounding image or information indicating whether the surrounding image is the latest information;
Information visualization method. Acquiring a surrounding image showing the situation of an object around the observation point in real space and adding it to the collected information ;
Information of a three-dimensional virtual space corresponding to the real space is registered in the space model information, a desired space model is selected from the space model information, and the selected space model is selected from the collected information. a step of synthesizing a surrounding image, associating a virtual space image showing a virtual structure in the three-dimensional virtual space with the synthesized image , and displaying the situation of the object;
A program for causing a computer to execute
The collected information includes:
image information of the surrounding image;
identification information that associates the spatial model with the surrounding image;
information indicating the date and time of the acquired surrounding image or information indicating whether the surrounding image is the latest information;
program .

Images