WO2020241331A1 - Three-dimensional data management method for building and mobile terminal for implementing same - Google Patents

Abstract

Provided is a method for relatively simply generating a three-dimensional space model of a building by using a mobile terminal. This mobile terminal comprises: a camera unit which captures an image; a plane detection unit which detects a plurality of planes from the image captured by the camera unit, and generates a plurality of plane polygons; a three-dimensional shape creation unit which generates a three-dimensional space model by performing an image process; and a display unit, wherein the three-dimensional shape creation unit receives a selection of at least one plane polygon from the plurality of plane polygons, acquires a second image corresponding to the selected plane polygon image-captured by the camera unit, and attaches a texture image based on the second image to the selected plane polygon, thereby generating the three-dimensional space model.

Description

3D data management method for buildings and mobile terminals that realize this

[Related application]
This application claims the priority of Japanese Patent Application No. 2019-102781 entitled "Three-dimensional data management method for buildings and mobile terminals that realize this" filed on May 31, 2019. The disclosure is incorporated herein by reference in its entirety.
The present invention relates to a three-dimensional data management method for buildings and a mobile terminal that realizes the method.

As a background technology in this technical field, there is Japanese Patent Application Laid-Open No. 2013-225245 (Patent Document 1). In this publication, "the actual image is based on the position of an image acquisition unit that acquires an input image generated by imaging a real space using an imaging device and one or more feature points reflected in the input image. A recognition unit that recognizes the relative position and orientation between the space and the image pickup device, an application unit that provides an augmented reality application using the recognized relative position and orientation, and an execution unit by the recognition unit. Provided is an image processing device including a display control unit that superimposes a guidance object that guides a user who operates the image pickup device on the input image according to the distribution of the feature points so that the recognition process is stabilized. ".

Further, there is Japanese Patent Application Laid-Open No. 2014-002645 (Patent Document 2). In this publication, "The composite image display system of the present invention is composed of a portable imaging terminal and a data server, and a three-dimensional computer graphic image of a building under construction is pre-written and stored in the data server. An imaging unit in which a portable imaging terminal captures an image of a building and outputs it as an captured image, a display unit that displays the captured image, the captured image, and the same position and imaging direction in which the captured image is captured in a computer graphic image. In the virtual image of, the position control unit that compares the coordinate positions of the same feature points and the comparison result of the coordinate positions of the feature points perform coordinate conversion in which the coordinate positions of the feature points overlap, and the virtual image is added to the captured image. It is provided with an image processing unit that is superimposed and displayed on the display unit. "

Japanese Unexamined Patent Publication No. 2013-225245 Japanese Unexamined Patent Publication No. 2014-002645

The above-mentioned Patent Document 1 describes an environment recognition technique based on a group of feature points in an image. Further, Patent Document 2 describes an image processing technique for superimposing a virtual image on an captured image and displaying it on a display unit. However, these techniques described in the literature do not take into account the generation of three-dimensional spatial models of buildings.

Therefore, the present invention provides a method for relatively easily generating a three-dimensional space model of a building using a mobile terminal.

In order to solve the above problems, for example, the configuration described in the claims is adopted.
The present application includes a plurality of means for solving the above problems. For example, a mobile terminal detects a plurality of planes from a camera unit that captures an image and an image captured by the camera unit. It has a plane detection unit that generates a plurality of plane polygons, a three-dimensional shape creation unit that generates a three-dimensional space model by performing image processing, and a display unit, and the three-dimensional shape creation unit is the said. It accepts the selection of at least one plane polygon from a plurality of plane polygons, acquires a second image corresponding to the selected plane polygon photographed by the camera unit, and obtains a texture image based on the second image. The three-dimensional space model is generated by pasting on the selected plane polygon.

According to the present invention, a three-dimensional space model of a building can be generated relatively easily by using a mobile terminal.
Issues, configurations and effects other than those described above will be clarified by the description of the following embodiments.

It is an example of the block diagram of the whole management system 1. This is an example of the hardware configuration of the integrated management system 101. This is an example of the hardware configuration of the construction company terminal 102. This is an example of the hardware configuration of the resident terminal 105. This is an example of human-related master information 500. This is an example of the thing-related master information 600. This is an example of the thing-related master information 700. This is an example of the phase-related master information 800. This is an example of transaction information 900 stored in the main DB 220. This is an example of transaction information 900 (continued) stored in the main DB 220. This is an example of excerpting a part of the information stored in the transaction information 900. This is an example of the phase start / end condition information 1200. This is an example of phase progress determination information 1300. This is an example of the data registration processing flow 1400. This is an example of the phase update processing flow 1500. This is an example of the input screen 1600 at the time of drafting a project. This is an example of the input screen 1700 for adding a new work item. This is an example of the screen 1800 when carrying out a field survey. This is an example of screen transition when the person in charge conducts a field survey. This is an example of an input screen when carrying out construction. This is an example of the screen 2100 when the matter list is displayed on the Web browser. This is an example of another screen 2200 when the matter list is displayed on the Web browser. This is an example of the screen 2300 when the case list is displayed on the Web browser at the time of acceptance / completion. This is an example of the dashboard screen 2400. This is an example of the screen 2500 of the process chart. This is an example of the matter search screen 2600. This is an example of the data analysis processing flow 2700. This is an example of the common factor analysis processing flow 2800. It is an example of the block diagram of the whole management system 1. This is an example of the hardware configuration of the integrated management system 101. This is an example of the hardware configuration of the construction company terminal 102. This is an example of the hardware configuration of the worker terminal 108. This is an example of the data structure 3300 of the building data. This is an example of building data 3400. This is an example of partition data 3500. This is an example of block data 3600. This is an example of equipment data 3700. This is an example of markup data 3800. This is an example of 3D shape data 3900. This is an example of communication data 4000. This is an example of the camera markup display screen 4100. This is an example of the pin markup display screen 4200. This is an example of a display screen of a three-dimensional space model. A plane polygon is an example of an explanatory diagram. This is an example of a texture. This is an example of an explanatory diagram of projective transformation. This is an example of an explanatory diagram for updating a plane polygon. This is an example of an explanatory diagram for updating a plane polygon. This is an example of the building list screen 4900. This is an example of the building addition / edit screen 5000. This is an example of the building detail screen 5100. This is an example of the block addition / edit screen 5200. This is an example of the block detail screen 5300. This is an example of the markup detail screen 5400. This is an example of the three-dimensional space model viewing screen 5500. This is an example of the AR information editing screen 5600. This is an example of the camera markup creation processing flow 5700. This is an example of the pin markup creation processing flow 5800. This is an example of the three-dimensional space model creation processing flow 5900. This is an example of the plane polygon creation processing flow 6000. This is an example of the texture image generation processing flow 6100. This is an example of the plane polygon update processing flow 6200. This is an example of the three-dimensional space model browsing processing flow 6300. This is an example of the AR information editing process flow 6400. This is an example of communication using a three-dimensional space model.

Hereinafter, examples will be described with reference to the drawings.
"Safety,""safety," and "appropriateness" that arise from "trust" in construction / real estate are obtained by accumulating past information, but information is cut off between services, for example, past defect information at the time of mediation. It is impossible to know. And since the owners, residents, management companies, craftsmen, managers, and people involved in the building all change over time, the current situation is that accurate information cannot even be grasped by the government.

"Intermediary services,""materialservices," and "process control," which are overflowing in the streets, are "instantaneous" connections between emerging inventories, demand, and phenomena.
For example, with an intermediary service, it was not possible to know whether a resident had no trouble in the past or a well-managed management company. Even in the case of construction work such as repairs, there was no way to be reassured whether there was a problem that the building had continuously, or whether the craftsman would politely do the proper work when requesting work.

The "people," "things," and "money" that surround the building are characterized by high liquidity, and if you focus on these factors, the owner will change, the lessee will change, and the craftsman will change. The previous information will be cut off.

Therefore, in the building management system in this embodiment, by designing a data model centered on the building, fair information on "real estate", for which it was difficult to grasp the whole picture due to disconnection, and based on this It makes it possible to create matching.

In this embodiment, an example of a management system that centrally manages all related information from the time of new construction of a building to the time of demolition will be described.
FIG. 1 is an example of a configuration diagram of the entire management system 1.
The management system 1 includes a plurality of construction company terminals 102, a plurality of manufacturer terminals 103, a plurality of management company terminals 104, a plurality of resident terminals 105, and a plurality of other vendor terminals 106, each of which is an integrated management system via a network. It is connected to 101. The network may be wired or wireless, and each terminal can send and receive information via the network.

Each terminal of the management system 1 and the integrated management system 101 may be, for example, a mobile terminal such as a smartphone, a tablet, a mobile phone, or a personal digital assistant (PDA), or a wearable terminal such as a glasses type, a wristwatch type, or a clothing type. Good. It may also be a stationary or portable computer, or a server located in the cloud or on a network. Alternatively, it may be a combination of these plurality of terminals. For example, a combination of one smartphone and one wearable terminal can logically function as one terminal. Further, it may be an information processing terminal other than these.

The construction company terminal 102 is a terminal used by developers, construction companies, construction shops, craftsmen, and the like.
The maker terminal 103 is a terminal used by a material maker, a material wholesaler, or the like.
The management company terminal 104 is a terminal used by a management company or an intermediary.
The resident terminal 105 is a terminal used by the resident or the owner of the building.
The other contractor terminal 106 is a terminal used by a contractor other than the above, which intervenes between the time of new construction of the building and the time of demolition.
The integrated management server receives input of various information generated from the time of new construction of the building to the time of demolition from each of the above terminals, and stores these in the main DB 220.

Each terminal and integrated management system 101 of the management system 1 has a processor that executes an operating system, an application, a program, etc., a main storage device such as a RAM (RandomAccessMemory), an IC card, a hard disk drive, and an SSD (Solid). State Drive), auxiliary storage devices such as flash memory, communication control units such as network cards, wireless communication modules, and mobile communication modules, touch panel, keyboard, mouse, voice input, input by motion detection by imaging the camera unit, etc. It includes an input device and an output device such as a monitor or a display. The output device may be a device or a terminal for transmitting information for output to an external monitor, display, printer, device, or the like.

Various programs and applications (modules) are stored in the main memory, and each functional element of the entire system is realized by executing these programs and applications by the processor. In addition, each of these modules may be implemented by hardware by integrating them. Further, each module may be an independent program or application, but may be implemented in the form of a part of a subprogram or a function in one integrated program or application.
In this specification, each module is described as a subject (subject) that performs processing, but in reality, a processor that processes various programs, applications, and the like (module) executes processing.

Various databases (DB) are stored in the auxiliary storage device. A "database" is a functional element (storage unit) that stores a data set so that it can handle arbitrary data operations (for example, extraction, addition, deletion, overwriting, etc.) from a processor or an external computer. The method of implementing the database is not limited, and may be, for example, a database management system or a text file such as XML.

FIG. 2 is an example of the hardware configuration of the integrated management system 101.
The integrated management system 101 is composed of, for example, a server arranged on the cloud.
Programs and applications such as a data registration display module 210, a status determination module 211, and a data analysis module 212 are stored in the main storage device 201, and the integrated management system 101 is executed by the processor 203 to execute these programs and applications. Each functional element of is realized.

The data registration display module 210 cooperates with the construction company terminal 102, the manufacturer terminal 103, the management company terminal 104, the resident terminal 105, and the other contractor terminal 106 to store the information received from these terminals in the main DB 220, and also indicate. The data registration display module 210 registers and displays data together, but these may be separately separated and implemented as a data registration module and a data display module.

The state determination module 211 determines the current state of each project, construction, or work by analyzing the transaction information 900 stored in the main DB 220, and outputs / displays it to each terminal or output device 205.
The data analysis module 212 extracts various useful information by analyzing the information stored in the main DB 220, and outputs / displays it to each terminal or the output device 205.

The auxiliary storage device 202 includes databases such as a main DB 220, a human-related DB 221 and a thing-related DB 222, and a thing-related DB 223.
The main DB 220 stores information received from the construction company terminal 102, the manufacturer terminal 103, the management company terminal 104, the resident terminal 105, and the other contractor terminal 106 as transaction information 900 for the building.
The human-related DB 221 stores information about a person, that is, information related to a "human", such as information about a user and information about a craftsman, and stores human-related master information 500.
The thing-related DB 222 stores information about the building, that is, information related to the "thing", such as the type of building, layout, configuration in the section, and internal equipment, and stores the thing-related master information 600. ..

The thing-related DB223 is related to information related to a project, construction, work (sometimes collectively referred to as construction content) such as project contents, actions for the project, construction type, work contents, that is, "things". The information is stored, and the project-related master information 700 is stored. A "project" is sometimes called a "construction project" and usually includes a plurality of operations.
In addition, the thing-related DB 223 also stores phase-related information indicating the progress (phase) of the project / construction / work (construction content), and stores the phase-related master information 800.

FIG. 3 is an example of the hardware configuration of the construction company terminal 102.
The construction company terminal 102 is composed of, for example, a smartphone.
The integrated management system cooperation module 310 and the repair management module 311 are stored in the main storage device 301, and each functional element of the construction company terminal 102 is realized by executing these programs and applications by the processor 303.

The integrated management system linkage module 310 obtains repair management data 320 stored in the auxiliary storage device 302 and other information acquired and managed by the construction company on a regular basis, whenever the information is updated, or arbitrarily. It is transmitted to the integrated management system 101 at the timing of. Further, instead of a form like an application, the integrated management system 101 may be simply accessed via a Web browser and linked with the integrated management system 101 to transmit information using the function of the Web browser. The auxiliary storage device 302 stores the repair management data 320 for the building managed by the construction company.

FIG. 4 is an example of the hardware configuration of the resident terminal 105.
The resident terminal 105 is composed of, for example, a smartphone.
The integrated management system cooperation module 410 and the resident maintenance module 411 are stored in the main storage device 401, and each functional element of the construction company terminal 102 is realized by executing these programs and applications by the processor.

The integrated management system linkage module 410 periodically or updates the resident information management data 420 stored in the auxiliary storage device 402 and other information acquired and managed by the resident terminal 105. It is transmitted to the integrated management system 101 at an arbitrary timing. Further, instead of a form like an application, the integrated management system 101 may be simply accessed via a Web browser and linked with the integrated management system 101 to transmit information using the function of the Web browser. The auxiliary storage device 402 stores the resident information management data 420 of the resident.

FIG. 5 is an example of human-related master information 500.
The human-related master information 500 is master information of information related to all "people" from the time of new construction of a building to the time of demolition. The master ID 501, the category ID 503, and the individual ID 505 are foreign keys of the main DB 220, and values are entered in the corresponding parts of the transaction information 900. The respective contents are stored in the reference information 502, 504, 506.

By using this human-related master information 500, regarding the craftsman,
-What type of craftsman-What type of craftsman can handle-What construction skills can be handled-What kind of work can be handled? Information such as what can be stored and managed in transaction information 900 of the main DB 220. Can be done.

In the example of FIG. 5, information about the craftsman is stored, but in addition to this, master information such as information about the user, information about the craftsman, information about the manager, information about the construction company, and information about the management company is stored. can do.

FIG. 6 is an example of the thing-related master information 600.
The thing-related master information 600 is master information of information about "things" such as all buildings and equipment related from the time of new construction of a building to the time of demolition. The master ID 601 and the category ID 603 and the individual ID 605 are foreign keys of the main DB 220, and values are entered in the corresponding places of the transaction information 900. The respective contents are stored in the reference information 602, 604, and 606.

By using this thing-related master information 600, regarding the building,
・ What is the type of building (property)? ・ What is the floor plan (section) of the building? ・ What is the layout? ・ Is it a private part or a common part? ・ What are the components (blocks) in the section? Information such as which facility is related to the transaction information 900 of the main DB 220 can be stored and managed.

In the example of FIG. 6, information on buildings and equipment is stored, but in addition to this, master information such as information on equipment, information on parts, information on materials, and information on materials can be stored.

FIG. 7 is an example of the thing-related master information 700.
The thing-related master information 700 is master information of information related to "matters" such as all constructions and work contents related from the time of new construction of a building to the demolition. The master ID 701, category ID 703, and individual ID 705 are foreign keys of the main DB 220, and values are entered in the corresponding places of the transaction information 900. The respective contents are stored in the reference information 702, 704, and 706.

By using this Koto-related master information 700, regarding construction and work, etc.
-What is the content of the project? -What is the action for the project? -What is the content of the construction? -What is the content of the work? Transaction information 900 of the main DB 220 Can be stored and managed in.
Note that FIG. 7 is an example, and master information regarding other "things" may be stored.

FIG. 8 is an example of the phase-related master information 800.
The phase-related master information 800 is master information for managing the progress of the matter stored in the transaction information 900 of the main DB 220. The master ID 801 and the category ID 803 and the individual ID 805 are used as the values of the corresponding parts of the transaction information 900 of the main DB 220. The respective contents are stored in the reference information 802, 804, and 806.

By using this phase-related master information 800, regarding the progress of the matter,
-Which of the seven phases 821 of "request / investigation / arrangement / field work / completion report / request / completion" is currently in (phase_name811)
・ What is the situation in each phase (status_name812)?
・ What is the state of each of the above situations (status_item813)?
Can be managed in detail.

By combining status_name812 and status_item813, it is possible to specify all the statuses of the matter. Further, from the situation specified by this combination, it is possible to determine which of the seven phase lists 821 described in phase_name811 corresponds to the progress at that time.
The value of the individual ID 805 of status_name812 is stored in the status name 911 (status_name) of the transaction information 900 of the main DB 220, and indicates which status in the status name list 822 the transaction record is.
The value of the individual ID 805 of the status_item 813 is stored in the status item 912 (status_item) of the transaction information 900 of the main DB 220, and the details further indicate which status of the status item list 823 in the status of 822.

For example, in the transaction information 900, when the status name 911 (status_name) is project_assign_status and the status item 912 (status_item) is completed, it indicates that the request is in the completed state. When the status name 911 is tachiai_status and the status item 912 is assign_done, it indicates that the attendance has been requested. When the status name 911 is inspection_status and the status item 912 is ongoin, it indicates that the field survey (current condition) is being carried out.

9 and 10 are examples of transaction information 900 stored in the main DB 220.
The integrated management system 101 is connected to the construction company terminal 102, the manufacturer terminal 103, the management company terminal 104, the resident terminal 105, and the other contractor terminal 106 via a network, and accepts input from these terminals to receive input from the building. All transactions related from the time of new construction to the demolition are stored in the transaction information 900 of the main DB 220.

The data of the transaction information 900 is basically not updated, and records are continuously added in chronological order each time a transaction occurs.
As managed by the human-related master DB221, the thing-related master DB222, and the thing-related master DB223, all the events related to "human", "thing", and "thing" are stored as transaction information 900, and "who" and "where". "What you did" is expressed in one record. For example, more than 100 transaction data are generated and stored in one room restoration work project.

The information stored in the transaction information 900 has 100 or more items such as item numbers 903 and 913, but the information is not limited to those listed here, and other information may be stored.
The objects 901, 902, 911, and 912 to be described are information on the project, information on the timing, information on things (buildings and equipment), information on things (construction content: situation, response content, money), and people (requester and requester). Memorize information about the request destination).
The attributes 904 and 914 are the contents of each memorable information, and are defined as attribute types (class names) 905 and 915, respectively. Data types 906 and 916 are determined for each.

FIG. 11 is an example of excerpting a part of the information stored in the transaction information 900.
As the information 1103 about the thing, the building ID, the building name, the building address, and the section ID (room number) are stored. In addition, as the block ID and block type, Western-style rooms, toilets, entrances, etc., which are the sections to be surveyed on-site, are stored. Further, as the facility type ID, facility ID, model number, and manufacturer, the air conditioner, faucet, key, its model number, manufacturer, etc., which are the facilities (facility) to be investigated, are stored. 1101 corresponds to item numbers 903 and 913 in FIG. 10, and 1102 corresponds to attribute types 905 and 915.

The status of each record can be specified by the values of the status name 1111 (status_name) and the status item 1112 (status_item) and the status_name812 and status_item813 of the phase-related master 800 which is the reference destination thereof. The example of FIG. 11 is an excerpt of a part of each transaction when the case of the patent apartment room 101 is started, witnessing, investigation, construction arrangement, material arrangement, on-site response, and case approval are carried out.

FIG. 12 is an example of the phase start / end condition information 1200.
The phase number 1201, the phase name 1202, and the phase item 1203 indicate which of the seven phase lists 821 shown in phase_name811 of the phase-related master 800 corresponds to.
A start condition 1204 and an end condition 1205 are defined for each phase, and the current progress status (phase) of each record described in the transaction information 900 is determined by a combination of these conditions.

For example, even if one considers the restoration work, it is normal that multiple works and works are carried out in parallel within a limited period of time, and the progress is made by grasping all the situations of each. It was difficult to manage.

On the other hand, in this embodiment, when some event (action) occurs, it is accumulated in the transaction information 900 as a data record of each line, and the record of the transaction information 900 and the phase start / end condition 1200 are set. By comparing, it is possible to determine the phase to which each transaction belongs and display this in an easy-to-understand manner for the user.

FIG. 13 is an example of the phase progress possibility determination information 1300.
The phase progress possibility determination information 1300 is a table that defines whether or not a plurality of works and works that occur in parallel may be shifted from the current phase to the next phase.
The vertical axis of the table shows the current phase 1302 and its status 1303, and whether or not this can be transferred to the next phase status shown on the horizontal axis is indicated by "◎", "○", "×", and "-". It is shown by.

The meaning of each code is as follows.
“◎” indicates the most common progress that can be progressed to the phase on the horizontal axis.
"○" indicates that it is possible to proceed to the phase on the horizontal axis.
"X" indicates that it is not possible to proceed to the phase on the horizontal axis.
“Return” indicates a phase that is expected to return to a phase earlier than the current state (direction in which the phase number of 1301 is smaller).
"-" Indicates that it is not usually prohibited to return to the phase before the current state, but it does not specifically prohibit the return.

For example, from 2-1 deportation witness and 2-2 field survey, it generally proceeds to 2-3 work approval. After 2-3 work approval of the survey, it is common to go to 3-1 construction arrangement or 3-2 material arrangement, but if you go to 4 site work, you will get 5 completion report or 6 request as it is. There is also. 4-1 After the on-site response, the 4-2 work report will be made, but in some cases, it may be necessary to return to the 2-2 on-site investigation again.
In this way, it is possible to prevent the progress of an erroneous work process by determining whether or not the progress of each phase is possible based on the phase progress possibility information 1300.

FIG. 14 is an example of the data registration processing flow 1400.
The data registration process occurs from the data registration display module 210 of the integrated management system 101 in cooperation with the construction company terminal 102, the manufacturer terminal 103, the management company terminal 104, the resident terminal 105, and the other contractor terminal 106. This is a process for receiving and storing information (action content) related to an event.

The data registration display module 210 receives the action content from the terminal (step 1410). Instead of receiving the input contents from each terminal, the user may directly input the action contents by the input device 204 of the integrated management system 101.
Each action content is generated, for example, when the integrated management system cooperation module 310 of the construction company terminal 102 receives an input from the user. Alternatively, for example, the integrated management system cooperation module 410 of the resident terminal 105 is generated by receiving an input from the user.

The data registration display module 210 performs kinsoku check by inquiring the state determination module 211 whether or not the status change due to the received action content does not correspond to kinsoku processing (step 1420).

If it is determined that the prohibition processing is not applied, one record is added to the transaction information 900 of the main DB 220 (step 1440), and the status status related to this action content is described as the status name 911 and the status item 912. Write to (step 1450). The content to be written is selected from the status name list 822 and the status item list 823 described in the phase-related master information 800.

If it is determined that the prohibition processing is applied, an error to that effect is transmitted to each terminal that has received the action content, and each terminal displays the error content (step 1460). Alternatively, the error content is displayed on the output device 205 of the integrated management system 101.

FIG. 15 is an example of the phase update processing flow 1500.
The phase update process is a process in which the state determination module 211 of the integrated management system 101 determines the current phase according to the entry of the transaction information 900.

The state determination module 211 accepts the designation of the matter for which the phase is to be determined (step 1510).
Then, the module acquires the record of the transaction information 900 of the designated matter, and acquires the latest status information described in each record (step 1520).
For example, in the entry example of FIG. 11, the matter ID 1113 is “11” and the same items are extracted, and the status information (status name 1111 and status item 1112) of these records is acquired.

The state determination module 211 determines the current phase by comparing the acquired latest status information with the phase start / end conditions.
Specifically, the acquired status information (status name 911 and status item 912) of the module satisfies the phase end condition of the phase start / end condition (step 1530 is Yes) or satisfies the start condition of the next phase. If yes (step 1540 is Yes), it is determined that the next phase may proceed.

In this case, the state determination module 211 further refers to the phase progress possibility determination information 1300, and determines whether or not the process may proceed to the next phase (step 1550).
If it is determined that progress is possible, the phase is updated to the next phase (step 1560).

If the current phase end condition is not satisfied and the next phase start condition is not satisfied due to the phase start / end condition (No in both steps 1530 and 1540), it is assumed that there is no phase progress and the current phase remains as it is. (Step 1570).
Further, even if step 1530 or 1540 is Yes and the next phase may be proceeded, if the result of the phase progress propriety determination (step 1550) by the phase progress propriety determination information 1300 is No. , The current phase remains (step 1570).

The status determination module 211 displays the latest phase information updated as a result on each terminal or the integrated management system 101. Alternatively, the latest phase information is stored in the transaction information 900 (step 1580).

In this flow, the phase is determined for each specific matter, and the designation of the matter is accepted in step 1510, but the phase can be determined not only for each matter but also for any information specified by the user. It is possible.
For example, when it is desired to determine a phase related to a specific work, a record having the specific work ID may be extracted and the same phase determination process may be executed. If you want to determine the phase related to a specific building or equipment, sort by building name, division ID, facility ID, etc. described in transaction information 900 item 1103, extract records, and perform the same phase determination. The process may be executed.

16 to 25 are examples of display screens of each terminal and the integrated management system 101.
An example of an input screen from the terminal when entering the transaction information 900 and a screen for displaying the information acquired from the transaction information 900 is shown.

FIG. 16 is an example of the input screen 1600 at the time of drafting a project.
This is an example of a screen that a user who manages a project, such as a user of the integrated management system 101, a user of the management company terminal 104, and a user of the construction company terminal 102, inputs when drafting a project. In the case of the integrated management system 101, the data registration display module 210 displays a screen on the output device 205 and receives input from the user via the input device 204. In the case of each terminal such as the construction company terminal 102 and the management company terminal 104, the integrated management system cooperation module of each terminal displays the screen in cooperation with the data registration display module 210 of the integrated management system 101. The integrated management system linkage module may be in the form of a smartphone application or in the form of a Web browser that displays screen information received from the integrated management system 101.

Reference numeral 1601 is a selection field for selecting the type of the matter to be drafted.
Reference numeral 1602 is a field for inputting information about the building.
Reference numeral 1603 is a field for inputting information about the management company.
Reference numeral 1604 is a field for inputting information regarding the contact information.
1605 is a field for inputting the contract contents.
1606 is a field for inputting the details of the matter.
1607 is a field for inputting the corresponding schedule.
1608 is a column for entering the construction request destination.

When the save button of 1609 is clicked, the data registration display module 210 displays each of the input information for each of the "project", "time", "thing", "thing", and "human" of the transaction information 900 of the main DB 220. Store in the corresponding location. Further, as in the record ID 100004 of FIG. 11, the information "project_assign_statu" and "completed" indicating that the "request" of the project is "completed" are entered in the status information 1111 and 1112.

FIG. 17 is an example of the input screen 1700 for adding a new work item.
Reference numeral 1701 is a selection field for accepting the selection of the matter, and the information of the matter list already stored in the Koto-related DB 223 and the transaction information 900 is displayed.
Reference numeral 1702 is a selection field for accepting selection of a work type. The work list information already stored in the Koto-related DB 223 and the transaction information 900 is displayed.
Reference numeral 1703 is a selection field for accepting selection of the type of construction. The information of the construction list already stored in the Koto-related DB 223 and the transaction information 900 is displayed.
In addition, when the input of information related to work or construction is received and the save button 1704 is clicked, the data registration display module 210 adds a one-line record to the transaction information 900 based on the input information.
For example, when the move-out witness agency is selected in 1702, the scheduled date and time, etc. are input and the save button 1704 is pressed, the "movement witness" is "requested" in the status information 1111, 1112 as in the record ID 10007 of FIG. The information "tachiai_status" and "assign_done" indicating that "" is entered.

FIGS. 18 to 20 are examples of screens displayed by each terminal such as a construction company terminal 102 and a management company terminal 104 that carry out site surveys and construction work. For example, the integrated management system linkage module of a terminal such as a smartphone operated by the person in charge displays the screen. The integrated management system cooperation module may be in the form of a smartphone application or in the form of a Web browser that displays screen information received from the integrated management system 101.

FIG. 18 is an example of the screen 1800 when carrying out a field survey.
Information about the matter is displayed in 1801. The integrated management system linkage module cooperates with the integrated management system 101 to acquire information on the matter stored in the transaction information 900 and information on the target building with reference to the thing-related DB223 and the thing-related DB222, and the number is 1801. indicate.
The key information is displayed on the map (1802) based on the address information (item number 26) stored in the transaction information 900, or the key information is displayed based on the key information (item numbers 73 to 76, etc.). 1803) It is also possible.

A list of field surveys conducted is displayed in 1811. The information displayed here is obtained by acquiring and displaying the information of the field survey (current condition) described in the transaction information 900. For example, a photograph or other information can be displayed by displaying information such as item Nos. 50-72 that describes in detail what has been done as information about things.

Further, when adding the field survey information, 1812 is clicked, and the screen transitions to the screen for adding the current condition information as shown in FIG.
When all the field surveys are completed, the completion report 1813 is clicked and the data registration display module 210 stores all the information in the transaction information 900.

FIG. 19 is an example of screen transition when the person in charge conducts a field survey.
1901 is a field for inputting information about the room to be constructed. The integrated management system cooperation module pulls down the information of the room already registered based on the transaction information 900 and the thing-related master information 600. If there is no corresponding room, the selection of "Add new" is accepted and a new addition screen such as 1910 is displayed in a pop-up.

1902 is a field for inputting information about the construction site. The integrated management system linkage module pulls down the information of the already registered parts (sections and equipment) based on the transaction information 900 and the thing-related master information 600. If there is no corresponding part, the selection of "Add new" is accepted and a new addition screen such as 1920 is displayed in a pop-up.

1903 is a field for inputting details such as the type of construction. When entering more detailed information, the entry of equipment details such as 1930 is accepted.

The integrated management system cooperation module can register a photograph of a part requiring construction and a photograph of the current equipment in the transaction information 900 at the time of conducting a site survey (1904).
Further, at this time, by searching the past transaction information 900 relating to the same room or the same construction site, it is possible to acquire the past situation, the past photograph, and the like and display it as 1905.

FIG. 20 is an example of an input screen at the time of construction implementation.
The screen 2000 on the left side is an example of an input screen in which the person in charge inputs information at the time of construction. The right screen 2010 is an example of an input screen when the inspector performs acceptance inspection in response to the input.
In 2001, past photographs of the same equipment stored in the transaction information 900 are searched and displayed.
In 2002, information about the target of this construction is acquired and displayed from information on things, things, people, and timing of transaction information 900.

2003 is an entry field for the person in charge of construction to enter the status of construction, and as in 2004, a photo of the current equipment can be uploaded and stored in the photo field (item numbers 66 to 68, etc.) of transaction information 900. it can. A URL or the like is actually described in the photo column of the transaction information 900, and the substance of the photo data is stored in the auxiliary storage device of the integrated management system 101, and this data is referred to by the URL. There is.

When the work is completed, the click of the 2005 button is accepted, and the data registration display module 210 stores the entered work information in the transaction information 900.

Next, the inspector of the work displays the information about the equipment of the project for which the construction has been completed. The integrated management system cooperation module of the terminal such as a smartphone used by the inspector displays the construction information stored in the transaction information 900 of the main DB 220 by linking with the integrated management system 101.

For example, in 2011, the content and cause of the trouble (items 53, 54, etc.) of the transaction information 900 are acquired and displayed.
In 2012, the photo before the construction was carried out is posted as Before, and in 2013, the photo after the construction is completed is posted as After. All of them are displayed by acquiring them from the attached photographs (items 66 to 68) of the transaction information 900.

The inspector carries out the acceptance inspection work, taking into consideration both the Before and After photographs displayed in a comparable state, writes a comment in 2014, and clicks the completion button 2015 of the acceptance report.
Then, the data registration display module 210 of the integrated management system 101 adds a one-line record to the transaction information 900 of the main DB 220 for the entered information.

21 to 26 are examples of screens when displaying an item on a Web browser.
The data registration display module 210 in the case of the integrated management system 101 displays a screen on the output device 205 by acquiring various information such as transaction information 900, human-related DB221, thing-related DB222, and things-related DB223, and is displayed by the user. Accepts the operation of. The integrated management system cooperation module of each terminal such as the construction company terminal 102 and the management company terminal 104 displays a screen in cooperation with the data registration display module 210 of the integrated management system 101. The integrated management system cooperation module may be in the form of a smartphone application or in the form of a Web browser that displays screen information received from the integrated management system 101.

FIG. 21 is an example of the screen 2100 when displaying the matter list on the Web browser.
The data registration display module 210 of the integrated management system 101 calls the information stored in the transaction information 900 of the main DB 220 with reference to the human-related DB221, the thing-related DB222, and the thing-related DB223, generates various screens, and outputs the information. Displayed on device 205. Moreover, you may display on each terminal via the Internet.

2101 and 2102 contain basic information about the matter.
For example, information about the building, information about the management company, information about the construction company, information about the contact information, the estimated order received, the reported defect, etc. are described, and these information are from the transaction information 900 and the information of various master DBs. It can be displayed by acquiring it.

When the "field survey" of 2103 is selected, the information of the "field survey" related to the matter displayed in 2101 is acquired from the transaction information 900, and the information related to the field survey is displayed in a list like 2104.
For each construction content, information such as where the room or location (section or equipment) is, what the construction is to be done, why the construction is to be done, etc. is obtained from the transaction information 900, such as 2105. Can be displayed on.
Similarly, regarding the information related to the items selected in 2103, the data registration display module 210 can acquire the relevant information from the transaction information 900 while referring to the human / mono / koto master DB, and display the information in the 2104 column. ..

In 2112, the current status of each work (requested and will be implemented) is determined and displayed based on the status information 911 and 912. The overall status of these plurality of operations performed in parallel is displayed in 2113. In addition, as the overall status, the latest status may be displayed, or both the most delayed and the most advanced of multiple works are acquired, and "Survey in progress-construction request". It may be displayed in a range such as "Done".

The status of work and construction carried out in parallel is determined by the status determination module 211, the progress phase of the entire project is specified, and it is displayed in 2111. In the example of 2111, it can be seen that the request and investigation have been completed and are currently in the arrangement phase.

The pencil icon under the action is a symbol indicating that it can be edited. By clicking this pencil icon, the user can go to the edit screen of each item and edit the contents. The trash can icon to the right of the pencil icon below the action is an icon for deleting individual work.

FIG. 22 is an example of another screen 2200 when displaying the matter list on the Web browser.
Here, a list of work to be ordered / arranged is displayed.
For example, each work such as air conditioner work, carpentry work, and acceptance inspection work is described, and a plurality of works are registered in each work.

In the example of carpentry work 2201, this work must be completed between August 9th and 16th (2205), and when expanded and displayed, four work 2202s are displayed.
The target and cause of each work, the request destination, the request content, etc. are acquired from each item described in the transaction information 900 and displayed as 2203.
For the two tasks 2204 that have not been requested yet, the input screen can be displayed and the arrangement can be proceeded by clicking the arrange button 2206.

In the construction period 2205, the information of the designated period is stored in the transaction information 900 or the thing-related DB 223, and this information is read out and displayed as a line table.
In this, each work to be arranged must be set between the beginning and the end of this construction period, and if the arrangement specifies a schedule that exceeds the construction period 2205, An error may be displayed.
Further, for each work schedule, the current status information and the progress of each work may be determined, and the optimum work schedule may be displayed in advance as a candidate date.

FIG. 23 is an example of the screen 2300 when the case list is displayed on the Web browser at the time of acceptance / completion.
The acceptance inspection result is displayed on 2301.
The completion report is displayed in 2302, and the status of settlement negotiations is displayed in 2303.
The data registration display module 210 displays the current status as the status 2311, 2321, 2322 of each item based on the contents of the status information 911, 922 of the transaction information 900.

FIG. 24 is an example of the dashboard screen 2400.
The data registration display module 210 collectively acquires the work information of the related matter from the transaction information 900 and sorts it by the status information 911 and 912, so that the work waiting for the quotation submission and the work waiting for the arrangement, as in 2401 Count and display the number of unreported on-site response, waiting for submission of completion report, waiting for approval of completion report, and scheduled on-site response.

In 2402, transaction information 900 is sorted by the ID of the logged-in user, and information on the on-site response schedule for the next week is acquired and displayed.
In 2403, the information about the same matter written in the transaction information 900 is displayed as a log.

FIG. 25 is an example of the screen 2500 of the process chart.
The data registration display module 210 displays a process chart for all the matters handled by the user. Usually, at least 3 to 10 craftsmen are involved in one project, and it is necessary to arrange a large number of craftsmen within the deadline for construction arrangements, and deep experience is required for project management.
However, as shown in FIG. 25, by visualizing the list of construction works / work that is currently in progress and that must be arranged in the future, it is possible to facilitate the support for arrangements and prevent omission of arrangements.

The data registration display module 210 acquires information about each construction / work from the transaction information 900 with reference to the master DB related to people / things / things, and displays a line chart for each case as shown in FIG. 25.

For example, the move-out work project 2501 in Room 101 needs to be carried out between August 9th and 16th (2502), but when this project is expanded and displayed, one survey and three works are listed. Has been (2503).
Two works are required for the survey, three works for the air conditioner work, and one work for the carpentry work, and each work schedule is disassembled and described as shown in the line chart 2504. The data registration display module 210 displays the number of days required for each work and the scheduled work date by acquiring the transaction information 900 or the information described in various master DBs.

FIG. 26 is an example of the matter search screen 2600.
For example, it is an example of a screen displayed on the management company terminal 104 or the construction company terminal 102. Both the management company and the construction company have a large number of projects, and it is necessary to manage the complicated progress of each project. By providing the function to search all the work information stored in the transaction information 900 as shown in FIG. 26, it is possible to display the state of the matter in charge of each company and what action should be taken next. It becomes possible to do.

2601 is a field for entering conditions for searching. It is possible to accept the selection of whether the project is restored to its original state or repair while moving in, and to search by phase, construction period, status, keywords, etc.

The search results are displayed in a list, and the summary of each case is displayed in 2604. If you click on each item, you can check the detailed information of the item.
Further, for each case, the current phase determined by the phase determination processing means is displayed in 2602, and the current status information is displayed in 2603.

In this way, by performing the search and the list display of the results as shown in FIG. 26, the current status of each project can be easily understood, and the outline and details of each project can be displayed, and the management company and the construction company can be displayed. Will be able to easily grasp the progress of the project.

As explained above, according to this embodiment, a management system that centrally stores and manages all related information from the time of new construction of a building to the time of demolition is realized. By the management system of this embodiment, the information that was conventionally managed and disconnected by each management company, each construction company, etc. is centrally managed in chronological order by one main DB 220, and the resident and the resident, The information that will be the basis for providing useful information to owners, management companies, construction companies, and all other related companies will be accumulated.

As described above, the transaction information 900 of the main DB 220 centrally stores all related information from the time of new construction of the building to the time of demolition in chronological order. By analyzing these accumulated information, it is possible to extract and display various useful information.
In this embodiment, a method of performing data analysis and extracting / displaying useful information based on the information of the main DB 220 accumulated in the first embodiment will be described.

FIG. 27 is an example of the data analysis processing flow 2700.
The data analysis module 212 of the integrated management system 101 acquires the information of the item to be searched from the main DB 220 (step 2710).
The data analysis module 212 analyzes the acquired data (step 2720) and displays the analysis result on the output device 205 of the integrated management system 101, or sends it to each terminal and displays it on each terminal (step 2730).

FIG. 28 is an example of the common factor analysis processing flow 2800.
This is a flow for explaining one of the use cases of the data analysis process 2720 in the data analysis module 212, and extracts, discovers, and displays a common factor from a large amount of information accumulated in a time series.

For example
1. 1. Discovery of common factors by sort analysis of building units 2. Discovery of common factors by sorting analysis of each section 3. Discovery of common factors by sort analysis of each facility 4. Discovery of common factors by sort analysis for each management company in charge 5. Items, evaluations, contents, etc. that occur in common in each unit are displayed as peculiar information by discovering common factors by sort analysis of each vendor.

Further, in the example of FIG. 28, paying attention to the fact that these transaction information 900s include information of a plurality of items having some hierarchical structures, the hierarchical items are common in order from the one with the smallest particle size to the one with the largest particle size. By discovering the factors, it is sequentially determined in which layer of the item the peculiar information appears.

The data analysis module 212 first acquires all the data of a specific item among the hierarchical items (step 2810).
The module analyzes the data and detects factors common to those data (step 2820).

When a common factor is found, the data having the common factor is output and displayed as singular information (step 2840).
If the common factor is not found, the item is moved to the item of the next higher hierarchy (2860), all the information of the next higher hierarchy is acquired again, and the same analysis process is performed.

The process is repeated until the analysis process is completed for all the items in the hierarchy (step 2850), and the singular information is sequentially discovered for each hierarchy.
For example, the transaction information 900 stores information on building troubles in hierarchical items such as buildings, sections, blocks, and equipment. Specifically, we are accumulating information on whether there is a problem with the equipment DD of the room CC of room BB of a certain building AA.

Information with such a hierarchical structure is traced for common factors in ascending order. For example, the information of the equipment DD (DD is a toilet) of the room CC of the BB room of the building AA having the transaction information 900 is traced back to the past, and all the records are acquired.
Here, if similar problems are discovered many times in the past, peculiar information that there is a problem in the toilet itself of the room CC of room BB of this certain building AA is extracted.

Next, the room CC of room BB of the building AA, which is one level higher, acquires all the same past records. By analyzing this, it is confirmed whether the peculiar information is extracted. Similarly, the analysis is performed one layer at a time.

Here, for example, the same analysis was performed on all records with the same building AA, and it was discovered as a common factor that toilet troubles also occurred in other rooms of the same building AA where the defect occurred. To do. In this case, it can be found that not only the equipment DD of the room CC of the room CC of the building AA has a toilet trouble in the past, but also the toilet of some other rooms of the building AA actually has a trouble. , There is a possibility of trouble around the water in the building AA itself.

In this way, by sequentially finding out the common factors of the information stored in the transaction information 900 for the hierarchical items, it is possible to comprehensively find out the problems and feature points of the entire building.

The hierarchical common factor analysis processing flow is effective for investigating when troubles are found. The cause of the equipment (equipment, etc.) in which the trouble was found is searched for in transaction information 900, and the common factor (cause of the trouble) is found by sequentially searching the equipment from the equipment in order. By doing so, it becomes possible to efficiently identify whether this trouble is caused by equipment, a section on the upper floor, or a building on the upper floor. You can accurately find the location of the problem.

In addition, the following data analysis use cases can be considered based on the time-series information from the time of new construction to the demolition of the building stored in the transaction information 900. This data analysis is realized by the data analysis module analyzing the data as shown in FIG. 27. The “item number” of the transaction information 900 indicates the item number 903 in FIG. 9 and the item number 913 in FIG.

1. 1. Monitoring the quality of the entire building Sorts the presence or absence of troubles in item 52 for each building ID in item 24, and totals the troubles by facility symptom type in item 50 every year.
As a result, it is possible to grasp the aging and construction quality of the building itself beyond the rooms and sections.

2. 2. Monitoring the quality of goods By sorting the presence or absence of troubles in item 52 by model year of item 39, manufacturer of item 40, and model year of item 41, regardless of the building or division, the model number is aggregated in chronological order. Calculate the trouble frequency for each manufacturer and model year.
For example, it is possible to obtain information that the 2000 model year YY air conditioner manufactured by XX is often out of order every two years.
In addition, by looking at the area where the air conditioner is out of order (item 26, building address), it is possible to output analysis results such as the frequency of failure of the above air conditioner in cold regions.

Since failure forecasts can be made from this information, it is possible to provide product demand forecasts to manufacturers, and it is also possible to have wholesalers make demand forecasts.
In addition, by ordering and replacing products of the same model number and the same manufacturer at a certain time, it is possible to reduce costs by purchasing in large quantities.

3. 3. Monitor the state of the area By looking at the correlation between the presence or absence of trouble in item 52, the facility ID in item 37, the facility symptom type in item 50, and the building address in item 26, for example, a foul odor that exceeds one section trouble It will be possible to detect problems with local infrastructure such as pest damage.
Normally, the area is managed by owned lots, and it is extremely difficult to know the problem of common infrastructure in the town in a unified manner. However, this data format makes it possible to know and foresee the problems facing the region beyond the units of rooms and buildings.

4. Quality of work = monitoring craftsmen It is said that the people who actually cure the building are "craftsmen", but these "craftsmen" are mainly divided into some genres such as electrical work, water supply work, and interior work. ..
Sorting by the name of the person in charge of the request in item 102, and counting the craftsmen who have NG in the work acceptance judgment of item 60, whether or not the craftsman does a sloppy work that can not finish the requested work at once The evaluation can be output.
In addition, if the time until the next problem occurs is short (calculated from the presence or absence of trouble in item 52 and the time stamp in item 21) at the location (facility ID of item 37) that the craftsman corresponds to, the appearance will be Even if the completion of construction is well accepted, it is possible to output the evaluation that the craftsman does a sloppy job that will soon be useless.

5. Monitor appropriate consideration Obtain the mean and variance of similar construction amounts.
By sorting by the facility type ID of item 33, the work name of item 58, the person in charge of the request of item 102, and the cost information of item 90, the costs related to the construction target part for a specific craftsman are totaled. Therefore, it is possible to output an appropriate price for each work.
Usually, the construction estimate is often passed as a set, and the work particle size is large, so it was not possible to analyze the price of individual work. Also, I didn't know how much the craftsman was working elsewhere.
However, in the transaction information 900 of this embodiment, all the operations are stored in a fine particle size. Therefore, by analyzing these individually, for example, even if the packing of the same water pipe is replaced, the craftsman A averages. It is possible to visualize that the average price of craftsman B is 100,000 yen for 30,000 yen.

6. Human scoring It is possible to score people and things by combining multiple analyzes.
For example, it is possible to score each craftsman by parameterizing the three parameters of "work price", "work quality", and "response speed" of the craftsman and evaluating each craftsman.
As for the "work price", as derived in Analysis 5, the degree of deviation from the average price can be scored. For example, since the price difference differs greatly depending on the type of work, the deviation value is used for scoring.
“Construction quality” can be scored based on the quality of work as described in Analysis 4.
The "response speed" can be scored by approximating the "site" and "whether the craftsman's office is close", for example. At this time, the availability of the schedule may be taken into consideration.
According to these, for example, the parameters of each function are optimized for a combination such as price (the deviation value that becomes higher when it is cheaper) x quality (score that becomes lower when it is worse) / response speed (score that rises slowly as it gets farther). As a result, it is possible to put out a craftsman who is currently suitable for the construction with one score.

7. Human score based on relationships between humans Although the "human score" was calculated in Analysis 6 above, it can also be scored in consideration of the relationships between multiple tasks.
The main DB 220 stores information indicating the relevance of the work, such as "who" in items 92 to 99 and "who" in items 100 to 103. By using this information and the contents of the work of item numbers 57 to 83, for example, information that A to B "ordered", information that A to B "obtained information", and A to B You can get information such as "evaluated".

It is possible to perform graph analysis of the importance of each node and edge by giving the relationship between A and B as a directed graph and performing numerical processing. For example, according to the idea of Pagerank of Google (registered trademark), the correlation between each person is recursively calculated assuming that the evaluation from highly reliable humans is high and conversely the evaluation from unreliable humans is low. It is possible to know with higher reliability that you are a "reputable" resident or a craftsman who "does a reliable job".
As a result, instead of the craftsmen, landlords, residents, and traders who are pushed by the "recommendations" of "unreliable" organizations, the craftsmen, landlords, residents, and traders who are "reliable or trusted by everyone" It is possible to build a fair evaluation system that raises the evaluation value.

8. Derivation of the predicted construction period For each item ID of item No. 10, the difference between the time stamp at the time of initial creation of item No. 22 and the time stamp of item No. 21 in which the completion report ID of item No. 15 is described is obtained to "actually". It is possible to calculate the "construction period".
Here, it is possible to calculate the scale and cost of construction work by aggregating the bid price (billed amount) of item No. 86 and the cost of item No. 90 for each project.

By sorting this actual construction period by the creator ID (person in charge of arranging construction) of item number 95, list data of project ID, start time, actual construction period, person in charge, offer price, and cost is calculated. It becomes possible.
If you analyze these factors and find that there is a correlation between the person in charge and other factors, for example, if you enter the name of the person in charge, the construction period, the offer price, the cost, etc. will be predicted accordingly. It is possible to find the value. In addition, if the correlation between the person in charge and other factors is obtained by taking into account the time factor, the person in charge and the start time can be entered to enter the construction period and start time with the seasonal factor taken into consideration. It is possible to obtain predicted values such as values and costs.

9. Derivation of construction period and cost trends The enormous number of construction cases stored in transaction information 900 is categorized, and the “construction period” and “cost” trends for each construction type are derived. For example, by performing unsupervised learning (clustering) when the relationship between these factors is unknown, it is possible to know the tendency of construction that differs greatly depending on the seasonality and the dependence of other factors, and to obtain an appropriate construction period and price. It can be derived.

10. Outlier detection For the above results such as construction period and cost, derive the standard threshold value of the factor that has a great influence on the result in the main DB 220, and constantly monitor whether this threshold value is exceeded. It is possible to create a mechanism.

For example, the threshold value X such as "If the construction arrangement is not completed for X weeks or more, the human has forgotten the arrangement" is clarified by the above analysis, and the difference between a certain item and a certain item in the main DB 220 is clarified. It can continue to monitor and display an alert if it exceeds the threshold X. In this mechanism, since the threshold value X itself is acquired from the analysis of transaction information 900, it is possible to set a threshold value close to the value that must be conscious in actual construction and work, and a service close to the actual human sense can be provided. It will be possible to provide.

In this embodiment, a method of generating a three-dimensional space model of a building relatively easily using a mobile terminal will be described. The same components as those in the previous embodiment are designated by the same reference numerals, and only the differences between them will be described.

With AR (Augmented Reality) technology, which superimposes and displays additional information in real space, characters, images, and images related to the object being viewed by the user are displayed on a transparent display or smartphone screen. Etc. can be superimposed and displayed. Further, it is possible to add related information (metadata) such as text, icon or animation called an annotation to the data as an annotation, and virtual objects of various forms can be used as the annotation.

Annotations are usually placed in the AR space based on the recognition of the three-dimensional structure of the real space reflected in the image by the SLAM (Simultaneous Localization and Mapping) method and the SfM (Structure from Motion) method. In the SLAM method, recognition of the position of the feature point and recognition of the position and orientation of the camera in the environment are simultaneously executed by using a set of feature points that are dynamically updated in response to a change in the input image. In the SfM method, the parallax is calculated from the positions of the feature points appearing in a plurality of images captured from different viewpoints, and the environment is recognized based on the calculated parallax.

This system provides a mechanism that can centrally manage all information related to people, things, and things from the time of new construction of a building to the demolition. For example, regarding a series of construction projects, transaction information such as where, what equipment, what is happening, what should be done, who requested what, when, and how much is sometimes It is equipped with a database that accumulates in a series, and provides a mechanism for accumulating and sharing information by accepting input from management companies, construction shops, material wholesalers, tenants, intermediaries, construction managers, developers, workers, call centers, etc. There is.

Information on things, which is information about buildings, is a three-dimensional structure, and it may be easier to understand visually if it is shared, viewed, and edited as a three-dimensional space model.
For buildings, three-dimensional modeling software called BIM (Building Information Modeling) data is used to memorize building shapes, spatial relationships, geographical information, and the quantity and characteristics of building members. Based on this BIM data. A three-dimensional space model of a building does not always exist for all buildings.

Therefore, in this embodiment, a mobile terminal having a camera unit is used to recognize the three-dimensional structure of the building, and although it is not a complete three-dimensional model of the building, it is relatively easy to model the three-dimensional space of the building. Provide a system to generate and share.

FIG. 29 is an example of a configuration diagram of the entire management system 1.
The management system 1 includes a plurality of construction company terminals 102, a plurality of manufacturer terminals 103, a plurality of management company terminals 104, a plurality of resident terminals 105, a plurality of other contractor terminals 106, a plurality of call center terminals 107, and a plurality of worker terminals. 108 are provided, each of which is connected to the integrated management system 101 via a network. The network may be wired or wireless, and each terminal can send and receive information via the network.

Each terminal of the management system 1 and the integrated management system 101 may be, for example, a mobile terminal (mobile terminal) such as a smartphone, a tablet, a mobile phone, or a personal digital assistant (PDA), or may be a glasses type, a wristwatch type, a clothing type, or the like. It may be a wearable terminal of. It may also be a stationary or portable computer, or a server located in the cloud or on a network. Further, the function may be a VR (Virtual Reality) terminal, an AR terminal, or an MR (Mixed Reality) terminal. Alternatively, it may be a combination of these plurality of terminals. For example, a combination of one smartphone and one wearable terminal can logically function as one terminal. Further, it may be an information processing terminal other than these.

The construction company terminal 102 is a terminal used by developers, construction companies, construction shops, craftsmen, and the like.
The maker terminal 103 is a terminal used by a material maker, a material wholesaler, or the like.
The management company terminal 104 is a terminal used by a management company or an intermediary.
The resident terminal 105 is a terminal used by the resident or the owner of the building.
The other contractor terminal 106 is a terminal used by a contractor other than the above, which intervenes between the time of new construction of the building and the time of demolition.
The call center terminal 107 is a terminal used by the call center.
The worker terminal 108 is a terminal used by a worker who works at a building such as a developer, a construction company, a construction shop, or a craftsman.

FIG. 65 is an example of communication using a three-dimensional space model. Each terminal shares a three-dimensional space model via a communication module, and can communicate as if they were in the same three-dimensional space. For example, the resident terminal 105 creates a simple three-dimensional space model for a malfunction of equipment in the building and shares it with the call center terminal 107. The resident and the person in charge of the call center can consult about the problem while looking at the 3D space model at the same time.

Each terminal of the management system 1 and the integrated management system 101 have a processor that executes an operating system, an application, a program, etc., a main storage device such as a RAM (RandomAccessMemory), an IC card, a hard disk drive, and an SSD (Solid). StateDrive), auxiliary storage devices such as flash memory, communication control units such as network cards, wireless communication modules, and mobile communication modules, touch panel, keyboard, mouse, voice input, input by motion detection by imaging the camera unit, GPS , Gyro, acceleration sensor and other input devices, and monitor, display, speaker, headphone, vibrator and other output devices. The output device may be a device or a terminal for transmitting information for output to an external monitor, display, printer, device, or the like.

FIG. 30 is an example of the hardware configuration of the integrated management system 101.
The BIM management module 3013 inputs, edits, and manages BIM data 3052 such as markup data 3800 and 3D shape data 3900.
The central communication module 3014 transmits and receives information stored in building data 3051, BIM data 3052, transaction information 900, and various DBs to and from communication modules of other terminals, and cooperates with each other.

The building data 3051 is building data 3400, section data 3500, block data 3600, equipment data 3700, and the like, and attribute information of the building is stored.
The BIM data 3052 is markup data 3800 and 3D shape data 3900, and stores three-dimensional related data for displaying a three-dimensional space model.

FIG. 31 is an example of the hardware configuration of the construction company terminal 102.
The markup management module 3121 inputs, edits, and manages the markup data 3800.
The 3D shape management module 3122 inputs, edits, and manages 3D shape data 3900.

The communication module 3123 transmits and receives building data 3151 and BIM data 3152 to and from the central communication module 3014 and other communication modules, and cooperates with each other.
The BIM data 3152 is markup data 3800 and 3D shape data 3900, and stores three-dimensional related data for displaying a three-dimensional space model.
The manufacturer terminal 103, the management company terminal 104, the other contractor terminal 106, and the call center terminal 107 have the same configuration as the construction company terminal 102.

FIG. 32 is an example of the hardware configuration of the worker terminal 108.
The worker terminal 108 is composed of, for example, a smartphone or a tablet terminal.
The main storage device 3201 stores the integrated management system cooperation module 3210, the SLAM module 3211, the plane detection module 3212, the markup creation module 3221, the 3D shape creation module 3222, and the communication module 3223, and stores these programs and applications. Each functional element of the worker terminal 108 is realized by the execution by the processor.
The resident terminal 105 has the same configuration as the worker terminal 108 or the construction company terminal 102.

The integrated management system cooperation module 3210 has terminal information management data 3220, environmental map data 3240, building data 3251, BIM data 3252, and other information acquired and managed by the worker terminal 108 stored in the auxiliary storage device 3202. Is transmitted to the integrated management system 101 regularly, every time there is an update of information, or at an arbitrary timing. Further, instead of a form like an application, the integrated management system 101 may be simply accessed via a Web browser and linked with the integrated management system 101 to transmit information using the function of the Web browser. ..

The SLAM module (self-position estimation / environment map creation module) 3211 recognizes the position of a feature point and uses a set of feature points that are dynamically updated in response to changes in the input image to recognize the position of the feature point and the camera unit 3206 in the environment ( Alternatively, the position and posture (orientation) of the worker terminal 108) are recognized.
The plane detection module 3212 detects a plane from the feature points extracted from the image taken by the camera unit 3206, and creates a plane polygon. As the captured image changes, the detected plane changes, and the plane polygons are sequentially updated.

The 3D shape creation module (3D shape creation module) 3222 performs image processing based on the image taken by the camera unit 3206 and the information of the SLAM module 3211 and the plane detection module 3212, and performs image processing on the building (building, section, block, etc.). Create a 3D shape and 3D space model of equipment, etc.).
The markup creation module 3221 creates and manages markup data created in association with 3D shape data.

The communication module 3223 transmits and receives environment map data 2340, building data 3251, and BIM data 3252 to and from the central communication module 3014 and other communication modules, and cooperates with each other.
The environment map data 3240 stores the coordinates in the three-dimensional space as common world coordinates between different terminals.

FIG. 33 is an example of the data structure 3300 of the building data.
The building data has a hierarchical structure, and there are a plurality of sections 3302 in the building 3301, a plurality of blocks 3303 in the section 3302, and a plurality of facilities 3304 in the block 3303. Building is a term that includes buildings, plots, blocks, and equipment. In addition, these data may be collectively referred to as building data or building data.
The connection of the hierarchical structure between the data may be described in each data, or may have information for managing the hierarchical structure different from each data.

Further, the building data is not necessarily limited to these four layers, and may have a structure in which some of the layers are not present, or conversely, a hierarchical structure having more than four layers may be used.
Further, the names are not limited to buildings, sections, blocks, and equipment, and may be called differently, or may be further divided within a specific floor. For example, even if there are divisions for each floor such as the 1st floor, 2nd floor, and 3rd floor in the building, and there are divisions for each room such as room 101, room 102, and room 103 below that. Good.

FIG. 34 is an example of building data 3400.
The building data 3400 stores information about the building.
For the building type of item No. 3, the information of the thing-related master information 600 is referred to.
In the latitude, longitude, and altitude of items 5 to 7, the latitude, longitude, and altitude of the reference position (for example, the entrance) of the building are described. For example, the latitude, longitude, and altitude can be input based on the map information, and the latitude, longitude, and altitude acquired from GPS or the like of the mobile terminal can be input.
As for the front direction of item No. 8, for example, if the entrance of the building is used as a reference, the direction of entering the entrance is the front direction, and the direction of the front is remembered clockwise with the north as 0 degrees. To do.

FIG. 35 is an example of partition data 3500.
The section data 3500 stores information about a section one level below the building.
For the section types and floor plans of items 3 and 4, refer to the information of the mono-related master information 600.
The positions XYZ of item numbers 5 to 7 store the relative positions with respect to the reference position (for example, the entrance) of the building data 3400 one level higher.
As the front direction of the item No. 8, the relative angle from the front direction of the item No. 8 which is the reference direction of the building data 3400 one level higher is stored as a reference.

FIG. 36 is an example of block data 3600.
The block data 3600 stores information about a block one level below the block.
For the block type of item No. 3, the information of the thing-related master information 600 is referred to.
The positions XYZ of item numbers 4 to 6 store relative positions with reference to the reference position of the division data 3500 one level higher.
As the front direction of the item No. 7, the relative angle from the front direction of the item No. 8 which is the reference direction of the division data 3500 one level higher is stored.

FIG. 37 is an example of equipment data 3700.
The equipment data 3700 stores information about equipment one level below the block.
For the equipment type of item No. 3, the information of the thing-related master information 600 is referred to.
The positions XYZ of item numbers 8 to 10 store relative positions with reference to the reference position of the block data 3600 one layer above.
As the front direction of the item No. 11, the relative angle from the front direction of the item No. 7 which is the reference direction of the block data 3600 one layer above is stored as a reference.

Each item of the building data of FIGS. 34 to 37 may be stored in the transaction information 900, or may be constructed as data separate from the transaction information 900 and referred to from the transaction information 900. May be good.

FIG. 38 is an example of markup data 3800.
The markup data 3800 stores the position and orientation of a specific object in the three-dimensional space model, and related images and comments. In this system, the information is stored in the form of a JSON file, but other formats may be used.
In the example of FIG. 38, a mark is made for a camera markup indicating the position of the camera in which the image was taken in the 3D space model of a specific block (for example, a room) or a pin markup for marking in the 3D space model. An example of storing the updater is shown.

The type of item No. 2 indicates whether the markup is for the camera markup or the pin markup.
Items 3 and 4 store information about equipment related to markup, and the equipment ID refers to mono-related master information 600. Equipment that is not stored in the product-related master information 600 can be entered in the other equipment names in item No. 4.
The positions XYZ of item numbers 6 to 9 store relative positions with reference to the reference position of the block one level above.

The rotations of items 10 to 14 store the quaternion so that the three-dimensional direction can be expressed.
An explanation for each markup can be entered in the explanation of item No. 18.
Information for a plurality of images taken in association with the markup is stored in the attached images of items 20 to 30. For example, for an air conditioner, which is a facility in a three-dimensional space model, a plurality of images such as an entire image, a partial image, and a model number portion of the air conditioner can be stored in association with each other.

FIG. 39 is an example of 3D shape data 3900.
The 3D shape data 3900 is information for expressing a 3D shape, and is stored in, for example, a glTF format for expressing a 3D model or a scene by JSON.
Information about the transaction information 900 managed by the integrated management system 101 is described in the extension area of the gITF format.
The nodes of item numbers 12 to 16 store information about a plurality of plane polygons generated by the plane detection module 3212. The AR plane ID of item No. 16 stores the IDs assigned to each of the plurality of plane polygons.

FIG. 40 is an example of communication data 4000.
Communication data 4000 shows an example of a data structure shared between terminals.
Information on the 3D space model is shared among a plurality of terminals, and interaction is possible as if the user using each terminal is in the same 3D space model. When performing such communication, information on which position and which direction each user is looking at in the three-dimensional space model is shared and displayed. The communication data 4000 stores information such as item numbers 3 to 16 for displaying an avatar in order to express the user's viewpoint and movement.

Next, a process of creating camera markup by the worker terminal 108 will be described.
FIG. 41 is an example of the camera markup display screen 4100.
The worker terminal 108 displays an image (real-time moving image) 4102 captured by the camera unit 3206. When the orientation of the camera unit is changed, the image is updated in real time accordingly, and the image is displayed on the screen of the output device 3205.
The feature points 4103 detected by the SLAM module 3211 and the plane information detected by the plane detection module 3212 from these feature points are superimposed and displayed on the video. The video may be a high-speed moving image of 30 frames or 60 frames per second, or may be a frame-by-frame advance of a still image of about 2 or 3 frames per second. Further, it may be simply a still image instead of a moving image, or it may be a still image and may be configured to be updated when the position or orientation of the camera is changed by a predetermined value or more.

FIG. 57 is an example of the camera markup creation processing flow 5700.
When the camera markup creation button 4101 is selected by tapping the screen by the user or the like, the markup creation module 3221 starts the camera markup creation process 5700.
The markup creation module 3221 receives an image shooting instruction by the user tapping the shooting button 4104 (step 5710), shoots an image by the camera unit 3206, and saves the image (step 5720).

The markup creation module 3221 acquires information indicating the shooting position and orientation in the three-dimensional space of the camera unit 3206 (or the worker terminal 108) at the time of shooting from the SLAM module 3211 (step 5730).
The markup creation module 3221 displays the camera icon 4105, which is a 3D icon image indicating the shooting position and orientation in the corresponding positions and orientations in the three-dimensional space (step 5740). In the example of the display screen 4100 of FIG. 41, the camera icon 4105 in the shape of a quadrangular pyramid is displayed, indicating that the image was taken from the position of the apex of the head of the quadrangular pyramid toward the bottom surface.

The markup creation module 3221 stores the markup data in the auxiliary storage device 3202 in association with the captured image (step 5750).
The markup creation module 3221 further accepts input from the user of a comment on the captured image and the equipment shown therein, and stores this information in the auxiliary storage device 3202 in association with the markup data (step 5760). ..

By displaying the camera markup, it is possible to understand where and in what direction the captured image was taken in the 3D space model.
For example, in the input screen at the time of construction implementation in FIG. 20, after the person in charge inputs information on the input screen 2000 before the construction implementation, the construction completion person uploads a photograph to the input screen 2000 for acceptance inspection. In 2012, the photo before the construction was carried out is posted as Before, and in 2013, the photo after the construction is completed is posted as After.

Here, the person who completed the construction is urged to take a photo for acceptance inspection at almost the same position and shooting direction as the photo 2012 before the construction, while looking at the camera markup of the three-dimensional space model. Going further, from the information on the shooting position and shooting direction of the camera unit 3206 acquired by the SLAM module 3211, the shooting position is almost the same as the shooting position of the previously shot image and almost the same as the shooting direction of the previously shot image. The configuration may be such that a photograph for acceptance inspection is automatically taken when facing the same direction.
Alternatively, the work completion report button 2005 may be clickable for the first time when photographs are taken at substantially the same position and in substantially the same shooting direction.

Note that the substantially same position is determined to be almost the same position when the shooting position of After comes within a predetermined range such as within the range of a sphere having a radius of 1 m from the shooting position of Before. Further, the substantially same direction is determined to be substantially the same angle when the shooting direction of After comes within a predetermined angle such as within a range of an angle of 10 degrees from the shooting direction of Before. Alternatively, a mechanism may be used in which photographs are taken at exactly the same position and in the same shooting direction.

The images taken before and after the construction are sent to the integrated management system 101 by the integrated management system cooperation module 3210, and are stored in association with the transaction information 900 and various DBs.
In addition, for example, in the phase start / end condition information 1200 of FIG. 12, when images at substantially the same position and in substantially the same direction are registered before and after the construction, the phase may be started or ended.

Next, a process of creating pin markup by the worker terminal 108 will be described.
FIG. 42 is an example of the pin markup display screen 4200.
The worker terminal 108 displays an image (real-time moving image) 4202 taken by the camera unit 3206. When the orientation of the camera unit is changed, the image is updated in real time accordingly, and the image is displayed on the screen of the output device 3205.
The feature points 4203 detected by the SLAM module 3211 and the plane information detected by the plane detection module 3212 from these feature points are superimposed and displayed on the video.

FIG. 58 is an example of the pin markup creation processing flow 5800.
When the pin markup creation button 4201 is selected by tapping the screen by the user or the like, the markup creation module 3221 starts the pin markup creation process 5800.
The worker terminal 108 displays an image (real-time moving image) 4202 captured by the camera unit 3206 (step 5810).

The feature points 4203 detected by the SLAM module 3211 and the plane information detected by the plane detection module 3212 from these feature points are superimposed and displayed on the video (step 5820).
The markup creation module 3221 displays a pin icon 4204, which is a 3D icon image pierced perpendicularly to the plane on the detected plane. The pin icon 4204 is movable on a plane and displays a concentric animation 4205 to prompt the user to identify the position.

When the user accepts the selection of a part of the plane by moving the pin icon 4204 (step 5830), the markup creation module 3221 displays the pin icon 4204 in the direction perpendicular to the plane at the position of the selected plane (step 5840). ).
The markup creation module 3221 acquires information indicating the position and orientation of the pin icon 4204 in the three-dimensional space model in addition to the plane information from the plane detection module 3212 (step 5730). It should be noted that the SLAM module 3211 may provide information on the position and orientation of the pin icon 4204 instead of the plane detection module 3212.

The markup creation module 3221 stores the markup data together with the information of the pin icon 4204 in the auxiliary storage device 3202 (step 5850).
The markup creation module 3221 further accepts input from the user of a comment on the captured image and the equipment shown therein, and stores this information in the auxiliary storage device 3202 in association with the markup data (step 5860). ..

FIG. 43 is an example of a display screen of a three-dimensional space model.
A simple three-dimensional space model can be created by pasting an image (texture) taken by the camera unit 3206 on the plane polygon detected by the plane detection module 3212.

In the example of the three- dimensional space model 4300 and 4350, a plurality of plane polygons 4302 and 4352 are displayed as wire frames. On some of these plane polygons, images that have been projected and transformed from the captured image are pasted as textures as textures such as 4303, 4304, 4353, and 4354.
For example, in the example of the three-dimensional space model 4350, the wall texture image 4353 is pasted on the plane polygon of the back wall, and the floor texture image 4354 is pasted on the floor plane polygon.

The integrated management system cooperation module 3210 uploads the three-dimensional space model created through the network to the integrated management system 101.
The construction company terminal 102, the manufacturer terminal 103, the management company terminal 104, the resident terminal 105, the other contractor terminal 106, the call center terminal 107, etc. receive the three-dimensional space model from the integrated management system 101 and display it in 3D on a Web browser or the like. As a result, it becomes possible to grasp the three-dimensional space without going to the site. In addition, by displaying the markup according to the model, it provides the same viewpoint as the site.

Also, by embedding the markup data 3800 and its additional information in the 3D model itself, this simple 3D space model can be used as BIM (Building Information Modeling). This is an approach in the opposite direction to the general use case in which a 3D CAD drawing is created with BIM software when designing a new property. It is an effective means to create a simple 3D space model and utilize it as BIM data even for most properties for which BIM data does not exist.

FIG. 59 is an example of the three-dimensional space model creation processing flow 5900.
When the 3D space model creation button 4301 is tapped, the 3D space model creation process 5900 is started.
The camera unit 3206 captures an image (real-time moving image) (5910).
The plane detection module 3212 detects a plane in the image and creates and displays a plane polygon (step 5920). In the example of the three- dimensional space models 4300 and 4350, the plane polygons 4302, 4352 and the like are represented by wire frames.

In the example of the three- dimensional space models 4300 and 4350, the image captured by the camera unit 3206 is not displayed, and the wire frame is displayed on a black background. On the other hand, it may be configured to display the image being shot.
The 3D shape creation module 3222 accepts the selection of some plane polygons from the user (step 5930), and when it receives the image shooting instruction, shoots an image toward the selected plane (step 5940).

For the selection and shooting of the plane polygon, for example, when the user moves the shooting direction of the camera unit 3206, the plane polygon in the shooting direction becomes active (highlighted) on the screen, so that the plane polygon is selected. After that, it is detected that the shooting buttons 4305 and 4355 are tapped, and an image is shot.
In addition, as a configuration in which an image is taken by tapping one of a plurality of plane polygons displayed on the screen, that is, a plane polygon is selected and a shooting instruction is simultaneously performed with one tap from the user. May be good.

The 3D shape creation module 3222 projects and transforms the captured image to generate a texture image (step 5950), and then pastes the texture image onto the selected planar polygon (step 5960).
The 3D shape creation module 3222 stores 3D shape data 3900 (plane polygon information and texture image information) in the auxiliary storage device 3202.

FIG. 44 is an example of an explanatory view of a plane polygon.
In 3D drawing by GPU (Graphics Processing Unit), polygons are mostly represented by a set of multiple triangles, and the plane detection module 3212 assumed here also outputs plane polygons as a set of multiple triangles. To do. However, it is not limited to this.
One plane polygon can be represented by the following four configurations.
-Three-dimensional vector array (V0 to V3) that stores vertex coordinates (XYZ coordinates)
-Two-dimensional vector array (U0 to U3) that stores texture coordinates (UV coordinates) at each vertex coordinate.
-Scalar array (T0, T1) that stores indexes that store three vertex coordinates in a clockwise (or counterclockwise) order to create a triangle.
・ Texture image

FIG. 60 is an example of the plane polygon creation processing flow 6000.
The camera unit 3206 captures an image (step 6010).
The plane detection module 3212 detects a plane from the feature points extracted from the image taken by the camera unit 3206, and projects the vertices of all the triangles constituting the plane from the three-dimensional coordinates to the two-dimensional coordinates on the camera image ( Step 6020).
The plane detection module 3212 divides one triangle into a plurality of polygons by geometric calculation on a two-dimensional plane (step 6030).
The plane detection module 3212 divides one polygon into a plurality of triangles (step 6040).

The plane detection module 3212 obtains the coordinates (viewport coordinates) on the camera image for the vertices of each triangle (step 6050).
The plane detection module 3212 projects all the vertex coordinates on the two dimensions onto the original three-dimensional plane and returns them to the three-dimensional coordinates (step 6060).
The plane detection module 3212 creates a plane polygon with three-dimensional coordinates (step 6070).

FIG. 45 is an example of a texture.
The image taken from the camera is a deep "perspective image", with the near one being large and the distant one being small. When this perspective image is pasted on a 3D polygon as it is, even if the UV coordinates of the triangle vertices are specified correctly, the perspective correction function provided in the GPU that performs 3D drawing is incorrect. Processing is performed, resulting in an unnatural 3D display.
For example, when the perspective image 4501 is pasted as a texture image on a plane polygon as it is, the quadrangle in the image becomes a distorted shape like 4502, resulting in an unnatural 3D display.

Therefore, the image captured by the camera unit 3206 is subjected to a projective transformation (homography) process in which a portion close to the camera is reduced and a portion far from the camera is enlarged, and then the image is pasted as a texture on a 3D polygon.
For example, a texture image 4503 is generated by performing a projective transformation process on the perspective image 4501. By pasting this texture image 4503 on a plane polygon, a correct 3D display 4504 without distortion can be obtained.

FIG. 61 is an example of the texture image generation processing flow 6100.
The 3D shape creation module 3222 acquires the captured image (step 6110) and transforms the image so as to project the image plane of the camera onto the plane polygon surface in the three-dimensional space (step 6120).
The 3D shape creation module 3222 stores the converted image as a texture image (step 6130).

FIG. 46 is an example of an explanatory diagram of the projective transformation.
(1) The viewport coordinates of the four corners of the unconverted image (original image) obtained by camera shooting are
P _TR = (1, 1) P _BR = (1, 0) P _BL = (0, 0) P _TL = (0, 1)
It is represented by.

(2) Extend the four corners in the direction of the camera's line of sight, and multiply the world coordinates that intersect the plane in the 3D space to which the target triangle (target plane polygon) belongs by the View matrix and Projection matrix of the camera, respectively. Find the W value of (the farther the world coordinates are, the larger the W value becomes).
W _TR W _BR W _BL W _TL
(3) PWn is a value obtained by multiplying the relative coordinates from the viewport coordinates (0.5, 0.5), which is the center of the image, by Wn.
PW _n = (P _n- (0.5, 0.5)) × W _n

(4) P _TR ', P _BR ', P _BL ', P _TL'with the following formula, with PW _{MIN at} the lower left and MW _{MAX at} the upper right of the quadrangle surrounding PW _TR , PW _BR , PW _BR , and PW _BL. Find and normalize to a value between 0 and 1.
_{P n '= ((PW n} .x - PW MIN .x) / (PW MAX .x - PW MIN .x), (PW n .y - PW MIN .y) / (PW MAX .y - PW MIN. y))
(5) Perform image projection transformation (homography) so that P _TR → P _TR ', P _BR → P _BR ', P _BL → P _BL ', and P _TL → P _TL '.
In this way, the converted image without perspective is generated and stored as a texture image.

FIG. 47 is an example of an explanatory diagram for updating a plane polygon.
Since the plane detection module 3212 sequentially adds and expands the detected planes each time the mobile terminal (camera unit 3206) is moved, it is necessary to sequentially update the polygons to which the photographs taken by the camera unit 3206 are pasted. is there.

FIG. 62 is an example of the plane polygon update processing flow 6200.
As the captured image of the camera unit 3206 changes, the plane detection module 3212 expands the plane 4701 to which the texture image is already attached and generates the expanded plane polygon 4702 (step 6210).
When an image is taken for the expanded planar polygon 4702, the 3D shape creation module 3222 generates a new texture image (step 6220).

The 3D shape creation module 3222 adds a new texture image to the extended plane polygon camera imaging range 4704 (step 6230).
Further, the 3D shape creation module 3222 generates an image 4705 in which the camera shooting range is deleted from the plane polygon to which the previous texture image is pasted (step 6240).
The 3D shape creation module 3222 stores 3D shape data 3900 (plane polygon information and texture image information) regarding the updated plane polygon 4706 (step 6250).

FIG. 48 is an example of an explanatory diagram for updating a plane polygon.
Since one polygon is composed of a plurality of triangles, polygon addition and polygon deletion processing are performed for each triangle.
(1) Polygon addition processing For the triangular polygon of the new detection plane shown in the camera, the texture image generated from the captured image is pasted on the part in the camera image, and then divided into multiple triangular polygons. ..

The example of the additional processing screen 4800 is within the camera shooting range in the expanded planar polygon and shows the added additional planar polygon 4805. The three-dimensional space screen 4801 displays the additional plane polygon 4805 from a bird's-eye view in the three-dimensional space. The camera shooting range display 4802 shows an image shot by the camera, and the magnified camera shooting range display 4803 is an enlarged display of the image of the camera shooting range display 4802.
In the magnified camera shooting range display 4803, the right side and the lower side of the original triangular shape of the additional plane polygon 4805 are outside the camera shooting range, and the pentagonal shape portion within the camera shooting range is added as the additional plane polygon 4805. Will be done. On the three-dimensional space screen 4801, the pentagon of the additional plane polygon 4805 is stored as a plane polygon divided into three triangles.

(2) Polygon deletion process The existing triangular polygons captured by the camera are divided into a plurality of triangular polygons by removing the part in the image.
The example of the deletion processing screen 4850 shows the deleted plane polygon 4855 in which the camera shooting range is deleted from the expanded plane polygon. The three-dimensional space screen 4851 displays the deleted plane polygon 4855 from a bird's-eye view in the three-dimensional space. The camera shooting range display 4852 shows an image shot by the camera, and the enlarged camera shooting range display 4853 is an enlarged display of the image of the camera shooting range display 4852.
The deleted plane polygon 4855 captures a part of the area overlapping the upper side of the original triangular shape in the magnified camera shooting range display 4853, and the region of this camera shooting range is deleted as a heptagonal plane. The polygon is stored. On the three-dimensional space screen 4851, the heptagon of the deleted plane polygon 4855 is stored as a plane polygon divided into five triangles.

As explained above, it is possible to generate a 3D space model of a building relatively easily using a mobile terminal. The 3D space model generated here is uploaded to the integrated management system 101, and the markup data of the 3D space model is associated with various information stored in the transaction information 900, the human-related DB221, the thing-related DB222, and the thing-related DB223. It is stored as BIM data 3052 including 3800 and 3D shape data 3900.

The 3D space model is downloaded from the integrated management system 101, and can be shared, displayed, and edited between terminals. 49 to 56 are examples of screen transitions in which, for example, the worker terminal 108 displays building information and browses and edits a three-dimensional space model. An example of the case where the worker terminal 108 displays is described below, but the same processing is performed even when the construction company terminal 102 or the call center terminal displays.

FIG. 49 is an example of the building list screen 4900.
The integrated management system cooperation module 3210 acquires information from the integrated management system 101 and displays list information 4902 of a plurality of buildings. Buildings can be searched by keywords in the search area 4901. When 4903 is selected, the screen transitions to the building details screen 5100. When 4904 is selected, the screen transitions to the building addition / edit screen 5000.

FIG. 50 is an example of the building addition / edit screen 5000.
The integrated management system cooperation module 3210 accepts the input of the information of 5001 to 5006, adds the building information, and transmits / registers it to the integrated management system 101. The latitude and longitude information 5003 of the building is acquired from GPS. The building address 5004 may be entered by the worker himself, or the building address information acquired from the map information by GPS information may be entered by tapping the "from position" button.

FIG. 51 is an example of the building detail screen 5100.
The integrated management system cooperation module 3210 acquires information from the integrated management system 101 and displays the building detail screen 5100. Detailed information 5101 of the building and a photograph or three-dimensional image 5102 of the building are displayed on the upper part. In addition, the list information 5105 of the section for a specific building is displayed. The sections are grouped into several groups 5104, and are displayed in groups such as the first floor and the second floor.
When 5103 is selected, the screen changes to 5000 on the building edit screen. Selecting 5106 transitions to the building details screen. When 5107 is selected, the screen transitions to the section addition / edit screen.

The integrated management system cooperation module 3210 acquires information from the integrated management system 101 and displays a list information of blocks for a specific section in the same manner as when the building detail screen 5100 is displayed. Detailed information of the section and a photograph or a three-dimensional image of the section are displayed at the upper part. The blocks are grouped together. When a specific block is selected from the block list information, the block detail screen 5300 is displayed.

FIG. 52 is an example of the block addition / edit screen 5200.
The integrated management system cooperation module 3210 accepts the input of the information of 5201 to 5205, adds the block information, and transmits / registers it to the integrated management system 101. Information on the upper section 5202 of the block is also registered so that the hierarchical structure can be understood.

FIG. 53 is an example of the block detail screen 5300.
The integrated management system cooperation module 3210 acquires information from the integrated management system 101 and displays the block detail screen 5300. The detailed information 5301 of the block and the photograph or the three-dimensional image 5302 of the block are displayed on the upper part. In addition, the list information 5305 of the equipment for a specific block is displayed. The equipment is grouped into several groups 5304, which are grouped and displayed, for example, interior and equipment.

Select 5303 to move to the block edit screen. When 5306 is selected, the screen transitions to the markup detail screen 5400 showing the details of the equipment. When the AR edit button 5307 is selected, the screen transitions to the AR information edit screen 5600. When the 3D browsing button 5310 is selected, the screen transitions to the 3D space model browsing screen 5500.

FIG. 54 is an example of the markup detail screen 5400.
The integrated management system linkage module 3210 acquires information from the integrated management system 101 and displays the markup detail screen 5400. Detailed markup information 5401 is displayed at the top. Detailed information about the markup 5401 and a photograph or three-dimensional image of the markup 5402 are displayed at the upper part. It also displays list information 5405 of images and comments stored for a particular markup. Selecting 5403 transitions to the markup edit screen. When 5406 is selected, for example, a stored image is read out and displayed.

FIG. 55 is an example of the three-dimensional space model viewing screen 5500.
The integrated management system cooperation module 3210 acquires BIM data 3052 including building data 3051 and 3D space model markup data 3800 and 3D shape data 3900 from the integrated management system 101, and displays the 3D space model viewing screen 5500. indicate.

A plurality of camera markups 5502 and 5503 and a plurality of pin markups 5504 and 5505 are displayed on the three-dimensional space model 5501. Markup details 5510 displays comment and photo information related to markup. For example, when pin markup 5504 is selected, comments about pin markup and corresponding to it are displayed as in the display example. A plurality of photographs 5512 and 5513 stored in the above are displayed.
When the VR button 5520 is selected, the mode shifts to the VR viewing mode, and the three-dimensional space model can be displayed in VR. For example, the screen is divided into two left and right, and images with parallax for the left eye and the right eye are displayed respectively.

In FIG. 28, the transaction information 900 is searched, common factors are discovered in order of hierarchical items from the one with the smallest granularity to the one with the largest granularity, and it is determined in which layer the items the singular information appears. explained.
Here, for a plurality of data of the items in which the common factor is found, the building data 3051 and the BIM data 3052 are acquired and displayed in one three-dimensional space model. By doing so, it becomes possible to identify which of the plurality of items in which the common factor was found was the problem while browsing the three-dimensional space model.
This is a problem especially when identifying singular information such as problems in the same three-dimensional space model for multiple common factors that are separated in time, or for a remote location such as a call center. This is effective when displaying peculiar information such as points.

FIG. 56 is an example of the AR information editing screen 5600.
The AR information editing is executed by the markup creation module 3221 and the markup management module 3121 in cooperation with the 3D shape creation module 3222 and the 3D shape management module 3122.
For example, a pin markup 5602 is displayed on the three-dimensional space model 5601. When this pin markup 5602 is selected, the markup details 5650 are displayed as a pop-up screen in the AR information editing screen 5600, and editing is possible.
When the color change button 5630 is selected, a pop-up screen for changing the color displayed on the pin markup is displayed. When the equipment selection button 5620 is selected, the equipment can be selected from the information stored in the thing-related DB 222.
The markup details 5650 displays comment and photo information related to the markup. For example, when pin markup 5602 is selected, a comment about the pin markup and a correspondence to the comment are displayed as in the display example. A plurality of photographs 5651 stored in the above are displayed.

FIG. 63 is an example of the three-dimensional space model browsing processing flow 6300.
Although it is described as being executed by the construction company terminal 102, it may be executed by another terminal.
The integrated management system linkage module 310 acquires building data 3051 and BIM data 3052 from the integrated management system 101 (step 6310).
The integrated management system linkage module 310 displays list information regarding buildings, sections, blocks, and equipment based on the acquired building data 3051 (step 6320). It accepts a selection from the user and displays a 3D spatial model based on the BIM data 3052 for the selected item (step 6330).

FIG. 64 is an example of the AR information editing process flow 6400.
Although it is described as being executed by the construction company terminal 102, it may be executed by another terminal.
The integrated management system linkage module 310 acquires building data 3051 and BIM data 3052 from the integrated management system 101 (step 6310).
The integrated management system linkage module 310 accepts the selection of the three-dimensional spatial model data and / or related information to be edited (step 6420).
The markup management module 3121 and / or the 3D shape management module 3122 displays an edit screen for BIM data and / or related information for the selected item (step 6430), and stores the edited contents (step 6440).

The present invention is not limited to the above-mentioned examples, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means, etc. may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

In addition, the control lines and information lines indicate what is considered necessary for explanation, and not all control lines and information lines are necessarily shown on the product. In practice, it can be considered that almost all configurations are interconnected.

101 ... Integrated management system, 102 ... Construction management company terminal, 103 ... Manufacturer terminal, 104 ... Management company terminal, 105 ... Resident terminal, 106 ... Other vendor terminal, 107 ... Call center terminal, 108 ... Worker terminal, 210 ... Data Registration display module, 211 ... Status determination module, 212 ... Data analysis module, 220 ... Main DB, 221 ... Human-related DB, 222 ... Mono-related DB, 223 ... Things-related DB, 500 ... Human-related master information, 600 ... Mono-related Master information, 700 ... Koto-related master information, 800 ... Phase-related master information, 900 ... Transaction information, 1200 ... Phase start / end condition information, 1300 ... Phase progress / rejection judgment information, 3210 ... Integrated management system cooperation module, 3211 ... SLAM module , 3212 ... Plane detection module, 3221 ... Markup creation module, 3222 ... 3D shape creation module, 3223 ... Communication module

Claims (21)

1. It ’s a mobile device,
   The camera unit that takes images and
   A plane detection unit that detects a plurality of planes from an image taken by the camera unit and generates a plurality of plane polygons,
   A 3D shape creation unit that generates a 3D space model by performing image processing,
   Display and
   Have,
   The three-dimensional shape creation unit
   Accepting the selection of at least one plane polygon from the plurality of plane polygons,
   The second image corresponding to the selected plane polygon taken by the camera unit is acquired, and the second image is acquired.
   A mobile terminal characterized in that a three-dimensional space model is generated by pasting a texture image based on the second image on the selected plane polygon.
2. The mobile terminal according to claim 1.
   The camera unit captures a third image and
   In association with the third image, the shooting position and shooting direction of the camera unit in the three-dimensional space are stored.
   A mobile terminal characterized in that an icon image indicating the shooting position and the shooting direction is displayed on the three-dimensional space model.
3. The mobile terminal according to claim 2.
   A mobile terminal characterized in that the third image is displayed in association with the icon image indicating the shooting position and the shooting direction.
4. The mobile terminal according to claim 2 or 3.
   When the camera unit faces a position substantially the same as the shooting position and a direction substantially the same as the shooting direction, the camera unit shoots a fourth image.
   A mobile terminal characterized in that the third image and the fourth image are stored in association with each other.
5. The mobile terminal according to claim 2 or 3.
   When the position of the camera unit is within a predetermined range from the shooting position and the direction of the camera unit is within a predetermined angle from the shooting direction, the camera unit captures a fourth image.
   A mobile terminal characterized in that the third image and the fourth image are stored in association with each other.
6. The mobile terminal according to any one of claims 1 to 5.
   Accepts the selection of a part of the detected multiple planes,
   A mobile terminal characterized in that the position in the three-dimensional space of the selected part is stored in association with the input information.
7. The mobile terminal according to any one of claims 1 to 6.
   The three-dimensional shape creation unit
   A mobile terminal characterized in that the second image is converted to generate the texture image so that the imaging surface of the camera unit is projected onto the surface of the selected plane polygon in a three-dimensional space.
8. The mobile terminal according to any one of claims 1 to 7.
   The three-dimensional shape creation unit
   A mobile terminal characterized in that an image in a range corresponding to the selected plane polygon is cut out from the texture image and pasted on the plane polygon.
9. The mobile terminal according to any one of claims 1 to 8.
   The three-dimensional shape creation unit
   When the plane polygon to which the texture image is already pasted is expanded by the plane detection unit, a new texture image is pasted in the shooting range of the camera unit within the expanded plane polygon. Mobile terminal.
10. The mobile terminal according to any one of claims 1 to 9.
    The three-dimensional shape creation unit
    When the plane polygon to which the texture image is already attached is expanded by the plane detection unit, the camera unit is included in the expanded plane polygon among the plane polygons to which the texture image is already attached. A mobile terminal characterized by deleting the plane polygon in a portion overlapping the shooting range of.
11. It is a 3D data management method for buildings.
    Detects multiple planes from the image taken by the camera unit, generates multiple plane polygons, and creates
    Accepting the selection of at least one plane polygon from the plurality of plane polygons,
    The second image corresponding to the selected plane polygon taken by the camera unit is acquired, and the second image is acquired.
    A three-dimensional data management method characterized in that a three-dimensional space model is generated by pasting a texture image based on the second image on the selected plane polygon.
12. The three-dimensional data management method according to claim 11.
    The camera unit captures a third image and
    In association with the third image, the shooting position and shooting direction of the camera unit in the three-dimensional space are stored.
    A three-dimensional data management method characterized in that an icon image indicating the shooting position and the shooting direction is displayed on the three-dimensional space model.
13. The three-dimensional data management method according to claim 12.
    A three-dimensional data management method characterized in that the third image is displayed in association with the icon image indicating the shooting position and the shooting direction.
14. The three-dimensional data management method according to claim 12 or 13.
    When the camera unit faces a position substantially the same as the shooting position and a direction substantially the same as the shooting direction, the camera unit shoots a fourth image.
    A three-dimensional data management method characterized in that the third image and the fourth image are stored in association with each other.
15. The three-dimensional data management method according to claim 12 or 13.
    When the position of the camera unit is within a predetermined range from the shooting position and the direction of the camera unit is within a predetermined angle from the shooting direction, the camera unit captures a fourth image.
    A three-dimensional data management method characterized in that the third image and the fourth image are stored in association with each other.
16. The three-dimensional data management method according to any one of claims 11 to 15.
    Accepts the selection of a part of the detected multiple planes,
    A three-dimensional data management method characterized in that the position of the selected part in the three-dimensional space is stored in association with related information.
17. The three-dimensional data management method according to any one of claims 11 to 16.
    Three-dimensional data management characterized in that the second image is converted to generate the texture image so that the imaging surface of the camera unit is projected onto the surface of the selected plane polygon in the three-dimensional space. Method.
18. The three-dimensional data management method according to any one of claims 11 to 17.
    A three-dimensional data management method characterized in that an image in a range corresponding to the selected plane polygon is cut out from the texture image and pasted on the plane polygon.
19. The three-dimensional data management method according to any one of claims 11 to 18.
    When the plane polygon to which the texture image is already pasted is expanded, the three-dimensional data management characterized by pasting a new texture image in the shooting range of the camera unit within the expanded plane polygon. Method.
20. The three-dimensional data management method according to any one of claims 11 to 19.
    When the plane polygon to which the texture image is already pasted is expanded, the plane polygon to which the texture image is already pasted overlaps with the shooting range of the camera unit within the expanded plane polygon. A three-dimensional data management method characterized by deleting the plane polygon of a portion.
21. It is an information management program for buildings
    A program for causing a mobile terminal to execute each step of the three-dimensional data management method according to any one of claims 11 to 20.
    
    

Images