JP2018128894A - Image display controller and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display controller capable of intuitively recognizing a delay in an enforcement plan.SOLUTION: An image display controller 1 includes: a data acquisition unit 111 configured to acquire data of a three-dimensional object stored in a storage device; a captured image acquisition unit 114 configured to acquire an image captured by an imaging apparatus; a display control unit 115 configured to synthesize the image acquired by the captured image acquisition unit 114 with an image of the three-dimensional object obtainable based on the data acquired by the data acquisition unit 111 and cause a display device to display the synthesized image; and a date/time setting unit 112 configured to set date and time. The data of the three-dimensional object includes data representing each constituent element. The data of each constituent element includes three-dimensional position data and date-and-time data. The display control unit 115 synthesizes the image of the three-dimensional object on the basis of the constituent elements in which date/time data indicates date and time preceding the date and time set by the date/time setting unit 112.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image display control apparatus that displays an image based on augmented reality, and a program for realizing the image display control apparatus.

  A technique called augmented reality (AR) has been proposed in which a composite image obtained by combining an object that does not actually exist with a real landscape image captured by a camera is displayed in real time. For example, Patent Document 1 describes a simulation apparatus that combines and displays an image of a building under consideration for a landscape image including a planned building location of the building.

  In construction work and civil engineering work, it is necessary to manage delays in implementation plans and costs. By using the AR technology using the data of a three-dimensional model such as a building created at the design stage, it is possible to synthesize and display an image of the building planned for construction on the scene image at the site. However, it is not clear from the composite image whether the current construction is proceeding as planned, how late it is, and whether the cost to date is reasonable. It is necessary to confirm the schedule of the execution plan and the estimate.

JP 2014-115957 A

  The present invention has been made in view of the above problems, and provides an image display control device and an image display control device capable of intuitively recognizing a delay in an execution plan and a large amount of costs. The purpose is to provide a program for realizing this.

  The program provided by the first aspect of the present invention includes a data acquisition unit that acquires data of a three-dimensional object stored in a storage device, and a captured image acquisition unit that acquires an image captured by the imaging device. And a display control means for combining the image acquired by the captured image acquisition means with the image of the three-dimensional object based on the data acquired by the data acquisition means and displaying it on the display device, and setting the date and time or the total cost The data of the three-dimensional object includes data for each component, and the data of each component includes three-dimensional position data and date / time data or cost data, The display control means is the date and time set by the date and time data and the setting means, or set by the expense data and the setting means. Based on the cost, and wherein the combining images of the three-dimensional object.

  The “component” of the present invention is an element that constitutes a three-dimensional object, and indicates, for example, a polygon that is a minimum unit of drawing, such as a polygon. An element obtained by combining a plurality of polygons is also included in the “component”. Each point constituting the three-dimensional object is also included in the “component”.

  In a preferred embodiment of the present invention, the display control unit synthesizes the image of the three-dimensional object based on a component indicating that the date / time data is before the date / time set by the setting unit.

  In a preferred embodiment of the present invention, the data of each component further includes number data indicating the order, and the display control means accumulates the cost data of the components in the order of the number data. The image of the three-dimensional object is synthesized based on each component before the integrated value exceeds the total cost set by the setting means.

  In a preferred embodiment of the present invention, the program causes the computer to further function as imaging information acquisition means for acquiring imaging information indicating a position and an imaging direction of the imaging device, and the display control means includes the imaging A two-dimensional image obtained by projecting the three-dimensional object is synthesized based on the information.

  In a preferred embodiment of the present invention, the date and time set by the setting means is the current date and time.

  In a preferred embodiment of the present invention, the image of the three-dimensional object synthesized with the image acquired by the captured image acquisition means is translucent.

  In a preferred embodiment of the present invention, the three-dimensional object is a building.

  The image display control device provided by the second aspect of the present invention is a program storage unit storing the program provided by the first aspect of the present invention, and processing based on the program stored in the program storage unit And a control unit for executing the above.

  According to the present invention, the image of the three-dimensional object is combined with the image captured by the imaging device based on the date / time data and date / time, or the cost data and the total cost, and displayed on the display device. Therefore, an image of a three-dimensional object corresponding to the set date and time or total cost can be combined with an actual image and displayed. If the date and time data of each component is set according to the enforcement plan, and the current date and time are set, the image of the part that is scheduled to be completed by the current date and time is combined with the actual image and displayed. The From the difference between the image of the planned completion part and the actual image, it is possible to intuitively recognize the delay in the execution plan. In addition, if the cost data of each component is set according to the estimate and the total cost that has been taken up to now is set, the image of the part that is scheduled to be completed by the total cost will be the actual image Is synthesized and displayed. From the difference between the image of the planned completion part and the actual image, it is possible to intuitively recognize the amount of the cost.

  Other features and advantages of the present invention will become more apparent from the detailed description given below with reference to the accompanying drawings.

It is a figure for demonstrating the synthesized image which the image display control apparatus which concerns on 1st Embodiment produces | generates. It is a functional block diagram which shows an example of an internal structure of the image display control apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the three-dimensional image data which concerns on 1st Embodiment. It is an example of the flowchart which shows AR display processing. It is another example of the flowchart which shows AR display processing. It is a functional block diagram which shows an example of an internal structure of the image display control apparatus which concerns on 2nd Embodiment. It is a figure which shows an example of the three-dimensional image data which concern on 2nd Embodiment. It is a figure which shows an example of the display screen of an image display control apparatus. It is an example of the flowchart which shows a cost calculation process. It is a functional block diagram which shows an example of an internal structure of the image display control apparatus which concerns on 3rd Embodiment. It is a figure for demonstrating the system containing the image display control apparatus which concerns on 4th Embodiment, (a) is a functional block diagram which shows an example of the internal structure of the said system, (b) is an example of a head mounted display It is a figure which simplifies and shows.

  Hereinafter, a preferred embodiment of the present invention will be specifically described with reference to the accompanying drawings.

  The image display control apparatus 1 according to the first embodiment uses the AR technology to synthesize and display an image of a building planned for construction on a scene image at the site. In the present embodiment, an image of only a part scheduled to be completed at the current stage of the building is synthesized and displayed. For example, the 8-story building scheduled for construction shown in Fig. 1 (a) is scheduled to be completed on the first to third floors at the current stage (see Fig. 1 (b)). When only the first and second floor portions are completed (see FIG. 1C), an image in which the first to third floor portions are completed is combined with the actual first and second floor portions (FIG. 1D). reference). This composite image makes it possible to intuitively recognize that the 3rd floor part, which should have been completed by the schedule, is not yet completed, and that the actual construction is delayed compared to the implementation plan. . In addition, in FIG. 1, although the simplified building is taken as an example, it has a more complicated structure in practice. Moreover, the enforcement plan may be set not for each floor but for each section on the same floor and for each element (wall, floor, ceiling, window, door, etc.) constituting the building. In this case, only the sections and elements scheduled to be completed in the implementation plan are synthesized.

  The image display control device 1 is realized by installing an AR display program in a general-purpose computer. The AR display program is a program for causing the control unit 11 to execute an AR display process described later. In the present embodiment, an image display control device 1 is an image display control device 1 in which an AR display program is installed in a tablet computer equipped with an imaging device such as a camera, a display device such as a liquid crystal screen, and various sensors.

  FIG. 2 is a functional block diagram illustrating an example of the internal configuration of the image display control apparatus 1. The image display control device 1 includes a control unit 11, a storage unit 12, an operation unit 13, an imaging unit 14, a sensor unit 15, a display unit 16, a communication unit 17, and a bus 18. The control unit 11, the storage unit 12, the operation unit 13, the imaging unit 14, the sensor unit 15, the display unit 16, and the communication unit 17 are connected to each other via a bus 18 so that data transmission is possible.

  The image display control device 1 downloads the AR display program from the server 3 via the communication line 2 and reads it into the storage unit 12. The program may be read from another recording medium such as a flash memory or a CD-ROM.

  The storage unit 12 stores various information. The storage unit 12 includes a read only memory (ROM) and a random access memory (RAM) as a primary storage unit, and a secondary storage unit. The ROM is a read-only storage medium and stores a basic program. The RAM is a rewritable storage medium and provides an area for storing application programs and a work area for processing the programs. The secondary storage unit is, for example, a flash memory or a hard disk, and stores downloaded application programs and various data. In the present embodiment, the AR display program downloaded from the server 3 is stored in the secondary storage unit of the storage unit 12. In addition, three-dimensional image data, which will be described later, is also stored in the secondary storage unit. The storage unit 12 corresponds to a “program storage unit” and a “storage device” according to the present invention.

  The operation unit 13 receives an operation input from the operator, and includes various operation buttons and a touch panel. The operation unit 13 inputs a corresponding operation signal to the control unit 11 when the operation button is pressed by the operator. The operation unit 13 calculates a contact position on the panel surface based on touch information input from the touch panel (change information on minute capacitors or minute resistances arranged in a grid pattern on the panel surface of the touch panel), and the contact position. Is input to the control unit 11 as an operation signal. The touch panel is disposed on a display screen of a display device described later.

  The imaging unit 14 includes an imaging device (camera) having an imaging element such as a CCD (Charge Coupled Device). The imaging unit 14 inputs still image or moving image data captured by the imaging device to the control unit 11.

  The sensor unit 15 includes various sensors, and inputs detection signals detected by the sensors to the control unit 11. In the present embodiment, the sensor unit 15 includes a triaxial acceleration sensor, a triaxial gyro sensor, an orientation sensor, and a GPS (Global Positioning System). The triaxial acceleration sensor is a sensor that detects acceleration in each direction of three orthogonal axes. The triaxial gyro sensor is a sensor that detects angular velocities around the three axes that are orthogonal to each other. The direction sensor is a sensor that detects the direction. The GPS receives a signal transmitted from a satellite and detects position information. Note that all of these sensors may not be provided, or other sensors may be provided.

  The display unit 16 includes a display device such as a liquid crystal display device. The display unit 16 displays the image input from the control unit 11 on the display device.

  The communication unit 17 communicates with an external device via the communication line 2 and includes communication means such as a wireless LAN module. In addition, the communication part 17 is not limited to what performs wireless communication, You may perform wired communication. In the present embodiment, the communication unit 17 receives the AR display program from the server 3. In addition, 3D image data is received from another computer or server 3.

  The control unit 11 performs overall control of the operation of each unit included in the image display control device 1, and is realized by a CPU (Central Processing Unit). The control unit 11 includes at least a data acquisition unit 111, a date and time setting unit 112, a captured information acquisition unit 113, a captured image acquisition unit 114, and a display control unit 115 as functional blocks. The units 111 to 115 as functional blocks are realized by the control unit 11 executing the AR display program stored in the storage unit 12.

  The data acquisition unit 111 acquires 3D image data stored in the storage unit 12. The three-dimensional image data is acquired in advance and stored in the storage unit 12. In the present embodiment, the three-dimensional image data is data of a three-dimensional model of a building, and data for each component obtained by dividing a wall, a floor, a ceiling, a window, a door, and the like constituting the building into a plurality of parts. As prepared. In the present embodiment, the constituent elements are polygons that are polygons that are the minimum unit of drawing in three-dimensional computer graphics. That is, the three-dimensional image data includes a plurality of polygon data. FIG. 3 shows an example of 3D image data. As shown in FIG. 3, the three-dimensional image data includes a number, position data, texture data, date / time data, and a display flag for each polygon. The number is a number assigned to each polygon. The position data is coordinates indicating the position of each vertex of the polygon. In this embodiment, since the polygon is a triangle, the coordinates of three vertices are included. Each coordinate is a coordinate on the three-dimensional virtual space when the three-dimensional model is designed. In addition, the coordinates (coordinates based on latitude, longitude, and height) indicating the actual position may be used. The texture data is data indicating the texture pasted on the polygon. The date / time data is data indicating the date on which the portion including the polygon is scheduled to be completed. Date and time data is set according to the implementation plan. The date / time data is not included in the data of the 3D model at the time of design, and is added after the enforcement plan is determined. Note that the data acquisition unit 111 may create 3D image data from the data of the 3D model at the time of design and the data of the enforcement plan. The display flag represents whether or not to display a polygon, and is displayed when “1” (when the flag is set). In FIG. 3, for the nth polygon, coordinates (X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3, Z3) of three vertices are set as position data, and Tn as texture data. Is set, Dn is set as date and time data, and “0” is set in the display flag. The data acquisition unit 111 outputs the acquired 3D image data to the display control unit 115.

  The date and time setting unit 112 sets the date and time. In the present embodiment, since the current date is set, the date / time setting unit 112 acquires the current date from a clock or calendar provided in the image display control device 1 and outputs the current date to the display control unit 115. Note that the date and time setting unit 112 is not limited to setting the current date, and may set a desired date. In this case, a message prompting the user to input a desired date may be displayed on the display device, and the date may be set based on an operation signal input from the operation unit 13 by the operator's operation.

  The imaging information acquisition unit 113 acquires imaging information indicating the position and imaging direction of the imaging device. The imaging information acquisition unit 113 calculates the position and imaging direction of the imaging device based on various detection signals input from the sensor unit 15 and outputs the calculation result to the display control unit 115 as imaging information. The position of the imaging device is calculated based on the GPS detection signal, the direction of the imaging direction on the horizontal plane is calculated based on the detection signal of the direction sensor, and the inclination of the imaging direction with respect to the horizontal plane is a triaxial acceleration sensor or a triaxial gyro sensor. Is calculated based on the detected signal. Each calculation may be corrected by detection signals of other sensors. The imaging information calculation method and the sensor used for the calculation by the imaging information acquisition unit 113 are not limited as long as the position and imaging direction of the imaging device can be calculated.

  The captured image acquisition unit 114 acquires an image captured by the imaging apparatus. The captured image acquisition unit 114 outputs the image data input from the imaging unit 14 to the display control unit 115.

  The display control unit 115 generates a display image to be displayed on the display device of the display unit 16. The display control unit 115 includes 3D image data input from the data acquisition unit 111, date information input from the date setting unit 112, imaging information input from the imaging information acquisition unit 113, and a captured image acquisition unit 114. Based on the input image data, image data of the display image is generated. The display control unit 115 includes an extraction unit 115a, a projection processing unit 115b, and a synthesis unit 115c.

  The extraction unit 115a extracts a portion to be displayed on the display image from the three-dimensional image based on the three-dimensional image data. The extraction unit 115 a receives 3D image data from the data acquisition unit 111 and receives date and time information (hereinafter, “date and time information”) from the date and time setting unit 112. Then, the date / time data included in the 3D image data is compared with the date / time information input from the date / time setting unit 112. The extraction unit 115a extracts only polygons whose date and time indicated by the date and time data is earlier than the date and time indicated by the date and time information. In this embodiment, the date / time data indicates the date on which the portion including the polygon is scheduled to be completed, and the date / time information indicates the current date. Therefore, only the polygon of the portion scheduled to be completed up to now is included. Extracted. Only these extracted polygons are displayed in the display image. The polygon data extracted by the extraction unit 115a is distinguished from polygon data that has not been extracted by setting a display flag ("1" is set). Note that whether or not the data has been extracted may be distinguished based on the polygon data number. Of the three-dimensional image data, the extracted polygon data is output to the projection processing unit 115b.

  The projection processing unit 115b converts the coordinates of each vertex of the polygon, which is three-dimensional data, into two-dimensional data for display on the display screen. The projection processing unit 115 b receives polygon data extracted from the extraction unit 115 a and receives imaging information from the imaging information acquisition unit 113. The projection processing unit 115b first converts the coordinates of each vertex of the extracted polygon data input from the extraction unit 115a into coordinates indicating the actual position. That is, when a three-dimensional model is constructed as an actual building on a planned construction site, it is converted into position coordinates where each vertex is located. For the conversion, information such as the position coordinates of the planned construction site, the direction of the building, and each dimension stored in the storage unit 12 together with the three-dimensional image data is used. If the coordinates of each vertex of the polygon data are set as coordinates indicating the actual position, the conversion is omitted. Next, the projection processing unit 115b converts the coordinates of each vertex converted into the actual position coordinates into two-dimensional coordinates that are perspective-projected on the display screen. Perspective projection is a technique for drawing a three-dimensional object on a two-dimensional plane in three-dimensional computer graphics, extending the line of sight from the viewpoint to the three-dimensional object, and creating a virtual space between the viewpoint and the object. The object is drawn at the intersection of the screen and the line of sight arranged in the screen. The perspective of the perspective projection and the direction of the line of sight are set based on the imaging information input from the imaging information acquisition unit 113. That is, the position of the imaging device included in the imaging information is set as the perspective of perspective projection, and the imaging direction included in the imaging information is set as the direction of the line of sight of perspective projection. Polygon data obtained by converting the coordinates of each vertex is output to the synthesis unit 115c.

  The synthesizing unit 115c generates a display image. The composition unit 115c receives polygon data obtained by converting the coordinates of each vertex from the projection processing unit 115b, and receives image data from the captured image acquisition unit 114. The synthesizing unit 115c synthesizes the image based on the image data input from the captured image acquisition unit 114 by drawing the polygon input from the projection processing unit 115b, and generates a display image. Polygon drawing is performed by drawing a triangle composed of the coordinates of three vertices and pasting a texture. At this time, drawing may not be performed for polygons that cannot be seen by the polygons existing on the viewpoint (position of the imaging device) side. The texture is a translucent texture so that an actual image can be seen through. The generated display image is output to the display unit 16 and displayed on the display device. Note that the processing of the projection processing unit 115b and the combining unit 115c may be performed by a drawing processor provided separately from the CPU.

  The processing in these control units 11 is performed in real time. That is, the captured image acquisition unit 114 acquires the current image captured by the imaging device, and the imaging information acquisition unit 113 acquires the imaging information at that time. Then, the synthesizing unit 115c synthesizes each polygon that is projected by the projection processing unit 115b based on the imaging information. Therefore, a building that does not actually exist is combined with the current actual image captured by the imaging device, and displayed as if the building actually exists. In the present embodiment, the image data input from the captured image acquisition unit 114 is moving image data. In this case, each time image data of each still image constituting the moving image is input, the imaging information acquisition unit 113 acquires the imaging information at that time, and each of the polygons projected by the projection processing unit 115b based on the imaging information. Are repeatedly synthesized by the synthesizing unit 115c into a still image. Therefore, when the operator who has the image display control device 1 moves or changes the imaging direction of the imaging device, the position and orientation of the building change accordingly. Thereby, a display image can be displayed as if the building actually exists and is imaged by the imaging device.

  FIG. 4 is a flowchart for explaining the AR display process performed by the control unit 11. The AR display process is started, for example, when the operator selects AR display on the menu screen. Before starting the AR display process, it is necessary to obtain three-dimensional image data and store it in the storage unit 12.

  First, the data acquisition unit 111 acquires 3D image data stored in the storage unit 12 (S1). Next, the date and time setting unit 112 acquires date and time information indicating the current date and time (S2). Then, the extraction unit 115a compares the date and time data included in the 3D image data with the date and time information, and extracts only polygon data whose date and time indicated by the date and time data is earlier than the date and time indicated by the date and time information (S3). ). In the subsequent processing, only the extracted polygon data is used.

  Next, the imaging information acquisition unit 113 acquires imaging information based on various detection signals input from the sensor unit 15 (S4). Next, the projection processing unit 115b converts the coordinates of each vertex of the extracted polygon data into coordinates indicating the actual position, and further converts them into two-dimensional coordinates that are perspective-projected on the display screen (S5). .

  Next, the captured image acquisition unit 114 acquires image data of an image captured by the imaging device (S6). Next, the synthesizing unit 115c synthesizes the captured image by drawing a polygon in which the coordinates of each vertex are converted, and generates a display image (S7). Next, the display control unit 115 outputs the generated display image to the display unit 16 for display on the display device (S8). Then, returning to step S4, steps S4 to S8 are repeated.

  The AR display process is terminated when the operator selects termination (for example, when the operator touches the termination button displayed on the display screen and the touch panel reads the touch). Note that the flowchart shown in FIG. 4 is an example of the AR display process, and the AR display process performed by the control unit 11 is not limited to this.

For example, instead of extracting the corresponding polygon data first (see S3 in FIG. 4), the extracted data is extracted while extracting the corresponding polygon data as shown in the flowchart of FIG. On the other hand, projection processing and image synthesis may be performed. In the flowchart shown in FIG. 5, the control unit 11 acquires the three-dimensional image data, the date information, the imaging information, and the captured image (steps S1, S2, S4, and S6), and then increases the variable n indicating the polygon number. However, steps S9, S5, and S7 are repeated by the number of polygons (S3 ′). In step S9, the control unit 11, time data Dn corresponding polygon to determine whether the smaller date information D ₀ (before the date indicating the date indicated by the date data Dn is the date and time information D _0). In the case of _{Dn <D 0 (S9: YES} ) performs the synthesis of projection process (S5) and the image (S7), if the _{Dn ≧ D 0 (S9: NO} ) , the perform synthesis of projection processing and image Absent. This is repeated for the number of polygons, and then displayed on the display device (S8).

  Next, operations and effects of the image display control apparatus 1 according to the present embodiment will be described.

  According to the present embodiment, the extraction unit 115a compares the date / time data included in the 3D image data input from the data acquisition unit 111 with the date / time information input from the date / time setting unit 112, and indicates the date / time indicated by the date / time data. Extract only those polygons whose date is before the date and time indicated by the date and time information. Since the date and time data indicates the date when the part including the polygon is scheduled to be completed, and the date and time information indicates the current date, only the polygons of the part scheduled to be completed by the current date and time are extracted. . Then, only the extracted polygon is combined with the image captured by the imaging device and displayed on the display device. Therefore, only the part of the building that is scheduled to be completed by the current date and time is synthesized and displayed on the actual image. Thereby, it is possible to intuitively recognize the delay of the execution plan from the difference between the image of the planned completion part and the actual image. In the present embodiment, since the texture to be pasted on the polygon is made translucent, the building made of the polygon is displayed translucently. Thereby, since the actual image can be seen through, it is possible to easily distinguish the difference between the image of the part to be completed and the actual image.

  Further, according to the present embodiment, the captured image acquisition unit 114 acquires the current image captured by the imaging device, and the imaging information acquisition unit 113 acquires the imaging information at that time. Then, the synthesizing unit 115c synthesizes each polygon that is projected by the projection processing unit 115b based on the imaging information. As a result, a building made of polygons is synthesized with the current actual image captured by the imaging device according to the current position and imaging direction of the imaging device, and the building actually exists. Is displayed. Therefore, the synthesized image can be shown in real time at the construction site.

  According to the present embodiment, the image data input from the captured image acquisition unit 114 is moving image data. Then, every time image data of each still image constituting the moving image is input, the control unit 11 acquires imaging information at that time, and converts each polygon that is perspective-projected based on the imaging information into a still image. Repeat the synthesis. Therefore, when the operator who has the image display control device 1 moves or changes the imaging direction of the imaging device, the position and orientation of the building change accordingly. Thereby, the image which looked at the building from various directions can be displayed.

  In the first embodiment, the case has been described in which the date when the part including the polygon is scheduled to be completed is set as the date / time data of the 3D image data. However, the present invention is not limited to this. For example, instead of the date, the number of days from the start of construction may be set (for example, after 40 days). In this case, the date / time information set by the date / time setting unit 112 must also be set by the number of days since the start of construction. Further, not only the date but also the time may be specified and set. In this case, the date and time set by the date and time setting unit 112 may be set including the time. In this case, the enforcement state can be confirmed in more detail.

  In the first embodiment described above, a case has been described in which an image in which an entire building made up of polygons (the entire part to be completed) is combined with an actual image is displayed. However, the present invention is not limited to this. . For example, only a portion of a building made of polygons that is not displayed in an actual image (a portion that should have been completed but not actually completed) may be synthesized. Specifically, a common part between a building made of polygons and a building of an actual image may be detected by image processing so that the polygon corresponding to the common part is not displayed. In the example of FIG. 1, in FIG. 1D, only the third floor portion composed of polygons is combined with the actual image of the first and second floor portions.

  In the first embodiment, the case where the three-dimensional image data is data for each polygon has been described. However, the present invention is not limited to this. For example, the three-dimensional image data may be data for each element (wall, floor, ceiling, window, door, etc.) comprising a plurality of polygons and constituting a building. Moreover, it is good also as the data for every point of the three-dimensional model of a building. In this case, even if the points are the same on the coordinates, points with different date / time data need to be set as different points. For example, the point of the part scheduled to be completed at the date and time D1 and the point of the part scheduled to be completed at the date and time D2 are set as different points even if they are the same point on the coordinates.

  In the first embodiment, the case where AR display is performed by a so-called sensor method has been described, but the present invention is not limited to this. The AR display may be performed by other methods (for example, a marker method or a markerless method).

  In the first embodiment described above, a case has been described in which a building made up of a perspective-projected polygon is combined with a current image captured by an imaging device in real time, but is not limited thereto. You may make it synthesize | combine with the image imaged by the imaging device in the past. In this case, it is necessary to perform perspective projection based on the imaging information acquired when the image is captured. Moreover, in the said 1st Embodiment, although the imaging device imaged the moving image and demonstrated the case where the moving image which synthesize | combined the building was displayed on a display apparatus, it is not restricted to this. The imaging device may capture a still image, and the still image obtained by combining the buildings may be displayed on the display device.

  In the first embodiment, it has been described that the 3D image data includes date / time data in addition to position data and the like, and the image display control device 1 is suitable for confirming the progress of the enforcement plan. Further, a case where the progress of the budget can be confirmed by making the three-dimensional image data include data related to construction costs will be described below as a second embodiment.

  FIG. 6 is a functional block diagram showing an example of the internal configuration of the image display control device 1 ′ according to the second embodiment. In FIG. 6, the storage unit 12, the operation unit 13, the imaging unit 14, the sensor unit 15, the display unit 16, the communication unit 17, and the bus 18 are omitted, and only the control unit 11 is described. In FIG. 6, the same or similar elements as those in the image display control apparatus 1 (see FIG. 2) according to the first embodiment are denoted by the same reference numerals. As shown in FIG. 6, the image display control device 1 ′ includes an image display according to the first embodiment in that it includes a selection receiving unit 116, a deletion processing unit 117, and a cost calculation unit 118 as functional blocks. Different from the control device 1.

  In addition to the AR display program, the image display control device 1 ′ downloads a cost calculation program from the server 3 and reads it into the storage unit 12. The selection reception unit 116, the deletion processing unit 117, and the cost calculation unit 118, which are functional blocks, are realized by the control unit 11 executing the cost calculation program stored in the storage unit 12.

  The cost calculation program calculates the planned cost of the current completed part, deletes the unimplemented part from the display image created by the AR display program, and calculates the planned cost of the remaining part (actual completed part). Calculate and display. For this reason, the 3D image data according to the second embodiment includes cost data for calculating the planned cost.

  FIG. 7A shows an example of 3D image data according to the second embodiment. The 3D image data is obtained by adding cost data to the 3D image data according to the first embodiment. The cost data is obtained by dividing the construction cost of a portion (for example, a wall, a floor, a ceiling, a window, or a door) including the polygon by the number of included polygons. For example, if the construction cost of a wall is 1 million yen and the wall is composed of 100 polygons, 10,000 yen (= 1 million yen / 100 pieces) is set as cost data for each polygon. . Cost data is set according to the estimate. Note that the cost data is not included in the data of the three-dimensional model at the time of design, and is added later according to the estimate. Note that the data acquisition unit 111 may also create 3D image data from the 3D model data at the time of design, the data of the execution plan (used for setting date / time data), and the data of the estimate sheet. Good.

  FIG. 8 shows an example of the display screen 41 of the image display control device 1 ′, and shows a state where the display image 42 created by the AR display program is displayed. A plurality of function buttons 43 are displayed on the right side of the display image 42. Since the touch panel is arranged on the display screen 41, when the operator touches the display screen 41, an operation signal at the touch position is input from the touch panel to the control unit 11. Thereby, the control part 11 can recognize the position which the operator touched and can recognize what was selected. As the function buttons 43, a “display start” button for starting AR display, a “display end” button for ending AR display, a “pause” button for temporarily stopping AR display, and a portion to be deleted from the display image 42 are selected. A “selection start” button for starting and a “selection end” button for ending selection are arranged. Further, a part to be deleted can be selected by touching the display image 42. The selection may be performed for each polygon, or may be performed for each portion where a plurality of polygons are collected. The layout of the display screen 41 and the types and number of function buttons 43 to be displayed are not limited. Further, the selection method on the display image 42 is not limited.

  The selection accepting unit 116 accepts designation of a part selected by the operator for deletion from the display image 42 created by the AR display program. Based on the operation signal input from the operation unit 13, the selection receiving unit 116 recognizes the polygon corresponding to the portion selected on the display image 42 and outputs the number of the corresponding polygon to the deletion processing unit 117. The selection receiving unit 116 starts receiving the selection after the AR display is temporarily stopped and the selection start input is performed.

  The deletion processing unit 117 deletes the selected polygon from the 3D image data. The deletion processing unit 117 receives the polygon data (three-dimensional image data reflecting the display flag) extracted by the extraction unit 115 a and the polygon number from the selection receiving unit 116. The deletion processing unit 117 sets the polygon display flag corresponding to the number input from the selection receiving unit 116 to “0” in the three-dimensional image data input from the extraction unit 115 a. As a result, the polygon deleted by the deletion processing unit 117 is not displayed on the display image 42. The deletion processing unit 117 outputs the three-dimensional image data after performing the polygon deletion processing to the cost calculation unit 118.

  The cost calculation unit 118 calculates the planned cost of the part displayed on the display image 42. The cost calculation unit 118 receives the 3D image data after the deletion processing from the deletion processing unit 117. The cost calculation unit 118 includes the cost of the polygon (the polygon displayed in the display image 42) with the display flag set (“1” is set) among the input three-dimensional image data after the deletion process. Sum the data. Thereby, the estimated cost of the actual completion part currently displayed on the display image 42 is computable. The calculated scheduled cost is displayed on the display screen 41 (see FIG. 8).

  FIG. 9 is a flowchart for explaining a cost calculation process performed by the control unit 11. The cost calculation process is started when the AR display is temporarily stopped and selection start input is performed.

  First, the control unit 11 determines whether or not a selection end input has been performed (S11). Specifically, it is determined whether or not an operation signal indicating touch of the “selection end” function button 43 is input from the operation unit 13. When the selection end input is not performed (S11: NO), the selection receiving unit 116 determines whether or not a selection is specified (S12). Specifically, it is determined whether or not an operation signal indicating selection on the display image 42 is input from the operation unit 13. If there is a selection designation (S12: YES), the selected polygon is deleted. Specifically, the selection receiving unit 116 outputs the number of the selected polygon to the deletion processing unit 117, and the deletion processing unit 117 inputs the 3D image data input from the extraction unit 115a from the selection receiving unit 116. The display flag of the polygon corresponding to the number is set to “0”. Then, the process returns to step S11. Also in step S12, when there is no selection specification (S12: NO), it returns to step S11. In step S11, when the selection end input is performed (S11: YES), the process proceeds to step S14. In other words, steps S11 to S13 are repeated until selection completion is input, and if a selection is made, the selected polygon is deleted.

  Then, the cost calculation unit 118 calculates the planned cost of the portion displayed on the display image 42 based on the three-dimensional image data after the deletion process (S14). Specifically, the cost calculation unit 118 totals the cost data of polygons for which display flags are set out of the three-dimensional image data after the deletion process. Then, the display control unit 115 displays the calculated total cost on the display screen 41 (S15), and the cost calculation process ends.

  The flowchart shown in FIG. 9 is an example of the cost calculation process, and the cost calculation process performed by the control unit 11 is not limited to this.

  In the second embodiment, the same effect as that of the first embodiment can be obtained. Furthermore, in the second embodiment, the planned cost of the current completed part can be calculated by the cost calculation process. By comparing the planned cost of the current completed part with the actual cost, it is possible to know whether it is as planned and how much the budget has been exceeded. Moreover, since the display image 42 created by the AR display program is only operated, the progress of the budget can be easily confirmed on site.

  In addition, as shown in FIG.7 (b), a construction cost may be subdivided into items, such as material cost and a labor cost, and you may set for every item as expense data. In this case, the planned cost for each item can be calculated and compared with the actual cost for each item.

  In the second embodiment, the case where the operator selects and deletes an unexecuted portion from the display image 42 created by the AR display program has been described. However, the present invention is not limited to this. The unexecuted portion may be automatically deleted by image processing. Specifically, a difference portion between the display image 42 and an actual image (image captured by the imaging device and acquired by the captured image acquisition unit 114) is detected by image processing, and the detected difference portion is handled. Polygons can be deleted automatically. In this case, since the selection by the operator is not necessary, the progress of the budget can be confirmed more easily. Further, the difference portion detected by the image processing may be changed in color or blinked to assist the selection by the operator.

  In the first or second embodiment, the case of extracting the polygon to be displayed based on the date / time data has been described, but the present invention is not limited to this. For example, polygons to be displayed may be extracted based on cost data. The case of extracting polygons to be displayed based on cost data will be described below as a third embodiment.

  FIG. 10 is a functional block diagram illustrating an example of the internal configuration of the image display control device 1 ″ according to the third embodiment. In FIG. 10, the storage unit 12, the operation unit 13, the imaging unit 14, the sensor unit 15, The display unit 16, the communication unit 17, and the bus 18 are omitted, and only the control unit 11 is described, and in Fig. 10, the same as the image display control device 1 (see Fig. 2) according to the first embodiment. Alternatively, similar elements are denoted by the same reference numerals. As shown in FIG. 10, the image display control device 1 ″ includes a cost setting unit 112 ′ instead of the date / time setting unit 112 as a functional block, It differs from the image display control apparatus 1 according to the first embodiment in that an extraction unit 115a ′ is provided instead of the extraction unit 115a.

  In the third embodiment, three-dimensional image data according to the second embodiment shown in FIG. 7A is used as the three-dimensional image data. The date / time data may not be provided. That is, the 3D image data according to the third embodiment includes at least a number, position data, texture data, cost data, and a display flag. Numbers are assigned in the order they are actually built.

  The cost setting unit 112 'sets the total cost. In the present embodiment, a message prompting the input of the total amount of expenses spent up to now is displayed on the display device, and the cost setting unit 112 ′ is based on an operation signal input from the operation unit 13 by an operator's operation. To get the total cost. The cost setting unit 112 ′ outputs the acquired total cost to the display control unit 115. Note that the expense setting unit 112 ′ may automatically read the expense spent so far from the accounting system and calculate and set the total amount. The total cost up to the present time is not limited, and a desired total cost may be set.

  The extraction unit 115a 'extracts a portion to be displayed on the display image from the three-dimensional image based on the three-dimensional image data. The extraction unit 115a 'receives the three-dimensional image data from the data acquisition unit 111 and the total cost from the cost setting unit 112'. Then, the extraction unit 115a 'integrates the cost data included in the three-dimensional image data in the order of the numbers, and compares the integrated value with the total cost input from the cost setting unit 112'. The extraction unit 115a 'extracts polygons before the integrated value exceeds the total cost. In the present embodiment, the cost data indicates the construction cost for each polygon, and the total cost indicates the total amount of expenses that have been spent so far. Only polygons of are extracted. Only these extracted polygons are displayed in the display image. Polygon data extracted by the extraction unit 115a 'is distinguished from polygon data that has not been extracted by setting a display flag ("1" is set). Of the three-dimensional image data, the extracted polygon data is output to the projection processing unit 115b.

  According to the third embodiment, the extraction unit 115a 'integrates the cost data included in the three-dimensional image data in the order of the numbers, and extracts polygons until the integrated value exceeds the total cost. Therefore, only the polygons that are scheduled to be completed at the expense spent up to now are extracted, combined with the image captured by the imaging device, and displayed on the display device. Therefore, on the display device, only the portion of the building that is scheduled to be completed at the expense spent up to now is synthesized and displayed on the actual image. Thereby, it is possible to intuitively recognize the amount of the cost that is being taken from the difference between the image of the planned completion part and the actual image. That is, when the process of the image of the part to be completed is progressing from the actual image, for example, when the display is as shown in FIG. 1D, it can be understood that the cost is higher than the plan. On the contrary, if the actual image is processed more than the image of the part to be completed, it can be understood that the expense spent is less than the plan.

  In addition, although the said 1st-3rd embodiment demonstrated the case where the image display control apparatus 1 was implement | achieved using the tablet computer, it is not restricted to this. For example, a smartphone or a notebook computer may be used. Further, the image display control apparatus may be realized using a desktop computer. A case where a desktop computer is used will be described below as a fourth embodiment.

  FIG. 11A is a functional block diagram illustrating an example of an internal configuration of a system including the image display control device 6 according to the fourth embodiment. In FIG. 11A, the same or similar elements as those in the image display control apparatus 1 (see FIG. 2) according to the first embodiment are denoted by the same reference numerals. As shown in FIG. 11A, the image display control device 6 does not include the imaging unit 14 and the sensor unit 15, but instead includes the second communication unit 19. This is different from the image display control device 1.

  The image display control device 6 is obtained by installing an AR display program on a desktop computer. The image display control device 6 is generally installed on a desk and is not taken by the operator to the site. The image display control device 6 does not include the imaging unit 14 and the sensor unit 15 in the image display control device 1 according to the first embodiment, and the display unit 16 is also a stationary liquid crystal display device or the like. In the third embodiment, for example, a head mounted display 5 is used as a substitute for the imaging unit 14, the sensor unit 15, and the display unit 16 of the image display control device 1 according to the first embodiment.

  The second communication unit 19 performs wireless communication such as Bluetooth (registered trademark) with the communication unit 59 (described later) of the head mounted display 5. In addition, the 2nd communication part 19 and the communication part 59 are not limited to what performs wireless communication, You may perform wired communication.

  FIG. 11B is a diagram schematically illustrating an example of the head mounted display 5. The head mounted display 5 is a glasses-type display device worn by an operator on the head, and includes a control unit 51, an imaging unit 54, a sensor unit 55, a display unit 56, a communication unit 59, and a bus 58. .

  The imaging unit 54 includes an imaging device (camera). The imaging unit 54 inputs still image or moving image data captured by the imaging device to the control unit 51. The sensor unit 55 includes a triaxial acceleration sensor, a triaxial gyro sensor, an orientation sensor, and a GPS. The sensor unit 55 inputs detection signals detected by the sensors to the control unit 51. The display unit 56 includes a display device such as a liquid crystal display device. The display unit 56 displays the image input from the control unit 51 on the display device. The communication unit 59 performs wireless communication with the second communication unit 19 of the image display control device 6. The control unit 51 performs overall control of the operation of each unit included in the head mounted display 5, and is realized by a CPU. The control unit 51 causes the image display control device 6 to transmit the image data input from the imaging unit 54 and the detection signal input from the sensor unit 55 via the communication unit 59. In addition, the control unit 51 causes the display unit 56 to display the image received by the communication unit 59 from the image display control device 6.

  That is, the head mounted display 5 is a device having the functions of the imaging unit 14, the sensor unit 15, and the display unit 16 in the image display control device 1 according to the first embodiment, and is movable with the movement of the operator. It is a thing. The head mounted display 5 and the image display control device 6 transmit and receive various signals between the communication unit 59 and the second communication unit 19, so that functions similar to those of the image display control device 1 according to the first embodiment as a whole. Have

  In the fourth embodiment, the same effect as that of the first embodiment can be obtained.

  In the first to fourth embodiments, the image display control device 1 (1 ′, 1 ″, 6) is realized by installing the AR display program (and cost calculation program) in a general-purpose computer. However, the present invention is not limited to this, for example, the dedicated image display control device 1 (1 ′, 1 ″, which stores the AR display program (and the cost calculation program) from the beginning (from the time of manufacture). 6).

  In the said 1st-4th embodiment, although the case where this invention was utilized for building construction was demonstrated, it is not restricted to this. The present invention can be used for construction of highways and bridges, and can also be used for civil engineering work.

  The present invention can also be used for other than construction work and civil engineering work. For example, the present invention can be used when an object to be synthesized changes according to the date and time when a synthesized image obtained by synthesizing an object that does not exist in an actual image is displayed in real time using the AR technology.

  The image display control apparatus and program according to the present invention are not limited to the above-described embodiments. The specific configuration of each part of the image display control device and the program according to the present invention can be varied in design in various ways.

1, 1 ′, 1 ″, 6: Image display control device 11: Control unit 111: Data acquisition unit 112: Date and time setting unit (setting means)
112 ': Cost setting section (setting means)
113: Imaging information acquisition unit 114: Captured image acquisition unit 115: Display control unit 115a, 115a ′: Extraction unit 115b: Projection processing unit 115c: Composition unit 116: Selection reception unit 117: Deletion processing unit 118: Cost calculation unit 12: Storage unit (storage device, program storage unit)
13: Operation unit 14: Imaging unit 15: Sensor unit 16: Display unit 17: Communication unit 18: Bus 19: Second communication unit 2: Communication line 3: Server 41: Display screen 42: Display image 43: Function button 5: Head mounted display 51: Control unit 54: Imaging unit 55: Sensor unit 56: Display unit 58: Bus 59: Communication unit

Claims (8)

Computer
Data acquisition means for acquiring data of a three-dimensional object stored in a storage device;
Captured image acquisition means for acquiring an image captured by the imaging device;
Display control means for combining the image acquired by the captured image acquisition means with the image of the three-dimensional object based on the data acquired by the data acquisition means, and displaying it on a display device;
A setting means to set the date and time or total cost,
To function,
The data of the three-dimensional object includes data for each component,
The data of each component includes three-dimensional position data and date / time data or cost data.
The display control unit synthesizes the image of the three-dimensional object based on the date and time set by the setting unit or the total cost set by the cost data and the setting unit.
A program characterized by that. The display control means synthesizes the image of the three-dimensional object based on a component indicating the date and time before the date and time set by the setting means;
The program according to claim 1. The data of each component further includes number data indicating the order,
The display control means integrates the cost data of the constituent elements in the order of number data, and based on the constituent elements until the integrated value exceeds the total cost set by the setting means, the three-dimensional object Combining images of
The program according to claim 1. Causing the computer to further function as imaging information acquisition means for acquiring imaging information indicating a position and an imaging direction of the imaging device;
The display control means synthesizes a two-dimensional image obtained by projecting the three-dimensional object based on the imaging information;
The program according to any one of claims 1 to 3.   The program according to any one of claims 1 to 4, wherein the date and time set by the setting means is a current date and time.   The program according to any one of claims 1 to 5, wherein an image of the three-dimensional object combined with an image acquired by the captured image acquisition unit is translucent.   The program according to any one of claims 1 to 6, wherein the three-dimensional object is a building. A program storage unit storing the program according to claim 1;
A control unit that executes processing based on a program stored in the program storage unit;
An image display control device comprising:

Images