JP2016053961A - Information processing device, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correct a positional deviation occurring when a perspective image of a building scheduled for construction and a photographed image of the planned site for the building are synthesized on a real time basis.SOLUTION: An information processing device comprises; a memory in which three-dimensional data representing planned position coordinates of a building scheduled for construction and the pertinent building and topographic data including the heights in different positions in a prescribed range covering the planned position coordinates are stored; a camera; a position sensor; a direction sensor; a display unit; and a control unit. The control unit draws a perspective image of the building from the viewpoint of the camera on the basis of the three-dimensional data, the planned position coordinates, current position coordinates and the direction, causes a synthesized image resulting from superposing the perspective image over the photographed image to be displayed on the display unit, and can use, for drawing the perspective image, a value resulting from adding the average height of a human viewpoint to a height in the topographic data as the correcting value in the perpendicular direction of the current position coordinates.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an information processing apparatus capable of displaying an image for examining a surrounding landscape when a building is constructed, and an information processing method and program in the information processing apparatus.

  2. Description of the Related Art Conventionally, there is a technique for displaying an image of a building planned for construction on a photograph of an actual construction planned site and allowing the user to examine the scene of the site.

  In Non-Patent Document 1 below, the viewpoint position and the camera angle at which the background site photograph was taken are restored from the subject (corresponding to the orientation of the aerial photograph), and at the same angle from the position at which the photograph was taken, It describes that a perspective is created from three-dimensional data of a building to be constructed.

  Specifically, first, the shape of the building is prepared as three-dimensional data. Next, from the subject of the background photo, find four or more points whose position can be specified on the map, and the same coordinate system that describes the 2D coordinates [u, v] on the photo and the building as 3D data. Prepare four or more pairs of coordinates [x, y, z] in the real space described using. Using this, the position where the photograph was taken and the camera angle are restored. This calculation is performed using the “viewpoint extraction” operation screen. By creating a perspective of the building from the calculated viewpoint position and camera angle and displaying it in combination, it is possible to combine the photos at the correct position.

  This enables photo composition based on accurate alignment, which is different from a montage photo in which images are simply superimposed. At the time of this photo composition, in order to develop an image photographed with a film camera, create background image data with a scanner, and compose one photo, preparation work including the above-mentioned orientation calculation was required.

  On the other hand, real-time photo composition that superimposes and displays a perspective of a building or the like at a speed of about 30 times per second using a viewpoint position measured by a sensor on a background image dynamically acquired by a camera is a so-called “composite”. It has been used since about 2000 as "Reality" (MR). In the early days, a special display device (HMD: Head Mount Display) was used, but from around 2010, mobile terminals with built-in photographing cameras, GPS (Global Positioning System) sensors, orientation sensors, gravity sensors, etc. have become widespread. However, a technical and social environment that can be implemented at low cost has been prepared, and several applications using these have been developed and published (for example, see Patent Documents 1 and 2 below).

JP 2004-102835 A JP 2006-011970 A

Hideyuki Kobayashi, “Scenery Examination System for Civil Engineering Facilities by 3D CG-Prototype Version” Ver.1.0, Ministry of Construction, Architectural Research Materials No.85, September 1995

  However, since the positional accuracy acquired by GPS has an error of several meters or more, it is possible to display composite images on distant mountains, etc., but real-time photo composition for houses in urban areas is obtained from sensors. It is not yet practical to perform only from the obtained position information because of position shift or the like.

  In view of the circumstances as described above, an object of the present invention is information capable of correcting a positional deviation between both images when combining a perspective image of a building planned for construction and a captured image of the planned construction site in real time. To provide a processing device, an information processing method, and a program.

  In order to solve the above-described problem, an information processing apparatus according to an embodiment of the present invention includes a storage unit, a camera, a GPS sensor, a direction sensor, a display unit, an operation reception unit, and a control unit. The said memory | storage part can memorize | store the three-dimensional data which show the plan position coordinate of the construction plan of a building, and the said building. The camera can capture an image of a planned construction site of the building. The GPS sensor can detect the current position coordinates of the information processing apparatus. The direction sensor can detect the direction in which the camera is facing. The operation accepting unit can accept an operation of a user holding the information processing apparatus. The control unit draws a perspective image of the building from the viewpoint of the camera based on the three-dimensional data, the planned position coordinates, the current position coordinates, and the orientation, and the perspective image is displayed. It is possible to display a composite image synthesized on the captured image on the display unit. In addition, when the first operation is received from the user, the control unit stores the current position coordinates at the time of receiving the first operation, and displays the perspective image in the display state at the time of receiving the first operation. It is possible to fix and update the captured image in accordance with changes in the current position coordinates and the orientation. Further, the control unit, when a second operation is received from the user after the first operation, a current position coordinate when the first operation is received and a current position coordinate when the second operation is received Can be used as a correction value for adding the difference to the current position coordinates to generate the perspective image.

  With this configuration, the information processing apparatus calculates a correction value based on the first operation and the second operation by the user, and synthesizes the perspective image of the building planned for construction and the captured image of the planned construction site in real time. It is possible to correct the misalignment between the two images at the time. The user simply performs the first operation, and then simply moves the information processing device until the perspective image is correctly displayed at the planned construction position in the captured image, and then performs the second operation. The image can be corrected.

  When the first operation is received from the user, the control unit stores the azimuth at the time when the first operation is received, and when the second operation is received, the control unit receives the first operation. A difference between the azimuth at the time and the azimuth at the time of accepting the second operation may be calculated, and the difference may be used as the correction value for the azimuth to generate the perspective image.

  As a result, the information processing apparatus can easily correct not only the current position coordinates but also the positional deviation regarding the azimuth (rotation direction and vertical direction).

  When the first operation and the second operation are accepted after the calculation of the correction value, the control unit calculates a new correction value and calculates the new correction value to the previously calculated correction value. The correction values may be cumulatively added and used to generate the perspective image.

  Thereby, the information processing apparatus can improve the accuracy of the correction value by adding the correction value additionally and cumulatively every time the first operation and the second operation are received from the user.

  The azimuth sensor may be capable of detecting an inclination angle of the information processing apparatus with respect to a horizontal axis. In this case, the control unit calculates a correction value in the vertical direction of the current position coordinates based on the inclination angle detected after the first operation and the distance between the current position coordinates and the planned position coordinates. May be.

  As a result, the information processing apparatus can easily correct the positional deviation in the vertical direction in the composite image.

  The operation reception unit may be a touch sensor on the display unit, and the display unit may be capable of displaying a predetermined button image. In this case, the touch sensor may be capable of receiving a button image pressing maintenance operation as the first operation and a button image pressing release operation as the second operation.

  Thereby, the user can correct the composite image by a simple operation of a so-called long press (press hold) operation and press release (release) operation of a button on the display unit.

  The control unit, when the direction in which the perspective image is to be synthesized is out of the display range of the captured image, indicates information indicating the direction in which the perspective image is to be synthesized on the captured image. You may control the said display part so that it may display.

  Thereby, the information processing apparatus can make the user easily grasp the planned building position of the building.

  The current position coordinates may include latitude, longitude, and altitude. In this case, the storage unit may store terrain data including the height of each point in a predetermined range including the planned position coordinates. In this case, the control unit calculates a value obtained by adding an average height of the human viewpoint to the height in the topographic data corresponding to the latitude and longitude in the current position coordinates in the vertical direction of the current position coordinates. The correction value may be used to generate the perspective image.

  Thereby, the information processing apparatus can correct the positional deviation in the vertical direction in the composite image with higher accuracy by using the terrain data.

  The said memory | storage part may memorize | store the adjacent surface three-dimensional data which show the surface facing the said building of the adjacent building adjacent to the said building, and the adjacent surface position coordinate of the said surface. In this case, before the composition of the perspective image, the control unit, based on the adjacent surface three-dimensional data and the adjacent surface position coordinates, of the adjacent building serving as the foreground of the perspective image on the captured image. A transparent surface may be drawn at a position corresponding to the adjacent surface position coordinate.

  As a result, the drawing of the adjacent building surface is given priority for the part where the image of the adjacent building that is the foreground of the perspective image overlaps with the perspective image. As a result, the portion of the perspective image corresponding to the surface part is not drawn. As a result of the synthesis, the image of the adjacent building that was captured is displayed for that part, and the foreground of the perspective image is displayed correctly. The Therefore, the information processing apparatus can easily execute the foreground drawing process of the perspective image by simply holding the three-dimensional data of the surface without performing complicated calculations.

An information processing method according to another aspect of the present invention includes:
Based on the three-dimensional data indicating the building planned to be built, the planned position coordinates where the building is scheduled to be built, the current position coordinates detected by the GPS sensor, and the camera orientation detected by the direction sensor, Drawing a perspective image of the building from the perspective of the camera;
Displaying a synthesized image obtained by synthesizing the perspective image on an image of a building planned site of a building imaged by the camera;
Storing the current position coordinates when the first operation is received when the first operation is received from the user;
Fixing the perspective image to a display state at the time of accepting the first operation, and updating the captured image according to changes in the current position coordinates and the orientation;
When a second operation is received from the user after the first operation, a difference between the current position coordinate at the time of the first operation reception and the current position coordinate at the time of the second operation reception is calculated. And
And releasing the fixing and using the difference as a correction value to be added to the current position coordinates for generating the perspective image.

A program according to another embodiment of the present invention is provided in an information processing device.
Based on the three-dimensional data indicating the building planned to be built, the planned position coordinates where the building is scheduled to be built, the current position coordinates detected by the GPS sensor, and the camera orientation detected by the direction sensor, Drawing a perspective image of the building from the viewpoint of the camera;
Displaying the composite image obtained by combining the perspective image on the image of the planned construction site of the building imaged by the camera;
Storing a current position coordinate when the first operation is received when the first operation is received from the user;
Fixing the perspective image to a display state at the time of accepting the first operation, and updating the captured image according to changes in the current position coordinates and the orientation;
When a second operation is received from the user after the first operation, a difference between the current position coordinate at the time of the first operation reception and the current position coordinate at the time of the second operation reception is calculated. Steps,
The fixing is released, and the step of generating the perspective image as a correction value for adding the difference to the current position coordinates is executed.

  As described above, according to the present invention, it is possible to correct a positional shift between both images when a perspective image of a building planned for construction and a captured image of a construction planned site are combined in real time. The effects described here do not necessarily limit the present invention, and the present invention only needs to achieve any of the effects described in the present disclosure.

It is the figure which showed the hardware constitutions of the portable terminal which concerns on one Embodiment of this invention. It is the figure which showed the structure of the screen displayed during execution of the application which concerns on one Embodiment of this invention. It is the flowchart which showed the flow of the screen transition operation | movement at the time of starting of the said application. It is the flowchart which showed the flow of the real-time photo compositing process by the said application. It is the flowchart which showed the flow of the real-time photograph composite display process by the said application. It is the figure which showed typically the error correction process using the movement of the user by the said application. It is the figure which showed the example of the real-time photograph synthetic | combination screen in executing the error correction process using the user's movement by the said application. It is the figure which showed the relationship between a geoid and a local coordinate system. It is the figure which showed the geoid figure of Japan vicinity. It is the figure which showed the example of the topography data used in order for the said application to correct | amend the error of height. It is the flowchart which showed the flow of the three-dimensional model drawing process by the said application. It is the figure which showed the relationship between the perspective image in the said photograph composition, and its foreground. It is a figure explaining the foreground process in the conventional photo composition. It is the figure which showed the flow of the drawing process of the perspective image by the said application, and its foreground. It is the figure which showed typically the drawing process of the perspective image and its foreground by the said application. It is the figure which showed the coordinate axis of the site of the construction schedule of the building which the said application uses. It is the figure which showed the coordinate axis of the portable terminal which the said application uses.

  Hereinafter, embodiments according to the present invention will be described with reference to the drawings.

[Hardware configuration of mobile terminal]
FIG. 1 is a block diagram showing a hardware configuration of the mobile terminal according to the present embodiment.

  This portable terminal is used for real-time photo composition processing for studying the landscape of a building after construction.

  Here, specifically, the portable terminal is, for example, a smartphone, a mobile phone, a tablet PC (Personal Computer), a digital still camera, a PDA (Personal Digital Assistant), a portable AV player, a portable game machine, an electronic book, or the like. This is a portable information processing apparatus.

  As shown in the figure, a mobile terminal 100 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a nonvolatile memory 13, a display unit 14, a communication unit 15, a camera 16, a GPS sensor 17, a magnetic sensor ( An electronic compass) 18 and an acceleration sensor 19.

  The CPU 11 appropriately accesses the RAM 12 or the like as necessary, and comprehensively controls each block of the mobile terminal 100 while performing various arithmetic processes. The RAM 12 is used as a work area for the CPU 11 and temporarily holds the OS, various applications being executed, and various data being processed (three-dimensional data, captured image data, coordinate data, correction values, etc., which will be described later). .

  The nonvolatile memory 13 is, for example, a flash memory or a ROM (Read Only Memory), and fixedly stores an OS to be executed by the CPU 11, a program such as a software module described later, and firmware such as various parameters.

  In particular, in the present embodiment, the nonvolatile memory 13 stores the real-time photo composition processing application. The non-volatile memory 13 includes, as data used for execution of the application, image data (photo data) captured by the camera 16, three-dimensional data (geometric shape data, surface finish data, etc.) of a building to be built. Attribute data), position coordinate data of the planned construction site, and the like.

  The display unit 14 is, for example, an LCD (Liquid Crystal Display) or an OELD (Organic ElectroLuminescence Display), and corrects an image captured by the camera 16, a composite image obtained by combining the captured image with a perspective image of a building, and the composite image. A user interface or the like is displayed. Typically, the display unit 14 is integrated with a touch panel and can accept a user's touch operation. That is, the display unit 14 also functions as an operation detection unit.

  Although not shown, the mobile terminal 100 also includes hardware buttons such as a power button as the operation detection unit.

  The communication unit 15 communicates with other devices using a wireless LAN (IEEE802.11 or the like) such as WiFi (Wireless Fidelity) or a 3G or 4G network for mobile communication.

  The camera 16 captures a still image (photograph) and a moving image by an imaging element such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) sensor. The camera 16 has various functions such as a selection function of various shooting modes such as a night view mode and a portrait mode, a panoramic shooting function, a 3D shooting function, a white balance adjustment function, an aperture adjustment function, and the like. These functions can be set by operating 14 (touch panel) or the like.

  The GPS sensor 17 captures radio waves from three or more satellites, calculates coordinates describing the position where the mobile terminal 100 exists in the earth coordinate system based on the distance to each satellite, and calculates the coordinates of the ellipse. The position information (latitude / longitude / elevation information) of the mobile terminal 100 is detected by applying it on the body.

  Instead of using the GPS sensor 17, the position information may be detected by performing trilateral surveying with the base station by wireless communication by the communication unit 15.

  The magnetic sensor (orientation sensor) 18 can detect the direction of magnetic north as viewed from the mobile terminal 100. The acceleration sensor 19 can detect the vertical (gravity) direction. From the detected magnetic north orientation and vertical orientation, the attitude of the mobile terminal 100 (the angle of the camera 16) is calculated.

[Operation of mobile terminal]
Next, the operation of the mobile terminal 100 configured as described above will be described. The operation is executed in cooperation with hardware such as the CPU 11 and various sensors of the mobile terminal 100 and software stored in the nonvolatile memory 13. In the following description, for the sake of convenience, the CPU 11 is an operation subject.

(Application screen structure)
FIG. 2 is a diagram showing a configuration of a screen displayed during execution of the real-time photo composition application.

  As shown in the figure, the application can display five screens: a startup screen, a display model selection screen, a real-time photo composition screen, a playback selection screen, and a playback screen.

  The startup screen has selection buttons for each screen of “display model selection”, “reproduction display”, and “end”, and transitions to each screen according to the user's selection.

  The display model selection screen displays a list of three-dimensional models of buildings such as houses to be built, and selects one three-dimensional model by a user operation.

  The real-time photo composition screen is a composite photo that displays a composite photograph of a perspective image of a building drawn from the selected three-dimensional model and an image of a planned construction site (hereinafter, a background image) captured by the camera 15. It has a display area, a “calibration” button for correcting the positional deviation between the perspective image and the background image, and a “shutter” button for recording the background image.

  As will be described in detail later, when the “calibration” button is pressed while the perspective image and the background image are displayed, the perspective image is fixed to the screen when it is pressed, and only the background image is displayed on the mobile terminal 100. It will be updated as it moves.

  When the “shutter” button is operated, the background image is displayed in combination with the background image, the position coordinates acquired from the GPS sensor 17 at that time, the posture (azimuth and inclination) of the camera 16, time, other environmental conditions such as weather, and the background image. Identification information of a perspective image (three-dimensional model) of a building is also recorded.

  For example, the “calibration” button is displayed at the lower left of the display unit 14 and the “shutter” button is displayed at the lower right of the display unit 14, for example.

  The reproduction selection screen displays, for example, a matrix of reduced images of the background image recorded by the shutter, and selects one background image by a user operation.

  The reproduction screen reproduces the photo composition under the same conditions as the real-time photo composition in the field, using the background image selected by the reproduction selection screen, the corresponding perspective image, and the correction value stored at the time of calibration.

(Screen transition operation at application startup)
Next, the screen transition operation when starting the application will be described. FIG. 3 is a flowchart showing the flow of the screen transition operation.

  As shown in the figure, first, the CPU 11 determines whether or not the model selection button has been pressed by the user on the model selection screen (step 31).

  If it is determined that the model selection button has been pressed (Yes), the CPU 11 transitions to real-time photo composition processing (step 32).

  On the other hand, if it is determined that the playback selection button has been pressed on the playback selection screen (Yes in step 33), the CPU 11 transitions to playback photo composition processing (step 34).

  When it is determined that the end button has been pressed (Yes in step 35), the CPU 11 ends the application.

(Real-time photo composition processing)
Next, the real-time photo composition process will be described. FIG. 4 is a flowchart showing the flow of the real-time photo composition process.

  As shown in the figure, when displaying the model selection screen (step 41), the CPU 11 determines whether or not a three-dimensional model of a specific building has been selected from the list by the user (step 42).

  When determining that the three-dimensional model has been selected (Yes), the CPU 11 reads the selected three-dimensional model file from the nonvolatile memory 13 (step 43).

  Subsequently, the CPU 11 executes real-time photo composition display processing using the read file (step 44).

  Here, details of the real-time photo composition display processing will be described. FIG. 5 is a flowchart showing the flow of the real-time photo composition display process.

  As shown in the figure, the CPU 11 acquires a background image by the camera 16 and displays it on the display unit 14 (step 51).

  Then, CPU11 acquires the present position coordinate of the portable terminal 100 by the GPS sensor 17 (step 52).

  Subsequently, the CPU 11 determines whether or not the terrain data is used for the process of correcting the displacement of the altitude in the position coordinates (step 53). If it is determined that terrain data is to be used (Yes), calibration processing using the terrain data is executed (step 54). This process will be described later. Whether or not to use the terrain data may be set in advance by the user in the application, or the terrain data may be used when the application determines the complexity of the terrain and the terrain is more than a certain degree.

  If it is determined that the topographic data is not used (No), the CPU 11 acquires the attitude data (camera angle) of the mobile terminal 100 from the detection values of the magnetic sensor 18 and the acceleration sensor 19 (step 55).

  Subsequently, the CPU 11 creates a camera parameter cam including the current viewpoint coordinates, gaze vector, upper vector, focal length, and the like from the acquired current position coordinates and posture data (step 56).

  And CPU11 performs the drawing process of the three-dimensional model of a building using the said camera parameter cam (step 57).

  Here, the details of the three-dimensional model drawing process will be described.

  In the 3D model drawing process, if an error occurs between the position of the 3D model (perspective image) of the building using the current camera parameter cam and the planned building position of the building in the background image, Processing for correcting the error (calibration processing) is executed.

  In the present embodiment, the mobile terminal 100 can use movement by the foot of the user carrying the mobile terminal 100 as one of the error correction processes. Hereinafter, the error correction process by the movement of the user will be described.

-Error correction processing by user movement-
The position information acquired by the mobile terminal 100 from the GPS satellite can be used in the application together with a function provided by an operating system on the mobile terminal 100 such as Android or iOS. The coordinate system of the GPS system is a three-dimensional orthogonal coordinate system with the center of the earth as the origin. However, the position information provided by such a function is already in the latitude, longitude as described above for convenience of use. And the calculation result processed into the form of elevation.

  In order to grasp the measurement error of the horizontal position, conventionally, the portable terminal is placed at a point where the latitude and longitude are known, and the deviation of the measured value from the positive value is calculated. Thereafter, an error is corrected (calibrated) by using the value obtained by adding the deviation value to the measured value for the necessary processing.

  However, the above work requires skill and labor for searching for a point where the position coordinates can be specified on the map or for obtaining the difference by going to that point. According to the method according to the present embodiment, this correction can be performed by a very simple method.

  FIG. 6 is a diagram conceptually showing the error correction processing by the movement of the user. FIG. 7 is a diagram showing an example of a real-time photo composition screen during execution of the error correction process.

  First, as shown in FIGS. 6 and 7A, the user refers to the photograph composition screen 70 of the building perspective image P and the background image B displayed by the mobile terminal 100, and arranges the perspective image P and the image. Both the site C to be constructed (such as an open space) are moved to a place displayed on the screen.

  Next, the user presses the calibration button 71 on the screen 70. Then, the display of the perspective image P using the position coordinates immediately before the button 71 is pressed is fixed in that state, and only the background image B is updated. At this time, measurement values from the sensors such as the GPS sensor 17 when the button is pressed are stored.

  Next, the user moves while holding down the calibration button 71 and walks around to find a place where the perspective image P is synthesized and displayed at the correct position and orientation. When the combined display that is sufficiently matched is obtained, the user releases the calibration button 71 at that location (see FIG. 7B).

  Then, the portable terminal 100 calculates the difference between the measured value from each sensor such as the GPS sensor 17 at the moment when the button is released and the measured value at the moment when the calibration button 71 is pressed. calculate.

  Thereafter, the portable terminal 100 draws the perspective image P using the difference as a correction value and adding the correction value to the measurement value from each sensor such as the GPS sensor 17. When the viewpoint position changes, the display of the perspective image P changes so that the basket is fixed to the site C.

  When this correction operation is further performed in a state where a correction value has already been given, the mobile terminal 100 additionally adds the new correction value to the correction value so far and cumulatively adds it to the perspective image P. Used for drawing.

  Here, when the orientation in which the perspective image P is to be synthesized is out of the display range of the background image B on the screen 70, the mobile terminal 100 synthesizes the perspective image P on the background image B. Information (for example, right and left texts and arrow images indicating them) indicating the position where they should exist can be displayed.

  The portable terminal 100 also performs correction by the same method as the correction by the movement with respect to the rotation (the detection value of the magnetic sensor 18). That is, when the user presses the calibration button 71 to change the horizontal direction without moving and releases the calibration button 71 in a state where the perspective image P is displayed at the correct position, the correction value in the rotation direction is calculated. Is done.

  Regarding the correction in the vertical direction (elevation), for example, when the building is displayed above the position where it should originally be built (when the elevation of the viewpoint position is measured lower than the actual position by the GPS sensor 17), The vertical rotation (with respect to the horizontal axis) of the mobile terminal 100 is used to calculate the correction value.

  That is, the user presses the calibration button 71 in a state where the perspective image P is displayed at an angle looked up by the user, rotates the portable terminal 100 up and down to a horizontal angle, and releases the calibration button 71.

  At this moment, the perspective image P of the building is displayed in a state where the building is inclined to the ground. Next, when a change in the vertical angle is included in the difference from when the calibration button 71 is pressed to when it is released, the mobile terminal 100 determines the angle as the current position coordinates of the mobile terminal 100, and Based on the horizontal distance between the position coordinates of the building and the planned construction position, it is converted into a vertical height error corresponding to the vertical angle error of the portable terminal 100, and the value is corrected. As a result, after this conversion is completed, the perspective image P recovers the vertical posture.

  By realizing the mechanism as described above, the user does not need to read or input numerical values necessary for correction, and can quickly acquire and use correction data.

-Height correction using position and topographic data-
Next, the height correction process using the topographic data shown in step 53 and step 54 of FIG. 5 will be described.

  Compared with the latitude / longitude data obtained from the GPS sensor 17, the altitude data tends to have a larger error. As a factor of this, the international geoid data used for calculating the altitude inside the processing system of the GPS sensor 17 is more accurate than the geoid that the Geospatial Information Authority of Japan has measured and published in the vicinity of Japan. Inferred to be low.

  FIG. 8 is a diagram showing the relationship between the geoid and the local coordinate system. FIG. 9 is a diagram showing a geoid diagram around Japan.

  In the local coordinate system, the tangent plane to the geoid surface is the XY plane, and the altitude (geoid height) from the geoid surface corrected for unevenness of the isogravity surface with respect to the earth ellipsoid corresponds to the altitude obtained from the GPS coordinates ( FIG. 8). However, the irregularities of the geoid surface are complex near Japan, especially on the Pacific side of eastern Japan, while the highest point in the country exists near the Abukuma Mountains, there is a deep valley near the Japan Trench, between which the steep slope of the geoid Therefore, the resolution and accuracy of the geoid greatly affect the height error (FIG. 9).

  As described above, in order to correct the height error, it is possible to execute correction by rotating the mobile terminal 100 in the vertical direction. In the present embodiment, in addition to this, the ground height of the region that is the range of the viewpoint movement or the height of the road or the like in the city area is stored in the mobile terminal 100 as data. Using this data, the mobile terminal 100 obtains the height of the ground at that point from the latitude and longitude, and adds the average pedestrian viewpoint height (for example, 1.5 m) to the viewpoint position. This can be used for real-time photo composition instead of the above-described height error correction by the rotation of the mobile terminal 100.

  This topographic data can be created from digital map elevations in metropolitan areas and disaster areas. FIG. 10 shows an example of this terrain data. The topographic data can also be created from the building data.

  The ground height mesh data used to determine the viewpoint height is expressed as a 256-tone monochrome image with 0.1m accuracy height data if the elevation difference between the highest and lowest points is within 25.6m. For example, it can be stored in a relatively versatile format such as a JPEG format, for example, in a server on the cloud with a relatively small capacity. Therefore, the data can be distributed to the mobile terminal 100 in a short time, restored at the mobile terminal 100, and used for position coordinate calculation.

  If the past townscape created as three-dimensional data or the ground surface of the future housing complex is different from the current situation, by using the ground data corresponding to each data, walking around the current site, the kite is also past or planned It becomes possible to walk around the ground surface of the plan.

  FIG. 11 is a flowchart showing the flow of the three-dimensional model drawing process. Of these steps, steps 81 to 88 show the flow of the error correction process by the movement.

  As shown in the figure, the CPU 11 determines whether or not the calibration button 71 is pressed (step 111).

  When determining that the calibration button 71 is pressed (Yes), the CPU 11 determines whether or not the hold flag indicating whether or not the button 71 is continuously pressed is true at that time (step 112). ).

  When determining that the hold flag is false (No), the CPU 11 sets the hold flag to true (step 113).

  Subsequently, the CPU 11 stores the position coordinate and posture state m at that time as the position coordinate and posture state h when the calibration button 71 is pressed (step 114).

  On the other hand, if it is determined in step 111 that the calibration button 71 has not been pressed (No), the CPU 11 determines whether or not the hold flag is true at that time (step 115).

  When determining that the hold flag is true (Yes), the CPU 11 sets the hold flag to false (step 116). That is, the CPU 11 determines that the user's finger is released from the calibration button 71 at this time.

  Subsequently, the CPU 11 calculates a difference between the stored state h and the current state m, and adds the value to the correction term d (step 117).

  Then, the CPU 11 adds the correction term d to the current value m and draws a three-dimensional model (perspective image P) of the building (step 118).

-Real-time foreground processing-
Next, foreground processing of the perspective image P will be described.

  When compositing a perspective image P created from 3D data describing a planned building and the current background image, the subject in front of the building is hidden by the perspective image P of the building. It is necessary to deal with it. 12A shows a composite photo in which the foreground subject F is hidden by the perspective image P, and FIG. 12B shows a composite photo in which the foreground subject F is correctly displayed without being hidden by the perspective image P.

  FIG. 13 shows foreground processing in a conventional photo composition method. In the conventional photo compositing process, since the viewpoint position is fixed, in order to hide the building using this, the subject on the image existing in front of the building is displayed on the building. In addition to the background image, a foreground image is prepared in advance, and a method of overdrawing and displaying in the order of the background image → the building → the foreground image is used. The foreground image used here is time-consuming and time-consuming preparatory work so as not to hide the building on the back side by making the part other than the subject in front of the building transparent by image processing. It was processed by hand.

  However, in the real-time photo composition in this embodiment, it is not realistic to create such a foreground image F every time the viewpoint is moved for the background image B that changes constantly.

  Therefore, in the real-time foreground processing according to the present embodiment, the foreground image is not used, and instead, an existing building (or a part thereof) that is likely to be a foreground of a building to be built is used as a foreground model in advance. It is created as three-dimensional data for this purpose, and this is drawn prior to drawing the perspective image of the building.

  FIG. 14 is a diagram showing a flow of drawing processing of a perspective image and its foreground in the present embodiment. FIG. 15 is a diagram schematically showing the drawing process of the perspective image and its foreground.

  FIG. 14A shows a state in which a background image B is captured, FIG. 14B shows a state in which a foreground model is drawn, and FIG. 14C shows a state in which a perspective image of a building is drawn.

  As shown in FIG. B, by rendering a transparent image W in which the colors of all the faces of the three-dimensional data of the foreground model are transparent, the display screen does not change at this stage, and the background image Is still displayed.

  However, as an effect of this, the transparent drawing process of the foreground model F is performed on the portion behind the foreground model F as viewed from the viewpoint position in the perspective image P of the building to be three-dimensionally drawn next. As a result of being prioritized, the image is hidden without being drawn, and as a result, the background image of that portion (in this case, the actual image of the front building) is displayed (see FIG. 14C).

  The three-dimensional data of the building serving as the foreground model F need not be created in detail for the entire building. As shown in FIG. 15, for example, it is assumed that there is an open space for one house in a group of houses that form a townscape, and a new building is planned there. When creating the three-dimensional model of the planned building, the mobile terminal 100 creates only data Wa and Wb adjacent to (facing) the planned building among the walls of the existing buildings on both sides. To do. And the portable terminal 100 can achieve the intended purpose by attaching a transparent attribute to this and drawing it prior to drawing the perspective image P of the building.

  Returning to FIG. 11, the flow of the real-time foreground processing will be described.

  First, the CPU 11 clears the depth buffer (step 119), and draws the three-dimensional model of the wall surface of the adjacent building on the screen which is not desired to be hidden by the perspective image P as a transparent object in the depth buffer (step 120).

  Then, the CPU 11 draws the model of the perspective image P as an opaque object in the depth buffer (step 121).

  Thus, the real-time photo composition process ends (step 44 in FIG. 4).

  Returning to FIG. 4, thereafter, the CPU 11 determines whether or not the measured value of any of the sensors has changed (Yes in step 45).

  If it is determined that the value of any of the sensors has changed (Yes), the CPU 11 executes the real-time photo composition process using the changed value (step 44).

  If it is determined that the value of any sensor has not changed (No), the CPU 11 determines whether the shutter button 72 has been pressed (step 46).

  If it is determined that the shutter button 72 has been pressed (Yes), the CPU 11 uses the background image, the position coordinates acquired from the GPS sensor 17 at that time, the orientation (azimuth and inclination) of the camera 16, correction values, and photo composition. The camera parameters after correction, date and time, other environmental conditions such as weather, and identification information of the perspective image (three-dimensional model) of the building synthesized and displayed on the background image are recorded (step 47).

  If it is determined that the shutter button 72 has not been pressed (No), the CPU 11 determines whether an application termination request has been input from the user (step 48), and if it is determined that the termination request has been input. (Yes), the application is terminated.

(Calculation method in error correction)
Here, an example of the calculation method in the error correction will be described in detail.

  In the present embodiment, the mobile terminal 100 is treated as a rigid body having a total of six degrees of freedom, ie, three related to the position and three related to the posture.

  In order to express the position and orientation of the mobile terminal 100, in this processing system, as shown in FIG. 16, three-dimensional coordinates are fixed on the site, with the east as the X axis, the north as the Y axis, and the top as the Z axis. Use (x, y, z).

  In a coordinate system fixed on the ground, the east is the x-axis and the north is the y-axis. In surveying, the north is the x-axis and the west is the y-axis. Also, in the CG field that started with photography, there is a custom that the east is the x axis and the top is the y axis.

  The position, direction, and shape of the building to be displayed are expressed using the coordinate system of the site.

  On the other hand, as shown in FIG. 17, as coordinate axes fixed to the mobile terminal 100, in a posture in which the user holds the screen (display unit 14) horizontally and horizontally in front, W on the right, U on the top, and V on the front. Use coordinate axes.

  Of the above six degrees of freedom, three are coordinate values related to the position.

  The viewpoint position P (x, y, z) is expressed using coordinate values obtained by converting the latitude and longitude elevation obtained from the GPS sensor 17 into the coordinate system of the site.

  The origin of the site coordinate system is the basic position.

  For the latitude and longitude elevation of the site origin, the numerical value obtained by reference point surveying etc. is used in building design.

  The purpose of this embodiment is to simply and correctly correct the deviation D between the coordinate value Pm calculated from the measured value of the GPS sensor 17 and the coordinate value Pt measured from the site origin.

  For example, when a measurement coordinate value measured by GPS at the position of the site origin is a correction value D1, a value obtained by subtracting D1 from the measurement value is a correct site coordinate value.

  For example, when the coordinate value measured from the site origin where the GPS measurement value indicates the origin is the correction value D2, the value obtained by adding D2 to the measurement value is the correct site coordinate value.

  Even without moving to the vicinity of the origin, it can be corrected by the deviation measured by the calibration operation on the photo composition screen, and the accuracy of the correction value already obtained can be improved.

  That is, if the movement during the holding by the calibration button 71 is d, the correction value can be acquired and improved by subtracting d from D1 or adding it to D2.

  Of the above six degrees of freedom, three are values related to the posture.

  The portable terminal 100 (rigid body) includes the acceleration sensor 19 and the magnetic sensor 18 as described above.

  The measured value of the acceleration sensor 19 is (0, 0, 0) in the coordinate system of the terminal 10 when it is free-falling or in a weightless space. When resting on the desk, it receives acceleration upward from the desk. Therefore, it can recognize itself whether it is placed horizontally or vertically. (In the coordinate system fixed to the portable terminal 100, it can be seen where the top is).

  The gravitational field on the earth is not uniform and is distorted. For example, in the vicinity of the epicenter of the Great East Japan Earthquake (trench, plate boundary), there is a dense material in the basement, and the isogravity surface, that is, the sea surface is recessed by several tens of meters from the earth ellipsoid. In the vicinity of Mt. Abukuma, the isogravity surface is the highest in the vicinity of Japan.

  For this reason, the mountains in this area are higher in geometric shape than the altitude, and the sea level is inclined eastwardly downward near the Pacific coast of Tohoku Japan. As described above, the uneven state of the isogravity surface is measured and disclosed as a geoid, and more accurate height and upper vector can be obtained by adding this data.

  By obtaining the geoid of the place from the position coordinates obtained by the GPS sensor 17, it is possible to obtain the accurate altitude (= height from the distorted isogravity surface) of the mobile terminal 100 and the orientation of the center of the earth. Survey data before the year 2000 AD began work near Tokyo in the Meiji period, and the iso-gravity plane (horizontal perceived by the level) tilted eastward relative to the earth ellipsoid was Since the surface is considered to be in contact with the circular body, it cannot be connected to the world coordinate system measured by the GPS sensor 17 unless complicated conversion is performed.

  The magnetic sensor 19 detects magnetic north as described above. Magnetic north is not aligned with the north pole (true north) of the earth's axis. In addition, the deviation varies depending on the location due to the distribution of iron, etc. in the basement. The geographical distribution of the deviation between magnetic north and true north is also measured by the Geographical Survey Institute and published as data. From the position coordinates and distribution map obtained by GPS, the deviation from true north of the magnetic north at that location can be known and corrected.

  In addition, the magnetic field on the ground in three dimensions is not horizontal. However, since the vertical direction can be measured by the acceleration sensor 19 as described above, it is possible to calculate the horizontal magnetic north by correcting the vertical direction.

  The coordinate system fixed to the portable terminal 100 is the W axis when the screen is held horizontally, the V axis is the front, the U axis is the top, and the attitude of the portable terminal 100 is measured directly above this coordinate system ( Self-recognition is possible without receiving external information transmission from the direction of acceleration≈Z axis of site coordinate system) and true north (magnetic north≈Y axis of site coordinate system).

  Therefore, it can be known in which direction each axis of the site coordinate system is viewed from the coordinate system of the mobile terminal 100. Conversely, it is possible to reversely calculate the orientation of the mobile terminal 100 according to the coordinate system of the site.

  In general, the posture of a rigid body such as the mobile terminal 100 is an x, y, z component measured in the site coordinate system of each unit vector in the W axis, V axis, and U axis directions which are coordinate axes fixed to the device. It is often expressed by a 3 × 3 matrix indicating Although the calculation process is simple, the number of calculations is large and the rotational degree of freedom is only 3. Therefore, the data storage method is redundant.

  On the other hand, rotation can also be expressed by using a quaternion Q (scalar component is 1, vector component is 3) obtained by expanding a complex number into four dimensions.

When the four components constituting Q are represented as (t, x, y, z), t is referred to as a scalar component of Q, and (x, y, z) is referred to as a vector component.
The length of the vector component of Q is √ (x ² + y ² + z ² ).
The length of Q is √ (t ² + x ² + y ² + z ² ).
The sum of the two quaternions (t ₁ , x ₁ , y ₁ , z ₁ ) and (t ₂ , x ₂ , y ₂ , z ₂ ) is (t ₁ + t ₂ , x ₁ + x ₂ , y ₁ + y ₂ , z ₁ + z ₂ ).
The product of two quaternions (t ₁ , x ₁ , y ₁ , z ₁ ) and (t ₂ , x ₂ , y ₂ , z ₂ ) is (t ₁ * t ₂ -x ₁ * x ₂ -y ₁ * y ₂ -z ₁ * z ₂ , t ₁ * x ₂ + x ₁ * t ₂ + y ₁ * z ₂ -z ₁ * y ₂ , t ₁ * y ₂ -x ₁ * z ₂ + y ₁ * t ₂ + z ₁ * x ₂ , t ₁ * z ₂ + x ₁ * y ₂ -y ₁ * x ₂ + z ₁ * t ₂ ).

The reciprocal (1 / Q) of the quaternion Q = (t, x, y, z) is (t / r ² , -x / r ² , -y / r ² , where Q is r. , -z / r ² ).
In particular, when the length of Q is 1, the reciprocal of Q can be obtained by simply inverting the sign of the vector component while leaving the scalar component unchanged. Q * (the reciprocal of Q), that is, Q / Q is (1,0,0,0).

  The rotation of a quaternion V by the quaternion Q can be obtained by calculation (ie, Q * V / Q) Q * V * (the reciprocal of Q). At this time, the vector component of V rotates around a certain rotation axis by a certain angle.

  When the vector component of Q representing rotation is viewed as a vector, the direction of this vector is equal to the direction of the rotation axis.

  A quaternion Q = (1,0,0,0) with a scalar component of 1 and a vector component of all zero means no rotation.

In addition, when the scalar component is zero and the length of the vector component is 1, it represents 180 degree rotation.
Q = (0, x, y, z), x ² + y ² + z ² = 1

  In that case, the rotation -Q = (0, -x, -y, -z) in which the vector component represents the rotation axis and the signs of the vector components are all reversed yields the same result.

In other cases, it represents a rotation of a certain angle Ψ (0 to 180 °) about the vector component. In this case, the scalar component t of Q is cos (ψ / 2), and the length of the vector component is r = sin (ψ / 2).
Q = (cos (Ψ / 2), Px sin (Ψ / 2), Py sin (Ψ / 2), Pz sin (Ψ / 2))
Here, P is a vector representing a rotation axis having a length of 1, and Ψ is a rotation angle.

For example, when the mobile terminal rotates horizontally (panning in a video camera), the rotation axis is a vertical line.
P = (0,0,1)
If the rotation angle is 60 degrees, Q = (about 0.866, 0, 0, 0.5)
(About 0.866 of real component is cos30 °, (√3) / 2, z component is 1/2 at 1 × sin30 °)
The reciprocal of Q is (about 0,866, 0, 0, -0.5)
It is.

The scalar component t is derived from the vector component,
t = cos (Ψ / 2) = √ (1-x ² -y ² -z ² )
Therefore, it is not necessary to store the scalar component as data.

  The attitude of the mobile terminal 100 can be expressed as a rotation with respect to a certain basic attitude (for example, an attitude held horizontally toward true north). Therefore, it can be expressed by one quaternion Q representing this rotation. Furthermore, since the generality is not lost even if the length of Q is limited to 1, it is sufficient to store the numerical value of 3 as its vector component as data. If Q is determined, Q * (0,0,1,0) / Q: each component Q * (0,0,0,1) of the Y-axis vector (gaze vector) of the mobile terminal 100 viewed from the site coordinate system ) / Q: The attitude of the mobile terminal 100 is determined by each component of the Z-axis vector (upper vector) of the mobile terminal 100 as viewed from the site coordinate system.

If the mobile terminal 100 in the posture expressed by Q1 is further rotated by Q2 and the posture is Q3,
Q3 = Q2 * Q1, and therefore, for example, the gaze vector V in the basic posture is
Q2 * (Q1 * V / Q1) / Q2 = Q3 * V / Q3 is converted.

  By using a method of storing only three values constituting a vector component of a quaternion representing such rotation as data, when the rotation is repeated, an object other than an error related to rotation is deformed (enlarged, reduced, (Affine transformation) can be prevented. Further, the number of calculations can be reduced as compared with the case of using a matrix.

  In the present processing system, the measured attitude of the mobile terminal 100 is transmitted to the display system in the form of a gaze vector and an upper vector expressed in the site coordinate system.

The posture of the mobile terminal 100 can also be expressed by using the Euler angle and rotating the yaw angle (Y) horizontally from the basic posture, rotating the pitch angle (P) up and down, and rotating the roll angle (R) sideways. Using the roll R, pitch P, and yaw Y calculated from the magnetic sensor 19 and the acceleration sensor 18, the quaternion Q representing the rotation can be further calculated as follows.
Q = (cos (R / 2), 0, sin (R / 2), 0) * (cos (P / 2), sin (P / 2), 0,0) * (cos (Y / 2), (0,0, sin (Y / 2))
This rotation corresponds to the procedure of rotating the azimuth around the U axis by the yaw angle from the basic posture, then rotating the pitch angle around the W axis, and finally rotating the roll angle around the V axis.

  It is an object of the present embodiment to correctly correct the deviation Qd between the attitude Qm calculated from the acceleration sensor 19 and the magnetic sensor 18 and the attitude based on the site coordinate system. For example, if the posture calculated with the measured value from the sensor when the mobile terminal is placed in the basic posture of the site coordinate system is the error Qd1, the posture Qm obtained from the measured value of an arbitrary posture is corrected to Qm / Qd1. The correct posture can be obtained.

  Also, assuming that the attitude obtained by sensor measurement values corresponding to the basic attitude is Qd2 as the attitude measured in the site coordinate system, the correct attitude is obtained by correcting Q obtained from the measured value of any attitude to Q * Qd2. That is, a more accurate display is obtained.

  Qd1 is a quaternion indicating the rotation from the correct basic posture to the basic posture on the measurement value. Qd2 is a quaternion indicating rotation from the basic posture on the measurement value to the correct basic posture. Qd1 and Qd2 have a relationship of Qd1 * Qd2 = 1, in other words, Qd2 = 1 / Qd1.

  If the rotation of Qd is performed while the hold operation for performing correction by looking at the synthesized display result is performed by the user, Qd1 is divided by Qd or Qd2 is multiplied by Qd. The accuracy of the correction value Qd1 or Qd2 can be further increased.

  U1 and V1 represent the measured values of the upper vector and gaze vector before the hold operation as quaternions, and represent the measured values of the upper vector and gaze vector in quaternions when the hold was released. Is U2, V2, the right direction vectors are obtained as W1 = V1 * U1 and W2 = V2 * U2, respectively.

Using these, the square of the rotation Qd during the hold period is
Qd ² = (W2 * W1 + V2 * V1 + U2 * U1 + (1,0,0,0)) * (-0.5,0,0,0)
Can be obtained as

However, when the scalar component of Qd ² is −1, that is, when the rotation is 180 °, the vector components of Qd ² are all zero, so that Qd cannot be obtained. In such a case, the vector component of Qd is the length of the vector component of (V2-V1) * (U2-U1) or the vector component of (W2-W1) * (V2-V1). The one with a larger value is directly obtained by normalization, and the scalar component is set to zero.

Qd from Qd ^2, without the use of trigonometric functions can be calculated by the following procedure.
The components of ^{Qd 2, (T, X,} Y, Z), the component of Qd (t, x, y, z) When, t is √ ((1 + T) / 2).
If k = (√ ((1-T) / 2)) / (√ (1-T ² )),
x = kX, y = kY, z = kZ.
However, when T is 1, k = 0. Further, when T is −1, the process of [0176] is completed, so this process is not necessary.

[Modification]
The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present disclosure.

  In the above-described embodiment, the perspective image is fixed when the calibration button 71 is pressed, and after the pressing is maintained (long touch), the correction value is calculated when the finger is released. However, the perspective image may be fixed simply by touching the button 71, and the correction value may be calculated by the next touch operation.

  In addition, a button for fixing a perspective image and a button for calculating a correction value may be prepared separately. That is, the operation for fixing the perspective image and the operation for calculating the correction value do not have to be linked.

  In the above-described embodiment, each value of the GPS sensor, the magnetic sensor, and the acceleration sensor is corrected. However, any one or two of these values may be corrected.

  The processing shown in each flowchart of the above-described embodiment is an example, and the object of the present invention can be achieved by other processing.

11 ... CPU
12 ... RAM
DESCRIPTION OF SYMBOLS 13 ... Non-volatile memory 14 ... Display part 15 ... Communication part 16 ... Camera 17 ... GPS sensor 18 ... Magnetic sensor 19 ... Acceleration sensor 70 ... Photograph composition screen 71 ... Calibration button 71 ... Button 72 ... Shutter button 100 ... Portable terminal

Claims (7)

An information processing apparatus,
A storage unit capable of storing planned position coordinates of a building to be built and three-dimensional data indicating the building, and terrain data including the height of each point in the viewpoint movement range including the planned position coordinates;
A camera capable of capturing an image of the building planned site of the building;
A position sensor capable of detecting current position coordinates including latitude, longitude and altitude of the information processing apparatus;
An orientation sensor capable of detecting the orientation in which the camera is facing;
A display unit;
Based on the three-dimensional data, the planned position coordinates, the current position coordinates, and the azimuth, a perspective image of the building is drawn from the viewpoint of the camera, and the perspective image is captured. Display the synthesized image synthesized above on the display unit,
A value obtained by adding an average height of the human viewpoint to the height in the terrain data corresponding to the latitude and longitude in the current position coordinates is used as a value in the vertical direction of the current position coordinates. An information processing apparatus comprising: a control unit that can be used for drawing. The information processing apparatus according to claim 1,
The storage unit stores adjacent surface three-dimensional data indicating a surface facing the building of an adjacent building adjacent to the planned construction site of the building and adjacent surface position coordinates of the surface,
The control unit, before synthesizing the perspective image, based on the adjacent surface three-dimensional data and the adjacent surface position coordinates, on the captured image, the adjacent building that is the foreground of the perspective image. An information processing apparatus that draws a transparent surface at a position corresponding to surface position coordinates. The information processing apparatus according to claim 1,
The control unit includes information indicating an orientation in which the perspective image is to be synthesized on the captured image when the orientation in which the perspective image is to be synthesized is out of a display range of the captured image. An information processing apparatus that controls the display unit to display. The information processing apparatus according to claim 1,
An operation receiving unit capable of receiving an operation of a user holding the information processing apparatus;
When the first operation is received from the user, the control unit records at least the current position coordinates, the azimuth, and the identification information of the three-dimensional data together with the captured image. Information processing device. The information processing apparatus according to claim 4,
The control unit receives the current position coordinates and the orientation recorded at the time of the first operation when the user's operation of selecting one image from the plurality of recorded images is received, and the recording An information processing apparatus that causes the display unit to display a composite image obtained by combining the perspective image with the selected image based on the three-dimensional data having the identified identification information. Three-dimensional data indicating the building planned to be built, the planned position coordinates of the building planned to be built, the current position coordinates including the latitude, longitude, and altitude of the information processing device detected by the position sensor, and the direction sensor Draw a perspective image of the building from the viewpoint of the camera based on the orientation of the camera,
Displaying a synthesized image obtained by synthesizing the perspective image on the image of the planned construction site of the building imaged by the camera;
Based on the terrain data including the height of each point in the viewpoint movement range including the planned position coordinates, the average human viewpoint in the height in the terrain data corresponding to the latitude and longitude in the current position coordinates An information processing method in which a value obtained by adding the height of the current position is used for drawing the perspective image as a value in the vertical direction of the current position coordinates. In the information processing device,
Three-dimensional data indicating the building planned to be built, the planned position coordinates of the building planned to be built, the current position coordinates including the latitude, longitude, and altitude of the information processing device detected by the position sensor, and the direction sensor Drawing a perspective image of the building from the viewpoint of the camera based on the orientation of the camera,
Displaying a synthesized image obtained by synthesizing the perspective image on an image of a planned construction site of a building imaged by the camera;
Based on the terrain data including the height of each point in the viewpoint movement range including the planned position coordinates, the average human viewpoint in the height in the terrain data corresponding to the latitude and longitude in the current position coordinates And a step of using the value obtained by adding the height of the current position coordinates for drawing the perspective image as a value in the vertical direction of the current position coordinates.