JP2010128799A - Composite media synthesizing apparatus, composite media display system, composite media synthesizing method, and composite media synthesizing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To create a useful three-dimensional map of a region including places whose three-dimensional shapes are difficult to calculate. <P>SOLUTION: A three-dimensional map creation part 123 of a composite media combining apparatus 120 constructs the three-dimensional map of the periphery of a measurement device using any of data to be output by a camera, a laser scanner, and stereo-camera mounted on a mobile measurement device. A media combining part 125 extracts the video of any region other than a region in which any three-dimensional map cannot be created by a three-dimensional map creation part 123 (region that is captured by a camera or a stereo-camera with low image precision, a region where a laser beam of the laser scanner cannot be received) from video data to be output by the camera, and combines the extracted video with the three-dimensional map created by the three-dimensional map creation part 123. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a composite media composition device, a composite media display system, a composite media composition method, and a composite media composition program. In particular, the present invention relates to a composite media composition device that synthesizes three-dimensional data acquired from a movable measuring device equipped with a camera and video data, and a composite media display device that displays the composite media. .

  Conventionally, there is a technique for constructing three-dimensional data of a townscape. For example, in Patent Document 1, street-level three-dimensional data is constructed by installing a laser measurement device on a vehicle and traveling. The self-position of the vehicle is acquired by GPS (Global Positioning System), and the shape of the object at the distance acquired by the laser measurement device from the self-position is measured and recorded as a three-dimensional model.

  Conventionally, there is a technique in which a traveling device estimates its own position from an image acquired by a camera mounted on the device and a distance acquired by a laser measuring instrument. For example, in Patent Document 2, the reference point is determined from the image acquired by the camera, and the distance from the traveling device to the reference point is measured by a laser measuring instrument, so that the self-position is determined based on the distance to the reference point. I guess.

  Conventionally, there is a technique for displaying an image according to a position. For example, in Patent Document 3, based on pre-recorded road network data, an image in which a direction toward a road branch point is captured in advance, or an image in which a direction bent from a road branch point in a specific direction is captured in advance. Provide and display.

Conventionally, there is a method of calculating the three-dimensional shape of an object shown in a two-dimensional image using epipolar geometry (see, for example, Non-Patent Document 1).
JP 2002-31528 A JP 2006-31642 A JP 2005-3752 A Co-authored by Xugang and Saburo Saburo, “3D Vision”, Kyoritsu Shuppan, April 1998

  In the conventional technology for constructing three-dimensional data of a townscape, there is a problem that a three-dimensional model cannot be constructed indoors or in a valley of buildings where GPS cannot be received because the self-position is acquired by GPS.

  In the conventional self-position estimation technology, since the laser measuring device is directed toward the target and the distance is measured, a driving device that directs the measuring device toward the target direction of the top, bottom, left, and right is necessary, and it is difficult for third parties to notice it for security purposes. As described above, there is a problem that the apparatus cannot be reduced in size.

  In the conventional video display technology according to the position, since the video is selected and displayed based on the position information, there is a problem that an appropriate video cannot be displayed in a place where the exact position is not known. In addition, there is a problem that even if there is a 3D model that reproduces the landscape other than the video, it cannot be used simultaneously with the video.

  According to the conventional method, when calculating the three-dimensional shape of an object shown in a two-dimensional image, the three-dimensional shape may not be calculated depending on the surrounding environment (brightness, width, height, etc.) of the object. There was a problem.

An object of this invention is to create a useful thing as a three-dimensional map of the area | region with the location where it is difficult to calculate a three-dimensional shape, for example. Another object of the present invention is to provide a system including a composite media composition device and a composite media display device having at least one of the following features, for example.
(1) A measurement device that travels indoors or in a valley of a building has an external positioning system (a system installed in addition to the measurement device for positioning. For example, a GPS satellite, a wireless LAN (Local Area Network), etc.) Get travel position without need.
(2) A machine that changes the direction of the measuring device is not mounted on the traveling measuring device.
(3) Even if the information collected from the traveling measuring device is an image or three-dimensional data, it can be displayed spatially continuously.

A composite media composition device according to an aspect of the present invention includes:
Shape data acquisition for acquiring shape data observed by the sensor during movement of the measuring device from a movable measuring device having a sensor for observing the shape of the peripheral region and a camera for capturing images of the peripheral region And
A three-dimensional map creating unit that creates a three-dimensional map of the region where the shape is observed by the sensor from the data obtained by the shape data obtaining unit;
A video data acquisition unit that acquires data of video captured by the camera during movement of the measurement device from the measurement device;
From the data acquired by the video data acquisition unit, the video of the region other than the region where the 3D map creation unit created the 3D map is extracted, and the extracted video and the 3D map created by the 3D map creation unit are extracted. And a media composition unit for synthesizing the above.

  According to one aspect of the present invention, in the composite media composition device, the three-dimensional map creation unit 3 of the region where the shape is observed by the sensor from the data of the shape observed by the sensor during the movement of the measurement device. A three-dimensional map is created, and the media composition unit extracts video of a region other than the region where the three-dimensional map creation unit has created the three-dimensional map from the data of the video captured by the camera while the measuring device is moving. By combining the extracted video and the 3D map created by the 3D map creation unit, for example, a useful 3D map of a region where it is difficult to calculate a 3D shape is obtained. It becomes possible to create.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Embodiment 1 FIG.
FIG. 1 is a diagram showing an outline and configuration of a composite media display system 100 according to the present embodiment.

  In FIG. 1, the composite media display system 100 includes a measurement device 110, a composite media composition device 120, and a composite media display device 130.

  The operator moves the measuring device 110 from a remote place. As will be described later, the measuring device 110 has a camera (other sensors may be mounted). The measuring device 110 sends the video data acquired from the camera to the composite media composition device 120 once every 1 second (an example of unit time) or once every 1 meter (an example of unit distance), for example (if any) Laser data obtained from the sensor is also sent).

  The composite media composition device 120 constructs a peripheral three-dimensional map (three-dimensional model) from the video data acquired by the measurement device 110 (from laser data, if any, or from video data and laser data, etc.). Recorded as three-dimensional data. The three-dimensional data is electronic data that three-dimensionally represents the shape of an object such as a structure (for example, a floor, a wall, a ceiling, or a pillar) or an installation (for example, a desk, a box, or a shelf). Specific examples include VRML (Virtual, Reality, Markup, Language). In the case of a square wall, the positions of the four vertices at the four corners are expressed in three-dimensional coordinates. The three-dimensional coordinates are such that the ground is the XY plane, the height is the Z direction, and the scale is, for example, 1.0 for 1 millimeter. The composite media composition device 120 records a video when a surrounding 3D map cannot be constructed from the video data acquired by the measurement device 110. At this time, the composite media composition device 120 estimates and records the moving path of the camera from the video. The composite media composition device 120 calculates the distance and direction that the measurement device 110 has moved since the previous time when the position of the measurement device 110 (which may be a relative position with respect to the movement start point or the like) is calculated. Record. The composite media composition device 120 synthesizes the three-dimensional data and the video data. Here, “combining 3D data and video data” means, for example, combining 3D data and video data into one media data (for example, one file) that can be reproduced continuously, or 3 This refers to recording link data (to be described later) on a recording medium together with the dimension data and video data.

  The composite media display device 130 displays a three-dimensional map at an arbitrary position designated by the operator. If there is no three-dimensional data corresponding to the position designated by the operator but only video data, the video at that position is displayed. For example, when a certain point A is designated, the composite media display device 130 displays an image of the point A if there is no 3D data for the point A. In addition, when the 3D map is displayed, the composite media display device 130 synthesizes and displays the video on the 3D map when there is video data acquired for a portion where the 3D data does not exist. For example, if there is a portion where data is missing in the three-dimensional map of the point A, the composite media display device 130 displays the three-dimensional map of the point A and displays the video of the place in the portion. To do. In addition, when the video is displayed, the composite media display device 130 synthesizes and displays a 3D map if there is 3D data constructed for a portion where the captured video does not exist. For example, when there is a portion where nothing is reflected in the video at the point A, the composite media display device 130 displays the video at the point A and also displays a three-dimensional map of the place in the portion.

  As described above, the three-dimensional data is three-dimensional map data. The video data is data of a moving image or continuous photos at intervals of 1 second, for example. The video data is recorded by being divided into a plurality of electronic files, for example. Link data is data indicating the association between video data and three-dimensional data. For example, when video is connected to a part of a 3D map at a certain location, link data that defines which part of the video is connected to which part of the 3D map is recorded in the link data. The In the link data, the difference between the location where the 3D map is switched to the video and the location where the video is switched to the 3D map is also recorded. In the link data, attribute information indicating the electronic file name of the video data, the frame number, the time stamp (may be an elapsed time from the shooting start time), the position of the measuring device 110 corresponding to the video data, and the like is also recorded.

  FIG. 2 shows an example of link information of a place where a video is switched to a three-dimensional map. FIG. 3 shows an example of link information of a place where the 3D map is switched to the video.

  An example of link information is a frame-shaped vector. The shape of the door, gate, etc. frame in the video frame can be indicated by a vector and recorded in the link data. In this case, in the link data, a frame-shaped vector is recorded in association with each frame of the video data. In the link data, the position of the frame in the three-dimensional data (three-dimensional map) corresponding to the frame shape vector in the video data is also recorded. Further, in the link data, the coordinates of the shooting position of the video frame corresponding to the position of the frame in the three-dimensional data and the direction vector of the shooting direction are recorded.

  As described above, as an example of link data, an electronic file name of video data, a frame number (or a time stamp of video data), a frame shape vector in an image (frame), and a frame shape vector in a related three-dimensional map It is conceivable to associate video shooting position coordinates, video shooting direction vectors, and the like in a three-dimensional map.

  4, 5, and 6 are diagrams illustrating a configuration example of the measurement device 110.

  In the example of FIG. 4, the measuring device 110 includes a carriage 111 and can move. In addition, the measurement apparatus 110 includes a camera 112 (monocular camera) that captures an image of the peripheral area. The camera 112 is mounted on the carriage 111. The camera 112 is an example of a sensor that observes the shape of the peripheral area. A computer 201 is also mounted on the carriage 111. The computer 201 is equipped with a communication device in order to transmit video data photographed by the camera 112 to a remote place. The communication device is a device that performs wireless LAN communication, for example.

  In the example of FIG. 5, the measurement device 110 includes a laser scanner 113 that measures the three-dimensional coordinates of the peripheral region in addition to the camera 112. The laser scanner 113 is also an example of a sensor that observes the shape of the peripheral region. Data of three-dimensional coordinates measured by the laser scanner 113 is also transmitted to a remote place by a communication device mounted on the computer 201.

  In the example of FIG. 6, the measurement device 110 includes a stereo camera 114 instead of the camera 112. The stereo camera 114 is also an example of a sensor that observes the shape of the peripheral area. Video data captured by the stereo camera 114 is also transmitted to a remote location by a communication device mounted on the computer 201.

  7 and 8 are diagrams showing examples of configurations of the composite media composition device 120 and the composite media display device 130. FIG.

  In the following description, it is assumed that both the composite media composition device 120 and the composite media display device 130 are software on a computer, but a combination of software and hardware may be used.

  As shown in FIG. 7, when the composite media composition device 120 is in the computer 201 on the measuring device 110, the three-dimensional data, video data, and link data output from the composite media composition device 120 are sent to the computer 202 at a remote location. It is transmitted via a network such as a wireless LAN and stored in a recording medium. The three-dimensional data, video data, and link data stored in the recording medium by the computer 202 are used by the composite media display device 130 in the computer 202.

  As shown in FIG. 8, when the composite media composition device 120 and the composite media display device 130 are on a computer 202 at a remote location away from the measurement device 110, the computer 201 on the measurement device 110 and the computer 202 on the remote ground are located. The acquired data of the camera 112, the laser scanner 113, and the stereo camera 114 and the operation instruction signal from the remote computer 202 to the measuring device 110 are transmitted and received via the wireless LAN. Three-dimensional data, video data, and link data created by the composite media composition device 120 from data acquired by the camera 112, laser scanner 113, and stereo camera 114 are stored in a recording medium by the computer 202 and used by the composite media display device 130. The

  FIG. 9 is a diagram illustrating a function of the operation instruction device 203 for the operator to instruct the operation of the measurement device 110 from the computer 202 at a remote place.

  In the following description, the operation instruction device 203 is assumed to be software on a computer, but may be a combination of software and hardware.

  As shown in FIG. 9, the operation instruction device 203 in the computer 202 at a remote location issues a traveling instruction to the traveling measuring device 110. Specifically, the operation instruction device 203 arranges left turn, forward, right turn, and reverse buttons on the travel instruction screen 204 on the computer 202. Then, for example, when the operator presses and releases one of the buttons for one second, the operation instruction device 203 gives an instruction to perform the operation described in the button for one second to the computer 201 on the measurement device 110 via the wireless LAN. send. In response to this instruction, the measurement device 110 performs a driving operation. Specifically, the computer 201 on the measuring device 110 rotates the left and right drive wheels of the carriage 111 during the instructed time. The forward movement is performed by rotating the two wheels in the forward direction. Reverse is done by rotating the two wheels in the opposite direction. The left turn is performed by increasing the rotation speed of the right drive wheel rather than the left drive wheel. Turn right by increasing the rotational speed of the left drive wheel rather than the right drive wheel. It is assumed that the rotational acceleration is constant for both acceleration and deceleration, and when reaching a constant speed, the computer 201 on the measuring device 110 rotates the driving wheel at a steady speed. The computer on the measuring device 110 obtains as data the moving distance and moving direction from the position before driving to the position after driving from the time when both wheels are operated. In addition, you may install a rotary encoder in a wheel. In this case, since the rotary encoder detects the rotation angle of the wheel, the movement distance and movement direction from the position before driving to the position after driving can be obtained as data from the rotation angle.

  FIG. 10 is a block diagram showing the configuration of the composite media composition device 120.

  In FIG. 10, the composite media composition device 120 includes a shape data acquisition unit 121, a video data acquisition unit 122, a three-dimensional map creation unit 123, a movement amount estimation unit 124, and a media composition unit 125.

  The shape data acquisition unit 121 acquires, from the measurement device 110, data on the shape of the surrounding area observed by the sensor during the movement of the measurement device 110 (at the same time, the data indicating the sensor direction, characteristics, etc. may be acquired). Good). When the measurement apparatus 110 is configured as in the example of FIG. 5 (including the camera 112 and the laser scanner 113), the shape data acquisition unit 121 moves from the measurement apparatus 110 to the laser scanner 113 while the measurement apparatus 110 is moving. The data of the three-dimensional coordinate measured by is acquired. Assuming that the data when the measuring device 110 moves for one unit time or one unit distance is one data, the shape data acquisition unit 121 may acquire one data from the measuring device 110 in real time, The data of the times may be acquired periodically or at an arbitrary timing. Here, the data of the shape observed by the sensor when the measuring device 110 starts to move is referred to as initial data (if data indicating the direction, characteristics, etc. of the sensor is acquired, the data Also included). Thereafter, the data of the shape observed by the sensor every time the measuring device 110 moves by unit time or unit distance is referred to as update data of the moving section (the section in which the measuring device 110 has moved by unit time or unit distance). (When data indicating the direction, characteristics, etc. of the sensor is acquired, the data is also included).

  The video data acquisition unit 122 acquires, from the measurement device 110, video data captured by the camera 112 (or the stereo camera 114) while the measurement device 110 is moving. Assuming that the data when the measuring device 110 moves one unit time or one unit distance is one data, the video data acquisition unit 122 may acquire one data from the measuring device 110 in real time, The data of the times may be acquired periodically or at an arbitrary timing. Here, video data captured by the camera 112 each time the measuring device 110 moves by unit time or unit distance is taken as video data of the movement section (section in which the measuring device 110 has moved unit time or unit distance). In addition, when the measuring apparatus 110 is configured as in the example of FIG. 4 (or FIG. 6) (no sensor separate from the camera 112), the camera 112 also serves as a sensor. In this case, since the video data acquisition unit 122 also serves as the shape data acquisition unit 121, the video data in the first movement section is the initial data described above. Then, the video data of each movement section after the second becomes update data of each movement section.

  The three-dimensional map creating unit 123 constructs a three-dimensional map of the peripheral area of the measuring device 110 using at least one of data output from the camera 112, the laser scanner 113, and the stereo camera 114 mounted on the measuring device 110. Specifically, the 3D map creation unit 123 creates a 3D map from the initial data acquired by the shape data acquisition unit 121 (or the video data acquisition unit 122). Thereafter, each time the shape data acquisition unit 121 acquires update data of one movement section, the three-dimensional map creation unit 123 updates the three-dimensional map with the update data of the movement section. That is, the 3D map creation unit 123 constructs a 3D map from the new update data, and synthesizes the already constructed 3D map and the newly constructed 3D map.

  The movement amount estimation unit 124 estimates the movement amount in the movement section every time the measuring apparatus 110 moves by unit time or unit distance. The movement amount estimation unit 124 may estimate the movement amount in each movement section from the data (for example, three-dimensional coordinate data) acquired by the shape data acquisition unit 121, or the data acquired by the video data acquisition unit 122. From this, the movement amount in each movement section may be estimated. In the latter case, the movement amount estimation unit 124 estimates the movement direction, orientation displacement, and movement distance of the camera 112 based on the video acquired by the camera 112 (or the stereo camera 114), thereby moving the movement amount of the measurement device 110. Is estimated.

  The media composition unit 125 extracts the video of the region other than the region where the 3D map creation unit 123 created the 3D map from the data acquired by the video data acquisition unit 122, and the extracted video and the 3D map creation unit 123. Is combined with the 3D map created by The region other than the region where the 3D map creation unit 123 created the 3D map is mainly a region where the 3D map creation unit 123 cannot create the 3D map. Examples of the area where the 3D map creation unit 123 cannot create a 3D map include, for example, an area where the video accuracy is poor (such as a dark area) even when the camera 112 (or the stereo camera 114) captures the image, and the laser of the laser scanner 113. And the like that are out of the range to reach (a wide region, a high region, etc.). When such a region exists, the 3D map creation unit 123 cannot update the 3D map with the update data of one or more moving sections acquired by the shape data acquisition unit 121 (or the video data acquisition unit 122). Become. Therefore, the media composition unit 125 determines the region (three-dimensional map creation unit) from the video data of the movement section acquired by the video data acquisition unit 122 based on the movement amount in the movement section estimated by the movement amount estimation unit 124. The image of the area where the 3D map created by 123 does not exist is extracted, and the extracted video and the 3D map created by the 3D map creation unit 123 are combined. That is, the media composition unit 125 records a 3D map (3D data), video (video data), and link information (link data) on a recording medium. As described above, when recording the link information, the media composition unit 125 calculates which part of the 3D map and which part of the video are to be displayed by calculating a frame shape vector or the like. There is a need to. Therefore, the media composition unit 125 detects, for example, a portion in which a deep opening (a frame such as an entrance door or a gate) is shown from the video data acquired by the video data acquisition unit 122, and displays the video of the detected portion. It is extracted as an image of an area where the 3D map created by the 3D map creation unit 123 does not exist. Then, the media composition unit 125 associates the extracted video with the already created three-dimensional map.

  FIG. 11 is a block diagram showing a configuration of the composite media display device 130.

  In FIG. 11, the composite media display device 130 includes a virtual position determination unit 131, a three-dimensional data display unit 132, a three-dimensional upper video composite display unit 133, a video data display unit 134, and a video three-dimensional composite display unit 135.

  The virtual position determination unit 131 determines the viewpoint position of the map displayed on the screen. Then, the virtual position determination unit 131 determines whether it is a range for displaying a three-dimensional map or a range for displaying an image.

  The three-dimensional data display unit 132 displays the three-dimensional data according to the viewpoint position.

  The three-dimensional upper video composition display unit 133 synthesizes and displays the video on the three-dimensional map.

  The video data display unit 134 plays back video.

  The on-video three-dimensional composition display unit 135 synthesizes and displays a three-dimensional map on the video.

  FIG. 12 is a diagram illustrating an example of a hardware configuration of the computers 201 and 202.

  In FIG. 12, computers 201 and 202 include an LCD 901 (Liquid / Crystal / Display), a keyboard 902 (K / B), a mouse 903, an FDD 904 (Flexible / Disk / Drive), a CDD 905 (Compact / Disk / Drive), and a printer 906. Hardware devices. These hardware devices are connected by cables and signal lines. Instead of the LCD 901, a CRT (Cathode / Ray / Tube) or other display device may be used. Instead of the mouse 903, a touch panel, a touch pad, a trackball, a pen tablet, or other pointing devices may be used.

  The computers 201 and 202 include a CPU 911 (Central Processing Unit) that executes a program. The CPU 911 is an example of a processing device. The CPU 911 includes a ROM 913 (Read / Only / Memory), a RAM 914 (Random / Access / Memory), a communication board 915, an LCD 901, a keyboard 902, a mouse 903, an FDD 904, a CDD 905, a printer 906, and an HDD 920 (Hard / Disk) via a bus 912. Connected with Drive) to control these hardware devices. Instead of the HDD 920, a flash memory, an optical disk device, a memory card reader / writer, or other storage medium may be used.

  The RAM 914 is an example of a volatile memory. The ROM 913, the FDD 904, the CDD 905, and the HDD 920 are examples of nonvolatile memories. These are examples of the storage device. The communication board 915, the keyboard 902, the mouse 903, the FDD 904, and the CDD 905 are examples of input devices. The communication board 915, the LCD 901, and the printer 906 are examples of output devices.

  The communication board 915 is connected to a LAN or the like. The communication board 915 is not limited to a LAN, but is an IP-VPN (Internet, Protocol, Private, Network), a wide area LAN, an ATM (Asynchronous / Transfer / Mode) network, or a WAN (Wide / Area / Network) It does not matter if it is connected to. LAN, WAN, and the Internet are examples of networks.

  The HDD 920 stores an operating system 921 (OS), a window system 922, a program group 923, and a file group 924. The programs in the program group 923 are executed by the CPU 911, the operating system 921, and the window system 922. The program group 923 includes programs that execute the functions described as “˜units” in the description of the present embodiment. The program is read and executed by the CPU 911. The file group 924 includes data, information, and signal values described as “˜data”, “˜information”, “˜ID (identifier)”, “˜flag”, and “˜result” in the description of this embodiment. And variable values and parameters are included as items of “˜file”, “˜database”, and “˜table”. The “˜file”, “˜database”, and “˜table” are stored in a storage medium such as the RAM 914 or the HDD 920. Data, information, signal values, variable values, and parameters stored in a storage medium such as the RAM 914 and the HDD 920 are read out to the main memory and the cache memory by the CPU 911 via a read / write circuit, and extracted, searched, referenced, compared, and calculated. It is used for processing (operation) of the CPU 911 such as calculation, control, output, printing and display. During the processing of the CPU 911 such as extraction, search, reference, comparison, calculation, calculation, control, output, printing, and display, data, information, signal values, variable values, and parameters are temporarily stored in the main memory, cache memory, and buffer memory. Remembered.

  The arrows in the block diagrams and flowcharts used in the description of this embodiment mainly indicate input / output of data and signals. Data and signals are recorded in memory such as RAM 914, FDD904 flexible disk (FD), CDD905 compact disk (CD), HDD920 magnetic disk, optical disk, DVD (Digital Versatile Disc), or other recording media Is done. Data and signals are transmitted by a bus 912, a signal line, a cable, or other transmission media.

  In the description of the present embodiment, what is described as “to part” may be “to circuit”, “to device”, “to device”, and “to step”, “to process”, “to”. ~ Procedure "," ~ process ". That is, what is described as “˜unit” may be realized by firmware stored in the ROM 913. Alternatively, what is described as “˜unit” may be realized only by software, or only by hardware such as an element, a device, a board, and wiring. Alternatively, what is described as “to part” may be realized by a combination of software and hardware, or a combination of software, hardware and firmware. Firmware and software are stored as programs in a recording medium such as a flexible disk, a compact disk, a magnetic disk, an optical disk, and a DVD. The program is read by the CPU 911 and executed by the CPU 911. That is, the program causes the computer to function as “to part” described in the description of the present embodiment. Or a program makes a computer perform the procedure and method of "-part" described by description of this Embodiment.

  FIG. 13 is a flowchart showing the operation of the composite media composition device 120.

  In FIG. 13, in step S101 (post-movement position data acquisition process), the moving distance and moving direction from the pre-driving position to the post-driving position measured by the computer 201 of the measuring apparatus 110 are three-dimensional. This is transmitted to the map creation unit 123. In addition, from the computer 201 of the measuring device 110, data acquired by the laser scanner 113 is sent to the shape data acquisition unit 121 of the composite media composition device 120, and data acquired by the camera 112 and the stereo camera 114 is received by the composite media composition device 120. To the video data acquisition unit 122. The transmission interval is the timing when the operator's movement operation is finished. Alternatively, the moving distance and the moving direction may be transmitted from the computer 201 of the measuring device 110 to the three-dimensional map creating unit 123 at regular intervals regardless of the operator's moving operation. In the 3D map creation unit 123, the position data obtained continuously are connected (after calculating the position of the measuring device 110 last time, the direction, direction change, and distance of the measuring device 110 are calculated). Thus, the coordinates and direction of the current location with the origin at the point where the measurement device 110 first moves are obtained as position data. The coordinates are expressed by, for example, XY coordinates where the coordinate value 1.0 is 1 millimeter with the direction of the first measuring device 110 as the Y axis and the right hand toward the direction as the X axis.

  In step S101, the input values from the computer 201 of the measuring device 110 to the three-dimensional map creating unit 123 are the moving distance and moving direction from the position before driving to the position after driving measured by the computer 201 of the measuring device 110. However, when a device for controlling the rotational speed of the wheel or obtaining the rotational angle of the wheel is not mounted on the measuring device 110, the following is performed.

  The video of the camera 112 (or stereo camera 114) is sent from the computer of the measuring device 110 to the movement amount estimation unit 124 of the composite media composition device 120, and the movement distance and movement direction calculated here are sent to the three-dimensional map creation unit 123. As an input value.

  In step S102 (peripheral three-dimensional data construction processing), the three-dimensional map creation unit 123 constructs the peripheral three-dimensional data at the time when the position data of the measuring device 110 is obtained. When the laser scanner 113 is mounted on the measuring device 110 as in the example of FIG. 5, the position on the plane where the wall or object is present can be acquired as a point sequence measured by the laser as shown in FIG. Therefore, this point sequence is converted into three-dimensional data as a preset ceiling height surface. A ceiling and floor surface with a size covering this surface may be constructed. Further, a texture of a plane with a wall or an object may be extracted from a video (photograph) taken by the camera 112 and pasted on the three-dimensional data. When the stereo camera 114 is used for the measuring apparatus 110 as in the example of FIG. 6, the three-dimensional coordinates of the feature points in the image can be obtained as shown in FIG. Ask. When the camera 112 (monocular camera) is mounted on the measuring device 110 as in the example of FIG. 4, the position data obtained at least from two points and the video obtained at each point are used. A point cloud similar to the stereo camera 114 is measured by the principle of the stereo camera. The three-dimensional map creation unit 123 records the three-dimensional data thus obtained as a three-dimensional map.

  The 3D map creation unit 123 may not be able to construct 3D data beyond the measurable distance of the laser scanner 113. In addition, the camera 112 and the stereo camera 114 may be difficult to measure depending on the lighting conditions. Therefore, steps S103 to S106 are selectively executed in the 3D map creation unit 123 depending on whether or not 3D data can be constructed.

  If 3D data has been constructed immediately before and 3D data has also been constructed this time, Step S103 (3D data synthesis processing) is executed.

  If the 3D data has been constructed immediately before and the 3D data has not been constructed this time, Step S104 and Step S105 (3D video connection processing) are executed.

  If 3D data cannot be constructed immediately before, and 3D data can be constructed this time, Step S103 and Step S106 (video 3D connection processing) are executed.

  If the 3D data cannot be constructed immediately before and the 3D data cannot be constructed this time, step S104 (movement amount estimation process) is executed.

  Hereinafter, step S103 (three-dimensional data composition processing) will be described.

  In step S103, the three-dimensional map creation unit 123 translates and rotates the surface constructed in accordance with the coordinates and direction in which the measurement device 110 exists when the three-dimensional data is constructed. When there is an overlapping surface between the 3D data constructed by the 3D map creation unit 123 and the already recorded 3D map, one surface of the overlapping portion may not be cut. In addition, the three-dimensional map creation unit 123 records the position and direction of the measurement device 110 at the time of acquisition as attributes in the three-dimensional data.

  Hereinafter, step S104 (movement amount estimation processing) will be described.

  In step S104, the input value to the movement amount estimation unit 124 is an image of the camera 112 (or the stereo camera 114) transmitted from the computer 201 of the measurement apparatus 110. As illustrated in FIG. 15, the movement amount estimation unit 124 uses an image before moving (referred to as image 1) and an image after moving (referred to as image 2) in the video as input values. The movement amount estimation unit 124 extracts corresponding points estimated to be the same place between the image 1 and the image 2, and obtains how much the corresponding points have moved. For this purpose, a method generally called the Lucas-Kanade method can be used. The movement amount estimation unit 124 obtains the moving direction of the camera 112 from the change in position on the screen of a plurality of corresponding points of two photos. Therefore, the method described in Non-Patent Document 1 can be used. The movement amount estimation unit 124 obtains a plurality of epipolar lines from the epipolar equation, and obtains an epipole point that is an intersection. The epipole point on the image 1 indicates that the position of the camera 112 at the time when the image 2 is captured in the direction of this point from the position of the camera 112 that captured the image 1. Next, the movement amount estimation unit 124 calculates the movement distance of the camera 112. Since the difference in the coordinate in the height direction between the ceiling and the floor is the same as the height of the ceiling, the movement amount estimation unit 124 obtains the coordinates of each feature point from the images 1 and 2, and the distance between the ceiling and the floor is determined in advance. The difference between the shooting points of image 1 and image 2 that is the same as the known height is obtained. The ceiling and floor are divided into points at the top and points at the bottom. The difference in direction as well as the distance is obtained. Since the direction from the shooting point of image 1 to the shooting point of image 2 is known from the epipole point, the movement amount estimation unit 124 may obtain the distance using the rotation speed of the wheel together with this direction. The movement amount estimation unit 124 records the obtained position and direction of the camera 112 as attribute information for the corresponding frame of the video data.

  In step S104, the movement amount estimation unit 124 may correct the movement amount calculated in the past. As shown in FIG. 16, it is assumed that a three-dimensional map has already been constructed, and images captured by the camera 112 in the optical axis directions indicated by arrows at the positions of points A, B, C, and D are acquired. At this time, the arrow becomes a movement vector. Here, it is assumed that the point of the frame 1 coincides with the point A. It is assumed that a frame is found in the video imaged from the point D by the operation in step S105 described later. At this time, it is assumed that the found frame and the existing frame 2 are close to each other, that is, a difference of Δx in the X-axis direction of FIG. 16 and Δy in the y-axis direction. If Δx and Δy are each equal to or less than a preset threshold value, it is determined that the found frame is frame 2. Next, let Xa be the vector component in the X-axis direction of A, and similarly extract xb, xc, and xd. Δx × xa / (xa + xb + xc + xd) is added to xa. Similarly, Δx × xb / (xa + xb + xc + xd) is added to xb, and the same addition is performed to xc and xd. A vector component in the Y-axis direction of A is also set to ya with respect to the Y-axis direction, and Δy × ya / (ya + yb + yc + yd) is subtracted from ya. The movement vector is adjusted in this way, and the shooting position of the video map portion is corrected so as to coincide with the frame of the three-dimensional map as shown in the corrected diagram of FIG.

  Hereinafter, step S105 (3D video connection processing) will be described.

  In step S105, the media composition unit 125 associates the video of the camera 112 mounted on the measurement apparatus 110 with the already acquired three-dimensional data. For this purpose, the media composition unit 125 records the link information shown in FIG. 3, for example.

  For example, it is assumed that the 3D map creation unit 123 tries to construct 3D data each time the measuring device 110 moves. It is assumed that the shape at the back of the vacant part (the opening having a depth) in the figure cannot be measured from the corridor-shaped branch point as shown in the upper left of FIG. At this time, the media composition unit 125 determines that it is a part for switching from 3D data to video. The media composition unit 125 uses the frame (bold line in the figure) of the frontage that can be measured as a switching frame from 3D data to video. The media composition unit 125 determines from the shooting position, direction, and angle of view of the video, and sets the frame in the video corresponding to this frame as the frame in the video. The media composing unit 125 links the frame (frame shape vector) in the three-dimensional data, the frame in the video (frame shape vector), the position and direction of the camera, together with information specifying the video data and the frame. Record as.

  Here, a method for specifying a frame (frame shape vector) in three-dimensional data and a frame (frame shape vector) in an image will be described in detail.

  A frame (frame shape vector) in the three-dimensional data is detected from, for example, laser scan data (data measured by the laser scanner 113).

  FIG. 17 shows a frame detection method when the carriage 111 of the measuring apparatus 110 moves from an area where a 3D map can be constructed to an area where construction is difficult.

  In FIG. 17, a 3D map can be constructed from the laser scan data at a point A. Next, since the straight line included in the laser scan data acquired at the point B where the turn is left has already been constructed at the point A, it is not reconstructed. At point C, a straight line is detected, but the part where the walls cannot be detected on the left and right is set as a range in which the landscape is expressed using video. At the same time, the 3D map already constructed from the point C and the nearest straight line end point are identified as a frame. The end portion of the wall which gives a height to this end point portion is a frame (frame shape vector) in the three-dimensional data.

  FIG. 18 shows a method for obtaining a frame (frame shape vector) region in an image from laser scan data.

  As an image acquisition condition, the camera 112 is installed horizontally, and the ceiling height hr, the camera h height hc, and the focal length f are known. A polar coordinate system with the optical axis as the reference direction is set on the horizontal plane with the position of the camera 112 as the origin, and the coordinates of the points where both ends of the gate (frame) are set with the ceiling and floor are obtained. In FIG. 18, θa and θb are observed values, and θr and θf are obtained from the equations θr = arctan ((hr−hc) / ra) and θf = arctan (hc / ra).

  FIG. 19 shows the x-axis and y-axis on the image plane.

  The coordinates of the four corners of the gate are obtained from θa, θb, θr, and θf. The expression xr = ftan (θr) indicates the x-axis distance of the ceiling on the image plane. The expression xf = f tan (θf) indicates the x-axis distance of the floor on the image plane. The expression ya = ftan (θa) represents the y-axis distance of the wall a. The coordinates on the image plane are obtained from the image sensor size and the number of acquired pixels. The wall b can be obtained by the same calculation.

  Hereinafter, step S106 (video three-dimensional connection process) will be described.

  In step S106, the media composition unit 125 associates the three-dimensional data newly constructed by the three-dimensional map creation unit 123 with the already acquired video data. For this purpose, the media composition unit 125 records the link information shown in FIG. 2, for example.

  If the 3D map creation unit 123 cannot build 3D data immediately before this time, and the 3D map creation unit 123 can build 3D data this time, the media composition unit 125 converts the 3D data at the time of measurement. When displayed according to the position and direction of the camera 112, a frame (frame shape vector) in a range in which three-dimensional data can be measured is recorded as link data. The media composition unit 125 further records the position and direction of the camera 112 as link data together with information specifying the video data and the frame.

  FIG. 20 is a flowchart showing the operation of the composite media display device 130. FIG. 21 is a diagram illustrating a screen display example of the composite media display device 130.

  In FIG. 21, the movement button 301 is the same as the button shown in FIG. 9, and is for operating the measuring device 110. The current landscape display 302 displays the image of the camera 112 (or the stereo camera 114) mounted on the measuring device 110 as it is. A map 303 displays three-dimensional data from directly above. On the map 303, a current position direction 304 that is a traveling direction at the current position and a virtual position direction 305 that is a position direction at the virtual position are displayed. The virtual landscape display 306 is a screen that displays three-dimensional data and video data at an arbitrary position and direction by an operation of an operator. The virtual movement button 307 is used to specify an arbitrary position and direction viewpoint different from the current position and direction. The 3D button 308 requests 3D data display, and the video button 309 requests video display.

  20, in step S201 (virtual position determination processing), the virtual position determination unit 131 performs post-movement position data acquisition processing (step S101 in FIG. 13) in the composite media composition device 120 while the measurement apparatus 110 is moving. ) Is the virtual position. However, when the measuring device 110 is not moving, the position and direction of the viewpoint are changed to forward, right turn, left turn, and reverse directions corresponding to the virtual movement button 307 pressed by the operator. The virtual position determination unit 131 sends the determined virtual viewpoint position and direction to the video data display unit 134 and the three-dimensional data display unit 132.

  In step S202 (three-dimensional data display process), the three-dimensional data display unit 132 displays any three-dimensional data within a predetermined range from the virtual viewpoint position.

  In step S203 (three-dimensional upper image composition display processing), if there is image data associated with the three-dimensional data displayed in step S202 by link data, the three-dimensional upper image composition display unit 133 displays the image as three. Combine and display on a 3D map.

  Here, FIG. 22 shows a video display example on a three-dimensional map. A square vector of links is shown at the end of the three-dimensional model of the corridor. The square vector is also set as a link as a video frame. The three-dimensional upper video composition display unit 133 is arranged so that they overlap, and reproduces the viewpoint in the hallway of the three-dimensional data. At this time, it is assumed that there are different frames of video depending on the position where the video of the same frame is captured. In this case, in accordance with the movement of the virtual position, the three-dimensional upper video composition display unit 133 changes the video data so that the frame of the video whose shooting position is closest to the virtual position is displayed in the frame of the three-dimensional data. Finally, when the three-dimensional data does not enter the field of view due to the movement of the virtual position, the three-dimensional upper video composition display unit 133 stops displaying the three-dimensional data and displays only the video data.

  In step S204 (video data display processing), the video data display unit 134 displays video data if there is no 3D data within a predetermined range from the virtual viewpoint position. The video data display unit 134 plays the video data if the operation of the virtual viewpoint is forward, and plays the video data in the reverse direction if the operation is backward. The video data display unit 134 changes the position and direction of the virtual viewpoint on the map based on the position information (attribute information) corresponding to the frame of the video data in conjunction with the previous and next playback. That is, the video data display unit 134 does not follow the button operation of the virtual viewpoint when reproducing the video data, but follows the position information that is the attribute of the video data.

  In step S205 (three-dimensional on-video composite display processing), if link information is present in the video frame during the display of the video data in step S204 (the video data displayed in step S204 is associated with the link data. As shown in FIG. 23, the on-video 3D composite display unit 135 fills the frame portion in the video with a transparent color and overlays the back of the video to display the corresponding 3D data. indicate. At this time, the three-dimensional data is combined and displayed so that the frame portions match.

  Here, FIG. 24 shows a display example of the map 303 when the virtual viewpoint is moved. When the composite media display device 130 starts displaying the virtual viewpoint from the point A and proceeds to the point B, the three-dimensional data cannot be constructed around the dotted line portion and there is no map, but the position where the measurement device 110 has moved is Since the direction is recorded, it is possible to display this area with a dotted line.

  As described above, according to the present embodiment, in the movement amount estimation process, the movement amount estimation unit 124 estimates the movement amount of the measurement apparatus 110 from the video acquired from the camera 112, and thus the movement amount is measured. Even when the device is not loaded on the measuring device 110, the video and the map can be associated with each other.

  In the 3D video connection process, the media composition unit 125 can associate the 3D data in the area where the 3D data can be constructed with the video data of the area where the 3D data cannot be constructed. Then, in the 3D image synthesis display process, the 3D image synthesis display unit 133 continuously displays images in the space where the 3D data cannot be acquired during the 3D data display by the 3D data display unit 132. Can be used to display. As a result, it is possible to record, reproduce, and transmit a space that is difficult to sense. FIG. 25 shows an example in which a three-dimensional map is displayed and continuously switched to a video. The left side of FIG. 25 is an example of displaying the inside of a room. On the right side of FIG. 25 is an example of display from the corridor to the large lobby.

  Further, in the video three-dimensional connection process, the media composition unit 125 can associate the video and the three-dimensional data by matching the positions. Then, in the on-image three-dimensional composition display processing, the on-image three-dimensional composition display unit 135 can continuously synthesize and display the three-dimensional data measured from a certain portion of the image. As a result, it is possible to record, reproduce, and transmit a space that is difficult to sense. FIG. 26 shows an example in which a video is displayed and continuously switched to a three-dimensional map. In FIG. 26, the lobby is displayed as an image, and is switched to the corridor of the three-dimensional map as it moves forward.

1 is a diagram illustrating a configuration and an outline of operation of a composite media display system according to Embodiment 1. FIG. 6 is a diagram illustrating an example of link information according to Embodiment 1. FIG. 6 is a diagram illustrating an example of link information according to Embodiment 1. FIG. 2 is a diagram illustrating a configuration example of a measurement device according to Embodiment 1. FIG. 2 is a diagram illustrating a configuration example of a measurement device according to Embodiment 1. FIG. 2 is a diagram illustrating a configuration example of a measurement device according to Embodiment 1. FIG. 2 is a diagram illustrating a configuration example of a composite media composition device and a composite media display device according to Embodiment 1. FIG. 2 is a diagram illustrating a configuration example of a composite media composition device and a composite media display device according to Embodiment 1. FIG. It is a figure which shows the function of the operation instruction apparatus which concerns on Embodiment 1. FIG. 1 is a block diagram showing a configuration of a composite media composition device according to Embodiment 1. FIG. 1 is a block diagram showing a configuration of a composite media display device according to Embodiment 1. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer according to Embodiment 1. FIG. 4 is a flowchart illustrating an operation of the composite media composition device according to the first embodiment. 3 is a diagram illustrating an example of a method for creating a three-dimensional map according to Embodiment 1. FIG. 6 is a diagram illustrating an example of a movement amount estimation method according to Embodiment 1. FIG. 6 is a diagram illustrating an example of a correction method in movement amount estimation according to Embodiment 1. FIG. 6 is a diagram illustrating an example of data and a frame extraction method measured by the laser scanner according to Embodiment 1. FIG. 6 is a diagram illustrating an example of a frame calculation method according to Embodiment 1. FIG. 6 is a diagram illustrating an example of a frame in a screen plane according to Embodiment 1. FIG. 3 is a flowchart illustrating an operation of the composite media display device according to the first embodiment. 3 is a diagram illustrating a screen display example of the composite media display device according to Embodiment 1. FIG. 6 is a diagram showing an example of video display on a three-dimensional map according to Embodiment 1. FIG. 6 is a diagram showing a 3D map display example on the video according to Embodiment 1. FIG. It is a figure which shows the example of a map display at the time of moving the virtual viewpoint which concerns on Embodiment 1. FIG. 6 is a diagram illustrating an example of switching from a 3D map to a video according to Embodiment 1. FIG. It is a figure which shows the example of a switch from the image | video concerning Embodiment 1 to a three-dimensional map.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 Composite media display system, 110 Measuring apparatus, 111 Dolly, 112 Camera, 113 Laser scanner, 114 Stereo camera, 120 Composite media composition apparatus, 121 Shape data acquisition part, 122 Image | video data acquisition part, 123 3D map creation part, 124 Movement amount estimation unit, 125 media composition unit, 130 composite media display device, 131 virtual position determination unit, 132 3D data display unit, 133 3D upper image composition display unit, 134 image data display unit, 135 3D image composition Display unit, 201, 202 computer, 203 operation instruction device, 204 travel instruction screen, 301 movement button, 302 current landscape display, 303 map, 304 current position direction, 305 virtual position direction, 306 virtual landscape display, 307 virtual movement button, 308 3D button 309 Video button, 901 LCD, 902 keyboard, 903 mouse, 904 FDD, 905 CDD, 906 printer, 911 CPU, 912 bus, 913 ROM, 914 RAM, 915 communication board, 920 HDD, 921 operating system, 922 window system , 923 program group, 924 file group.

Claims (11)

Shape data acquisition for acquiring shape data observed by the sensor during movement of the measuring device from a movable measuring device having a sensor for observing the shape of the peripheral region and a camera for capturing images of the peripheral region And
A three-dimensional map creating unit that creates a three-dimensional map of the region where the shape is observed by the sensor from the data obtained by the shape data obtaining unit;
A video data acquisition unit that acquires data of video captured by the camera during movement of the measurement device from the measurement device;
From the data acquired by the video data acquisition unit, the video of the region other than the region where the 3D map creation unit created the 3D map is extracted, and the extracted video and the 3D map created by the 3D map creation unit are extracted. And a media composition unit for synthesizing the media. The composite media composition device further includes:
A movement amount estimation unit that estimates a movement amount in the movement section every time the measuring device moves for a unit time or a unit distance,
The shape data acquisition unit acquires, as initial data, shape data observed by the sensor when the measurement device starts moving from the measurement device, and the measurement device moves by unit time or unit distance. Each time, the data of the shape observed by the sensor is acquired as update data of the movement section,
The 3D map creation unit creates a 3D map from the initial data obtained by the shape data acquisition unit, and then each time the shape data acquisition unit obtains update data of one movement interval, Update the 3D map with the update data,
The video data acquisition unit acquires, from the measurement device, video data captured by the camera each time the measurement device moves by unit time or unit distance as video data of the movement section,
When the 3D map creation unit cannot update the 3D map with the update data of one movement section acquired by the shape data acquisition unit, the media composition unit moves in the movement section estimated by the movement amount estimation unit. Based on the amount, from the video data of the moving section acquired by the video data acquisition unit, the video of the region where the 3D map created by the 3D map creation unit does not exist is extracted, and the extracted video and the 3D The composite media composition apparatus according to claim 1, wherein the composition is synthesized with the three-dimensional map created by the map creation unit.   The media composition unit detects a portion in which a deep opening is reflected from the video data acquired by the video data acquisition unit, and the 3D map created by the 3D map creation unit creates a video of the detected portion. The composite media composition apparatus according to claim 2, wherein the composite media composition apparatus extracts the video as an image of an area where no image exists.   4. The composite media composition apparatus according to claim 2, wherein the movement amount estimation unit estimates a movement amount in each movement section from the data acquired by the video data acquisition unit. The camera also serves as the sensor,
The video data acquisition unit also serves as the shape data acquisition unit,
When there is an area where the 3D map creation unit cannot create a 3D map, the media composition unit extracts a video of the area from the data acquired by the video data acquisition unit, and extracts the extracted video and the 3D map. The composite media composition device according to any one of claims 1 to 4, wherein the composite image composition unit synthesizes the three-dimensional map created by the map creation unit. The sensor is a laser scanner that measures a three-dimensional coordinate of a peripheral region,
The media synthesizing unit extracts a video of an area outside the range reached by the laser of the laser scanner from the data acquired by the video data acquisition unit, and the extracted video and the 3D map created by the 3D map creation unit 5. The composite media composition device according to claim 1, wherein the composite media composition device synthesizes the map. A composite media composition device according to any one of claims 1 to 6,
A composite media display system comprising: a composite media display device that displays on the screen data obtained by combining the video and the three-dimensional map. The shape data acquisition unit of the composite media synthesizer observes the sensor from a movable measuring device having a sensor for observing the shape of the peripheral region and a camera for capturing images of the peripheral region while the measuring device is moving. A first step of obtaining data of the shaped shape;
A second step in which the three-dimensional map creation unit of the composite media composition device creates a three-dimensional map of the region whose shape is observed by the sensor from the data acquired in the first step;
A third step in which a video data acquisition unit of the composite media synthesizing apparatus acquires, from the measurement apparatus, video data captured by the camera during the movement of the measurement apparatus;
The media composition unit of the composite media composition device extracts a video of an area other than the area where the 3D map is created in the second step from the data acquired in the third step, and extracts the extracted video and the first And a fourth step of synthesizing the three-dimensional map created in two steps. A video data acquisition unit of the composite media composition device acquires, from a movable measurement device including a camera that captures a video of a peripheral area, video data captured by the camera during the movement of the measurement device. Steps,
A second step in which a three-dimensional map creation unit of the composite media composition device creates a three-dimensional map of an area where an image is captured by the camera from the data acquired in the first step;
When there is an area where the media composition unit of the composite media composition apparatus cannot create a 3D map in the second step, the image of the area is extracted from the data acquired in the first step, And a third step of synthesizing the three-dimensional map created in the second step. Shape data acquisition for acquiring shape data observed by the sensor during movement of the measuring device from a movable measuring device having a sensor for observing the shape of the peripheral region and a camera for capturing images of the peripheral region Processing,
A three-dimensional map creation process for creating a three-dimensional map of a region where the shape is observed by the sensor from the data acquired by the shape data acquisition process;
Video data acquisition processing for acquiring data of video captured by the camera during the movement of the measurement device from the measurement device;
From the data acquired by the video data acquisition process, a video of an area other than the area where the 3D map creation process created the 3D map is extracted, and the extracted video and the 3D map created by the 3D map creation process are extracted. A composite media composition program for causing a computer to execute a media composition process for composing data. Video data acquisition processing for acquiring data of video captured by the camera during movement of the measurement device, from a movable measurement device equipped with a camera that captures a video of a peripheral region;
A three-dimensional map creation process for creating a three-dimensional map of a region where a video is shot by the camera from the data acquired by the video data acquisition process;
When there is an area where the 3D map creation process cannot create a 3D map, the video of the area is extracted from the data acquired by the video data acquisition process, and the extracted video and the 3D map creation process are created. A composite media composition program for causing a computer to execute media composition processing for compositing with a three-dimensional map.