JP7441579B1 - Information processing system and information processing method - Google Patents

Abstract

The present invention provides an information processing system and the like that can easily manage images photographed by a moving object by associating them with photographing target positions. According to an embodiment of the present invention, an actual imaging position information acquisition unit 110 that acquires actual imaging position information indicating an actual imaging position in real space associated with acquired imaging image data; , a virtual distance generation unit 120 that generates virtual distance information indicating a virtual distance from a virtual shooting position corresponding to an actual shooting position to a shooting target position, and angle of view information and focal length information when shooting image data. An information processing system is provided that includes a size identifying unit 130 that identifies the size of image data at a shooting target position based on virtual distance information. [Selection diagram] Figure 5

Description

The present invention relates to an information processing system and an information processing method.

Conventionally, it has been possible to autonomously control flying objects such as (Drones) and unmanned aerial vehicles (UAVs) (hereinafter collectively referred to as "flying objects"), and running objects such as unmanned ground vehicles (UGVs). BACKGROUND ART Mobile bodies have begun to be used in industry, and in particular, such mobile bodies are used to inspect the interior and exterior of buildings. Patent Document 1 discloses a system for inspecting indoors using a mobile object and managing images thereof.

JP2019-179422A

However, in the technique disclosed in Patent Document 1, it is unclear which position of the building and from which direction the image taken by the moving object is taken, and furthermore, it is unclear which position and direction of the building the image taken by the moving object is taken from, and furthermore, it is unclear whether the image taken by the moving object is the image taken at the actual site of the object to be photographed. It is also difficult to tell the size from the image. Therefore, it is difficult to manage images taken by a moving object in association with the position of the object to be photographed.

Furthermore, it is possible to create a virtual space that imitates the actual site by pasting the photographed images onto a 3D model, but as mentioned above, the size of the object being photographed at the actual site is unknown. If so, it is unclear at what size the photographed image should be pasted onto the three-dimensional model. Therefore, it is necessary for the worker to manually adjust the size of the photographed image and the pasting position on the 3D model using an image editing system etc. to perform the pasting work. The burden on the worker increases when pasting it on the model.

The present invention has been made in view of this background, and it is an object of the present invention to provide an information processing system etc. that can easily manage images taken by a moving object by associating them with the position and size of the object to be photographed. purpose.

The main invention of the present invention for solving the above-mentioned problems is to provide an actual photographing position information acquisition unit that acquires actual photographing position information indicating an actual photographing position in real space that is linked to acquired photographic image data; , a virtual distance generation unit that generates virtual distance information indicating the virtual distance from the virtual shooting position corresponding to the actual shooting position to the shooting target position, and the angle of view information and focal length information when shooting image data, and the virtual distance. The information processing system includes a size specifying unit that specifies the size of image data at a shooting target position based on information.

According to the present invention, in particular, it is possible to provide an information processing system and the like that can easily manage images photographed by a moving body in association with the photographing target position and its size.

1 is a diagram showing the configuration of an information processing system according to an embodiment of the present invention. FIG. 2 is a block diagram showing the hardware configuration of the management server in FIG. 1. FIG. FIG. 2 is a block diagram showing the hardware configuration of the user terminal in FIG. 1. FIG. FIG. 2 is a block diagram showing the hardware configuration of the mobile body in FIG. 1. FIG. FIG. 2 is a block diagram showing the functions of the management server in FIG. 1. FIG. FIG. 7 is a diagram for explaining an example of a process in which a size specifying unit specifies the size of image data at a shooting target position. 3 is a flowchart illustrating an information processing method according to the present embodiment.

The contents of the embodiments of the present invention will be listed and explained. An information processing system etc. according to an embodiment of the present invention has the following configuration.
[Item 1]
an actual photographing position information acquisition unit that acquires actual photographing position information indicating an actual photographing position in real space that is linked to the acquired photographic image data;
a virtual distance generation unit that generates virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the actual shooting position to a shooting target position in a virtual space;
a size identifying unit that identifies the size of the image data at the photographing target position based on the angle of view information and focal length information when the photographed image data was photographed, and the virtual distance information;
An information processing system equipped with.
[Item 2]
further comprising an image projection unit that projects the image data of the specified size onto a model to be photographed existing at the position to be photographed in the virtual space;
The information processing system described in item 1.
[Item 3]
further comprising a difference detection unit that detects a difference in shape between the object to be photographed reflected in the projected image data and the model to be photographed;
The information processing system described in item 2.
[Item 4]
a photographing target specifying unit that identifies at least one of a photographing target model existing at the photographing target position in the virtual space or a photographing target space based on a center position of the angle of view when photographing the photographed image data; further comprising;
The information processing system described in item 1.
[Item 5]
further comprising an image linking unit that stores the image data in association with at least one of the model to be photographed or the space to be photographed specified by the photographing target specifying unit;
The information processing system described in item 4.
[Item 6]
acquiring actual photographing position information indicating an actual photographing position in real space that is linked to the acquired photographic image data;
generating virtual distance information indicating a virtual distance from a virtual photographing position corresponding to the actual photographing position to a photographing target position in a virtual space;
specifying the size of the image data at the photographing target position based on the angle of view information and focal length information when the photographed image data was photographed, and the virtual distance information;
An information processing method performed by a computer, including:
[Item 7]
acquiring actual photographing position information indicating an actual photographing position in real space that is linked to the acquired photographic image data;
generating virtual distance information indicating a virtual distance from a virtual photographing position corresponding to the actual photographing position to a photographing target position in a virtual space;
specifying the size of the image data at the photographing target position based on the angle of view information and focal length information when the photographed image data was photographed, and the virtual distance information;
A program that causes a computer to execute.

<Details of embodiment>
An information processing system and the like according to an embodiment of the present invention will be described below. In the accompanying drawings, the same or similar elements are given the same or similar reference numerals and names, and redundant description of the same or similar elements may be omitted in the description of each embodiment. Furthermore, features shown in each embodiment can be applied to other embodiments as long as they do not contradict each other.

<Configuration>
As shown in FIG. 1, the information processing system according to the present embodiment includes a management server 1, one or more user terminals 2, and one or more mobile objects 4 (for example, a flying object, a traveling object, etc.). It has the above mobile object storage device 5. The management server 1, the user terminal 2, the mobile object 4, and the mobile object storage device 5 are connected to each other via a network so that they can communicate with each other. Note that the illustrated configuration is an example, and is not limited to this. For example, a configuration that does not include the movable body storage device 5 and is carried by the user may be used.

<Management server 1>
FIG. 2 is a diagram showing the hardware configuration of the management server 1. Note that the illustrated configuration is an example, and other configurations may be used.

As illustrated, a management server 1 is connected to a user terminal 2, a mobile object 4, and a mobile object storage device 5, and constitutes a part of this system. The management server 1 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

The management server 1 includes at least a processor 10, a memory 11, a storage 12, a transmitting/receiving section 13, an input/output section 14, etc., which are electrically connected to each other via a bus 15.

The processor 10 is an arithmetic device that controls the overall operation of the management server 1, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and/or a GPU (Graphics Processing Unit), and executes programs for this system stored in the storage 12 and developed in the memory 11 to perform various information processing. .

The memory 11 includes a main memory made up of a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory made up of a non-volatile storage device such as a flash memory or an HDD (Hard Disc Drive). . The memory 11 is used as a work area for the processor 10, and also stores a BIOS (Basic Input/Output System) executed when the management server 1 is started, various setting information, and the like.

The storage 12 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 12.

The transmitter/receiver 13 connects the management server 1 to the network. Note that the transmitting/receiving unit 13 may include a short-range communication interface of Bluetooth (registered trademark) and BLE (Bluetooth Low Energy).

The input/output unit 14 includes information input devices such as a keyboard and mouse, and output devices such as a display.

The bus 15 is commonly connected to each of the above elements and transmits, for example, address signals, data signals, and various control signals.

<User terminal 2>
The user terminal 2 shown in FIG. 3 also includes a processor 20, a memory 21, a storage 22, a transmitting/receiving section 23, an input/output section 24, etc., which are electrically connected to each other through a bus 25. Since the functions of each element can be configured in the same manner as the management server 1 described above, a detailed explanation of each element will be omitted.

<Mobile object 4>
The moving object 4 is a known moving object including a flying object such as a drone or an unmanned aerial vehicle, a running object such as an unmanned ground vehicle, and is particularly a moving object that can be autonomously controlled. As a specific example of such a moving body, the moving body 4 will be explained below. FIG. 4 is a block diagram showing the hardware configuration of the mobile object 4. As shown in FIG. Flight controller 41 may include one or more processors, such as a programmable processor (eg, a central processing unit (CPU)).

Further, the flight controller 41 has a memory 411 and can access the memory. Memory 411 stores logic, code, and/or program instructions executable by the flight controller to perform one or more steps. Further, the flight controller 41 may include a sensor 412 such as an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, a proximity sensor (eg, lidar), or the like.

Memory 411 may include, for example, a separable medium or external storage such as an SD card or random access memory (RAM). Data acquired from cameras/sensors 42 may be communicated directly to and stored in memory 411. For example, still image/video data taken with a camera or the like may be recorded in the built-in memory or external memory, but the data is not limited to this. It may be recorded in one of the user terminal 2, the mobile storage device 5, and the user terminal 2. A camera 42 is installed on the moving body 4 via a gimbal 43.

Flight controller 41 includes a control module (not shown) configured to control the state of the mobile object. For example, the control module may be configured to adjust the spatial position, velocity, and/or acceleration of a moving body having six degrees of freedom (translational motion x, y, and z, and rotational motion θ _x , θ _y , and θ _z ). , and controls the propulsion mechanism (motor 45, etc.) of the moving body via an ESC 44 (Electric Speed Controller). A propeller 46 is rotated by a motor 45 supplied with power from a battery 48, thereby generating lift of the moving body. The control module can control one or more of the states of the mounting section and sensors.

Flight controller 41 is a transceiver configured to transmit and/or receive data from one or more external devices (e.g., a transceiver 49, terminal, display, or other remote controller). It is possible to communicate with the unit 47. Transceiver 49 may use any suitable communication means, such as wired or wireless communication.

For example, the transmitter/receiver 47 uses one or more of a local area network (LAN), wide area network (WAN), infrared rays, wireless, WiFi, point-to-point (P2P) network, telecommunications network, cloud communication, etc. can do.

The transmitting/receiving unit 47 transmits and/or receives one or more of data acquired by the sensors 42, processing results generated by the flight controller 41, predetermined control data, user commands from a terminal or a remote controller, etc. be able to.

The sensors 42 according to the present embodiment include an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, a proximity sensor (for example, LiDAR (Light Detection and Ranging), etc.), or a vision/image sensor (for example, a camera). obtain. Hereinafter, the sensors 42 may be referred to as "camera 42". When the sensors 42 according to the present embodiment take an image with the camera 42 as a vision/image sensor, the actual shooting position of the camera 42 in real space at the time the image is taken, the optical center axis of the camera 42, etc. It is configured to acquire photographing condition information regarding the actual photographing direction, which is the direction in real space that the camera 42 faces, the focal length and angle of view of the camera 42, etc., in association with the image. Therefore, the image data of the image photographed by the camera 42 according to the present embodiment includes not only the data of the image but also the photographing condition information as described above.

<Management server functions>
FIG. 5 is a block diagram illustrating functions implemented in the management server 1. In the embodiment of the present invention, the management server 1 stores images taken by a camera included in the sensors 42 of the moving body 4 and photographing condition information associated therewith, and structures (for example, buildings such as buildings). various functional units that execute processing to place photographed images in a corresponding three-dimensional model based on three-dimensional model data indicating the arrangement of internal components (e.g., walls, columns, stairs, equipment, etc.) have.

In this embodiment, the management server 1 includes a processor 10, a transmitting/receiving section 13, and a storage section 200. The processor 10 includes an actual shooting position information acquisition section 110, a virtual distance generation section 120, a size specification section 130, an image projection section 140, a difference detection section 150, a shooting target specification section 160, an image linking section 170, and a movement execution section 180. ing. Furthermore, the storage unit 200 includes various databases such as a three-dimensional data storage unit 210, a photographed image data storage unit 220, and a movement information storage unit 230.

First, various databases in the storage unit 200 will be described. The three-dimensional data storage unit 210 stores three-dimensional model data indicating internal and external components of a structure and their arrangement positions. The three-dimensional model data is three-dimensional model data (more preferably having dimensional information) that indicates the arrangement of components within a structure and is created based on data created with CAD (Computer-Aided Design) design software. For example, a three-dimensional model reconstructed from BIM (Building Information Modeling) data, CIM (Construction Information Modeling) data, CAD data, BIM data, etc. It may be data or the like, or it may be three-dimensional model data obtained by generating a structure having a predetermined height based on two-dimensional blueprint data. Generation of three-dimensional model data such as reconstruction may be executed by the processor 10 of the management server 1, or the generated three-dimensional model data may be executed by an external device different from the management server 1. may be obtained. Further, the three-dimensional model data indicating the composition may be associated with information indicating the type, size, scale, etc. of the composition. In addition, as information regarding the three-dimensional model data, for example, a predetermined spatial region based on the three-dimensional model data (for example, a spatial region based on a component such as the spatial region between pillars A and B, or a three-dimensional Even if information indicating a spatial region defined by a three-dimensional coordinate system in which original model data is arranged (particularly a spatial region based on coordinates specified by user operation) is stored in the three-dimensional data storage unit 210. good.

Based on the three-dimensional model data stored in the three-dimensional data storage unit 210, the processor 10 of the management server 1 generates a virtual space expressed by the three-dimensional model data in a three-dimensional coordinate system. The arrangement information such as the arrangement position and arrangement direction of each three-dimensional model in the virtual space generated in this way is known.

The photographed image data storage unit 220 stores photographed image data photographed by the sensors 42 of the mobile body 4 and transmitted from the mobile body 4 to the management server 1 . As described above, the image data photographed and acquired by the sensors 42 of the moving body 4 includes not only the data of the image but also the actual photographing position of the camera 42 in real space at the time the image was photographed, and the image data of the camera 42 . Photographing condition information regarding the actual photographing direction, which is the direction in real space in which the optical center axis of the image is directed, the focal length of the camera 42, etc., is included in association with (linked to) the image. Therefore, the photographed image data stored in the photographed image data storage section 220 includes image data and photographing condition information linked thereto. In the photographed image data, the photographing condition information may be metadata associated with the image data.

The captured image data stored in the captured image data storage unit 220 further exists at a captured target position in the virtual space that is specified by a captured target specifying unit 160 of the processor 10 described later and linked to an image linking unit 170. The information may include information regarding at least one of the model to be photographed and the space to be photographed. Details of the above-mentioned identification processing and linking processing by the photographic target identification unit 160 and the image linking unit 170 will be described later. Furthermore, the photographed image data stored in the photographed image data storage unit 220 is related to the size of the image data at the position of the photographic subject, which is the position of the photographic subject in the virtual space, specified by the size specifying unit 130 as described later. Information may also be associated.

The movement information storage unit 230 stores movement information used in movement for the purpose of photographing actual images of various objects to be photographed inside and outside a building, for example. Movement information includes, for example, movement route information (including waypoint information), movement speed, flight altitude, imaging conditions (shooting direction, imaging angle of view, imaging focal length, overlap rate of captured images, etc.), and information acquired during movement. (For example, image data and photographing condition information associated with the image data, etc.).

The movement information can be generated, for example, by setting parameters of various information included in the movement information on the management server 1 or the user terminal 2. In addition, as a movement route, for example, a movement route passing through each waypoint may be generated with the position of the moving body storage device 5 as the movement start position and the movement end position, or conversely, a movement route may be generated that passes through each waypoint. 5, the position where the mobile body 4 is carried by the user is set as the movement start position (so-called home point), or the user collects the mobile body 4 at the movement end position (which may be returned to the home point). Alternatively, the location may be selected as the movement start position or movement end position based on the information of the mobile object storage device 5 managed by the management server 1 (for example, position information, storage status information, storage device information, etc.). The configuration may be such that the moving route is generated as a moving route including the location of the moving object storage device 5 that has been moved.

Next, each functional unit of the processor 10 will be explained. The actual photographing position information acquisition unit 110 is linked to photographed image data photographed by the camera 42 of the mobile body 4 and transmitted from the mobile body 4 to the management server 1. As the photographing condition information, in particular, actual photographing position information indicating the actual photographing position of the camera 42 in real space at the time when the image was photographed is acquired. The actual shooting position information acquisition unit 110 further determines the actual shooting direction, which is the direction in real space in which the optical center axis of the camera 42 faces, and the focal length of the camera 42, from the above shooting condition information linked to the acquired image data. and the angle of view.

The actual shooting position of the camera (sensors) 42 in real space is a position expressed in a three-dimensional coordinate system in real space. If the camera 42 of the mobile object 4 includes a GNSS sensor (such as a GPS sensor) and the GNSS sensor can obtain latitude and longitude information as the position information of the sensors 42, the camera (sensors) 42 can actually capture images in real space. The position may be expressed in latitude and longitude. Alternatively, when acquiring the position information of the sensors 42 with a non-GNSS sensor such as an inertial sensor (acceleration sensor, gyro sensor), the actual shooting position of the camera (sensors) 42 in real space is the reference position in real space ( For example, it may be expressed as a position in a three-dimensional coordinate system with the origin at the flight start position (home point) of the moving object 4.

The processor 10 of the management server 1 generates a virtual space expressed by the three-dimensional coordinate system based on the three-dimensional model data stored in the three-dimensional data storage unit 210, for example, based on the real space. By associating the three-dimensional coordinate system with the three-dimensional coordinate system of the virtual space, a process is performed to establish a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space. As an example, the processor 10 aligns a reference position in the three-dimensional coordinate system of the real space (for example, the flight start position of the moving object 4) with a reference position in the three-dimensional coordinate system of the virtual space that corresponds to that position. By associating them with each other, a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space is constructed. Thereby, the processor 10 determines the position of the moving object 4 flying in the real space in the three-dimensional coordinate system of the real space using the three-dimensional coordinate system of the virtual moving object (virtual object corresponding to the moving object 4 in the real space) in the virtual space. It is possible to convert and express the position in the original coordinate system. Note that if the scale of the three-dimensional model data in the virtual space is different from the scale of the corresponding structure (component) in the real space, the scale of the three-dimensional model data may be adjusted in conjunction with alignment.

In this way, by building a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space, for example, waypoints can be created based on the three-dimensional coordinate system of the virtual space displayed on the user terminal 2. Even when , it is possible to designate a position such as generating a movement route based on the three-dimensional coordinate system of the moving body 4 in real space. When the three-dimensional model data includes dimensional information, it becomes possible to generate a travel route using the actual scale, such as flying straight for 10 meters and turning to the right, based on the three-dimensional coordinate system of the virtual space.

The virtual distance generation unit 120 generates a virtual distance in the virtual space generated by the processor 10 from a virtual shooting position corresponding to the actual shooting position of the camera 42 in the real space to a shooting target position that is the position of the shooting target in the virtual space. Performs processing to generate virtual distance information indicating distance. The position of the object to be photographed in the virtual space can be determined using, for example, coordinate information in the virtual space, a three-dimensional model of the object to be photographed in the virtual space (virtual constructs such as walls, columns, stairs, equipment, etc. in a building in the virtual space). It may be a position or an arbitrary spatial region position within the virtual space.

The virtual distance generation unit 120 generates a virtual photographing position based on a virtual photographing position in the virtual space in the three-dimensional coordinate system of the virtual space and the photographing target position (the position of the three-dimensional model corresponding to the composition of the photographing target). A process is performed to calculate a virtual distance, which is the distance from to the shooting target position, and generate virtual distance information. The virtual distance, which is the distance from the virtual shooting position to the shooting target position, is, for example, a line segment that connects the virtual shooting position and the shooting target position in the three-dimensional coordinate system of the virtual space along the shooting direction (particularly the optical axis direction). It can be obtained by finding the length of .

The size specifying unit 130 uses the angle of view information and focal length information of the camera 42, which is shooting condition information included in the captured image data acquired when the camera 42 captures an image, and the virtual distance generated by the virtual distance generating unit 120. Based on the information, a process is performed to specify the size of image data at the position of the photographing target, which is the position of the photographing target in the virtual space.

Here, with reference to FIG. 6, an example of a process in which the size specifying unit 130 specifies the size of image data at a shooting target position will be described.

First, the size specifying unit 130 uses the angle of view information and focal length information of the camera 42 linked to the photographed image data to determine the focal length n from the virtual photographing position C in the virtual space corresponding to the actual photographing position of the camera 42. Calculate the size of a distant first plane (the "near plane" of the view frustum shown in FIG. 6). The size of the first plane corresponds to the size of the photographed image data of the photographed object acquired by the camera 42.

Next, the size specifying unit 130 uses the size of the first plane calculated as described above, the focal length information, and the virtual distance information generated by the virtual distance generating unit 120 to determine the virtual distance f from the virtual shooting position C. The size of the image data in a second plane (the "far plane" of the view frustum shown in FIG. 6) that is separated by the amount of time is calculated. The size of the second plane is the size obtained by enlarging the size of the first plane according to the ratio of the focal length n to the virtual distance f, and is the size of the image data at the shooting target position. When the three-dimensional model data includes dimension information and scale information, the size specifying unit 130 calculates the size of the image data at the photographing target position as described above to determine the actual size corresponding to the three-dimensional model at the photographing target position. It becomes possible to obtain the actual size of a spatial component from the size of its image data.

The image projection unit 140 performs a process of projecting image data of the size specified by the size specifying unit 130 onto a photographing target model that is a three-dimensional model of the photographing target existing at the photographing target position in the virtual space. Through this process, an image of a component in the real space corresponding to the model to be photographed is displayed at the corresponding position on the model to be photographed, which is a three-dimensional model of the target to be photographed, existing at the position of the target to be photographed, to the size of the model to be photographed. They are pasted in a matching size, and the texture of the real space composition (the object to be photographed) is given to the surface of the model to be photographed.

The difference detection unit 150 performs a process of detecting a difference in shape between an object to be photographed that appears in the image data projected in the virtual space by the image projection unit 140 and a model to be photographed that corresponds to the object to be photographed.

An example of the difference detection process will be described using an example in which the object to be photographed is a cupboard having a substantially rectangular parallelepiped shape. The outline of the cupboard or the corner where the outlines intersect with each other), and the characteristic parts corresponding to the characteristic parts of the photographed object in the photographed target model onto which the image data is projected (for example, the three-dimensional model of the cupboard) The shape of the object and the model reflected in the projected image data is extracted based on the positional relationship between these characteristic parts. Detect the difference in placement position. The difference information acquired as a result of the detection process by the difference detection unit 150 is used, for example, by correcting the position, shape, size, etc. of the image data projected in the virtual space by the image projection unit 140, and converting the image data into a model to be photographed. It can be used to properly overlap the top.

The photographing target specifying unit 160 identifies the photographing target model existing at the photographing target position in the virtual space or at least the photographing target space based on the center position of the angle of view of the camera 42 when the camera 42 photographs the photographed image data. Perform processing to identify either one.

More specifically, the photographing target specifying unit 160 first selects the photographed image data from among the photographing condition information that is acquired when the camera 42 photographs the photographed image data and is associated with the photographed image data. A virtual photographing direction in the virtual space corresponding to the actual photographing direction of the camera 42 is determined based on information regarding the actual photographing direction, which is the direction in the real space in which the optical center axis of the camera 42 faces at the time. The virtual photographing direction in the virtual space can be expressed by a virtual straight line extending from the virtual photographing position in the virtual space corresponding to the actual photographing position of the camera 42 to the virtual photographing direction corresponding to the actual photographing direction of the camera 42. The center position of the angle of view of the camera 42 may correspond to a virtual photographing position in the virtual space. Next, the photographing target specifying unit 160 identifies a photographing target existing at a photographing target position in the virtual space that is intersected by a virtual straight line extending from the virtual photographing position in the virtual space corresponding to the actual photographing position of the camera 42 in the virtual photographing direction. Identify the model or space to be photographed.

Through such processing, the photographing target specifying unit 160 identifies the photographing target model or photographing target existing at the photographing target position in the virtual space based on the center position of the angle of view of the camera 42 when the camera 42 photographs the photographed image data. Identify at least one of the spaces.

The image linking unit 170 performs a process of storing image data in the storage unit 200 by linking the image data to at least one of the model to be photographed and the space to be photographed identified by the photographic target specifying unit 160. More specifically, the image linking unit 170 attaches to the captured image data captured by the camera 42, based on the specific information acquired as a result of the specific processing by the captured target specific unit 160, The photographed image data in which the model to be photographed in the virtual space or the space to be photographed in the corresponding virtual space is associated with the object is stored in the storage unit 200. 220.

The movement execution unit 180 executes movement of the moving body 4 for the purpose of photographing actual images of various objects to be photographed inside and outside the building, based on various types of movement information stored in the movement information storage unit 230.

<An example of the information processing method according to this embodiment>
Next, with reference to FIG. 7, an information processing method (particularly a method for specifying image data size on a three-dimensional model) according to this embodiment will be described. FIG. 7 is a flowchart illustrating the information processing method according to this embodiment.

First, the actual photographing position information acquisition unit 110 in the information processing system of this embodiment acquires the photographed image data from the photographed image data photographed by the camera 42 of the mobile body 4 and transmitted from the mobile body 4 to the management server 1. The associated shooting condition information is acquired (step S101).

The photographing condition information is acquired when the camera (sensors) 42 as a vision/image sensor photographs an image, and includes the actual photographing position of the camera 42 in real space at the time the image is photographed, and the camera 42 It includes information regarding the actual photographing direction, which is the direction in real space in which the optical center axis of the camera 42 faces, the focal length and angle of view of the camera 42, and is associated with the photographed image data, which is the data of the photographed image. In the process of step S101, the actual photographing position information acquisition unit 110 particularly acquires information regarding the actual photographing position of the camera 42 in real space at the time when the image was photographed from the photographed image data.

Next, in the virtual space generated by the processor 10, the virtual distance generation unit 120 in the information processing system calculates the position of the object to be photographed in the virtual space from the virtual photographing position corresponding to the actual photographing position of the camera 42 in the real space. Virtual distance information indicating a virtual distance to a certain photographic target position is generated (step S102). As an example, the virtual distance generation unit 120 generates a line segment that connects the virtual imaging position in the virtual space and the imaging target position (the position of the three-dimensional model corresponding to the composition of the imaging target) in the three-dimensional coordinate system of the virtual space. By determining the length, it is possible to obtain the virtual distance to the photographing target position, which is the position of the photographing target in the virtual space.

Next, the size specifying unit 130 in the information processing system selects the angle of view information and focal length information of the camera 42 from among the photographing condition information included in the photographed image data acquired when the camera 42 photographs an image, and the virtual distance Based on the virtual distance information generated by the generation unit 120, the size of the image data at the shooting target position in the virtual space is specified (step S103).

As described with reference to FIG. 6 as an example, the size specifying unit 130 is configured to locate a first plane (viewed in FIG. Calculate the size of the "near plane" of the frustum, and then use the size of the first plane, focal length, and virtual distance to calculate the size of a second plane located a virtual distance away from the virtual shooting position. By this, the size of the image data at the position of the object to be photographed in the virtual space is specified.

In this way, by specifying the size of the image data at the shooting target position in the virtual space, the captured image data is projected to the shooting target position (the position of the three-dimensional model corresponding to the composition to be shot) in later processing. In some cases, it is possible to set the image data to a size appropriate to the position of the object to be photographed. Furthermore, if the 3D model data includes dimension information and scale information, it is possible to obtain the actual size of the structure in the real space corresponding to the 3D model at the shooting target position from the size of the image data. Become.

Next, the image projection unit 140 in the information processing system projects image data of the size specified by the size specifying unit 130 onto a photographing target model that is a three-dimensional model of the photographing target existing at the photographing target position in the virtual space. (Step S104). As a result, an image of a component in the real space corresponding to the model to be photographed is pasted onto the model to be photographed at the target position, with a size that matches the size of the model to be photographed. The texture of a real space component (object to be photographed) is added to the surface of the model.

Next, the difference detection unit 150 in the information processing system detects a difference in shape between the object to be photographed that appears in the image data projected in the virtual space by the image projection unit 140 and the model to be photographed corresponding to the object to be photographed. (Step S105).
For example, the difference detection unit 150 extracts a characteristic part of the object to be photographed that is reflected in the projected image data and a corresponding characteristic part in the model of the object to be photographed on which the image data is projected, and determines the positional relationship between these characteristic parts. Based on this, a difference in shape or placement position between the object to be photographed and the model to be photographed that appear in the projected image data is detected. By using the difference information acquired in this way, for example, the image projection unit 140 can correct the position, shape, size, etc. of the image data projected in the virtual space, and properly display the image data on the model to be photographed. It is possible to arrange them so that they overlap.

Next, the photographing target specifying unit 160 in the information processing system determines, based on the center position of the angle of view of the camera 42 when the camera 42 photographs the photographed image data, the photographing target model existing at the photographing target position in the virtual space, Alternatively, at least one of the spaces to be photographed is specified (step S106).

In the process of step S106, the photographing target specifying unit 160 first determines the actual photographing direction of the camera 42 based on information regarding the actual photographing direction, which is the direction in real space that the optical center axis of the camera 42 faces at the time when the photographed image data is photographed. Find the virtual shooting direction in the virtual space corresponding to the actual shooting position of the camera 42, and then find the shooting target in the virtual space that is intersected by a virtual straight line extending from the virtual shooting position in the virtual space corresponding to the actual shooting position of the camera 42 in the virtual shooting direction. A model to be photographed or a space to be photographed existing at a position is identified.

Finally, the image linking unit 170 in the information processing system stores the image data in the storage unit 200 by linking it to at least one of the shooting target model or the shooting target space specified by the shooting target specifying unit 160 (step S107 ). As a result, for example, a model to be photographed or a space to be photographed corresponding to the image data stored in the photographed image data storage section 220 can be searched, or a model or object to be photographed stored in the three-dimensional data storage section 210 or It becomes possible to search for image data corresponding to the imaging target space, and it becomes possible to automatically specify image data to be projected onto the imaging target position by the image projection unit 140 in step S104.

As described above, the present embodiment provides an information processing system and the like that can easily manage images photographed by the moving body 4 in association with the photographing target position and its size. In particular, the information processing method according to the present embodiment includes specifying the size of the image data at the imaging target position in the virtual space, thereby converting the captured image data into the imaging target position (three-dimensional When projecting the image data to the position of the model, it is possible to make the image data a size appropriate for the position of the object to be photographed.Furthermore, if the 3D model data includes dimension information and scale information, it is possible to It becomes possible to obtain the actual size of a component in real space corresponding to a three-dimensional model from the size of its image data.

The moving body 4 may further include a device, equipment, etc. used to inspect the inner wall and/or outer wall of the structure for the presence or absence of a predetermined event. More specifically, imaging devices (visible light cameras, infrared cameras, metal detectors, ultrasonic measuring devices, etc.), keying devices, detection devices (metal detectors), sound collection devices, odor measuring devices, gas detection devices, etc. All the devices necessary to know the condition of the structure to be inspected, such as air pollution measuring instruments, detection devices (devices for detecting cosmic rays, radiation, electromagnetic waves, etc.), can be employed.

Further, the information processing method according to the present embodiment may be executed, for example, during security or monitoring inside a structure, and may further include devices, equipment, etc. used for security or monitoring. More specifically, structures to be guarded and monitored, such as imaging devices (visible light cameras, infrared cameras, night vision cameras, metal detectors, ultrasonic measuring instruments, etc.) and sensor devices (motion sensors, infrared sensors, etc.) All devices necessary to image and detect abnormalities, intruders, etc. can be employed.

Furthermore, the mobile body 4 can be suitably used as a mobile body for photographing equipped with a camera, etc., and can also be used in the security field, infrastructure monitoring, surveying, and in buildings and structures such as sports venues, factories, and warehouses. It can also be used in various industries such as inspection and disaster response.

The embodiments described above are merely illustrative to facilitate understanding of the present invention, and are not intended to be interpreted as limiting the present invention. It goes without saying that the present invention can be modified and improved without departing from its spirit, and that the present invention includes equivalents thereof.

1 Management server 2 User terminal 4 Mobile object 5 Mobile object storage device

Claims (7)

Actual shooting position information acquisition that acquires actual shooting position information indicating an actual shooting position in the real space that is linked to the image data from shot image data including image data obtained by shooting in real space with a camera . Department and
a virtual distance generation unit that calculates a virtual distance from a virtual shooting position in a virtual space corresponding to the actual shooting position to a shooting target position in the virtual space and generates virtual distance information;
Using the angle of view information and focal length information when the photographed image data was photographed, calculate the size of the image data in a first plane separated by the focal length from the virtual photographing position, and using the virtual distance information. expand the size of the image data in the first plane according to the ratio of the focal length and the virtual distance, and expand the size of the image data in the first plane in accordance with the ratio of the focal length to the virtual distance, and expand the size of the image data in the second plane in the virtual space that is separated by the virtual distance from the virtual shooting position. a size identifying unit that identifies a plane size of the image data ;
An information processing system equipped with. further comprising an image projection unit that projects the image data of the specified plane size onto a model to be photographed existing at the photographing target position in the virtual space;
The information processing system according to claim 1. further comprising a difference detection unit that detects a difference in shape between the object to be photographed reflected in the projected image data and the model to be photographed;
The information processing system according to claim 2. Based on the center position of the angle of view included in the angle of view information when the photographed image data is photographed, at least one of the photographing target model existing at the photographing target position in the virtual space or the photographing target space is determined. further comprising a photographing target specifying unit for specifying;
The information processing system according to claim 1. further comprising an image linking unit that stores the image data in association with at least one of the model to be photographed or the space to be photographed specified by the photographing target specifying unit;
The information processing system according to claim 4. acquiring actual photographing position information indicating an actual photographing position in the real space that is linked to the image data from photographed image data including image data obtained by photographing the real space with a camera ;
calculating a virtual distance from a virtual shooting position in a virtual space corresponding to the actual shooting position to a shooting target position in the virtual space, and generating virtual distance information;
Using the angle of view information and focal length information when the photographed image data was photographed, calculate the size of the image data in a first plane separated by the focal length from the virtual photographing position, and using the virtual distance information. expand the size of the image data in the first plane according to the ratio of the focal length and the virtual distance, and expand the size of the image data in the first plane in accordance with the ratio of the focal length to the virtual distance, and expand the size of the image data in the second plane in the virtual space that is separated by the virtual distance from the virtual shooting position. identifying a plane size of the image data ;
An information processing method performed by a computer, including: acquiring actual photographing position information indicating an actual photographing position in the real space that is linked to the image data from photographed image data including image data obtained by photographing the real space with a camera ;
calculating a virtual distance from a virtual shooting position in a virtual space corresponding to the actual shooting position to a shooting target position in the virtual space, and generating virtual distance information;
Using the angle of view information and focal length information when the photographed image data was photographed, calculate the size of the image data in a first plane separated by the focal length from the virtual photographing position, and using the virtual distance information. expand the size of the image data in the first plane according to the ratio of the focal length and the virtual distance, and expand the size of the image data in the first plane in accordance with the ratio of the focal length to the virtual distance, and expand the size of the image data in the second plane in the virtual space that is separated by the virtual distance from the virtual shooting position. identifying a plane size of the image data ;
A program that causes a computer to execute.

Images